In the post Taxonomies for Describing Mixed and Alternate Reality Systems, we provided a framework for talking about the various physical components of an augmented reality system. But how should we talk about the different elements within the perceived augmented reality scene?
Milgram and Kishino (Milgram, Paul & Fumio Kishino, “A taxonomy of mixed reality visual displays”, IEICE TRANSACTIONS on Information and Systems 77, no. 12 (1994): 1321-1329) started by clarifying the notions of real and virtual in an augmented reality sense:
- Real objects are objects that have a physical, tangible existence, whereas virtual objects are purely digital representations, without a physical correlate, within the rendered visual scene (although they may be digital representations of things that do exist).
- An object viewed directly appears has an existence in the real world and is viewed as such by the viewer. A non-directly viewed object is one that has been sampled and re-presented to the viewer via a display medium, or a virtual object whose existence can only be viewed via such a medium. This is referred to as the image quality.
- A real image is one that has “some luminosity at the location at which it appears to be located”, such as a directly viewed object or an image viewed on a screen. Virtual images are produced by optical tricks, such as holograms and mirror images, and have no luminosity at the location at which they appear.
Whilst these distinctions are helpful when considering the representation of a single object, they may become confused when trying to analyse a view composed of multiple objects, both real and virtual. For example, in the Google Translate example described in Augmenting Reality With Digital Overlays, the screen is a physical display, that is, a real image, that provides a non-direct view. But is the text a real object or a virtual object?
To help us talk about objects within the augmented visual scene, we might add an additional correspondence dimension, that describes whether an object within the scene, or component of it, is presented as:
- a raw, otherwise untouched, part of the image (that is, a faithful re-presentation of the object represented in that part of the image);
- an overlay, where an additional layer of information is added to the scene, as in the case of a HUD dashboard;
- a re-touch, where the object is still recognisable but has been reshaped and/or recoloured;
- a replacement, where an object has been detected and then replaced.
We now have various tools at out disposal for helping us see – and talk about – the various components of a mixed reality system from a range of critical perspectives.