In the post Taxonomies for Describing Mixed and Alternate Reality Systems, we provided a framework for talking about the various physical components of an augmented reality system. But how should we talk about the different elements within the perceived augmented reality scene?
Milgram and Kishino (Milgram, Paul & Fumio Kishino, “A taxonomy of mixed reality visual displays”, IEICE TRANSACTIONS on Information and Systems 77, no. 12 (1994): 1321-1329) started by clarifying the notions of real and virtual in an augmented reality sense:
- Real objects are objects that have a physical, tangible existence, whereas virtual objects are purely digital representations, without a physical correlate, within the rendered visual scene (although they may be digital representations of things that do exist).
- An object viewed directly appears has an existence in the real world and is viewed as such by the viewer. A non-directly viewed object is one that has been sampled and re-presented to the viewer via a display medium, or a virtual object whose existence can only be viewed via such a medium. This is referred to as the image quality.
- A real image is one that has “some luminosity at the location at which it appears to be located”, such as a directly viewed object or an image viewed on a screen. Virtual images are produced by optical tricks, such as holograms and mirror images, and have no luminosity at the location at which they appear.
Whilst these distinctions are helpful when considering the representation of a single object, they may become confused when trying to analyse a view composed of multiple objects, both real and virtual. For example, in the Google Translate example described in Augmenting Reality With Digital Overlays, the screen is a physical display, that is, a real image, that provides a non-direct view. But is the text a real object or a virtual object?
To help us talk about objects within the augmented visual scene, we might add an additional correspondence dimension, that describes whether an object within the scene, or component of it, is presented as:
- a raw, otherwise untouched, part of the image (that is, a faithful re-presentation of the object represented in that part of the image);
- an overlay, where an additional layer of information is added to the scene, as in the case of a HUD dashboard;
- a re-touch, where the object is still recognisable but has been reshaped and/or recoloured;
- a replacement, where an object has been detected and then replaced.
We now have various tools at out disposal for helping us see – and talk about – the various components of a mixed reality system from a range of critical perspectives.
Digital Worlds - Distorted Reality 
Definitions of reality are frustratingly tricky. If I have a plastic apple, it is a real object, yet not a real object. A faithful representation is usually only a faithful representation of how a person with a good view, 20/20 vision and no colour blindness perceives visual reality in certain light conditions and in certain atmospheric conditions.
Magritte wrote some interesting notes on the relations between representations and reality (scroll down to below the inevitable picture of a ‘pipe’) http://ideophone.org/magritte-on-words-and-images/
@r3becca Agreed – there is a lot of scope for philosophical and artistic diversions across this series of posts. Maybe I need to do a couple of posts on that tack…? Or at least some stubs into relevant readings?
Illusions, hallucinations and delusions are another class of perceptions where the reality and the interpretation thereof become matters of debate.