Augmented Reality and Autonomous Vehicles – Enabled by the Same Technologies?

In Introducing Augmented Reality Apparatus – From Victorian Stage Effects to Head-Up Displays, we saw how the Pepper’s Ghost effect could be used to display information in a car using a head-up display projected onto a car windscreen as a driver aid. In this post, we’ll explore the extent to which digital models of the world that may be used to support augmented reality effects may also be used to support other forms of behaviour…

Constructing a 3D model of an object in the world can be achieved by measuring the object directly, or, as we have seen, measuring the distance to different points on the object from a scanning device and then using these points to construct a model of the surface corresponding to the size and shape of the object. According to IEEE Spectrum’s report describing A Ride In Ford’s Self-Driving Car, “Ford’s little fleet of robocars … stuck to streets mapped to within two centimeters, a bit less than an inch. The car compared that map against real-time data collected from the lidar, the color camera behind the windshield, other cameras pointing to either side, and several radar sets—short range and long—stashed beneath the plastic skin. There are even ultrasound sensors, to help in parking and other up-close work.”

Whilst the domain of autonomous vehicles may seem to be somewhat distinct from the world of facial capture on the one hand, and augmented reality on the other, autonomous vehicles rely on having a model of the world around them. One of the techniques currently used in detecting distances to objects surrounding an autonomous vehicle is LIDAR, in which a laser is used to accurately detect the distance to a nearby object. But recognising visual imagery also has an important part to play in the control of autonomous and “AI-enhanced” vehicles.

For example, consider the case of automatic lane detection:

Here, an optical view of the world is used as the basis for detecting lanes on a motorway. The video also shows how other vehicles in the the scene can be detected and tracked, along with the range to them.

A more recent video from Ford shows the model of the world perceived from the range of sensors one of their autonomous vehicles.

Part of the challenge of proving autonomous vehicle technologies to regulators, as well as development engineers, is the ability to demonstrate what the vehicle thinks it can see and what it might do next. To this extent, augmented reality displays may be useful in presenting in real-time a view of a vehicle’s situational awareness of the environment it currently finds itself in.

DO: See if you can find some further examples of the technologies used to demonstrate the operation of self-driving and autonomous vehicles. To what extent do these look like augmented reality views of the world? What sorts of digital models do the autonomous vehicles create? To what extent could such models be used to support augmented reality effects, and what effects might they be?

If, indeed, there is crossover between the technology stack that underpins autonomous vehicles, computational devices developed to support autonomous vehicle operation may also be useful to augmented and mixed reality developers.

DO: read through the description of the NVIDIA DRIVE PX 2 system and software development kit. To what extent do the tools and capabilities described sound as if they may be useful as part of an augmented or mixed reality technology stack? See if you can find examples of augmented or mixed reality developers using such toolkits originally developed or marketed for autonomous vehicle use and share them in comments below.

Using Cameras to Capture Objects as Well as Images

In The Photorealistic Effect… we saw how textures from photos could be overlaid onto 3D digital models as well as how digital models could be animated by human puppeteers: using motion capture to track the movement of articulation points on the human actor, this information could then be used to actuate similarly located points on the digital character mesh; and in 3D Models from Photos, we saw how textured 3D models could be “extruded” from single photograph by associating points on them with a mesh and then deforming the mesh in 3D space. In this post, we’ll explore further how the digital models themselves can be captured by scanning actual physical objects as well as by constructing models from photographic imagery.

We have already seen how markerless motion capture can be used to capture the motion of actors and objects in the real world in real time, and how video compositing techniques can be used to change the pictorial content of a digitally captured visual scene. But we can also use reality capture technologies to scan physical world objects, or otherwise generate three dimensional digital models of them.

Generating 3D Models from Photos

One way of generating a three dimensional model is to take a basis three dimensional mesh model and map it onto appropriate points in a photograph.

The following example shows an application called Faceworx in which textures from a front facing portrait and a side facing portrait are mapped onto a morphable mesh. The Smoothie-3d application described in 3D Models from Photos uses a related approach.

3D Models from Multiple Photos

Another way in which photographic imagery can be used to generate 3D models is to use techniques from  photogrammetry, defined by Wikipedia as “the science of making measurements from photographs, especially for recovering the exact positions of surface points”. By using taking several photographs of the same object and identifying the same features in each of them, and then align the photographs, using the differential distances between features to model the three-dimensional character of the original objects.

DO: read the description of how the PhotoModeler application works: PhotoModeler – how it works. Similar mathematical techniques (triangulation and trilateration) can also be used to calculate distances in a wide variety of other contexts, such as finding the location of a mobile phone based on the signal strengths of three or more cell towers with known locations.

“Depth Cameras”

Peripheral devices such as Microsoft Kinect, the Intel RealSense camera and the Structure Sensor 3D scanner perceive depth directly as well as capturing photographic imagery.

In the case of Intel RealSense devices, three separate camera components work together to capture the imagery (a traditional optical camera) and the distance to objects in the field of view (an infra-red camera and a small infra-red laser projector).

With their ability to capture distance-to-object measures as well as imagery, depth perceiving cameras represent an enabling technology that opens up a range of possibilities for application developers. For example, itseez3d is a tablet based application that works with the Structure Sensor to provide a simple 3D scanner application that can capture a 3D scan of a physical object as both a digital model and a corresponding texture.

Depth Perceiving Cameras and Markerless mocap

Depth perceiving cameras can also be used to capture facial models, as the FaceShift markerless motion capture studio shows.

Activity: according to the FAQ for the FaceShift Studio application shown in the video below, what cameras can be used to provide inputs to the FaceShift application?

Exercise: try to find one or two recent examples of augmented or mixed reality applications that make use of depth sensitive cameras and share links to them in the comments below. To what extent do the examples require the availability of the depth information in order for them to work?

Interactive Dynamic Video

Another approach to use video captures to create interactive models is a new technique developed by researchers Abe Davis, Justin G. Chen, and Fredo Durand at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL)  referred to as interactive dynamic video. In this technique, or few seconds (or minutes) of video are analysed to study the way a foreground object vibrates naturally, or when gently perturbed.

Rather than extracting a 3-dimensional model of the perturbed object, and then rendering that as a digital object, the object in the interactive is perturbed by constructing a “pyramid” mesh over the pixels on the video image itself (Davis, A., Chen, J.G. and Durand, F., 2015. Image-space modal bases for plausible manipulation of objects in video. ACM Transactions on Graphics (TOG), 34(6), p.239). That is, there is no “freestanding” 3D model of the object that can be perturbed. Instead, it exists as a dynamic, interactive model within the visual scene within which it is situated. (For a full list of related papers, see the Interactive Dynamic Video website.)

SAQ: to what extent, if any, is interactive dynamic video an example of an augmented reality technique? Explain your reasoning.

Adding this technique to our toolbox, along with the ability to generate simple videos from still photographs as described in Hyper-reality Offline – Creating Videos from Photos, we see how it is increasingly possibly to bring imagery alive simply through the manipulation of pixels, mapped as textures onto underlying structural meshes.

The Virtual Worlds Universe

In Have You Got a Second Life?, I introduced the 3D social network Second Life, and hinted that this was just one of many such virtual worlds.

To see just how big the universe of virtual worlds is becoming, the Association of Virtual Worlds recently published a directory describing over 250 virtual worlds: The Blue Book: A Consumer Guide to Virtual Worlds.

Download a copy of The Blue Book: A Consumer Guide to Virtual Worlds. What categories does The Blue Book use to classify each world? For a consumer guide, what other information would you find useful to know? Identify three or four combinations of category that interest you, and see if a virtual world is listed that matches those categories.

Several other sites also offer comparison charts for the increasing number of virtual worlds that are now in existence. For example, virtualenvironments.info provides a comparison matrix of fifteen or so of the larger virtual worlds.

Identify two or three different scenarios in which it might be appropriate to visit a virtual world (for example, a business meeting, a school ‘geography field trip’, or a ‘night on the virtual town’). To give you an example of how such worlds might be used in business, for example, read this article from Business Week: The (Virtual) Global Office.

Now identify a virtual world that looks like it might provide an appropriate setting for each activity. Now visit each virtual world (or at least its website). In what ways do the worlds meet – or fail to meet – your expectations? Feel free to write a post about the scenarios you chose, the criteria you used to select an appropriate world, the worlds you selected (and why), what you expected to find in those virtual worlds, and how those worlds met your expectations. If you can find a demo video, or video review, of the virtual worlds in question, embed it in your post.

How should I behave in a Virtual World?

As with all social situations mediated by communication technologies, there is often a right way and a wrong way to behave when entering a virtual world.

Many organisations have a code of conduct that regulates their employees’ behaviour. Suppose that you work for an organisation that makes use of 3D virtual worlds. Write down five areas of personal behaviour, presentation or activity that might be addressed by a code of conduct for working in virtual worlds.

Now read the IBM Research IBM Virtual World Guidelines. Did you identify similar issues in your own list?

Have You Got a Second Life?

Of all the 3D virtual worlds that can now be found on the internet, Second Life is arguably the one that has received the most popular press attention.

If you have ever been in to Second Life, then you will be familiar with the sort of things it can offer. If you have not visited Second Life – or indeed, never been into a 3D, avatar populated immersive world – here are a couple of quick tastes of what life is like “in-world”.

The first is a presentation about Second Life that has been uploaded to the social presentation sharing SlideShare – “An Introduction to Virtual Worlds: Second Life and Beyond. Even if you have been into Second Life, quickly flicking through the presentation may point out some features about it that you didn’t notice at the time.

The second is a user-generated movie about Second Life that I discovered YouTube…

How does Second Life differ from 3D worlds like Google Earth or Virtual Earth? How does it differ from 3D game worlds? In your opinion, is Second Life a game?

Now watch the following clip about the game “The Sims” (IGN Review) – what similarities and differences are there between Second Life and The Sims?

The most obvious difference to me is that in The Sims the player takes on a third person, God-like role, controlling the actions (to some extent) of their player characters, whereas in Second Life, the “player” becomes (or actually is) the avatar.

In the Sims, the game world is a self-contained fiction: the aim of the game, such as it is, is to help the player characters live out their lives in the Sims world. To a certain extent, there is an element of ‘progression’: players must look after characters within the game world that are dependent on them and help them keep up with Joneses – get a job, and education, a house and so on (every time I have tried to play the Sims the session has ended with my characters’ house burning down!)

In contrast, Second Life just provides a canvas for creativity and social interaction – Second Life is an online world (in contrast to the desktop or console bound Sims) within which you can chat and socialise with other people from all over the world.

Want to know more about Second Life?

We’ll look at worlds like Second Life again in later posts, in the contexts of community and making money in virtual worlds…

In the meantime, the following video replays a Google tech talk, recorded in March 2006, featuring a presentation from Glimpse Inside a Metaverse: The Virtual World of Second Life. Even though Second Life has moved on since the presentation was recorded, if you’re interested in hearing about Second Life from the insdie (including some insights about the techie stuff!) it’s well worth listening to:

If you want to try Second Life out for yourself, you can find it at Second Life – http://secondlife.com. If you would rather read about Second Life second-hand, then there’s always the book Second Lives: A Journey Through Virtual Worlds, by Tim Guest!

However, as we’ll see in further posts, there are plenty of virtual worlds other than Second Life, many of them popular with different age groups (Second Life is largely for the over-30s!). So don’t feel as if you have to join Second Life to experience a 3D virtual world – as you’ll see in the next post on this topic…

3D Worlds Fitness Test/Checklist

Going through some old notes I’d collected about potential ideas for exercises in 3D worlds, I came across the following checklist I’d scribbled down at some point that seemed like good things to know about when exploring a 3D digital world.

If you can think any other ‘need to know’ skills, please add them as a comment.

Can you:

• zoom in and out;
• tilt the view to a desired perspective;
• rotate a view;
• navigate to a particular location;
• search for a particular location;
• bookmark a particular location;
• add an information layer in a “mirror world”, such as one of the following:
  • Google Earth;
  • NASA WorldWind;
  • Virtual Earth 3D.

When it comes to controlling 3D avatars in a digital world, can you move around as easily as you can in the real world?

Give yourself 5 points if you get the joke ;-)

Our Heavens in 3D…

For the sake of completeness, as much as anything, I thought I’d just complement the Our World in 3D… post with a brief reference list of interactive online media and desktop applications that you can use to explore the night sky, and the objects contained within it.

There are several applications around that let you observe the night sky as if you were on earth, as well as letting you explore it in a 3D navigational way…

• Worldwide Telescope (Microsoft):

• Celestia:

• NASA Worldwind:

• Google maps for other worlds: Google Moon, Google Mars and Google sky (a map of the night sky); there is also Sky in Google Earth:

• Star viewer: a mashup based on Google Sky, that provides an interactive map of the night sky that includes embedded images and videos that offer a useful extra level of detail, or commentary, about celestial objects we can see from Earth.
• Wikisky: similar to Star Viewer, but more comprehensive.

Visit one or two of the online applications, or download one of the desktop applications. To what extent do you think that the power of interactivity brings the idea of the scale of the universe home, compared to descriptions one might find in a book, for example?

[The videos referred to in this post are available in the compilation Splashcast video show “Our Heavens in 3D” on the Digital Worlds Splashcast video channel.]