Hyper-reality Offline – Creating Videos from Photos

In Mediating the Background and the Foreground – From Green Screen and Chroma-Key Effects to Virtual Sets we saw how green screen/chroma key effects could be used to mask out part of one image so that it could be composited with another. In this post, you’ll see how we can also generate animation effects from a single image.

Many of you will recognise the following effect from television documentaries, as well as screen savers or photo-stories:

Know as the Ken Burns effect, named after the documentary maker who made extensive use of the technique, it allows a moving image to be generated from a still photograph by panning and zooming across the image.

But what happens if you take a flat, static image, separate out the foreground and background elements, and then apply the effect, panning and zooming foreground and background elements differentially to create a “2.5D” parallax effect?

These views can be created from a single, flat image by cutting the foreground component out into its own layer, and then inpainting the background layer;  when the foreground component moves relative to the background, the inpainted area hides the fact that that part of the original image was taken up by the foreground component.

The inpainting effect can be achieved by applying an image processing technique that works from the edge of a cropped area inwards, trying to predict what value each missing neighbouring pixel should be based on the actual set values of the surrounding pixels. More elaborate techniques allow for “content aware” fills, in which the patterns generated from the surrounding texture are used to fill in the missing area. The following video show how to apply such as content aware effect in a popular photo-editing tool.

An extension of the technique – content aware crop – automatically inpaints whitespace around the edge of an image when changing the aspect ration of an image, such as following a straightening of the horizon.

Developing algorithms for improved content aware fills is an active area of academic, as well as commercial, research (eg (Pathak, Deepak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. “Context Encoders: Feature Learning by Inpainting.” arXiv preprint arXiv:1604.07379 (2016).)).

Related techniques can be used to improve the quality of images, as demonstrated by the Magic Pony Technology company (MIT Technology Review – Artificial Intelligence Can Now Design Realistic Video and Game Imagery) or deep learning neural networks. For example, a project by David Garcia shows how deep learning can “upscale 16×16 images by a 4x factor. The resulting 64×64 images display sharp features that are plausible based on the dataset that was used to train the neural net.” Here’s an example of what the networks can do (“the first column is the 16×16 input image, the second one is what you would get from a standard bicubic interpolation, the third is the output generated by the neural net, and on the right is the ground truth”):

Additional 2.5d effects can be created by animating both the foreground and background elements. Alternatively, by associating a mesh with particular points in a photo, translating those points appropriately results in the animation of the meshed element .

These effects are all based on the manipulation of pixels within a static image. But as you’ll see in another post, flat images can also be used as the basis for generating three dimensional models.

Tuning the colour palette of an image can is another technique that can be used to make it feel hyper-real, or somehow sharper than the captured reality. Similar techniques can also be applied to video to create a stylised hyper-real video effect.

As you are perhaps beginning to realise, many mediated reality effects rely on a whole stack of technologies, techniques or other effects being available first. But this in turn means that many of today’s yet-to-be invented techniques are likely to be built from a novel combination of techniques that already exist, or that can be built on; and once those new techniques are identified, and tools built to implement them efficiently, they in turn will provide the basis for yet more techniques.