In the post From Magic Lenses to Magic Mirrors and Back Again we reviewed several consumer facing alternate reality phone applications, such as virtual make-up apps In this post, we’ll review some simple face based reality distorting effects with an alternative reality twist.
In the world of social networks, Snapchat provides a network for sharing “disposable” photographs and video clips, social objects that persist on the phone for a short period before disappearing. One popular feature of snapchat comes in the form of its camera and video filters, also referred to as SnapChat Lenses, that can be used to transform or overlay pictures of faces in all sorts of unbecoming ways.
As the video shows, the lenses allow digital imagery to be overlaid on top of the image, although the origin of the designs is sometimes open to debate as the intellectual property associated with facepainting designs becomes contested (for example, Swiped – Is Snapchat stealing filters from makeup artists?).
Behind the scenes, facial features are captured using a crude form of markerless facial motion capture to create a mesh that acts as a basis for the transformations or overlays as described in From Motion Capture to Performance Capture and 3D Models from Imagery.
Another class of effect supported by “faceswap” style applications is an actual faceswap, in which one person’s face is swapped with another’s – or even your own.
Indeed, New York songwriter Anthony D’Amato went one step further, using the app to swap his face with various celebrities to make a faceswapped video of him singing one of his own songs (/via Digital Trends (World’s first FaceSwap music video is equal parts creepy, impressive).
As well as swapping two human faces, faceswapping can be used to swap a human face with the face of a computer game character. For computer gamers wanting to play a participating role in the games they are playing, features such as EASports GameFace allow users to upload two photos of their face – a front view and a side view – and then use their face on one of the game characters models.
The GameFace interface requires the user to physically map various facial features on the uploaded photograph so that these can then be used to map the facial mesh onto an animated character mesh. The following article shows how facial features registered as a simple mesh on two photographs can be used to achieve a faceswap effect “from scratch” using open source programming tools.
DO: read through the article Switching Eds: Face swapping with Python, dlib, and OpenCV by Matthew Earl to see how a faceswap style effect can be achieved from scratch using some openly available programming libraries. What process is used to capture the facial features used to map from one face to the other? How is the transformation of swapping one face with another actually achieved? What role does colour manipulation play in creating a realistic faceswap effect?
If you would like to try to replicate Earl’s approach, his code is available on Github at matthewearl/faceswap. (A quick search of Github also turns up some other approaches, such as zed41/faceSwapPython and MarekKowalski/FaceSwap.)
Developing algorithms and approaches face tracking is an active area of research, both in academia and commercially. The outputs of academic research are often written up in academic publications. Sometimes, the implementation code is made available by researchers, although at other times it is not. Academic reports should also provide enough detail about the algorithms described for independent third parties to be able to implement, as is the case in Audun Mathias Øygard’s clmtrackr.
DO: What academic paper provided the inspiration for clmtrackr? Try running examples listed on auduno/clmtrackr and read about the techniques used in the posts Fitting faces – an explanation of clmtrackr and Twisting faces: some use cases of clmtrackr. How does the style of writing and explanation in those posts compare to the style of writing used in the academic paper? What are the pros and cons of each style of writing? Who might the intended audience be in each case?
UPDATE: it seems as if Snapchat may doing a line of camera enabled sunglasses – Snapchat launches sunglasses with camera. How much harder is it to imagine the same company doing a line in novelty AR specs that morph those around you in a humorous and amusing way whenever you look at them…?! Think: X-Ray spex adds from the back of old comics…
Digital Worlds - Distorted Reality 
Recent Comments