AI Can Turn A Photo Of Your Face Into An Uncanny 3D Model

Dec 08, 2016 at 4:45 PM ET

With the rise of AI and machine learning, the billions of photos uploaded to social media sites every day have become a trove of useful data, allowing advertisers, researchers and governments to automatically recognize faces, detect objects, and perform other kinds of advanced image analysis. Now, scientists at the University of Southern California have created a learning algorithm capable of building an eerily-precise 3D model of a person’s head using a single low-resolution photo of their face.

The facial reconstructions are done using an artificial neural network, a kind of algorithm that learns to accomplish a certain task (like identifying faces) by being “trained” on a large collection of example data. In this case, the researchers trained the neural network to look at 2D images and extrapolate a 3D texture mapping that approximates the subject’s facial dimensions with a high degree of realism.

“We demonstrate successful face reconstructions from a wide range of low resolution input images, including those of historical figures,” the researchers write in a paper submitted for peer review to the ArXiv pre-print repository, using old photos of Donald Trump and boxing legend Muhammad Ali as examples. “In addition to extensive evaluations, we validate the realism of our results using a crowdsourced user study.”

The findings have implications for the future of realistic avatars in virtual reality. At its VR developer-focused event in October, Facebook-owned Oculus introduced expressive, cartoon-like avatars that can emote in real-time based on hand gestures and the emotive qualities of the user’s voice.

“With virtual and augmented reality becoming the next generation platform for social interaction, compelling 3D avatars could be generated with minimal efforts and pupeteered through facial performances,” the researchers write. “Within the context of cultural heritage, iconic and historical personalities could be restored to life in captivating 3D digital forms from archival photographs.”

Others have demonstrated that the ability to replicate faces could also have major impacts on privacy and security. There would be nothing preventing someone with access to a person’s avatar from manipulating it to perform lewd or objectionable acts, for example. (There has already been at least one documented case of sexual assault in VR).

The impacts wouldn’t necessarily be limited to VR. Earlier this year, researchers at Stanford University demonstrated a system that can doctor videos by performing “re-enacted” facial and mouth movements in real-time. A similar research paper previously showed that it’s also possible to defeat the “face unlock” security feature offered by many smartphones by pasting someone’s Facebook photo onto a 3D model and manipulating it within a VR headset to unlock the device.