You’re Going To Look Silly When People Can See Your Face In VR

Google hopes to make virtual reality less isolating by showing our expressions under the mask

Feb 23, 2017 at 8:03 AM ET

One of the biggest critiques of virtual reality is that its fully immersive nature can be isolating.

For onlookers, it’s impossible to see what the user is experiencing, and clunky VR goggles completely obscure the top half of the face so it’s hard to even gauge facial expressions. Even when two people are put in the same virtual reality environment, it’s hard to feel like you’re experiencing it together when you can’t see one another. Google wants to change that.

Google software engineer Avneesh Sud recently wrote a blog post about a collaboration between Daydream Labs, Google Research, and YouTube that aims to use machine learning and 3-D computer vision in order to combat the problems critics say will keep virtual reality from becoming a mainstream consumer technology. He feels that the solution lies within “mixed reality.” Mixed reality, also called hybrid reality, merges augmented reality (like how Pokémon Go lets users “see” virtual creatures in real-life settings) with virtual reality…and plain old reality reality.

In order to address the face obfuscation issue, this setup creates a dynamic 3D model of users’ faces and uses eye-tracking software to capture their gaze so that it can project the person’s approximate facial expression on the outward-facing front side of the headset. To observers, it would therefore like the VR user is wearing a “translucent scuba mask.”

Then, by adding a green screen, tracking devices, and a video camera to the apparatus, viewers can virtually “see” from the perspective of their VR-using peers.

“Imagine being able to VR video conference and see the expressions and nonverbal cues of the people you are talking to, or seeing your friend’s reactions as you play your favorite game together,” Sud writes of the project, which he notes is still in its early stages.

While perhaps a step in the right direction of making virtual reality more of a group activity, one need only look at press images created by Daydream Labs to see how silly this experience actually looks in practice.