Facebook can project your eyes onto a VR headset, and it’s exactly as uncanny as it sounds
Facebook Reality Labs wants to help people see your eyes while you’re in virtual reality — even if the results sit somewhere between mildly unsettling and nightmarish. Earlier this week, FRL released a paper on “reverse passthrough VR,” a recipe for making VR headsets less physically isolating. Researchers devised a method for translating your face onto the front of a headset, although they emphasize it’s still firmly experimental.
“Passthrough VR” refers to a feature that displays a live video feed from a headset’s cameras, letting users see the real world while they’re still wearing the device. Facebook’s Oculus Quest platform, for instance, shows users a passthrough feed when they step outside their VR space’s boundaries. It’s useful for quickly dropping out of VR, and it can also enable a form of augmented reality by adding virtual objects to the camera feed. But as FRL notes, the people around a headset user can’t make eye contact, even if the wearer can see them perfectly. That’s awkward if bystanders are used to seeing their friend or co-worker’s uncovered face.
FRL scientist Nathan Matsuda decided to change this. A blog post explains that Matsuda started in 2019, when he mounted a 3D display onto an Oculus Rift S headset. The screen displayed a virtual rendering of his upper face, and custom-rigged eye-tracking cameras captured where Matsuda was looking, so his avatar’s eyes could point in the same direction. The result was basically Matsuda wearing a telepresence tablet showing a copy of his own face — which is arguably just as awkward but with a more intriguingly postmodern twist.
According to the blog post, FRL chief scientist Michael Abrash — quite understandably — didn’t find the idea very practical. “My first reaction was that it was kind of a goofy idea, a novelty at best,” he notes. “But I don’t tell researchers what to do, because you don’t get innovation without freedom to try new things.”
Matsuda ran with the concept, and over the next two years, he led a team in developing a svelter design. The team’s prototype headset — which it revealed ahead of next week’s SIGGRAPH conference — adds a stack of lenses and cameras to a standard VR headset display. The stereo cameras capture an image of the face and eyes inside the headset, and their motion is mapped onto a digital model of the face. Then, the image is projected onto an outward-facing light field display. That display creates the illusion of looking through the lenses of thick goggles and seeing a pair of eyes, although in reality you’re still seeing a real-time animated copy. If the wearer jumps back into full VR, the display can go blank to signal that they’re no longer engaging with the outside world.
The result is a pair of octagonal goggles that would look right at home in a Terry Gilliam film. FRL used a simple rendering of a virtual face, but it also showed off the system with its more realistic Codec Avatars, as seen below.
FRL acknowledges that the system’s individual components aren’t all revolutionary. HTC already has a face tracking add-on for its Vive Pro headsets; it maps movement onto an avatar inside VR, not an outward-facing screen, but the principle is similar. This week’s paper focuses on the potential of light field displays and the system’s opportunities for better in-person social interactions.
HoloLens-style projection glasses theoretically leave your face much clearer than passthrough displays — although a lot of those glasses have darkened lenses, and as Road to VR notes, projected light on clear lenses can also block your gaze. But as companies like Apple reportedly experiment with passthrough designs, Facebook’s new research shows that solid screen isn’t necessarily a barrier to eye contact… of a sort.
Facebook Reality Labs wants to help people see your eyes while you’re in virtual reality — even if the results sit somewhere between mildly unsettling and nightmarish. Earlier this week, FRL released a paper on “reverse passthrough VR,” a recipe for making VR headsets less physically isolating. Researchers devised a…
