Facebook wants its AR glasses to give wearers superhearing
A new Facebook experiment shows how sound could play a major role in augmented reality. Chief scientist Michael Abrash and his team at FRL Research (formerly Facebook Reality Labs) released details today about what the team calls “perceptual superpowers” — AR systems that figure out what you’re trying to hear, then amplify it and dampen background noise. Combined with spatial audio features, the device creates the aural equivalent of a hologram overlay in a pair of glasses.
Facebook has pursued high-quality virtual sound for years, largely through its Oculus virtual reality headsets. FRL Research’s latest work focuses on AR applications. To give one example, “imagine being able to hold a conversation in a crowded restaurant or bar without having to raise your voice to be heard or straining to understand what others are saying,” the company explains.
AR glasses could do this by picking up audio with microphones, using contextual clues to gauge which sounds are important, and feeding those sounds through a noise-canceling earpiece. Conversely, if you’re on a phone or video call, improved spatial sound could project participants’ voices or other audio to specific parts of the room, increasing the sense that you’re really with somebody else — or “audio presence,” in FRL Research’s terms.
As Facebook acknowledges, the lab’s “perceptual superpowers” pitch is very similar to the function of existing hearing aids, which also amplify sound and reduce background noise. (One experimental system even uses brain implants to focus on specific voices.)
AR glasses, however, present unique opportunities for people with and without impaired hearing. Directional sensors and outward-facing or eye-tracking cameras can collect detailed contextual information, including your bodily orientation and where your gaze is focused. That helps the earpieces pinpoint an “acoustic spotlight” to amplify. And, of course, an AR system can mix its own soundscape with the amplified audio.
Facebook could also complement the system with its other AR initiatives. LiveMaps, for instance, aims to create rich maps full of information about people’s surroundings. If LiveMaps detects that someone is wearing glasses in a restaurant, the glasses could do something like automatically flag and cancel clinking silverware sounds.
Facebook released a photo of a prototype in-ear monitor on a dummy head as well as a picture of team members wearing headphones that look more like Valve’s off-ear Index speakers. The Oculus Quest VR headset uses directed speakers, which are a convenient one-size-fits-all solution for projecting sound into an environment, but leak sound to bystanders — while in-ear monitors may be better for changing overall sound levels while wearing AR glasses in public. “What form factor to use to solve a problem ultimately depends on the application,” says FRL Research audio lead Ravish Mehra.
Other companies have experimented with aural augmentation — most prominently Bose, which revealed audio-only AR glasses in 2018. Bose’s glasses used spatial sensors to detect where wearers were looking and offer features like virtual street tours. The company failed to make them commercially viable, though, and it abandoned the project this June.
Unlike in previous years, Facebook isn’t holding conferences where it can show off experimental tech in 2020, so we haven’t heard its audio in person. FRL Research offered reporters a demonstration of its filtering system on a remote call. And any spatial audio improvements could soon be applied to VR headsets, including a rumored Oculus Quest update around this month’s Facebook Connect conference. “The work that we are doing and the work that we presented today definitely has applications for our VR line of devices,” says Mehra.
A full application of these capabilities could be years away. We don’t know much about Facebook’s roadmap for consumer AR glasses, even if we’re slowly learning more about what the company thinks they ought to do. A couple of pictures show plastic frames, but they’re designed to hold the microphone array, not work as full-fledged glasses. Facebook has previously said it’s built multiple prototype variants, including a recent combined AR / VR headset design that looks like a pair of sunglasses.
To sell a real commercial product, Facebook will need to allay concerns that go beyond the technical. That includes convincing people that “perceptual superpowers” don’t pose a privacy threat.
Without any limitations, sufficiently powerful microphones and filtering tech could let people imperceptibly eavesdrop on conversations across a public space. AI audio analysis presents even weirder and more alarming possibilities, like the ability to flag specific voices or conversation keywords in a crowded room. (It could also theoretically perform more innocuous, helpful tasks like real-time translation.) And, of course, the glasses would be recording your own conversations as well.
There’s no indication that FRL Research is considering deliberate surveillance. But many people already fear — probably incorrectly — that Facebook’s apps are recording them to target ads. So the mere possibility may alarm consumers. FRL Research says that captured audio is currently encrypted and stored on servers that a small number of researchers can access.
Abrash says that Facebook is attempting to build privacy into all aspects of its glasses design, echoing a refrain about its overall AR / VR plans. Among other things, the system could ask for permission from someone else’s glasses before amplifying a conversation, or it could have a limited range. “Rather than thinking of this as a magic flashlight we can point at anybody and hear what they’re saying, think about it more as people being able to participate in the conversation they’re in anyway,” he told reporters.
And for now, Abrash says, the tiny microphone arrays aren’t powerful enough to conduct long-range spying. “If you ever see anyone walking with glasses that are two feet wide,” he cautioned jokingly, “you should be suspicious.”
A new Facebook experiment shows how sound could play a major role in augmented reality. Chief scientist Michael Abrash and his team at FRL Research (formerly Facebook Reality Labs) released details today about what the team calls “perceptual superpowers” — AR systems that figure out what you’re trying to hear,…
