Facebook AR Glasses Make ‘Superhuman’ Hearing Possible

Abhinav Raj, Writer
@uxconnections

Research science arm of Facebook is developing AR glasses that produce sounds ‘perceptually indistinguishable from reality’ and grant ‘perceptual superpowers’

At the junction of the Allegheny, Monongahela, and Ohio rivers is the small city of Pittsburgh in the state of Pennsylvania; the city that harbours the Facebook Reality Labs, led by a cohort of pioneers in Augmented and Virtual Reality. 

The social media giant’s research science arm is pursuing a special endeavour this time: one in the shared realm of audio engineering, VR, and AR. The team is a conflux of engineers, UX researchers, research scientists and designers on a unique mission—to ‘create virtual sounds that are perceptually indistinguishable from reality and to redefine human hearing’, along with another intriguing goal: superhuman auditory perception. 

With one perseverant driving principle, Facebook marches forth: to give people the tools to feel connected anytime, anywhere.

The Real and the Virtually Real

A headset allows for an immersive VR experience. (Image: Jessica Lewis on Unsplash)

Facebook envisions a future where innovations in virtual and augmented reality will yield important tools which will integrate into the lives of people, allowing them to achieve great feats. These innovations, according to the social media giant, will become as ‘universal and essential’ as modern-day personal computers and smartphones. 

“In the future, our AR glasses will merge the physical and digital worlds, blending what’s real with what’s possible, resulting in the next mainstream, must-have, wearable consumer technology,” Facebook stated. 

At the interphase of VR, AR and true reality, Facebook Reality Labs have an ambitious goal: to obscure the line that demarcates what is real, and what is not. 

But what takes virtual reality a step closer to our own?

The Acoustics of Reality

Two FRL Research Scientists perform a demonstration. (Image: Facebook Reality Labs)

Could it be possible to focus on one singular source of sound, whilst simultaneously cancelling out the background noise, in a consumer electronic product?

9 months ago I would have said ‘Yes’, but with reservations. However, in October of 2019, Apple came out with a product that took away my inhibitions. The AirPods Pro featured Active Noise Cancellation, a feature that allowed Apple’s $249 earbuds to ‘cancel’ unwanted sounds, achieved by detecting external sound through an outward-facing microphone and subsequently countering it with anti-noise; by the application of a physical phenomenon called destructive interference of waves.

FRL researchers have found the key ingredients that make for ‘realistic audio’ and make it indistinguishable from sounds produced in the real world.

A noise and vibration engineer tests products in an anechoic chamber. (Image: Unsplash)

The proximity of a person’s left or right ear from the emitter, the acoustics of the room a person is in, and even the shape of the ears influence the signals of one’s brain in deducing the source of a sound. If the core components are replicated, the resultant will be sound ‘perceptually indistinguishable’ from any real-world source.

Spatialized audio—sound with specified vectors mimicking a directional nature that give a listener the impression of being in a three-dimensional environment and high-quality acoustic simulation technologies allow the generation of high fidelity sound.  

FRL Research believes that its retrofitted AR glasses can allow a user to selectively prioritise and zero in on one sound source, like picking a focal point to enhance image quality on a camera. 

“By using multiple microphones on your glasses, we can capture the sounds around you. Then by using the pattern of your head and eye movements, we can figure out which of these sounds you’re most interested in hearing, without requiring you to robotically stare at it. This lets us enhance the right sounds for you and dim others, making sure that what you really want to hear is clear, even in loud background noise.”  

Through intricately engineered in-ear monitors (IEMs), FRL aims to integrate active noise cancellation to drown the din and amplify sounds relevant to the user. 

The AR glasses will enhance human perception: not just visual and auditory, but also through relaying information to one’s mind. By integrating LiveMaps— a VR-enabled map, one can navigate the real world through auditory and visual cues. 

Each day pioneers in science and technology push the frontiers of knowledge a little ahead, and craft tools that enable us to do more, and be greater than ourselves. 

The time is now: to ask not what technology can do, but to ask what you can do with it.