Virtual reality’s new tricks: How it fools your brain into having a “real” experience

Imagine looking down at Earth while you float through space. Hearing a shell screaming overhead from inside a military bunker. Feeling an ocean breeze on your skin. Inhaling the scent of the forest after the rain. All while standing in the same spot.

Virtual reality technology is evolving from merely strapping on a headset for a visual encounter to truly immersive experiences incorporating every human sense. This type of VR experience not only makes you feel like you're in a different world — it also tricks the most primal parts of your brain into believing it. Your higher consciousness might know you're standing in a VR kiosk at a mall, but your senses see, hear, feel and even smell a completely different environment.

Tico Ballagas, senior manager in the Artificial Intelligence and Emerging Compute Lab at HP, says VR will soon better replicate how humans experience and interact with the world — all being made possible technology that works in harmony with the brain.

“One of the high-level ideas floating out there, even outside of VR, is that we’re entering a new age,” says Ballagas. “We shifted from the Industrial Age to the Information Age, and now we’re transitioning to the Experience Age.”

Multisensory memory

Conjuring lifelike experiences in virtual reality requires a suspension of disbelief that demands a technologically complex, biologically-based illusion. “Our brains are wired for evolutionary reasons to pick up some things very acutely and quickly,” says Mayank Mehta, a professor of neurophysics in the departments of neurology, neurobiology, physics and astronomy at the University of California, Los Angeles, who has conducted extensive studies about VR. “For other things, the brain has evolved not to worry so much about it. People creating [VR] experiences must understand deeply where the corners can be cut, and where your brain will notice the tiniest mismatch.”

Your laptop screen is actually flickering constantly, refreshing 30 to 60 times a second or higher on more advanced gaming laptops. While the neurons in the brain respond to this flicker, our minds don’t perceive the infinitesimal pause. But by contrast, anyone who has watched a movie with the audio track lagging slightly behind the visuals knows the dissonance it creates.

Our evolutionary instincts must be fooled or else our brains will notice any misstep immediately. That means spatial audio that places realistic sounds in the right spots in relation to your body. Controllers that allow you to reach out and feel as if you’re touching an object that isn’t really there. Olfactory and taste elements that add evocative scents and flavors to your encounter.

“When we’re talking about multisensory experience, all of this needs to come together in harmony, because one error can shatter the illusion,” Mehta says.

Multisensory harmony is key to making virtual encounters feel like fully lived experiences — even triggering the brain to encode the experience as a memory. Joanna Popper, HP’s global head of virtual reality for location-based entertainment, recalls trying out BBC and Rewind's “Home — a VR Spacewalk” experience, which enables viewers to feel what it's like to float around the International Space Station.

“Later I went to a dinner where an astronaut showed a video of his own experience,” Popper says. “And I thought, ‘Yup, I’ve been there too!’ It took me a minute to step back and realize, no, of course I haven’t actually been on the ISS. But the VR experience had created a memory. And that memory felt real.”

Feel the impact

Those experiences require virtual actions to “feel” the real world using haptics, feedback technology that uses vibrations or motion. HP is developing spatial actuators — little devices arranged around the body that create tactile experiences like the feeling of wind — to augment its Z VR Backpack, which users can wear to feel vibrations, shakes and bursts of air that simulate movement.  

“I can feel the vibration of an explosion, the wind that comes off that impact,” Ballagas says. “When you have the sights, the sounds and this haptic sensation all together in a spatially representative way — you’re aligned sensorially. That alignment is how you create these really immersive experiences.”

These spatial haptics can help a paramedic experience realistic first-responder training, or help college and pro athletes simulate the physicality of a football game without risking injury. The tactile facet also boosts pure entertainment experiences like games, he adds, making these environments more realistic.

Reach out and touch it

Many existing VR experiences use handheld controllers that allow users to point at virtual objects and “grab” them by clicking a button or pulling a trigger. But that isn’t how we reach for or feel things in the real world. That’s why HP developed “finger-first” controllers called VR Grabbers, handheld devices that create a haptic, or touch-based, illusion. The user can interact more naturally using their fingers directly to grab a virtual object; the way the Grabbers hit together in the physical world makes the user feel as if they are actually picking up and moving the item.

“We changed the rate at which the grabbers were converging in virtual reality, such that they looked like they were converging around the virtual objects, when in physical reality they were just joining together,” Ballagas explains. “So you attribute the force of your fingers squeezing the grabbers to the force of the virtual objects, and it’s very effective.” Essentially, your hands “feel” what your eyes see.

The illusion is valuable for not only elevating entertainment experiences but also for training in activities that require tools: assembling a circuit board, performing surgery or even deactivating a bomb.

Hear something in the (virtual) bushes

Audio is already part of most VR experiences, but truly realistic sound involves more than what can be broadcast through headphones. Immersion requires spatial audio, which simulates sounds coming from various directions and distances.

Ballagas echoes Mehta’s point about our brains acutely perceiving a mismatch in sound: “We’re very adept at localizing sounds and distinguishing subtle differences,” he says. “We know where a sound source should have come from, and if it isn’t right, we notice.”

Companies such as Google, Oculus and Adobe include spatial audio in their hardware and experiences where sound is localized correctly to match the way sound naturally travels from one ear to the other. But individual head width and ear shape can affect how each person receives these sounds.

HP is working on new technology to create even more personalized spatial audio, as part of software to be released later this year. 

While Ballagas can’t share details just yet, he says the technology is based on “specific attention to how we create HTRFs,” or head-related transfer functions, which determine how our ears receive a sound from a point in space.

Breathe the forest air

Our sense of smell is linked to the amygdala, the part of the brain that helps manage emotion and memory, which is why scents are so powerful. But creating the experience in VR can be challenging, in part because the chemicals required to create scents can linger or mingle with other scents. 

Grace Boyle, an artist who founded the London-based collective The Feelies that creates original multisensory XR (extended reality) content, says scent is an important part of the VR experience. “A scent can immediately transport us in place and time, Boyle says. “That power cannot be ignored.”

The Feelies work with perfumers to create bespoke scents that provide what they call “olfactory input.” Their scent units for “We Live in an Ocean of Air,” a deeply detailed virtual reality experience at London’s Saatchi Gallery earlier this year from the collective Marshmallow Laser Feast, are delivered at key points in the narrative. 

"We Live in an Ocean of Air" puts patrons in a redwood forest to explore a massive sequoia tree, hear a rainstorm and watch their breath transform from oxygen to carbon dioxide — all while inhaling the earthy scent of the forest. The scent units designed by Boyle were based around an olfactory analysis of the fragrant air in a sequoia grove and delivered across the experience using a machine the Feelies designed, with each smell a controlled release that didn’t interfere with the one before.

“As you rise up visually and the audio track swells and you smell the air, all of these senses come together. The tone of the scents also rise to end on an open, fresh, airy note, like a musical chord,” says Boyle. “We need to design experiences with every sense on an even playing field — visual should not be elevated over any other. This is what it takes to reach an altered state of being.”

New realities in training

That altered state can be a powerful tool. Imagine a firefighter in training: In the real world, the stakes could be high on that first call, attacking her senses and stoking the fear centers of her brain. Training with a VR experience can include all the intensity and detail of the real thing, without the danger. 

In VR, if she doesn’t get it right the first time, no one gets hurt. Yet she still retains an evocative, sense-laden memory that’s more vivid and instructive than what any written manual or traditional simulation could impart. 

Other types of workers benefit from VR training as well. Popper says that when manufacturing company Siemens used HP technology in a training experience for staffers working on manufacturing a gas turbine, they saw a 60 percent reduction in training time needed. HP itself is using VR technology to help improve the training of large-format printer technicians, with the goal of reducing training time and costs. 

“Data on training shows that people who read a manual retain about 5 percent, and those who listen to a presentation retain 10 percent,” Popper says. “With virtual reality, they retain 75 percent. Do you want the surgeon who is set to operate on you to recall 5 percent or 75 percent?”

The experience evolution

Every new technology, from the telephone to the internet, has changed the way humans understand and interact with the world around them. Mehta, the UCLA neurophysicist, says virtual reality is poised to create an even greater shift than the tech advances that came before it because of how it manipulates our brains’ responses and our minds’ perceptions throughout every second of the experience. 

“It’s an experiment that has never happened in the human experience — until now,” he says. 

Virtual reality taps into evolutionary biology to make our brains believe, with all five senses and every fiber of our beings.

“For the first time ever, you’re able to package, transmit and communicate using experiences as the medium,” says Ballagas. “VR has this ability to conjure experiences at any moment for anybody, and that creates a new way that we as a human race can interact.”

VR, AR and 3D printing are revolutionizing how doctors prepare for and perform surgery.