Michael Fassbender’s soft lilt is in my ear spouting nonsense about assassins and their creed. Around me is a whirl of action set against a cartoonish backdrop. I’m in the new Assassin’s Creed virtual experience, and thanks to a blend of low-res graphics and high-res people, it feels like I’ve found my way inside some lost Mortal Kombat game circa 1993.
I don’t have any control. The camera, (and I) race down the corridor of a medieval castle. We skitter to a stop at the edge of a turret. I don’t have a body—I hover over the ground like a spectre—so there’s no way to fully comprehend how close to the edge my virtual self is.
Then the camera is lurching forward as my ghostly form is thrown off the turret. I’m seated, but my stomach is in my throat. At the moment my lunch lasagna is about to reacquaint itself with my mouth, the screen goes black. The experience is over.
I rip off the headset and gulp in a huge lungful of air. The world stops spinning long enough for me to catch the eye of one of the creators. He’s grinning, ecstatic with his debut of the Assassin’s Creed experience, and I’m nauseous and trying to be polite.
This is a carefully designed virtual experience. It was shot on the same locations as the big budget movie, and intended to get the average person excited about virtual reality—a surprisingly daunting task. This is, in many respects, the best that VR has to offer. And as the nausea tries to settle in my stomach one thing is painfully clear.
Our bodies aren’t ready for it.
How you experience your place in the physical world is determined by your eyes and your vestibular system, which is primarily composed of a complex series of bone structures in your inner ear. Spinning quickly in a circle or standing up too fast can wreak havoc with this delicate arrangement. Damage to the system creates vertigo. It’s hyper sensitive and does not like to be tricked—yet that’s exactly what VR does.
It’s why my hosts had me sit down before I put on the headset to experience Assassin’s Creed. The experience mimics an extremely popular demo for VR: the roller coaster. It’s a method of showing off how immersive VR can be, because it directly affects the vestibular system. Plunging down a hill or racing around a sharp corner in VR puts your stomach in your throat almost as accurately as if you were on the real thing.
But while you might be exhilarated, your body is wildly compensating for what your vestibular system is perceiving. Sitting down reminds your body that no, you are not racing around a curve at 60 miles per hour. It also allows you to consciously ground yourself. Because if you don’t ground yourself, if your conscious and unconscious body is not prepared for the experience, then things can get messy quickly.
The poor Russian man above is riding a roller coaster in VR, and because VR is shockingly realistic to your vestibular system, he’s reacting to every climb, turn, and descent. He’s also wearing an Oculus Rift DK1. That’s the first major consumer VR headset, released back in 2013. It’s an extraordinary piece of gear, that, according to Oculus co-founder and former Chief Engineer Jack McCauley, is on par with today’s most popular headsets—the mobile kind like the Google Daydream and Samsung Gear VR. Unfortunately, both the DK1 and current mobile headset have one crucial problem: They’re lacking in what VR engineers like to call “localization.”
A far cry from how the term is used with software developers (it’s the procedure employed to translate software for specific countries and locales), VR localization is about tracking the body in a virtual space. Some current systems do it pretty well, tracking not only where your head turns and how fast it turns, but where you exist within the confines of the virtual experience. If shoved, like the terrified man above, you lurch forward in the virtual space too.
Screaming VR Man doesn’t have that luxury. When he’s pushed, his eyes don’t see a corresponding movement. Instead it’s like he’s strapped down, unable to move while being violently (from his perception) assaulted. VR is facing an uphill climb because no one has truly, perfectly unlocked localization, directly mimicking the relationship between the eyes and vestibular system in the real world.
“When I was working at Oculus, I took it upon myself to do research into the kind of affects of VR,” McCauley told Gizmodo. He was one of the few. The rest of the company, passionate gamers eager to realize the dreams promised by Star Trek’s Holodeck and William Gibson’s cyberpunk novels, were focused on creating a cool product that wowed the senses. The potential nausea, vomiting, sweating, fatigue, and disorientation were concerns but not a priority.
McCauley, who, as Chief Engineer on the DK1 and DK2, spent a lot of time trying to figure out the relationship between VR, our eyes, and our vestibular systems, noted that there still isn’t enough research being done outside the military.
“I don’t think we understand enough about how the brain works and the vestibular and ocular system,” McCauley said. Their relationship with VR remains a mystery, and until it’s solved there’s always going to be a threat to our bodies. One wrong slip off the stool or push by a friend and you could wind up like that poor Russian man, screaming into the virtual abyss.
As McCauley notes, different audiences have very different expectations when slipping on a VR headset: Non-gamers can better tolerate low frame rates and artifacting, while regular gamers won’t. Gamers make up the bulk of the people currently developing VR, and everything from mobile headsets for your phone to the $800 ones you plug into your computer are being tailored to them. As someone with three different consoles, too many hours thrown into Overwatch, and a fond memory of Virtual Mario, I was one of the gamers harboring an unconscious need for speed.
The first time I tried VR, I was using an old gaming laptop and an Oculus Rift DK2. I didn’t pay attention to minimum spec requirements for a PC. I didn’t think of how a slower PC might dramatically change the VR experience. In my mind it would be as simple as mirroring the screen on the laptop to the two in the headset—the chunky behemoth with its mobile video card would be enough.
Then I tried playing Alien: Isolation. The computer’s fans hummed angrily as it struggled to output video to the headset. It was having to power not one, but two 1080p displays inside the headset, while immediately rendering a whole new world with every quick turn of my head. The alien’s tail swished across the screen at 60 frames per second. That’s more than adequate when you’re sitting three feet back, but the headset was trying to replicate the real world from only a few inches away. That demands, for a gamer, at least 90 frames per second.
When the refresh rate goes below that, VR sickness can set in, and it’s directly tied to how often you move your head. While a high refresh rate might be coveted by gamers, regular joes will find it wanting, particularly if they plan to watch any live action content on it. That’s because all the content we watch on our phones and TVs is coming in at around 24 to 30 frames per second. It gives content, especially content in motion, a pleasant “cinematic” blur that’s impossible to replicate with current VR headsets. Increase that frame rate and actors look like goofy jackasses in a cheap soap opera.
With VR, we’re going to have to adjust our taste and reconfigure what our brains immediately consider “good” cinematography. Filmmakers are involved in this as well. Ang Lee shot this year’s Billy Lyn’s Long Half Time Walk in 120 fps, James Cameron is in talks to shoot the next Avatar in anything from 48 to 240 fps, and Peter Jackson famously got a critical trouncing for shooting The Hobbit in 48 frames per second. Unfortunately, audiences and critics don’t actually like watching the movies at these speeds, except as an oddity.
Images are so crisp and clear in higher frames per second, which is great for exploring a video game, but it makes live action content feel like it’s being performed on a stage. Translating that stage to the equivalent of a 105-inch 360-degree screen demands a whole other level of adjustment from audiences.
Yet it won’t just be eyes and tastes that need to adjust to make VR more viable. It’s the mind too.
Virtual reality sits on the cusp now of what is real and what we perceive to be real. It’s the uncanny valley writ large. An all-together disturbing experience too real for our bodies to ignore, but too fake for our minds to believe.
But things are changing. Back when the first full flight simulators for pilots went online, researchers noticed a problem. Pilots who had logged hours in an actual airplane were getting sick when flying in a sim, while pilots who had logged hours in a sim would get sick when they flew a plane. It’s called simulator sickness, is virtually identical to VR motion sickness, and might be the most profound problem for VR to overcome.
Our bodies are so incredibly in tune with our surroundings that they know, on a neurological level, when something is off. That’s what is at the core of simulator sickness. The pilots were so attuned to one particular extraordinary experience that when they moved to a similar, but just different enough one, they became ill.
Even if you do overcome simulator sickness, you could be in trouble when you return to the real world. “There seems to be some kind of training going on in your head,” McCauley told Gizmodo. If you spend enough time in VR your body could retrain itself. The real world would become the nauseating one.
The current VR market is being driven by gamers. Palmer Luckey, Jack McCauley, and their fellow Oculus founders developed the headset and transformed the VR industry into what it is today because they were enthusiastic gamers ready to drag the world into the virtual promised land.
The only problem is that this virtual promised land isn’t as ready as gamers would like it to be. It doesn’t matter how rarely you get motion sickness in the real world, the virtual one is an inescapable assault on the human body. Its successes come from tricking the mind and confounding the eyes. But our bodies can perceive the unnatural state of virtual worlds, and when they do they try to protect us, manifesting a feeling of wrongness that makes us feel sick. The problem lies in the VR technology itself. It’s just not good enough to make our bodies believe.