Facebook is really, really serious about augmented reality. Last year, the company showed off a frankly ambitious framework for the future of AR, and now the social media giant has revealed some of the advanced gadgets it’s designing to help people interact with the AR world.
The Facebook Reality Labs Research team’s biggest challenge is finding ways to interact with augmented reality the way we do with a PC. We have a number of headsets and glasses, but no AR equivalent to a mouse and keyboard.
So instead of trying to make existing devices work in AR, Facebook is looking to create new types of human-computer interfaces (HCIs) that are easy to use, reliable, and still provide some level of privacy. Facebook has said it envisions AI as a critical part of the formula to help provide you with the right tools or commands depending on the situation, which should help reduce friction or possible user confusion.
And while this tech is far from being polished, Facebook already has some ideas about how AR-based HCI devices might work in the future. Instead of relying completely on voice commands, Facebook sees wrist-mounted wearables as a good solution, offering a familiar and comfortable design not completely dissimilar to a standard wristwatch, but with new tech that can support various input methods.
Facebook says that by leveraging electromyography, it can use sensors to convert electrical signals that get sent from your brain to your hands into digital commands. Facebook claims EMG sensors are sensitive enough to detect movements of just one millimeter, with future devices potentially even being able to sense someone’s intentions without any actual physical movement. In essence, Facebook is looking to provide direct mind control of AR devices, but without the need for physical implants.
Further, with precise EMG sensors, Facebook can also support new gesture controls, like pinching your thumb and index fingers together to create a “click.” In this way, people can translate what they do on a regular PC into a new set of AR-based gestures that Facebook someday hopes to expand into all sorts of controls and movements. Facebook even hopes to reimagine typing with the help of AI to making writing essays or emails faster and more accurate.
Facebook said it knows that all these technologies will need to evolve with each other, because simply being able to click on an AR object won’t be enough if the rest of the AR interface is constantly getting in the way. And once again, Facebook thinks AI can help, by intelligently knowing when you want to switch virtual workspaces or focus on a specific tool or getting additional input from EMG sensors or even eye-tracking sensors.
Although touchscreens and virtual screens are useful, there’s simply no replacement for real physical stimulus. So in addition to touching something with your fingers, Facebook just showed off two different prototypes that deliver haptics in interesting ways.
With its “Bellowband” prototype, Facebook uses a string of eight pneumatic pumps attached to a wrist-mounted device that blow air and create various pressure and vibrations patterns. When combined with its Tasbi prototype (Tactile and Squeeze Bracelet Interface), Facebook has been able to create a device that squeezes your wrist to better mimic the sensation of moving or touching real objects.
The biggest issue, of course, is that Facebook’s track record on privacy is, well...we all know it’s not great. The company said safeguarding people’s data in AR is critically important, though Facebook Reality Labs science director Sean Keller added that “understanding and solving the full extent of ethical issues requires society-level engagement.” In short, Facebook needs feedback on how to improve privacy and security in AR (surprise, surprise), and is encouraging its researchers to publish relevant work in peer-reviewed journals.
Admittedly, while all of this does sound pretty far-flung, given the speed at which VR was adopted by certain sectors of business like engineering and design, it’s not that outlandish to imagine AR seeing similarly explosive growth over the next 10 to 15 years. And, as in other industries, if you’re the first company to define and control a market, there’s a good chance profits will follow. You can rest assured Facebook is going to do its best to try to stay ahead of competitors—but it sounds like Microsoft, Apple, and the rest all have the same idea. Let the games begin.