Forget virtual reality, for now at least, because augmented reality, or AR as it is commonly known, is powering the next batch of magic tricks heading to your phone. Apple and Google are pushing the tech hard, but what’s actually new about the next wave of AR? What’s changed since Pontiac stuff it in the ugly Aztek or Niantic had you catching Pokemon with it in Pokemon Go? And what are you going to be able to do with it...besides game?
AR, VR, and MR
Augmented reality (AR) is just that—reality augmented with some extra digital graphics, like the simple overlays you might see if you put on Google Glass. Virtual reality (VR), on the other hand, is all digital, a completely enclosed virtual space.
Mixed reality—a term which Microsoft is particularly keen on—is a mix of AR and VR, where those digital overlays are designed to look as if they’re part of the real world, though a combination of advanced sensors and high-end computing power. Think HoloLens, and you’ve got it.
Unfortunately for Microsoft and other fans of the mixed reality label, the term augmented reality is now being more broadly used to cover mixed reality too. Apple is using it, Google is using it, and most people just don’t care about the distinction between computer graphics that float in front of your face and computer graphics that look like they’re part of the scene in front of you.
Technically speaking, the new wave of AR hitting phones is MR or mixed reality—think Pokémon Go, but much more realistic. However, as Apple has ARKit, and Google has ARCore, we’ll stick to using the term augmented reality here.
Project Tango, ARKit, and ARCore
The first serious push for AR on mobile devices was with Project Tango, though Google got in the game so early that it took years for the relevant sensors and computing kit to become small enough to fit in a phone—and even then it was a bulky phone. Getting consumers to sacrifice a slim and light phone for a technology that wasn’t fully baked has been a tough ask so far.
Getting AR right to a high standard of quality means collecting a lot of data about the immediate environment. Fancy camera modules, like the one in Project Tango devices, help, but if a phone is powerful enough to crunch a lot of data in real time you don’t necessarily need the module for basic, cool, AR. That is why we’re just now seeing phones that are capable of advanced AR using standard-sized cameras and sensors.
While Google planted its AR flag in the sand first, it was Apple that sparked a new wave of interest in the tech with the launch of ARKit at WWDC 2017. ARKit is “a new framework that allows you to create unparalleled augmented reality experiences for iPhone and iPad”, which means it makes it easier for developers to get their 3D graphics into the real world.
How developers go about it is up to them, but Apple has already added some AR features to its Maps app, so you can stomp around your local city as if you were a 160-foot giant Japanese sea monster. The trick with pulling off good AR is having a phone camera that sees 3D space as well as 2D images, but Apple has managed to do the necessary magic in software—ARKit apps will work in the iPhone 6s and any newer iPhone running iOS 11 (essentially you need the A9 CPU or newer).
That said, it’s likely to be better with some cameras than others. Apple says the iPhone 8, iPhone 8 Plus, and iPhone X cameras are “calibrated” for use with AR. The module on the front of the iPhone X is perfectly capable of viewing subjects in 3D, but we’re not exactly sure yet how all the other 2D cameras on the new iPhones are calibrated for a better AR experience.
When images are captured in 3D the effect can be pretty fantastic. Take Snapchat face filters, for example: They already work with any camera, but at the iPhone launch Craig Federighi used the iPhone X to show off filters that looked like they were actually stuck to his face.
Of course the universe is in danger of overbalancing if Google or Apple have something that the other doesn’t, whether it’s mapping apps or mobile payment tech, and so we now have ARCore as well, which is the Android equivalent of ARKit. Again, it does the heavy lifting of working out where a phone is and what the environment is like around it, so that developers can plug into it. It’s currently in beta, with only the Google Pixel, Pixel XL, and Samsung Galaxy S8 supported for now.
What AR can do for you
AR apps aren’t new, but the technologies we’ve described above are about to make them much more effective at blending the real with the virtual. So while you can already get an app to drop a rolling car into the scene in front of you, soon that car will match the color and lighting of what your camera is showing—these apps will actually be able to see the flat surfaces and the shadows of the real world in front of you.
In our review of the Asus ZenFone AR we said that “every app feels like a tech demo”, and that’s the challenge for Android and iOS developers now, to create compelling AR experiences with the new technology. At its iPhone launch, Apple showed off an AR game where the action played out on a table—a fine demo, but would you rather play your mobile games walking around the room with your phone up to your face, or comfortably reclining on the couch?
Pokémon Go is a perfect example of a game that got the AR mix just right, and we also know Ikea is working on an ARKit app to let you see how furniture would look in your home before you actually take it there—apps like this already exist (Wayfair has been building AR furniture apps for a while), but the new wave of AR should make them much more accurate, accessible, and useful.
Apple’s Phil Schiller showed off an app that can map constellations to their correct position in the sky, another app trick we’ve seen before but which should improve with ARKit. For a better overview of what devs are already working on though, point your browser towards the Made With ARKit site.
You can see all kinds of ways in which the latest hardware and software makes AR elements more realistic than ever—from a car driving down a road to seeing menu items up close on your plate before you order. If you’ve got a classic 1980s music video that you want to see brought to life, well then ARKit can do that too.
The majority of these apps are still at the demo stage for now, but it’s not difficult to see the potential. Games can take place in the real world, whether that’s on your living room floor or around your neighborhood, while you’ll be able to imagine anything from a new sofa to a new house extension through your phone’s camera.
Further down the line you’ll be able to point your phone at anything and get information overlaid about what you’re seeing with a whole new level of accuracy—the distance to the pin on a golf course, the current stats for players out on a baseball field, or precise directions to the store you need in the mall you’ve just got lost in. You can measure distances, land a rocket in your backyard, and so much more.
Of course AR (or “mixed reality”) headsets are going to provide a much more immersive experience and are a more natural fit than having to hold your phone or tablet in front of you to see anything in AR land. Until those headsets hit the mainstream though, ARKit and ARCore are about to introduce millions of people to the potential of augmented reality. Instead of looking down at your phone you’re going to be holding it up to access all the information of the internet with a swipe.