In the week since its release as a free smartphone app, Pokémon Go has sent millions of people worldwide on digital scavenger hunts to hunt down and collect cartoon characters. It uses the camera and GPS system on an Android or iPhone handset to digitally superimpose these animated creatures on top of whatever scenery appears on the smartphone’s screen when its camera scans one’s surroundings. That might sound a lot like augmented reality, but one pioneer in that field prefers to call Pokémon GO and games like it “location-based entertainment.” Just about everything written about Pokémon GO since its July 6 release positions the game as a triumph of augmented reality—technology that overlays digital images atop the real world—on a smartphone. Perhaps it is just splitting hairs but Ken Perlin, a computer science professor and founding director of the New York University Media Research Lab, makes a distinction between simply dropping digital characters onto a screen based on a player’s location and integrating those characters into their surroundings so that they seem more real than virtual. Perlin’s experience with virtual and augmented reality extends back decades, and he has won several awards—including an Academy Award—for inventing ways to improve computer graphics. Perlin is not criticizing Pokémon GO, which has introduced a lot of people to the basic idea of augmented reality in a short period of time. Instead, he says interest in the game provides an opportunity to look at where augmented reality could go in the near future. [An edited transcript of the interview follows.] If Pokémon GO is not augmented reality, what is it? The term “augmented reality” has been applied to a number of different technologies over the past few decades—including Google Glass—that allow you to see text and images floating in front of you. There were likewise some popular smartphone apps from a few years ago that let you superimpose images over reality. But I think the push by Microsoft with its HoloLens technology as well as new headsets from companies including Meta and magic leap, [Inc.] will really show people how virtual objects can be integrated into real-world surroundings. To the point where users will, for example, see that there’s a virtual object sitting on a table and that object will remain where it is with respect to the user’s position as he or she moves. These new technologies are really raising expectations for augmented and mixed reality beyond holding your phone out to see a creature on your street corner. Games like Pokémon GO are fun but they’re really more location-based entertainment than augmented reality. In what other ways are the new augmented reality technologies distinct from what Pokémon GO offers? There is a fundamental difference between just slapping a label in front of something because I’m there and actually altering our perception of reality. If you or I are walking around and we’re seeing a creature through a wearable device over our eyes—as opposed to seeing it through our phone screens at arm’s length—then that creature has become part of our shared perception of reality. There’s a fundamental difference between our brains integrating objects into reality and simply being told that something is part of reality. What is mixed reality? There’s this debate going on right now about what to call augmented reality and what to call mixed reality. One reason it gets so complicated is that some of it is pure commercial marketing. Microsoft is trying to really push the fact that its HoloLens device is very accurate in where it overlays virtual objects on top of what a person sees through the headset. As a result Microsoft throws around the term “augmented reality” derisively to kind of say, “That’s what Google Glass did.” Instead, Microsoft is tying HoloLens to the term “mixed reality”. Mixed reality might actually win out because it’s less loaded. It’s also the term Jaron Lanier used back in the mid 1980s to describe a virtual reality that’s combined with what you see in the real world. I’ve started to switch over and use mixed reality because it confuses people less. What impact will this Pokémon GO phenomenon have on the future of augmented—or mixed—reality? Pokémon GO’s popularity is really important because [game developer Niantic, Inc.] is using a very major commercial brand that people know and love to make augmented reality—although I still prefer to call it location-based entertainment—accessible to anyone with a smartphone. We’ve seen this before: Some technological innovation happens and early adopters get all excited, but it’s not until some large brand gets involved that the technology grabs the attention of hundreds of millions of people. For example, we’ve had the illusion known as Pepper’s ghost since the 1800s. But it wasn’t until they used that same trick to put Tupac Shakur on stage [at the 2012 Coachella Music and Arts Festival] and started pretending it was a hologram that so many people became aware of Pepper’s ghost. And Ingress—the massively multiplayer online location-based game that Niantic released in 2013—has only a fraction of the number of players that Pokémon GO accumulated in just a few days. Pokémon GO uses essentially the same platform as Ingress, yet it has become a huge phenomenon.
Just about everything written about Pokémon GO since its July 6 release positions the game as a triumph of augmented reality—technology that overlays digital images atop the real world—on a smartphone. Perhaps it is just splitting hairs but Ken Perlin, a computer science professor and founding director of the New York University Media Research Lab, makes a distinction between simply dropping digital characters onto a screen based on a player’s location and integrating those characters into their surroundings so that they seem more real than virtual. Perlin’s experience with virtual and augmented reality extends back decades, and he has won several awards—including an Academy Award—for inventing ways to improve computer graphics.
Perlin is not criticizing Pokémon GO, which has introduced a lot of people to the basic idea of augmented reality in a short period of time. Instead, he says interest in the game provides an opportunity to look at where augmented reality could go in the near future.
[An edited transcript of the interview follows.]
If Pokémon GO is not augmented reality, what is it? The term “augmented reality” has been applied to a number of different technologies over the past few decades—including Google Glass—that allow you to see text and images floating in front of you. There were likewise some popular smartphone apps from a few years ago that let you superimpose images over reality. But I think the push by Microsoft with its HoloLens technology as well as new headsets from companies including Meta and magic leap, [Inc.] will really show people how virtual objects can be integrated into real-world surroundings. To the point where users will, for example, see that there’s a virtual object sitting on a table and that object will remain where it is with respect to the user’s position as he or she moves. These new technologies are really raising expectations for augmented and mixed reality beyond holding your phone out to see a creature on your street corner. Games like Pokémon GO are fun but they’re really more location-based entertainment than augmented reality.
In what other ways are the new augmented reality technologies distinct from what Pokémon GO offers? There is a fundamental difference between just slapping a label in front of something because I’m there and actually altering our perception of reality. If you or I are walking around and we’re seeing a creature through a wearable device over our eyes—as opposed to seeing it through our phone screens at arm’s length—then that creature has become part of our shared perception of reality. There’s a fundamental difference between our brains integrating objects into reality and simply being told that something is part of reality.
What is mixed reality? There’s this debate going on right now about what to call augmented reality and what to call mixed reality. One reason it gets so complicated is that some of it is pure commercial marketing. Microsoft is trying to really push the fact that its HoloLens device is very accurate in where it overlays virtual objects on top of what a person sees through the headset. As a result Microsoft throws around the term “augmented reality” derisively to kind of say, “That’s what Google Glass did.” Instead, Microsoft is tying HoloLens to the term “mixed reality”. Mixed reality might actually win out because it’s less loaded. It’s also the term Jaron Lanier used back in the mid 1980s to describe a virtual reality that’s combined with what you see in the real world. I’ve started to switch over and use mixed reality because it confuses people less.
What impact will this Pokémon GO phenomenon have on the future of augmented—or mixed—reality? Pokémon GO’s popularity is really important because [game developer Niantic, Inc.] is using a very major commercial brand that people know and love to make augmented reality—although I still prefer to call it location-based entertainment—accessible to anyone with a smartphone. We’ve seen this before: Some technological innovation happens and early adopters get all excited, but it’s not until some large brand gets involved that the technology grabs the attention of hundreds of millions of people. For example, we’ve had the illusion known as Pepper’s ghost since the 1800s. But it wasn’t until they used that same trick to put Tupac Shakur on stage [at the 2012 Coachella Music and Arts Festival] and started pretending it was a hologram that so many people became aware of Pepper’s ghost. And Ingress—the massively multiplayer online location-based game that Niantic released in 2013—has only a fraction of the number of players that Pokémon GO accumulated in just a few days. Pokémon GO uses essentially the same platform as Ingress, yet it has become a huge phenomenon.
