Apple gets patent for augmented reality on iPhones

It’s not a secret anymore. Apple plans for augmented reality are confirmed now with the US Patent Department granting it patent on “augmented reality maps”. This comes close on the heels of the IT giant obtaining a patent for mobile mapping on its iPhones on September 27.

(READ: Apple gets patent for mobile mapping in iPhones)

The US Patent and Trademark Office published Apple’s U.S. Patent No. 9,488,488 for “Augmented reality maps” on November 8 describing it as an mapping app capable of tapping into iPhone’s advanced sensor suite to present users with real-time augmented views of their surrounding environment.  It is said the technology will be used for navigation. Users will be able to choose the destination by watching the surrounding environment through their iPhone’s camera and calculate distance with the help of GPS coordinates.

Apple plans for augmented reality

The user can instruct the device to initiate a live video feed using an onboard camera and display the captured video images on a display. Using a GPS device, a digital compass, and optionally, an accelerometer, location, camera direction, and orientation information can be determined. Following this, the device can request data describing the surrounding areas and the objects therein. In some embodiments, this data includes map vector data. The can be requested from an onboard memory or a server. The data describing surrounding areas can further be requested in conjunction with a search request. The search request can also include a request for information about nearby places of interest.

The patent doesn’t name iPhone, but says by interpreting the data describing the surrounding areas, the device (presumably iPhone here) can determine what objects are presently being viewed on the display. A user can further use his smartphone to overlay information regarding the presently viewed objects, thus enhancing reality. In some embodiments, the device can also display search results overlaid onto the displayed video feed. Search results need not be actually viewable by a user in real life and instead can also include more-distant objects.

How the technology will work

The patent document describes the entire process: “a user points a handheld communication device to capture and display a real-time video stream. The handheld communication device detects geographic position, camera direction, and tilt of the image capture device. The user sends a search request to a server for nearby points of interest. The handheld communication device receives search results based on the search request, geographic position, camera direction, and tilt of the handheld communication device. The handheld communication device visually augments the captured video stream with data related to each point of interest. The user then selects a point of interest to visit. The handheld communication device visually augments the captured video stream with a directional map to a selected point of interest in response to the user input.”

The new technology will also enable the smartphone user to receive an input from the user requesting directions from a present location to a selected search result. Directions can be overlaid onto the video feed, thus showing a course and upcoming turns. As the user progresses along the given route, the overlaid directions can automatically update to show the updated path. 

It could also be possible for the display to include indicator graphics to point the user in a proper direction. “For example, if the user is facing south but a route requires the user to progress north, “no route” would be displayed in the display because the user would be looking to the south but the route would be behind him or her,” explains the patent document. In such instances, an indicator can point the user in the proper direction to find the route. 

Apple’s augmented reality initiatives

Apple is said to be working on augmented reality for some time now. While the company has been characteristically secretive about its plans, CEO Tim Cook has dropped hints about this. In a conference call with investors in July, Cook signalled that his company was is particularly interested in AR saying the technology had “huge potential”.

“AR can be really great,” Cook had said. “We have been and continue to invest a lot in this. We are high on AR for the long run, we think there’s great things for customers and a great commercial opportunity. So we’re investing.”

Further, experts had also pointed to how the new iPhone 7 Plus with its dual-camera technology is a move towards that direction. The two cameras allow the device to capture and collect in-depth mapping information of the surroundings.

Then of course what has been obvious and on the face is Apple’s high-profile purchases of AR companies such as Metaio (in 2015) and PrimeSense (in 2013).

Recently, KGI Securities analyst Ming-Chi Kuo said in a note to investors that AR would start to impact Apple products in a year or two. Kuo, who is widely considered one of the more accurate predictors of Apple’s moves, is of the view that AR will kick off on the iPhone and quickly make its way to Apple’s iPad, Macs, Apple Watch, Apple TV, and then automated driving systems.

“All of Apple’s past successes were related to human-machine interfaces, such as mouse for Mac, click wheel for iPod, and multi-touch for iPhone and iPad. Assuming Apple successfully develops AR, we predict the firm will enjoy the following competitive advantages: (1) redefining existing key products and leading competitors by three to five years. For instance, this could happen for iPhone, iPad and Mac; (2) eliminating obstacles of Apple Watch and Apple TV by offering an innovative user experience; and (3) entering new business fields, such as autonomous driving system,” Kuo had said in his note.