Artificial Intelligence and Digital Twins

Augmented reality in supermarkets

Have you ever wondered why supermarkets don’t offer augmented reality guides to the locations of their products? Instead of criss-crossing the store, hunting for the tomato juice or the paprika, why can’t you upload your shopping list into something like Google Maps and have it guide you on the most efficient route around the aisles?

It’s not because supermarket owners are afraid you won’t make impulse purchases. It’s because that sort of navigation (or wayfinder) technology is hard. But it’s coming soon.

Pebbles

In 2015, Emil Alon sold a company to Facebook for $60m. The company was Pebbles, and it enabled people to see their own hands while wearing virtual reality headsets. An array of lasers bounces infrared light off nearby objects, and algorithms create detailed 3D maps. They are calibrated to detect and image human skin, and the projection of your hands in front of your face enables you to control the images you see (pinch your fingers to reduce the size) and the devices you own (place a finger to your lips to reduce the volume of your music).

Pebbles is one of the many Israeli companies which have produced state-of-the-art computer vision technology, building on expertise spawned by the country’s defence industry. When you live in a country that is the size of Wales, and just 15km wide in parts, you’d better be good at spotting threats.

Resonai

Emil now runs Resonai, which has an even more ambitious remit: to give augmented reality its equivalent of the moment that smartphones experienced in 2007 when the iPhone was launched. The term “augmented reality” was coined in 1990 by Boeing researcher Tom Caudell, and since then the technology has experienced numerous false dawns, notably with Google Glass in 2012 and the Pokémon Go craze in 2016. We all know that one day it is going to be huge, and the tech giants are investing massively. Apple has its ARKit platform, Microsoft has Hololens, Google followed Cardboard with Daydream, and this year it paid $180m for North, a Canadian manufacturer of smart glasses. Facebook is still developing the Oculus range, and last year it launched a Second Life-style virtual environment called Horizon.

Six degrees of freedom

The reason for the false starts and the delay is that it is very hard for computers to map and represent the physical world, so the experiences which today’s augmented reality applications can provide are limited. They can overlay digital images onto the physical world, but these images incorporate very little information about the physical world. Emil thinks his new company has cracked the problem. Using highly accurate 6-DOF (degrees of freedom) tracking to capture every motion in six directions (up, down, forwards, backwards, left and right), he and his team have developed technology that creates a near-perfect simulation of any physical space in three dimensions. A digital twin, imbued with intelligence from the physical world.

The digital twin of, say, a building, is stored in Resonai’s cloud-based computer environment called Vera, and vast amounts of information about it can be made available to users. When you enter the building, the camera in your smartphone syncs with the digital twin, and renders the building intelligible to you. Making the physical world intelligible is part of the great promise of artificial intelligence.

Digital twins

The features that digital twins make possible are almost endless. As well as supermarket navigation, building owners can implement smart ticketing, control access to specific areas, govern or influence pedestrian flow – for example in response to social distancing requirements during the pandemic. They can offer ad hoc, interactive AR tutorials showing how to operate equipment like air-conditioning units or vending machines. They can log maintenance requirements, direct repair personnel to the fault location, and guide them through the repair process. Staff can find out what cleaning materials can be used safely on sensitive surfaces. This is Digital Real Estate – the Internet of Things in full swing – and it will generate revenue streams that no-one has yet thought of.

One of Resonai’s early customers is Moscow Trade Centre, one of the world’s largest wholesale shopping outlets, with 200,000 square metres and 6,000 stores. Resonai has created a digital twin for much of the estate, greatly easing the headaches of security and maintenance caused by the centre’s 1,500 monthly service requests from tenants. Resonai also has clients in Germany and Japan, and is on the cusp of a big announcement in healthcare.

Glassholes again

The obvious question to ask Emil is, when will everyone have a version of Google Glass that works well, and is socially acceptable? With the caveat that all forecasts are wrong, he replies that the first versions will probably appear in a couple of years, and the tipping point for mass adoption could well be around five years from now. They will probably be tethered to your mobile phone, since you can’t add much weight to a pair of glasses before they become uncomfortable, and you can’t allow them to get hot. They may provide audio with bone conduction technology, or maybe you will wear separate earpieces like Apple’s AirPods.

If you don’t like the sound of this, and the phrase “glasshole” has already sprung to your mind, well, of course nobody is going to force you to adopt this new platform. But refuseniks may start to feel “digitally blind” – disadvantaged by their inability to access the valuable intelligence about their environment which is available to their peers.

With all the massive players jostling for position in this new platform, the successor to the smartphone, a uniform protocol will be needed. The internet and the web could not have achieved mass adoption without TCP-IP and HTML, and the same requirement applies to virtual and augmented reality. This is the biggest prize for Resonai. Emil and his team are working to establish the handshakes and the rules of the road for this new world.

Augmented reality etiquette

There are many serious issues to resolve. Who determines what you see in virtual and augmented reality? If you enter a store, and take out your phone or flick a digital switch on your smart glasses, do you consent to the store owner influencing your perception, and if so, to what extent? Some nudges to your experience of colour, sound and smell could significantly influence your propensity to buy. Many people have painted hellscapes that this technology could create, like the shopping mall scene in the 2002 Tom Cruise film Minority Report. These issues will be resolved by some combination of regulation and etiquette.

Alexander Graham Bell was granted the first US patent for the telephone in 1876. He wanted the word “ahoy” to be used for telephonic greetings, but it was Thomas Edison’s suggestion of “hello” which won the day. In the late nineteenth and early twentieth centuries, a good deal of ink was spilled in the effort to work out the appropriate etiquette for making and receiving phone calls: what time of day you should call, how long you should talk for, and what subjects were appropriate uses of the device. Rather obviously, we are still working out the etiquette for the fast-moving world of social media. Virtual and augmented reality are going to give us a whole slew of new conundrums to play with.