The technology behind stores like Amazon Go

Amazon Go
Amazon has announced their cashier-less Amazon Go store, opening in beta, for selected Amazon employees to try out.  Upon entering the store, a shopper checks in using a mobile app.  Then, the shopper can simply gather all the items they need from the shelves and walk out.  Their Amazon account is charged for each item they remove from the store.

Observers were quick to make the connection between the descriptions of this store and an Amazon patent filed in 2014 which describes a number of technologies that could be used in such a store.  Let’s examine this patent and pick through the technologies that are described there.

Lots of Sensors
Of course, all this tracking of people and products requires the coordination of many different types of sensors.  This includes:

Cameras Gathering images for use in everything from facial recognition to image recognition of products
Depth Sensing Cameras Depth sensing cameras can accurately determine distances and relative position of customers to products, for example to determine if a customer is near enough to a product to be picking it up.
Pressure Sensors
Scales
Load Cells
Volume Displacement Sensors
All of these types of sensors can be use to determine if the current inventory of a product (in packaged or bulk form) has just changed.  In addition, sensors in the floor at the doorway can be used to basically ‘weigh’ the customer when they enter the store, and again when they leave, to validate the purchase
Infrared Sensors Infrared sensors can distinguish between (generally) cold products and warmer customers.  This can be used to tell when a customer’s hand is reaching into a product area and removing a product.
Light Curtains These simple sensors can detect interaction of customers with the products by detecting when a customers hand enters the region near a product.

Sensors in Action
An illustration from the patent filing below shows how all of these various sensors might collaborate to track products and customers.


fig2_patent_20150012396


Two Basic Questions
All of the tech in a cashier-less store like Amazon Go is focused on answering two basic questions:
 1) Who has entered the store (or who is now leaving)?
 2) What items has that customer interacted with, and which are they taking with them?

The ‘Who’ question is at least as hard as the ‘What’.  The current Amazon Go takes the easy way out on this one – customers are required to check in with an mobile application while they are standing in a ‘check in area’.  From here, it would be reasonable to assume that a combination of object tracking and facial recognition would be able to follow such customers throughout the store.  Although it would be fun to see if a human version of Three Card Monte shuffling in the aisle could confuse the tech.

The ‘What’ could quite reasonably be solved using RFID tags on every item, but where’s the fun in that?  Seriously though, the cost of RFID tags continues to be just too prohibitive for it to be placed on every single bannana, can of vegetables, or box of mac-n-cheese.  Plus, it requires either the stores to affix these tags, or the suppliers to include them.  This is why more ‘passive’ technologies which do not require modification to the packaging of each item are favored.

 

Brave New World?
Using cameras to track shopper behavior isn’t new.  For years, the same cameras that were used in retail stores to watch for shoplifting were also being used to gather retail analytics. Retailers can track which displays are receiving the most foot traffic.  They can measure ‘dwell time’ to determine whether customers are stopping to review a product, or just walking by.  Shoppers can be classified by age or gender and the effectiveness of displays judged on each classification.

Customer self-checkout is not new either.  Most retailers have some form of it, usually requiring interaction with an actual barcode scanner either at a standard checkout kiosk or as the customer places items in the cart

The Amazon Go experiment is taking each of these paths one more step, enabled by the newest sensor technologies, recognition algorithms, and computing power.  

Cashier-less stores may appear revolutionary today, but compare this to the first remote access self-service gas station, which opened on June 10, 1964.  The technology for remotely activating the pumps, maintaining safety, and controlling theft was primitive by today’s standards.  Few companies were convinced that drivers would be willing to pump their own gasoline.  But by 1982, 72% of all gasoline sold in the US was self-service.

Author: Ed Kuzemchak

Ed Kuzemchak is the Chief Technology Officer and Director of IoT and Embedded Systems Engineering at Software Design Solutions, an Applied Visions company. Prior to founding SDS in 2003, Ed gained his experience as chief software architect for the digital signal processing (DSP) tools group at Texas Instruments, as a member of the Tartan Laboratories highly optimizing compiler team, and as the lead for a compiler team at Raytheon. Ed holds an MS in Computer Science from the University of Pittsburgh and is the author several patents on embedded systems software. His Erdos number is 5.