When it comes to disruption of business as usual for supply chains, Amazon towers above the rest. They are often spoken about by competitors with a combination of reverence and fear. Amazon has achieved their almost mythic status by continually innovating at a pace that others find hard to keep up with. They have continually upped the ante on ease-of-use, selection, rapid delivery and the overall user experience. And they have branched out into other areas such as dominating the cloud infrastructure market with their AWS cloud services, a broad array of logistics/transportation services and assets, and bricks-and-mortar retailing with their Whole Foods acquisition and Amazon Go.

I visited the Amazon Go store while I was in Seattle recently. That is currently the only Amazon Go store in the world, although some sources say Amazon is planning to open as many as six new Go stores this year. Amazon declined to comment on whether and when they will open new stores, saying only, “It’s too early to speculate on that. Right now, we’re laser focused on customers and delivering good food to them fast.”

The Amazon Go store in Seattle is about 1,800 square feet. I might describe it as a checkout-less convenience store, with a good selection of prepared-onsite fresh food. Amazon described it to me this way: “We think of Amazon Go as the fastest place to grab hand-made meals and snacks, essential groceries, and fresh, seasonal meal kits to go. No checkout. No waiting in line. Just good food, fast. It’s the first store with Just Walk Out Technology, a new technology allowing customers to grab what they want and just walk out.”

The User Experience: Getting Started

Downloading and installing the ~11MB Amazon Go app was quick. Once installed, you login using your regular Amazon account ID and password. The first time you login, the app takes you through a 9-step animated tutorial which explains how to scan yourself into the store and how to shop. It took about a minute to go through the whole tutorial. I was impressed by how clear and easy-to-understand the tutorial was, especially with the simple animated graphics showing you exactly what to do. Since I had more than one credit card on my account with Amazon, it also asked me which one I wanted to use.

Next the app displays a 2D barcode to identify you, which you use to get into the store. After initial setup, the 2D barcode is the first thing displayed when you run the app (i.e. no login required). That fits with Amazon’s relentless pursuit of removing friction from the shopping process (“don’t make me login to my account just to get in your store!”), but it also means anyone who physically possesses your phone can use it to shop, without knowing your Amazon ID and password, if they can get into your phone. The app does have an optional setting to make it require a login upon launch, though it doesn’t do so by default. Also, the app was smart enough to prevent me from taking a screenshot of the 2D barcode, I assume so I wouldn’t share it on social media, possibly letting the whole world shop on my account. They redraw a new and different 2D barcode on the phone every 30 seconds, which I would imagine provides some degree of protection against people being able to shop on my account just by taking a picture of my barcode. 

Entering the Store

Getting into the store is somewhat similar to entering a mass transit system that has RFID readers with mechanized “saloon doors” (i.e. small swinging doors), except Amazon Go’s doors are clear plastic and you use your phone instead of an RFID card. It is somewhat similar to checking into an airplane using the airlines 2D barcode reader. So, for some people this will not be unfamiliar. At or near the store entrance (outside of the entry gates) there was the constant presence of one or two orange-shirted Amazon associates helping people with any issues. The most common was just explaining that the shopper needed to download the app and explaining to them how it worked.

Buying and Paying

You buy and pay for things by picking them up and walking out the store. That’s it. Thus, Amazon Go epitomizes the company’s obsession with removing friction from shopping.

How It Works: The Basics

Amazon Go embodies some stunning recent advancements in technology, particularly in sophisticated computer vision, sensor fusion, and deep learning. Amazon is understandably fairly tight-lipped about how this all works. They don’t want to make it easy for competitors to reverse engineer their technology. As a result, a fair amount of what has been written about how they perform this magic is speculation. In this article, we will endeavor to make clear distinctions between direct observations and speculation.

The easy part for the system is identifying you … or at least identifying your phone. The system uses the 2D barcode from the Amazon Go app on your phone to unambiguously link whoever has the phone to your Amazon account when they scan into the store.

As stated earlier, whoever has the phone (if the person can access and run the Amazon Go App) can shop at an Amazon Go store and all purchases they make will be charged to the Amazon account associated with the phone. An unprotected phone (or weakly protected phone) with Amazon Go is thereby almost like having a wallet full of cash, up to your credit card limit. People need to understand this and set up a strong password or biometrics or other means of preventing unauthorized access to their phones.

Now comes the hard part. From the moment you scan and enter into the store until you leave, the system needs to keep track of you, where you are, what items you pick up (and how many) and what items you set down. Thus, it keeps a running tally of exactly what items you walk out of the store with.

The Hardware

I’ve seen a lot of different assertions on what kinds of sensors Amazon is using in their store, with people talking about everything from lasers to infrared to RFID.

We are almost certain Amazon is not using RFID for tracking items in their Go store. First, I didn’t see any tags on any of the items. Grocery is a low-margin business where many items have just pennies of profit. Even the lowest-cost UHF RFID tags at high volumes — once you add up all costs it takes to produce, commission/serialize, apply, and test them — are about $0.07 to $0.12 each. In contrast, using video analytics, sensors, and machine learning adds $0.00 of variable cost to each item. There are also challenges using RFID around liquids and metals, which would make something like a can of soup or soda difficult to reliably read. Companies have found some ways of overcoming those limitations up to a point, but probably not good enough for the performance Amazon needs here.

One thing's for sure, there is a lot going on in that store.

Amazon is using a combination of computer vision, sensor fusion, and deep learning to accomplish the task of tracking individuals and what they pick up or put down. Specifically, they are inferring pose for articulated motion analysis in crowded scenarios. In other words, they are continuously analyzing the video streams to build a model of the position and pose of each person in the store; what position each person’s limbs and hands are in and how they are moving through space. This involves modeling the human body and its various joints to feasible positions. That is hard enough when there is only one person in the picture. It becomes really hard when there are many people, making it easy for different parts to be hidden or exposed and figure out what is what. I assume that is why there are so many cameras, to ensure there is no configuration of people that will fool the system.

They use deep learning algorithms to recognize products on and off the shelves, even in the presence of occlusion (blockages of the image). Image recognition is an area where we’ve seen tremendous advances from deep learning. Amazon can recognize an item even when it is partially hidden.

It is not clear how well the system knows if a product is handed from one person to another. Their stated policy, which appears right there in the introduction tutorial, is that the person who takes an item off of the shelf pays for it, even if they hand it to another person who walks out with it. However, a shopper doesn’t pay for an item if they pick it up, but then subsequently set it down before leaving the store.

Regarding facial recognition, Amazon said “We don’t identify customers through facial recognition. We use the QR code generated by our app to associate customers with their account when they enter the store.” So, it is clear that they do not use facial recognition to identify customers as they come into the store … they don’t “remember” what someone looks like between one visit to the store to the next visit. It is less clear whether they use facial recognition after a shopper has entered the store, during the single shopping visit, to distinguish one shopper from the other shoppers. In any case, privacy would not be compromised as Amazon is not associating the face with an identity.

When you step back and think about the different layers of intelligence here; how Amazon engineers have gotten the system to recognize people and their pose; how the store system stitches together inputs from thousands of different devices to form a cohesive understanding of each person and every item in the store; how they are able to deal with so many different potential scenarios and challenges of occlusion (blocked views); all of the work that must have gone into testing and tweaking and tuning and refining to get it to be nearly foolproof … it is extremely impressive.

Implications for Fixtures, Shelving, Packaging and Selection

It is amazing what Amazon is doing with this technology, but that doesn’t necessarily mean this solution is ready for all kinds of products and store formats. Here we speculate about its limitations. For example, the current setup has a dense array of cameras which we assume is probably required to make this all work. That setup may be difficult to duplicate in other settings, such as in a clothing store with lots of hanging items on racks or in the bulk produce section of a grocery store or the lumber section of a home improvement store. It is also unclear whether the system of scales on the shelves can reliably handle and price food that is sold by the pound.

Potential Impact on Loss Prevention and Shrink

It is interesting to consider the impact these systems will have on loss prevention and shrink. This system should eliminate sweet-hearting, which is where cashiers give items to people they know without charging them (often by doing “fake” swipes over the bar code reader to make it appear like they are ringing up the items without actually incurring a charge). It seems like it may also make many conventional forms of shoplifting or employee theft more difficult … possibly much more difficult.

Nevertheless, it could have a substantial impact on shrink. The average shrink rate for U.S. retail last year was about 1.4 percent of revenue. About 2/3 of this loss, or just under 1 percent of revenue on average, is due to theft (external and internal) with the other 1/3 from administrative errors, vendor fraud, and “unknown.” Considering that the average net profit for convenience stores is about 1.8 percent, if Amazon can eliminate most of that theft, they will have increased net profits by about 50 percent — and that is before considering any other impacts of this technology on profitability.

The Financial Model: Upfront Capital Costs, Store Velocity, Staffing and Profitability

The Amazon Go model changes the dynamics of the store’s financial model. There is a higher up-front capital expense compared with a typical convenience store. How much higher? We can guess — my gut says it is somewhere between $100,000 to $500,000 — but you shouldn’t put too much stock in that guesstimate. It is really hard to say with any level of confidence without more visibility into all the pieces involved. Regardless, there is some non-trivial additional upfront capital expense involved in rolling out one of these stores.

On the other side of the equation, an Amazon Go store should have a higher velocity of sales than a traditional store. There is no wait time in checkout lines, so people get in and out of the store faster. That means a given size store can support more traffic, more sales per hour. Another way to look at it is the space that would normally be needed for registers and lines can instead be dedicated to more shelves and products. Therefore, with more products in the same footprint, the store should produce higher sales per square foot.

Of course, that assumes that the same products with the same price and profit margins will be sold in an Amazon Go store as compared with an equivalent convenience store. But this is not your typical convenience store. A significant portion of the Amazon Go store products are fresh prepared-onsite items and meal kits that you won’t find in your typical 7-11 or service station convenience store. These are higher-margin products that should further enhance the revenue per square foot advantage of these stores.

The degree of labor savings is more of a question mark. True, there are no cashiers. But the higher velocity means the shelves need to be stocked more frequently, requiring slightly more labor. The alcoholic beverage section of the store has a dedicated employee to check IDs. There are one or two people out in front of the entrance gates at all times, helping people to download the app and answering questions. And there are quite a few people preparing the fresh food (though roughly the same amount of labor may be needed for offsite fresh food preparation, but it is not in-store labor). So, it is possible, maybe even likely, that this format actually requires more labor than the same size convenience store.

In summary, there is a significant additional upfront investment in capital required to put up one of these stores, but it pays off over time by higher revenue and profit per square foot. Labor costs may not be lower. Rather the higher profit is likely achieved by a combination of lower shrink, higher velocity of sales, and higher margin products.

Impact on the Broader Retail Landscape

While the technology in Amazon Go may be revolutionary, I believe the impact of these changes across retail will be evolutionary. It will take time. Amazon first needs to learn from their own efforts and indications are they will not roll out a lot of stores until they are ready. Furthermore, at this stage the technology seems only doable in small-format, convenience-type stores. Eventually it will expand to larger grocery stores, but that may take many years.

In most retail sectors, the speed of getting in and out of the store is not the first consideration. Shopper experience is key and that may mean personal attention, unique and engaging in-store experiences, opportunities for learning and discovery, attractive selections of merchandise, highly knowledgeable sales staff, and exuding exclusivity and class. None of that is accomplished by a grab-and-go approach. Yes, even luxury clothing stores may eventually adopt checkout-less technology, but it is pretty far down the list for them and that is likely many years (maybe even a decade or two) before it becomes common. The impact of Amazon’s entry is likely to be felt soonest in convenience stores and next in grocery. This includes online grocery; Amazon Go may actually displace some online grocery shopping for shoppers that place a premium on convenience and instant gratification.

There is the open question of where competitors would get the technology to deliver a checkout-less store. My working assumption is that Amazon will not be sharing this technology with competing retailers, much as they do not share their warehouse automation technology. I could be wrong — Amazon is more than happy to sell AWS services to competing retailers, but that does not so directly offer Amazon a competitive advantage as the Go store seems to do. We do see a number of startups aimed at providing Amazon Go-like technology and capabilities to competing retailers. However, it is really early and hard to tell which of these will succeed until we see some serious pilots and rollouts.

New Data — and How It Will Be Used

The Amazon Go store will generate a tremendous amount of new data. Amazon has revolutionized the use of online shopping data, following your every move online to make recommendations and try to get you to buy more. It would only make sense for Amazon to add into the mix the shopping behaviors for each shopper, as observed by its system running the Amazon Go store, to gain a fuller understanding of the shopper’s combined online and offline shopping persona, tastes, and shopping patterns. While other retailers have been striving to provide a unified omnichannel experience, Amazon has the opportunity to have a truly seamless connection between their physical stores and online shopping.

The Amazon Go data should also have a lot of value to CPG manufacturers, as well as to Amazon’s own merchants, planners, and store designers. They have the opportunity to observe customer engagement, items considered but not bought, shopping sequences, path to purchase, impact of different store layouts, impact of different planograms, product co-dependencies, influence of offline shopping on online shopping (and vice versa), and more. Of course, privacy concerns will need to be considered in all of this as well. It will be very interesting to see where Amazon goes with and all this new data.

The Future of Sentient Buildings

Looking beyond the impact on retail, Amazon Go represents the emergence of the “Sentient Building” that understands the actions and intentions of its occupants. This kind of awareness could be used for many different things. It might become a safety feature where a building “knows” when one of its occupants is in danger, or has been injured, or an unauthorized intruder has entered. Or it might help with evacuation in an emergency or direct emergency personnel to save trapped individuals. It could be used in a hospital setting to instantly locate key personnel or pieces of equipment that are needed in a hurry. The sentient home could help monitor elderly people to ensure their needs are met and/or they are compliant with treatment regimes. If opted-in, someone might want their building to act almost as a virtual life coach, gently reminding them to take a break and step away from their computer to take a walk.

Some of these scenarios, especially that last one require a high level of emotional intelligence or empathy to get right. There are technology companies attempting to provide that kind of emotional intelligence, such as Emotion Research Labs (based in Spain) who refers to themselves as the “artificial empathy” company (I assume that is AE to complement AI). They have algorithms that monitor facial muscles to determine the emotional state of a person. There are a handful of other companies working on emotion recognition as well. That will eventually become a key part of a building successfully anticipating and responding to the needs of its occupant. And not just buildings, but vehicles and other environments. Emotion Research Lab says they have the ability to measure attentiveness, which might be a great safety feature for ensuring that drivers and pilots of trucks, trains, planes, and ships are paying attention, potentially saving lives.

Thus, Amazon Go augers more than a new way of shopping. It is ushering in new ways of interacting with our environment, with more intelligent buildings and vehicles. We can only hope these technologies are used for good, to make things better for mankind.

Resources: ChainLink Research