Perhaps you’ve heard the rumors and even conspiracy theories about McDonald’s notoriously out-of-service McFlurry ice cream machines? For some customers, these broken ice cream machines are a mere inconvenience, while for other customers they can trigger a far more volatile reaction.
For many years, consumers were led to believe that the reason these machines were consistently offline was because they had to be run through a fragile four-hour cleaning cycle every night and that if the process was interrupted at any point, they would have to start the process all over again. However, upon closer inspection it turns out there is quite a bit more to this fascinating story, one that touches on a number of timeless business themes that run the gamut from monopolistic business practices to the right-to-repair.
So what does any of this have to do with edge computing in general and VMware specifically? More than you might think.
Broken McFlurry Machines are a Microcosm of Edge Computing
One of the first things that I was struck by when going down the broken McFlurry rabbit hole was that McDonalds franchise owners are emblematic of many of the most compelling edge computing use cases that we see every day here at Scale Computing. Most edge deployments, regardless of their industry, share many of the same traits as a McDonald’s franchise owner: they often operate as independent small business entities who are ultimately responsible for managing functions like IT that fall outside their core competency.
In the case of McDonalds, more than 80% of their stores are owned by an individual franchisee, meaning that each store owner is saddled with keeping a variety of Operational Technologies (everything from their deep fryers and HVAC systems to their ice cream machines) and IT systems (think point of sales, video surveillance, etc) running smoothly. As Andy Greenberg recounts in his feature article in WIRED:
Taylor sells a machine with these technical demands to businesses where they’ll ultimately be run by a bored teenager whose fast-food career is measured in weeks. So perhaps it’s no surprise that many McDonald’s restaurants’ ice cream machines seem to be as often broken as not.
Sound familiar? This is one of the defining characteristics of modern edge deployments. Remote or branch offices that rely on some type of on-premises IT infrastructure rarely have skilled technical support at their disposal to troubleshoot and mitigate issues when they occur. And as it is with the broken McFlurry machines, the typical edge customer maintains a service contract with a local IT solution provider and when something goes wrong, they will often be at their mercy and forced to wait days or even weeks until an authorized technician is available to come and fix it.
McFlurries & VMware Are Both Expensive and Unnecessarily Complex
The Taylor ice cream machines that McDonald’s franchise owners are contractually obligated to own and operate are expensive and highly complex pieces of industrial equipment whose core function - efficiently dispensing a serving of delicious ice cream - has become obfuscated by arcane and often unnecessary features that add little value to the business owner. As Greenberg goes on to explain:
But after years of studying this complex machine and its many ways of failing, O’Sullivan remains most outraged at this notion: That the food-equipment giant Taylor sells the McFlurry-squirting devices to McDonald’s restaurant owners for about $18,000 each, and yet it keeps the machines’ inner workings secret from them. What's more, Taylor maintains a network of approved distributors that charge franchisees thousands of dollars a year for pricey maintenance contracts, with technicians on call to come and tap that secret passcode into the devices sitting on their counters.
We hear similar rumblings about the expense and complexity of running VMware environments. In fact, a couple of months back we published the results of a survey entitled, The True Cost of VMware Deployments. After doing a deep dive analysis into our customer base, we realized that 70% of our customers had previously been VMware customers so we commissioned Spiceworks to interview these customers in order to better understand the various factors that led them to make the switch.
According to the survey of previous and current VMware end users, around 50 percent reported that they were not fully utilizing the technology’s available capabilities, which Scale Computing suggests shows the complexity of VMware may exceed the needs of its customers.
This is not a knock on VMware. Their solutions are extremely, if not exceedingly, feature-rich. But oftentimes, all of these new bundled capabilities become more of a burden than an actual benefit -- and increasingly, it becomes more difficult to rationalize the premium price tag when you’re only using a small fraction of product’s features. It’s like using a high-end ironclad cookware to make a grilled cheese sandwich. Sure, it’s more than capable of performing the task at hand, but do you really want to pay a thousand dollars to do so?
For many edge customers, the same holds true. They require baseline virtualization capabilities; they just don’t want to pay for all the extra bells and whistles that they’ll never use. I’m sure most McDonald’s franchise owners, if given the choice (a luxury they are currently not afforded), would happily choose a cheaper and more streamlined machine that was consistently operational.
The Break-Fix Model is Neither Sustainable for McFlurries or Virtualization
According to the website McBroken.com, 13% of all McDonald’s ice cream machines are inoperable at any given time. While that may not seem like a terrible number on first glance, consider that the standard for most industrial products is less than a 1% failure rate. Of course, the ideal target for mass industrial products is the vaunted ‘Five 9’s’ which holds that a product should be functioning 99.999% of the time, or approximately 3.14 errors for every one million opportunities. So this means that at any given time, McDonald’s ice cream machines experience some 130,000 failures per million opportunities - a rate that would likely give your average McKinsey consultant heart palpitations.
What this new wave of reporting has illuminated is that this isn’t necessarily a bug, it’s a feature. One that drives approximately 25% of recurring revenue to Taylor via its network of authorized repair technicians. As Greenberg explains:
Sell franchisees a complicated and fragile machine. Prevent them from figuring out why it constantly breaks. Take a cut of the distributors’ profit from the repairs. “It’s a huge money maker to have a customer that’s purposefully, intentionally blind and unable to make very fundamental changes to their own equipment… Meanwhile, many McDonald’s owners were paying thousands of dollars a month to Taylor distributors in service fees, often for making simple changes locked behind that menu.
In the domain of IT service, we are all familiar with the concept of ‘Break/Fix’ which refers to the fee-for-service method of providing IT services to businesses by which an IT solution provider performs services as needed and bills the customer only for the work performed. As companies grapple with the greater complexity of their IT infrastructure, the familiar break/fix way of doing things is becoming a less sustainable and appealing model, one that is especially germane to IT deployments at the network edge where skilled resources are scarce. From my conversation with CRN:
And because you’re not using it, then it becomes more complicated. And not just complicated in terms of stuff like setup and management, but this concept of, frankly, break-fix. So, something goes wrong, and now you’ve got a broken VMware environment in the back of a grocery store, and an IT guy – who’s maybe not even in the same country – is trying to talk the store manager through it. It’s a mess, right? It doesn’t make any sense. And that’s where these things sort of fall down and where Scale has been pretty good at.
VMware Licensing Fees & McDonald’s Service Contracts: Different Sides of the Same TCO Coin
The Taylor ice cream machines that McDonald’s franchise owners are forced to buy are themselves one of the most expensive pieces of equipment in the store. Each machine costs about $18,000 and more often overlooked, each franchisee’s agreement requires that they sign an exclusive service agreement with Taylor to repair the equipment when they invariably and inscrutable go dark.
What many franchise owners fail to appreciate when they sign on to become a franchisee is that these service fees can often dwarf the original capital expense, requiring them to allocate tens of thousands of dollars every year for a highly unpredictable and variable operating expenditure. So not only are franchisees losing revenue and frustrating their customers by not having a popular menu item available, they’re also devoting precious operating capital to a depreciating asset that cuts into their already thin profit margins.
In a similar (yet far less sinister) vein, our survey found that 30% of VMware admin time is spent on non-strategic maintenance tasks. Of course, certified VMware admins are highly trained specialists, who like the authorized Taylor technicians, are not always available when something goes wrong. Meanwhile, our survey also found that across organizations, an average of 12.3 hours per week is spent managing and patching VMware solutions. Worse still, enterprise IT admins estimate they spend 16 hours per week or close to 900 hours per year on routine maintenance tasks.
All of this to say that when many IT leaders choose a VMware virtualization deployment, there are a variety of hard and soft costs that are often overlooked at the outset and need to be considered when divining the TCO of their edge deployment. Because unlike a McDonald’s franchisee, you do have a choice.