- Cisco Live/Business Insider
Cisco is being publicly blamed for a bug in its switches that allegedly caused Peak Hosting to lose its biggest customer and, eventually, file for bankruptcy, according to legal documents and sources that discussed the situation with Business Insider.
In June, Peak Hosting filed for Chapter 11 bankruptcy after its largest customer, Machine Zone, decided to leave the hosting provider, refused to pay $96 million in fees, and filed a breach of contract lawsuit.
Machine Zone was using Peak to host its popular “Game of War: Fire Age” game. The game experienced a number of outages between April 2015 and November 2015, including one that brought the game down for 10 hours in October, according to the breach-of-contract lawsuit Machine Zone filed.
In the lawsuit, Machine Zone accused Peak of failing to meet its contractually obligated uptime and was suing to terminate the contract.
But Peak countered by blaming a bug in Cisco’s switch software, the legal documents say. It was this bug, and not Peak’s mismanagement of the network, that caused the most severe outages, it argued. Therefore Machine Zone shouldn’t be allowed to terminate the contract.
Machine Zone, as you might expect, didn’t buy that defense, and insisted that all sorts of other things were wrong with the network that couldn’t be blamed on a Cisco bug. It sent someone to Peak to audit the hosting provider and came up with a laundry list of issues from how the room’s fans were installed to how the facilities’ wiring was arranged.
And then, in December, Cisco publicly acknowledged that the bug did exist, reports Bloomberg’s Jordan Robertson. Unfortunately, that was a month after Machine Zone had sued, and Peak was headed down a financial spiral.
Machine Zone’s contract was for $4 million a month and accounted for about 80% of Peak Hosting’s total business. Peak offered a standard nearly “five 9’s” uptime guarantee, or 99.995% availability. That equates to downtime of less than 27 minutes per year, the legal filings say.
After its contract with Machine Zone was contested, Peak closed seven data centers and laid off most of its staff, going from 185 people down to 50, it said in legal documents.
No special treatment?
One former Cisco employee, who confirmed Peak Hosting was a Cisco customer, wondered how closely Cisco was helping Peak if it really was suffering outages from a critical bug in Cisco’s gear.
This person tells Business Insider that when a customer has a problem of this scale, such as bringing down the network or threatening its relationship with its customers, and it can’t be easily fixed by customer support, that problem can be escalated under the Customer Assurance Program, or CAP.
A CAP trouble ticket is for the most serious issues and it involves daily reports, and a lot of handholding, sometimes to the highest levels of the company. If the customer is big enough and the problem is severe enough, even Cisco’s CEOs is kept apprised of CAP customers, this person tells us.
To get this special treatment, our source told us, either the customer has to know about it and ask for it, or somebody within Cisco has to advocate for it.
The industry scuttlebutt is that if a bug in Cisco’s gear really caused Peak’s outage, and the situation went unsolved for months, Peak probably didn’t get the CAP treatment. Otherwise, this person believes, “In all likelihood the problem would have been resolved much sooner.”
Cisco declined comment on the bug and the customer’s level of support saying it does not comment on specific customer outages. Peak Hosting could not be reached for comment.