I have a BSEE, minor in economics and an MBA. I obtained the MBA before entering Corporate America. I didn’t get the MBA because I thought at the time the content of an MBA program would fill a void between engineering and economics. Instead, I thought it would give me a background in the terminology and types of analysis used by business types that would prove useful as I served as a liason between technical teams and non-technical teams.
As cynical as that motivation might have been, I was not prepared for what I actually saw in Corporate America. I started off in operations roles in telecommunications but once I started moving up, my roles were predominately tied to either product development or back-office ordering / provisioning / telemetry systems which all had to undergo enhancements to support new product launches. In my entire career, I didn’t see a SINGLE project that seemed to undergo basic ROI analysis on ONE sheet of paper accountable to ONE executive. Instead, a project was approved either because it was a regulatory requirement (THAT makes sense) or someone just decided this is something that should be done. At that point, marketing estimated its needs, product development estimated its needs, engineering estimated its needs, operations estimated its needs and customer care estimated its needs. SEPARATELY. Everyone turned in their estimates, funds were allocated, each group was held accountable for coming under its request but NO ONE ever reviewed the entirety of the project to realize the plan made no sense – or that even though we have to do SOMETHING, we don’t have to do THIS.
Okay, story time…
I worked for SBC from 1990 through January 2000. In early 1994, a new product was developed by another regional Bell company for providing voice-recognition based phone dialing. The physical product combined a Tandem computer with NonStop UNIX and dozens of Pentium based peripheral boards that ran recognition algorithms. This platform would be connected via trunks to a telephone switch like a 5ESS or DMS-100 to temporarily bridge a customer’s 2-wire connection into a channel into the Pentium peripherals. The customer would “train” the system by saying something like “Dominos” and map it to 816-555-1234. Later, as customers made calls, they would hear a short interval of dial tone, a BEEP, then get connected to this platform which would wait for them to say something, attempt to match it against prior samples, then dial the associated number.
At the time, I was working in a field operations role and headquarters operations planning people came out to visit our market and “partner” with us on launching a trial. Only the HQ people had no expertise in this system or anything vaguely related to it. I didn’t either but because I understood switching and understood computer systems, I wound up leading interactions with the vendor’s engineering team to plan the installation and turn-up of the trial launch. I apparently did so well at that that I wound up getting promoted into the very HQ group that was leading this deployment.
The product launched in my market right as I was promoted into the HQ group. At first, the product was provided for FREE to any customers that wanted it and proved wildly successful – at least in terms of take rate, when given away for free. After 3-4 months, given the FANTASTIC take rate, a decision was made to launch it across all of SBC – all five states – at a cost of roughly $70 million.
Uhhhhhhhhh… Remember, I mentioned that at this point, none of the TRIAL customers were actually PAYING for the product? Well, when product management decided to start collecting revenue, as soon as people started having to pay $6/month for a service that didn’t work that well — this was 1994, Pentiums at 60MHz was the state of the art for CPUs, there’s only so much algorithm logic that can run against 8kHz audio samples at those clock rates… – customers started churning out. NO PROBLEM, says Marketing, we’ll boost the advertising budget and sign up more people for 3 months free then pay. Well, that doesn’t solve the problem if the churn rate is nearly 50%. At some point, you will churn through the entire customer population and no one will be left.
Even THAT looming financial disaster was not enough to get executives to recognize the mistake and withdraw the product and stop spending money on deployments.
The product was finally axed after the vendor was informed that the Tandem platform they had selected had been manufacturer discontinued by Tandem and that the new version of Tandem NonStop UNIX on the new Tandem hardware was not backward compatible with the now-obsolete application software version we were currently running. That meant we would not only have to replace ALL of the Tandem computers we had JUST PURCHASED but our vendor would have to rewrite much of the software for the new NonStop UNIX version and re-test and we would have to chip in on all of those costs as well.
But even that combination of facts wasn’t recognized by execs. We had a meeting in late 1997 which was intended to review / confirm the next year’s spending plan for the product across Marketing, Product, Engineering and Operations. In that meeting, all of the teams were asked to summarize their ask and the talking stick was passed around the table. As teams were seated at the table, I was last in the circle along with my boss. Each team rattled off their plan, based on their own leadership’s marching orders. Each team provided their dollar figures and basically said “all in.” I was the last to speak – my boss just deferred to me and didn’t say a word. When asked by the EVP leading the meeting what our ask was, I basically said…
We think we should withdraw the product and kill it. Our vendor has already told us the core Tandem component is obsolete, cannot be supported and would require replacement across the footprint which would probably be $30 million alone. The vendor has also told us they have to rewrite their application software and could not support both the current and new versions in parallel without passing the duplicated costs on to us, so that’s probably another $10 million. And the PRODUCT DOESN"T WORK. Recognition rates as recorded by the telemetry are low which is supported by the the fact that customer churn is extremely high. Recognition will only improve with faster recognition processors, requiring all of the Pentium cards to be replaced for probably another $10 million after the vendor updates the software and training algorithms. And Marketing just told you they are cutting their advertising / promotion budget so we won’t be able to make up for the churn. And even if we did, we would churn through the entire base of customers.
DEAD SILENCE in the room. For probably 45-60 seconds. (That’s a looooooong time in a meeting with an EVP…)
The EVP then adjourned the meeting. A month later, a bland announcement appeared in some dull planning document saying that the product had been withdrawn as part of a product rationalization effort for the SBC / PacBell merger taking place at the time, allowing the company to take a write-off for capital assets tied to products that would not be kept as part of rationalizing operations. By my calculations, that write-off probably generated a one-time savings of $25-30 million dollars but no one on the team received even a note of thanks.
Back to the point of this thread. In my experience, it’s probably a tie between bean counters and “product development” types in the contest for who causes the most harm in large companies. The problem is worse in large “cash cow” companies that ARE otherwise profitable cuz profitability can hide many, many internal financial decisions. It can’t hide them forever, but long enough for those making the mistakes to move on to bigger, dumber things before the folly of their current mistake becomes unavoidable.
The case of Southwest Airlines’ week-long operational meltdown should be a wakeup call not only to other airlines but entire industries. The SWA meltdown is an example of a unique problem resulting from a system that has grown organically for DECADES, must process TENS OF THOUSANDS of continuous updates when functioning under NORMAL conditions and requires HUNDREDS OF THOUSANDS of updates to restore normal operations after major failures yet is subject to BOMBARDMENT from users trying to use the system during recovery, which further thrashes the system. This is a very familiar problem domain for people in “network” oriented industries – be they communications related or electricity generation / distribution.
In the case of SWA, their back-office has not been modernized to provide an efficient way for pilots and attendants to provide their status and location to the portion of the back-office that optimizes plane schedules. SWA knows the status of all of their planes and leased gates but status info for PEOPLE is pulled from a different system. That PEOPLE system has no modern “smartphone” app to let pilots and attendants click a button to efficiently push their status / location into a database. They have to CALL a human in an internal SWA call center who have to use an ancient application to update the status info. When a massive storm freezes 70% of flights, at least 70% of the required PEOPLE status data is obsolete and requires updating. Only the manual system for updating it was never sized to handle 70% of their pilots and attendants all calling in simultaneously to update their information. And if they cannot update their information, flights continue to encounter delays, which re-invalidates the status of other people in the system and it thrashes, unable to catch up.
As an aside, this exact scenario is exactly the danger we face with any massive failure in the electrical grid. If major portions of the grid are damaged, the entire grid can fail, requiring a LENGTHY restart sequence requiring gradual re-introduction of load that can take DAYS or WEEKS. We as a country should be paying far more attention to the “vandalism” incidents being reported at substations around the country. Those aren’t mere cases of vandalism – those are failed terrorist attacks and should be treated and investigated as such.
I can guarantee there are dozens of people in IT and operations at SWA who have been screaming about this problem for years asking for money to modernize the back-office and provide an app for operations employees to provide duty status. But hey, we’re making money… Why drop $70 million into this re-build? From where CEO Bob Jordon sits – which I’m pretty sure is more often seat 1, row 1 of a private jet than seat 24E of a Southwest 737 trying to board out of LGA – things look GREAT. Or at least, they did until they didn’t.