What IS it lately with manufacturing and technology firms lately? Here are two more recent tales of woe that are going to be very expensive / embarrassing for companies and a major pain for customers.
==========================
Toyota has finally announced its plan for dealing with a manufacturing flaw encountered with the new twin-turbo V6 engine it began putting in its Toyota Tundra pickups and Lexus LX SUVs beginning with 2022 and 2023 model years. Customers began reporting catastrophic engine failures with VERY little miles on the vehicles. Toyota traced the failures to millings left from machining of the cylnder heads that were not removed by washing prior to engine assembly. Up until the time the trend was confirmed, Toyota had put roughly 98,000 of these engines in Tundras and another 3800 in LX vehicles. The problem is that Toyota was unable to identify exactly WHICH of the 102,000 engines in total might encounter the failure.
Apparently, Toyota is STILL unable to narrow down the exact engines needing to be fixed. While they have only fielded about 980 requests from dealers to date for actual failed engines or warranty work related to this, Toyota bit the bullet and decided to replace ALL of the engines. All 102,000 of them.
For actual owners, this is actually a VERY good move. Replacing the ENTIRE engine is actually not only less labor than attempting to tear apart an existing engine and performing a small block replacement, it is more routine work requiring less expertise of the local dealer’s mechanics. It will ensure each owner gets a factory quality (***) engine rather than chancing any one of dozens of possible mistakes being made during reassembly.
*** Yes, one does have to look beyond the irony of that statement when remembering in this case “factory quality” led to this problem in the first place.
For Toyota, the cost of this will be staggering. I would estimate a brand new crated engine will cost maybe $9000 including shipping. Probably another $2000 in labor. $11,000 over 102,000 vehicles is over $1.1 billion dollars. In the larger scheme of things, Toyota has no choice. Having 102,000 owners driving around perpetually worrying their engine could experience a catastrophic failure at any moment would destroy the sense of trust Toyota owners feel and have been willing to pay a premium for over decades. While a jump in demand for 102,000 additional engines is going to constrain supply for additional new vehicles, it’s easy to argue that NOT replacing these engines would DESTROY any demand for new vehicles so it is better to do the smart thing for the long term, back the product already in the customers’ garage and fix the problem entirely.
As a loyal Toyota owner and one interested in the upcoming 4Runner which is going to use the same engine, Toyota’s handling of this failure has been of great interest.
==========================
Meanwhile, back in the computer industry, news stories from July 26, 2024 have reported that five different manufacturers of computer motherboards – including big names like Dell, Acer and SuperMicro – have found over 215 different models they produced were loaded with a cryptographic key protecting the UEFI layer of the motherboard BIOS that was compromised.
The private half of the key was published on a GitHub repository somewhere with only a four character password protecting clear text access to it. By brute force cracking access to that GitHub file, the original private key could then be used to sign any alternative BIOS content that could be pushed to any of these 200 motherboard models made in the last 2-3 years. These motherboards would accept that code and allow it to boot on the machine, giving an external party control over loading of additional layers of malware / spyware on the machine.
Sadly… maddingly… the cryptographic key in question was explicitly flagged as DO NOT TRUST. Apparently, the processes different makers use to combine a specific physical hardware design with a firmware release and a cryptographic key didn’t track the provenance of that private key to realize it traced back to this repository on GitHub that seemed to reflect the key had already been compromised. Those “build” processes seemed to simply key off the fact that someone else provided a key, the key seemed mathematically sound so out into the final build it went.
There have been prior UEFI related vulnerabilities identified in recent history as well, including an issue with Trusted Platform Module functionality in UEFI software made by Phoenix Technologies in April 2024 and an earlier flaw dubbed LogoFail identified in December 2023 stemming from a scheme hackers used to alter data used to render a hardware vendor’s logo during boot and corrupt that data to trigger failures that would allow bad code to take control of the machine.
This most recent key failure stands apart from those prior cases because it didn’t reflect “bad guys” being extremely clever and identifying a way to use bad data to crash code into a state allowing takeover. Instead, it reflects “good guys” blindly following flawed administrative practices that negated the entire chain of cryptographic protections at the very root of the process.
At this point, it isn’t known if this flaw was just a theoretical problem nipped in the bud before bad guys figured out how to exploit it or if exploits have already been deployed using it. At a minimum though, given the number of manufacturers involved and the number of motherboard models affected, it seems to affect a large portion of x86 based machines purchased in the last two years which will require many labor hours to discover and patch.
WTH
