I just got back from a meeting in South Korea. Several of my colleagues got caught in SK by the CrowdStrike disaster that affected computers worldwide. Luckily, my flight was scheduled after most affected computers were well on their way to recovery.
Microsoft is getting a lot of flack for the debacle, though really the issue seems to have originated with CrowdStrike itself, and it doesn’t affect PCs that don’t use CrowdStrike software. Personally, on the few Windows installations I use, I just use Windows Defender and common sense.
"Microsoft has blamed EU rules for enabling a faulty security update to cause the world’s biggest IT outage.
The software giant said a 2009 agreement with the European Commission meant it was unable to make security changes that would have blocked the CrowdStrike update that triggered widespread travel and healthcare chaos on Friday…
… In contrast, Apple blocked access to the kernel on its Mac computers in 2020, which it said would improve security and reliability."
Personally, I’ve never been in favor of the EU meddling in how technology companies build their technology. Non-tech-savvy bureaucrats as often as not have no idea of the consequences of changes they push on technology. While some may say that Apple’s insistence that such meddling interferes with Apple platform security is self-serving obfuscation, I think this worldwide debacle illustrates the point.
I’m hoping not only that Apple can use this event as a way to block the EU from meddling with Apple’s kernel-level software but also that Microsoft can effectively push back on the EU.
What I don’t understand is how a major company/distributor/updater, can ever blow out what must have been a major update, to all of it’s customers without testing in a smaller, well monitored segment… Or they did and missed the problem…
Part of my background was in doing major hardware, software updates to existing electronic telecom switches, as well as other toll gear, some handling massive numbers of live transactions, calls, even worldwide connections… One failure I hit was a shop wiring error on top of some primary timing sources being mis-routed by the telco folks, as we found out after recovery, it killed all services in 3 neighboring cities, zero access to call for help, however I had an old Motorola bag phone in my truck, made a call and we were able to back the update out, recover, but a couple hours of lost service… The inquisition afterward was fun, not sure if I’d get zapped, but instead, an attaboy for the recovery, some telco line assigners had their hands slapped… And then go back, repair the wiring error, finish the project… Factory checks should have caught the error AFAIK, this one frame was the only one with the error, Nationally, so just lucky I guess! Fun times, best not to panic!
Major? I guess the word is subject to interpretation, but it was routine, perhaps too routine. The people they are stopping are always coming up with new stuff to block, and Crowdstrike has to keep up.
The configuration files mentioned above are referred to as “Channel Files” and are part of the behavioral protection mechanisms used by the Falcon sensor. Updates to Channel Files are a normal part of the sensor’s operation and occur several times a day in response to novel tactics, techniques, and procedures discovered by CrowdStrike. This is not a new process; the architecture has been in place since Falcon’s inception.
Prepare, pre-test, pre-test, pre-test, with multi level sign-offs, monitoring…
We used a “Method Of Procedure” form, detailing every step we were going to take, and included methods of backing out of whatever the failure was, some MOPs were many, many pages, I innovated with additional sketches, images, (thanks to my personal Mac), to leave no doubt as to what was to be done… At times, weeks were spent on the procedure, with break points if possible…
One I recall, was a small system, TL microwave radio, booting it from a half watt to one watt, just me and my helper, plus the customer engineer… I’d promised to have it up before leaving, well, that turned into a 27 hour shift… the first amplifier failed, so back out, start over with a replacement, hoping the replacement was OK… Memorable…
Anyway, the dedication to never deny service to the end user, or the customer was top of everything, other than safety…
