The outage could have been far worse
I have a rough idea of why Crowdstrike’s update bricked countless Windows devices around the world, but it’s more important to discuss some reasons why we collectively found ourselves in a situation where a vendor, that most people hadn’t heard of, was allowed to take down so many services people depend on.
Surprisingly my organisation came out relatively unscathed, but not entirely. Restoring our servers and VMs is a trivial job in itself, but most things running on them can’t be taken offline without that quickly cascading into a much bigger problem. The next five hours were a mad rush to give all our client-facing services a basic function test, audit all our integrations and resync data for anything that might have failed overnight. The desktop support team are likely going to spend the week manually fixing all the bricked PCs across multiple sites.
As it turned out, the culprit wasn’t a failed Microsoft update, as was initially reported. The problem had something to do with Crowdstrike’s ‘Falcon Sensor’, which is an endpoint security program that runs in the background on corporate devices, usually on issued laptops. A defective software update for that caused such Windows devices to crash and prevent them from restarting properly. And, as we learned, all the corporate devices we previously considered non-critical, because they weren’t production servers, were collectively pretty damn critical, as new reports attest.
What I think happened is the .sys file is loaded into Windows’ kernel memory space, the kernel loads it as a driver, and it performs an illegal memory operation that crashes the kernel. It probably instructs the kernel to overwrite something in its own region of memory, though I’ve heard others mention something about a null pointer bug, and I’ve heard mention the .sys file wasn’t even a kernel driver (then why was it in kernel space?).
From that point, the kernel will attempt to load and execute the same driver every time the system is rebooted, with the kernel defaulting to an exception handler that gives us the famous blue screen. The only way out of that is to boot Windows into ‘safe mode’.
Fixing this would be a simple job, right? Well, unfortunately no. The client machines can’t be fixed remotely (as far as I know), yet most employees won’t be able to fix it themselves either. They wouldn’t have a BitLocker recovery key (needed to enter ‘safe mode’), the permissions to delete the .sys file, and maybe not the skills to perform the task to begin with. Basically desktop support teams need physical access to manually fix every bricked device, and that could take weeks for a large large organisation. And likely much longer for organisations that outsourced their IT support to some overseas company.
A security analyst and myself were both very surprised to discover how commonly used Cloudstrike is. What didn’t surprise me is that something like this happened globally. It was entirely predictable, and Tom Clarke was entirely correct when he said, in his analysis on the Sky News site, that the core problem is that our ’livelihood depends on IT products made by an increasingly powerful oligopoly of tech companies’. A handful of corporations have too much control over the systems we’re all dependent on, and that’s a disaster waiting to happen. The Crowdstrike incident merely gave us a taste of what such an outage would look like.
Countless organisations have gotten themselves into a situation where a handful of vendors now practically own their infrastructure and resources. If one of those vendors goes down, or gets compromised, it will take those client organisations down with it. Yes, the alternatives - managing IT resouces, development and security in-house, in theory, comes with more overhead, but that’s the price of having a society that’s so dependent on those resources.
The fact that Crowdstrike’s update didn’t just affect Windows devices with a specific configuration would suggest that that the kind of review, testing and deployment process, that every competent software engineer would insist on when deploying software, wasn’t followed. But the developer/engineer wasn’t solely responsible. There must have been senior engineers, product leads, DevOps and management responsible for enforcing safeguards that mitigate against the risk of defects being deployed onto production systems.
Crowdstrike probably has a EULA clause stating its products shouldn’t be used on critical systems, yet salespeople likely encouraged potential clients to do exactly that. Executives in client organisations pushed for Crowdstrike to be deployed on their own companies’ systems, because of compliance, legal and insurance demands - corporate security is largely about compliance these days. It’s hard to know if anyone could be held accountable when something like this happens.
In my professional opinion, having third-party vendor components loaded into kernel space, where any piece of code can cause the operating system to do anything, is intrinsically a bad idea. The vendor - especially a vendor that’s not vetting its own software thoroughly - would become an ideal channel for a supply chain attack. It’s entirely plausible that malicious code pushed into kernel space through a trusted vendor would evade most other security products - which typically provide considerably less visibility than network engineering tools - and remain undetected for weeks.