20 August 2024 · 6 min read

A content update is code

CrowdStrike staged its sensor code but shipped the Channel File 291 update, filed as data, with no canary. Anything that changes what a fleet executes is code and ships like code.

On 19 July 2024 at 04:09 UTC, CrowdStrike released a content update for its Falcon sensor on Windows, and at 05:27 it reverted it. The next day Microsoft estimated that the update had affected 8.5 million Windows devices, "or less than one percent of all Windows machines". The bug was small, a count off by one that no bounds check caught. What I keep coming back to is the route the update took. CrowdStrike's sensor code reaches customers through a staged rollout, and customers can hold back a version or two. The file that crashed the machines went through none of that. It travelled by a separate channel, for content CrowdStrike calls configuration data rather than code. Anything that changes what a fleet executes is code, whatever it is called, and it needs the same staged rollout, canaries and rollback as code.

Two routes to the same machine

CrowdStrike's preliminary post-incident review, published on 24 July, describes both routes. Sensor Content ships only inside sensor releases, which culminate in "a staged sensor rollout process that starts with dogfooding internally at CrowdStrike, followed by early adopters". Rapid Response Content, the other route, "is stored in a proprietary binary file that contains configuration data. It is not code or a kernel driver." CrowdStrike's technical details post of 20 July adds that updates to these Channel Files "occur several times a day".

The review's scope for the crash is a time window, not a ring: Windows hosts on sensor 7.11 or later that were online between 04:09 and 05:27 UTC and received the update. Its first fix for deployment implies there was no smaller group to watch: a staggered rollout "starting with a canary deployment". The later root cause analysis made the gap a finding: "Each Template Instance should be deployed in a staged rollout."

The Channel File 291 update window on 19 July 2024, in UTC A time axis from 03:40 to 06:40 UTC. A shaded band runs from 04:09, when the content update was released for the Windows sensor, to 05:27, when the defect was reverted: 78 minutes. Hosts in scope were those on sensor 7.11 and later that were online in that window and received the update. After 05:27, hosts coming online were not impacted, while a host that had already crashed might not stay up long enough to fetch the reverted file; CrowdStrike's first advice was to reboot it, preferably on a wired network, to give it the chance. Seventy-eight minutes on 19 July 2024 Channel File 291 update for Windows hosts, times in UTC from CrowdStrike's review 04:09 update released for the Windows sensor 78 minutes in scope: hosts on 7.11 and up online that received the update 05:27 defect reverted hosts coming online after this were not impacted a crashed host might not stay up long enough to fetch the fix 04:00 05:00 06:00 UTC Three pixels per minute; the band runs from release to revert.
Source: release and revert times, the scope of affected hosts and the note on hosts coming online later from CrowdStrike's preliminary post-incident review of 24 July 2024; the note on crashed hosts from the remediation guidance on the same page, which first advised a reboot to give a host "an opportunity to download the reverted channel file".

Twenty values for twenty-one fields

The root cause analysis CrowdStrike published on 6 August explains the crash. Sensor 7.11, released in February, added a Template Type for spotting abuse of named pipes and other interprocess communication (IPC), and Channel File 291 carried the instances that configure it. In the analysis's words: "The new IPC Template Type defined 21 input parameter fields, but the integration code that invoked the Content Interpreter with Channel File 291's Template Instances supplied only 20 input values to match against."

That mismatch sat on customers' machines for almost five months without harm, because nothing read the 21st field: the test instances and the first instances in production put a wildcard there. On 19 July two new instances went out, and one "introduced a non-wildcard matching criterion for the 21st input parameter". At the next IPC notification from the operating system, the Content Interpreter tried to read a 21st value from an array of 20, and no runtime bounds check stopped it. The crash dump in the analysis shows the index it asked for: 0x14, the 21st element.

A Content Validator checked the file before it shipped, and passed it. The analysis says it "based its assessment on the expectation that the IPC Template Type would be provided with 21 inputs". The validator compared the content with the definition of the type; the sensor ran the content against the code. The two disagreed, and the machines ran the code.

Twenty-one input fields defined, twenty supplied Two rows of boxes. The top row has 21 boxes: the IPC Template Type definition, which the Content Validator checked against, declared 21 input fields. The bottom row has 20 filled boxes and an outlined 21st position past the end of the array: the sensor code passed 20 input values to the Content Interpreter, and where a 21st would sit, the crash dump found a value that was not a valid pointer. An arrow points to that position from a label for the 19 July instance, which set a non-wildcard criterion in field 21, so reading index 0x14 hit a bad pointer past the end of the array. Earlier instances put a wildcard in field 21, so nothing read it. Twenty-one fields defined, twenty supplied The IPC Template Type behind Channel File 291, from CrowdStrike's root cause analysis Type definition, as the Content Validator checked it: 21 fields 1 20 21 Sensor code, values passed to the Content Interpreter: 20 19 July instance: a non-wildcard criterion in field 21 so reading index 0x14 hit a bad pointer past the end Earlier instances put a wildcard in field 21, so nothing read it. one box per input field; the 21st was defined but never supplied
Source: CrowdStrike's root cause analysis of 6 August 2024, its account of what happened, findings 1, 3 and 4, and the crash dump analysis.

The label did the damage

The code that crashed had been through the full release process. Sensor 7.11 became generally available on 28 February and "followed all Sensor Content testing procedures"; the Template Type passed a stress test on 5 March; four instances shipped between March and April and "performed as expected in production". None of that found the bug, because only a particular kind of content could reach it, and none had yet. The change that altered what the fleet executed on 19 July was the file, and it came by the one route with no canary in front of it.

That is why I think the line between configuration data and code was drawn in the wrong place. The review says a Template Type's "runtime behavior is configured dynamically by the Template Instance". The analysis describes the faulting code selecting an input "according to an index specified by Channel File 291", inside a kernel driver that is "loaded from an early phase of system boot". A file that tells a kernel driver which memory to read is a program, and the interpreter is the machine it runs on. The label also trusted the interpreter to make bad content harmless; the review says it "is designed to gracefully handle exceptions from potentially problematic content", and this one "could not be gracefully handled".

The review's account of why the new instances shipped is the clearest statement of the habit. They went out on the strength of "the testing performed before the initial deployment of the Template Type (on March 05, 2024), trust in the checks performed in the Content Validator, and previous successful IPC Template Instance deployments". Two of the three say nothing about the new file. The validator did check it, but only against the type's definition; the analysis's fifth finding is that instance validation should "include testing within the Content Interpreter".

Rollback cannot reach a machine that is down

The file was reverted 78 minutes after release, which is fast. But a revert reaches only the machines that stay up long enough to fetch it. CrowdStrike's remediation hub told administrators what to do when "hosts are still crashing and unable to stay online to receive the Channel File update": reboot to give the host "an opportunity to download the reverted channel file", and if it crashed again, use bootable recovery images, delete the Channel File by hand from a mounted disk, and find BitLocker recovery keys. By the hub's count, more than 97 percent of Windows sensors were online by 24 July compared with before the update, and the root cause analysis put it at about 99 percent by 29 July, against a usual weekly variance of about 1 percent. Seventy-eight minutes to undo the file; days to undo the damage.

A change that can take down the machine that receives it can also take down the channel you would use to undo it. For that class of change, rollback is a recovery plan, and the canary is the only defence that tries the real change on real machines before the fleet has it. On the analysis's account the crash needed nothing rarer than an IPC notification, and named pipes, the technical post notes, "are used for normal, interprocess or intersystem communication in Windows". A fault triggered that easily does not hide from a small sample. I would expect a ring of a few thousand machines to show it long before anyone promoted the file to a second ring.

How many devices each ring of a staged rollout would have exposed A grid of 1,000 squares standing for the 8.5 million devices Microsoft estimated were affected, so each square is 8,500 devices. Each ring includes the ones before it. One red square is a canary ring of 0.1 percent, 8,500 devices. A row of ten squares, the canary included, is a first ring of 1 percent, 85,000 devices. A block of 100 squares, the first ring included, is a second ring of 10 percent, 850,000 devices. The whole grid is everyone, 8.5 million. Halting at the canary exposes a thousandth of the devices. What a canary ring would have put at risk A thought experiment, with rings as shares of Microsoft's 8.5 million estimate Canary, 0.1% 8,500 Ring 1, 1% 85,000 Ring 2, 10% 850,000 Everyone 8,500,000 Halting at the canary exposes a thousandth of the devices, not all of them rings are cumulative each square is 8,500 devices 1,000 squares in all
Illustrative: the ring sizes are assumptions, not CrowdStrike's design; the whole grid is the 8.5 million devices Microsoft estimated in its post of 20 July 2024, so each square stands for 8,500 devices.

Speed is the honest objection. Rapid Response Content exists to respond, in the review's phrase, "at operational speed", several times a day. But speed argues for a short ladder, not for no ladder. The root cause analysis of 6 August describes one: instances "successively promoted to wider deployment rings or rolled back if problems are detected", with "bake-in time" after each promotion. Each ring that bakes for an hour delays a detection by an hour, so a short ladder costs a few hours. Against that, the recovery from 19 July took days, some of it by hand.

The worst case decides

The test I would apply is not what a change is called or what format it is in, but what its worst version could make a machine do. If the answer includes crashing the machine, hanging it or cutting it off from the network, the change is code. It gets a canary ring, a bake time long enough for the failure you fear to show itself, a halt condition written down before the release, and a rollback path that does not depend on the change having worked. Feature flags, routing tables, detection rules, model weights and regular expressions all go through that test, and most of them turn out to be code.

Microsoft's closing paragraph called the incident a reminder of how important it is "to prioritize operating with safe deployment and disaster recovery using the mechanisms that exist". The mechanisms existed at CrowdStrike too. They were applied to the sensor and not to the file, and the file was the program.

CrowdStrikeStaged RolloutsPostmortems
All writing

Written by Mohd Shayan

Get new posts by email

Occasional essays on engineering, AI, and building for the people technology leaves behind.

One email per new post. Unsubscribe any time.

Subscribe with RSS