The tarball is not the repository

Andres Freund's message to the oss-security list on 29 March 2024 gave its finding in one line: "The upstream xz repository and the xz tarballs have been backdoored." That sentence names two objects, and the gap between them is the lesson. The code that switched the backdoor on, a doctored copy of build-to-host.m4, a standard gnulib macro file that recent versions of gettext pull in, was never in the xz git repository. It existed only in the release tarballs for 5.6.0 and 5.6.1, which the attacker built and signed and distributions compiled. The payload was in git, packed into two binary test files of the kind reviewers skip. "The code is open" was true and beside the point: the artefact that was built was not the artefact anyone reviewed. The rule I take from it is to build from the tag, regenerate what the tag lacks, and treat test fixtures and generated files as code.

Two downloads on one release page

One portion of the backdoor, Freund wrote, "is solely in the distributed tarballs". The line he pointed to in build-to-host.m4 "is not in the upstream source of build-to-host, nor is build-to-host used by xz in git." The release pages offered both kinds of download: the uploaded, signed tarballs carried the trigger, and the "source code" archives, which he thought "github generates directly from the repository contents", did not. Lasse Collin, the project's original maintainer, confirmed on his incident page that those tarballs "were created and signed by Jia Tan", the attacker's alias, and that the test files are harmless "without the trigger code that was never included in the Git repository."

Russ Cox's walkthrough of the script shows the macro, during configure, finding tests/files/bad-3-corrupt_lzma2.xz by a marker string, swapping some characters back with tr, decompressing it and piping it to a shell. That script carves a second one out of good-large_compressed.lzma. The second checks, among other things, for x86-64 Linux, gcc, the GNU linker, GNU indirect-function support and a Debian or RPM package build, and if they hold, edits the generated Makefile so that make links an object file hidden in the test data into liblzma. OpenSSH did not use liblzma directly, but Debian and several other distributions patched it to notify systemd through libsystemd, which depended on liblzma.

Why nobody compared them

None of this needed the tarball to look odd, because autotools tarballs are meant to differ from the repository. "Usually, the configure script and its support libraries are only added to the tarball distributions, not the source repository," Cox writes. "The xz distribution works this way too." His timeline judges the extra macro "not suspicious by itself", since many legitimate ones arrive the same way. Against the standard copy the changes were small: the serial number went from 3 to 30, the copyright line gained 2024, and a handful of other changes did not, in Cox's words, "look terribly out of place". The serial bump mattered: Guillem Jover found that the autoreconf -f -i in Debian's build left alone macro files with a higher serial than the installed copy, so the tarball's newer gettext.m4 survived to pull in the doctored file, which serial 30 would have kept through a gettext upgrade.

The signature was no help either, because it answered a different question. Arch Linux's first build recipe for 5.6.1 fetched xz-5.6.1.tar.gz and its .sig from the GitHub release and listed two valid signing keys, Lasse Collin's and Jia Tan's. A signature binds a file to a key. It says nothing about whether the file matches the repository, and here the key was the attacker's own.

Seventeen months to own the release

The campaign ran for more than two years, and Cox's timeline of the attack shows it aimed at the one role a review of git cannot audit. Jia Tan's first patch reached the xz-devel list on 29 October 2021. From April 2022, accounts that Cox thinks were probably fakes complained about Jia Tan's stalled patch, and by June demanded a new maintainer: "Progress will not happen until there is new maintainer," one wrote. Collin replied that this was "an unpaid hobby project", and on 29 June called Jia Tan "practically a co-maintainer already". Jia Tan was committing directly by 30 December 2022. Collin built his last release, 5.4.1, on 11 January 2023, and Jia Tan built a first, 5.4.2, on 18 March. From then on, the person producing the signed tarballs was the attacker.

On 23 February 2024 the payload went into git as test files, and the next day 5.6.0 was tagged and its tarball published with the extra macro. The object code caused Valgrind errors, so on 9 March Jia Tan replaced the test files, with a commit message explaining that "a constant seed was used to recreate these files", and shipped 5.6.1 the same day.

Found by a stopwatch

It was not found by reading code. Freund was chasing "logins with ssh taking a lot of CPU, valgrind errors" on Debian sid; a failed login took 0.807 seconds instead of 0.299, and the slow step, he wrote, appeared to be the backdoor parsing symbol tables in memory. The damage was contained because the versions were new, and had "not yet widely been integrated by linux distributions, and where they have, mostly in pre-release versions." Debian's advisory lists its testing, unstable and experimental distributions, and Red Hat's alert lists Fedora 40 beta and Rawhide. The NVD entry for CVE-2024-3094 carries a CVSS base score of 10.0.

The rule

It has three parts, one for each gap.

Build from the tag. Fetch the source at the release tag and run the project's own bootstrap, autoreconf or autogen.sh, so every generated file comes from tools you installed, not from the maintainer's machine. Arch Linux did exactly this for xz on 28 March 2024, the day Freund privately warned distributions and the day before the public disclosure: its package switched from the tarball and signature to the git tag v5.6.1 and carried the message "improve reproducibility by running autogen.sh ourselves". That does not remove the payload, which was in git, but it leaves the payload inert, because the trigger never was.

Diff the tarball against the tag, and make the diff explain itself. A bare list of the files that exist only in the tarball would have shown build-to-host.m4 as one more macro added at packaging, which is Cox's point. The check that catches it is regenerating those files from the tag with the maintainer's tool versions, which the files record, and comparing contents, so a macro claiming to be the standard copy is compared with the standard copy.

Treat test fixtures and generated files as code. If the build can read a file, it is an input to the build, whatever directory it sits in. Freund wrote that the two files "were not even used for any 'tests' in 5.6.0", but make check fed every good-* and bad-* file to xz by glob, and 5.6.1 added a test aimed at one of them, so a rule that every fixture be tested would have passed both. Two checks would not have: a fixture described as generated must ship its generator, and a traced build that opens anything in tests/files/ outside make check fails. The README in tests/files said many of its files were made by hand, with no better source code "than the files themselves", which is exactly what an attacker wants in a hiding place.

What the rule does not fix

Building from the tag makes what runs the same as what can be read. It does not make anyone read it. The payload sat in git for five weeks, and an attacker with commit access can put a trigger in git too, dressed as plausibly as a version bump. What the rule changes is where that attacker has to work: every step becomes a public commit that anyone can diff and bisect, not a file that existed only on a release page. That is a smaller hiding place, and the only one review was ever going to cover.

"The code is open" is a statement about a repository. A build consumes whatever it is handed. The two are the same only when the builder makes them the same, and in March 2024 both Debian and Arch were building xz from the tarball.