XZ Outbreak

Lisa Mai Linh Glaziou

Lisa Mai Linh Glaziou

Created : April 11, 2024

I - Quick Presentation of the XZ Attack

A - What is XZ Utils Library ?

XZ Utils, and its underlying library liblzma, are open-source projects that implement the lzma compression and decompression. They are included in many Linux distributions out of the box, are very popular with developers, and are used extensively throughout the Linux ecosystem.

B - The outbreak

The CVE-2024-3094 is a vulnerability recently discovered in the open-source library XZ Utils that stems from malicious code that was pushed into the library by one of its maintainers.

It was originally reported as an SSH authentication bypass backdoor, but further analysis indicates that the backdoor actually enables remote code execution (RCE).
The threat actor started contributing to the XZ project almost two years ago, slowly building credibility until they were given maintainer responsibilities. Such long-term operations are usually the realm of state-sponsored threat actors, but specific attribution does not currently exist.

II - Timeline

2021 – GitHub user Jia Tan (JiaT75) account created. Started contributing to several projects with 546 commits done in 2021, of which the most suspicious one was made to libarchive. A more detailed account of this commit can be found below.

2022, February 6th – JiaT75 submits a first (legitimate) commit to the XZ repo. The commit adds arguments validation to the LZMA and LZMA2 encoders.

2022, November 30th – Lasse Collins, XZ Utils’ creator and sole maintainer so far, changes the bug reporting email to an alias that redirects emails to him and Jia Tan.

2023, January 11th – Lasse Collins releases his final version, 5.4.1.

2023, March 18th – Jia Tan builds and releases their first release, 5.4.2.

2023, June 27-28th – A series of changes were made to XZ Utils, possibly setting the ground for the attack. In these changes, support for ifunc implementation to crc64_fast.c, was added.

2023, July 8th – JiaT75 opens a Pull Request in oss-fuzz, a project that performs fuzz testing on XZ and many other OSS projects. The PR disables ifunc fuzzing, which effectively prevents oss-fuzz from finding the malicious changes done in XZ.

2024, February 15th – JiaT75 adds an ignore rule for build-to-host.m4 in the XZ repository, via its .gitignore file. This script file, soon to be included in actual release bundles, is executed during the package’s build, and contains the malicious M4 macros which initializes the backdoor’s installation on the victim’s machine.

2024, February 23rd – JiaT75 adds the obfuscated binary backdoor in two tests files in the XZ repository :

tests/files/bad-3-corrupt_lzma2.xz
tests/files/good-large_compressed.lzma

2024, February 24th – JiaT75 releases version 5.6.0 with the malicious build-to-host.m4. At this stage, the malicious payload is fully operational (any subsequent XZ version is compromised). Malicious xz-utils version 5.6.0 pulled by Debian, Gentoo and Arch Linux.

2024, February 27th – Malicious xz-utils version 5.6.0 pulled by Fedora.

2024, March 5th – Malicious xz-utils version 5.6.0 pulled by openSUSE.

2024, March 9th – JiaT75 updates the backdoor’s binaries to an improved version, and releases version 5.6.1. Malicious xz-utils version 5.6.1 pulled by Fedora, Gentoo and Arch Linux

2024, March 10th – Malicious xz-utils version 5.6.1 pulled by openSUSE.

2024, March 11th – Malicious xz-utils version 5.6.1 pulled by Alpine.

2024, March 26th – Malicious xz-utils version 5.6.1 pulled by Debian.

2024, March 29th – A detailed account of the malicious activity found in XZ utils was published on the oss-security mailing list by Andres Freund.

2024, March 30th – Lasse Collins, xz-utils original maintainer, made an official announcement regarding the project’s breach.

III - The backdoor itself

A - The vulnerability

The vulnerability lies within the XZ Utils, it’s compression utilities included in most Linux distributions. Versions 5.6.0 and 5.6.1 were affected.

When exploited, this flaw could allow a malicious actor to break SSH authentication and gain unauthorized access to the entire system remotely. Fortunately, widespread adoption of these versions had not yet occurred, minimizing the impact.

B - Backdoor mechanism

The backdoor was not directly inserted into the source code of liblzma that is visible in version control systems or utilized by XZ directly. Instead, it was hidden within binary test files in the XZ compressed format. These files appeared benign and were theoretically part of the library’s test suite.

The attackers used a sophisticated method that split the backdoor into parts, which were then concealed within two XZ compressed files. These files were disguised as ordinary test files, evading detection from casual inspection or automated tools that scan for malicious patterns.

This caused parts of the backdoor to remain relatively hidden, while still being used during the build process of dependent projects.

The shared object itself, compiled into liblzma, overrides the standard function name resolution process. When a process is loaded, function names are resolved into pointers pointing to the process memory, indicating the binary code. However, the malicious library disrupts this resolution process. It replaces the function pointer for the OpenSSH function RSA_public_decrypt with its own malicious function. This function, as detailed in research by Filippo Valsorda, extracts a command from the authenticating client’s certificate (after confirming it as the threat actor) and passes it to the system() function for execution, thus enabling Remote Code Execution (RCE) before authentication.

Fig. 1: The liblzma hooking process

C - Potential Impact

Currently, it appears as though the backdoor is added to the SSH daemon on the vulnerable machine, enabling a remote attacker to execute arbitrary code. This means that any machine with the vulnerable package that exposes SSH to the internet is potentially vulnerable.

The attackers came perilously close to gaining instantaneous access to any Linux machine running an infected distribution, encompassing Fedora, Ubuntu, and Debian. However, their plans were thwarted by Andres Freund. While probing a 500 ms latency anomaly following a software update, Andres meticulously traced the problem to the xz package, ultimately uncovering the backdoor.

IV - The discovery

On the 29th March 2024, Andres Freund posted this on Openwall :

“ After observing a few odd symptoms around liblzma (part of the xz package) on Debian sid installations over the last weeks (logins with ssh taking a lot of CPU, valgrind errors) I figured out the answer:

The upstream xz repository and the xz tarballs have been backdoored.

At first I thought this was a compromise of debian's package, but it turns out to be upstream.”

Andres Freund, a Microsoft engineer, noticed something peculiar. He was using a software tool called SSH for securely logging into remote computers on the internet, but the interactions with the distant machines were significantly slower than usual. So he did some digging and found malicious code embedded in a software package called XZ Utils that was running on his machine. This is a critical utility for compressing (and decompressing) data running on the Linux operating system, the OS that powers the vast majority of publicly accessible internet servers across the world. Which means that every such machine is running XZ Utils.

Freund’s digging revealed that the malicious code had arrived in his machine via two recent updates to XZ Utils, and he alerted the Open Source Security list to reveal that those updates were the result of someone intentionally planting a backdoor in the compression software.