The XZ-factor: social vulnerabilities in open source projects

By our experts

Blog post | 30-05-2024 | NCSC

A Microsoft consultant discovered, more or less by accident, that commonly used open-source software contained a backdoor [1]. He found a backdoor in liblzma, part of the xz compression tool, a widely used component within the Linux ecosystem. The discovery was made by chance when the consultant was experimenting with his own database tool on a test version of the Linux distribution Debian Sid. Debian Sid is the ‘unstable’ release of Debian – a very recent version with the latest code, not intended for production systems, but for testing purposes.

The discoverer was investigating performance issues on a system running the latest version of Debian when he noticed that his management connections via the SSH protocol were slower than usual. He eventually determined through further investigation that a new piece of software was performing extra checks when establishing an SSH connection. It then became clear that the piece of code had been maliciously added to grant SSH access at the system level using a specific key [2].

The malicious code had infiltrated Debian Sid through an ‘upstream’ code commit in XZ [1]. When building a Linux distribution, a lot of software is gathered from various open-source repositories. The code introduced in XZ was thus packaged into the Sid release of Debian. Other distributions such as Red Hat [2], OpenSUSE [3], Kali [4], Arch [5], and Ubuntu [6] were also found to have incorporated the code into their ‘unstable’ test versions.

Social Vulnerabilities in Small-Scale Open Source Projects

In January 2021, a volunteer named JiaTan joined the XZ project [7]. Initially, this user made contributions that seemed neither malicious nor particularly valuable. A number of other users then started a discussion with the main developer of XZ to expedite the implementation of JiaTan’s ‘quality of life’ fixes. Later, it turned out that these other users were probably linked to the attack operation. JiaTan defended the original developer of the project and advised the complainers not to rush, possibly to gain his trust.

After several months of such interactions, JiaTan was eventually trusted by the original developer. The developer mentioned experiencing social and mental problems and welcomed JiaTan’s help by officially granting him author status, allowing him to make his own code commits. This entire process took two years, eventually giving JiaTan the opportunity to make independent code changes to the project. This access was used carefully and cautiously: only about a year after gaining access did JiaTan use it to actually add malicious code. Who this JiaTan was and which actor organized the attack is (still) unknown.

The code was added in a sophisticated, obfuscated manner. It was added to parts of the software that are only seen and used when building new software, not as part of the code that is subsequently built. This code was then reused in the build process of the various mentioned Linux distributions.

Insight in Quality and Operations of Open Source Projects

The discovered backdoor emphasizes the importance of vigilance and thorough checks within open-source projects and among consumers of open-source software. In this blog, we discuss tools that supply chain specialists can use to identify similar security risks in other open-source projects. The goal is to form a picture of the available metrics to identify and manage threats associated with open-source software.

Identifying and mitigating security risks in open-source software requires a systematic approach. An effective way to assess and improve the security of open-source projects is by using the metrics developed by the CHAOSS (Community Health Analytics Open Source Software) organization [8]. CHAOSS is a Linux Foundation project that focuses on creating analytical and evaluation tools to assess the health of open-source projects. Below, we discuss some relevant CHAOSS metrics that can help identify security risks like the xz-backdoor.

1. Code Review

Efficiency The efficiency of code reviews is crucial for early detection of security vulnerabilities. Metrics such as Time to First Response and Time to Merge provide insight into how quickly code reviews take place and how fast changes are implemented. A slow response can indicate understaffing or insufficient security focus, increasing risks.

2. Issue Resolution

The speed and efficiency with which security issues are resolved is another important indicator. CHAOSS measures this through the Issue Resolution Time, which helps understand how actively a project addresses reported issues. A long Issue Resolution Time may indicate a lack of resources or priority for security issues.

3. Community Engagement

An active and engaged community can significantly contribute to better security. Metrics such as Contributor Growth and Organizational Diversity indicate how diverse and growing the community is. Projects with a high level of engagement and contributor diversity are often more robust, as more individuals from different backgrounds review the code.

4. Code Quality

Code quality is directly linked to software security. Metrics such as Code Changes, Code Churn, and Code Review Depth provide insight into how often code is revised and how thorough these reviews are. Projects with frequent, thorough code reviews and low churn rates are likely safer.

5. Risk Management

Finally, measuring the effectiveness of risk management practices is essential. CHAOSS offers metrics such as Risk Assessment and Security Response Efficiency that help determine how effectively a project identifies and responds to security threats.

CHAOSS Metrics in Practice

From the Software Supply Chain Security and Open Source factsheet [9], you may already be aware of the necessity to thoroughly understand the quality and reliability of open-source software projects. The factsheet advocates for a clear view of the supply chain, for both open-source and closed-source software. Developing supply chain expertise, or outsourcing it, is an important step. This helps to gain insight into the Software Bill of Materials (SBOM) and can assign responsibility for each component in the chain.

Those within the chain responsible for this insight need to work to understand the quality and reliability of the relevant components. To gauge the applicability of the CHAOSS Metrics, the NCSC examined a selection of several metrics, limited to characteristics that can be clearly derived from a public source:

Project availability on GitHub;
BUS factor [10]: the BUS factor is the smallest number of people responsible for at least 50% of the contributed code changes. For example, a factor of 1 means that more than half of the project is managed by one person.
Elephant factor [11]: how dependent is the project on large contributors/organizations. For example, a factor of 1 means that more than half of the project is contributed by one organization.
Project age
Number of authors
Number of contributors
First commit date
Last commit date

In this exploration, we selected software packages that are part of Buildroot, a commonly used system for building (mainly embedded) Linux distributions. Buildroot specifies software packages that form the core of many Linux distributions and provides direct links to the source code locations from which they are built. We then limited this selection to packages where the references to the source code involve a git repository. A git repository contains the complete development history of a project in a structured form called commit logs. We further limited the selection to git projects available on GitHub, to also utilize statistics generated by GitHub (outside the git protocol). This filtering step already significantly limited the search space to 241 packages out of the total 3115 used packages in the Buildroot Linux distribution.

Based on the selected metrics in this example, a Python script was used to query the development history of these packages on GitHub and generate an overview. The results were then filtered for projects with:

Age older than 12 years
A BUS factor less than 4, and/or an Elephant factor less than 4 Active
(last commit in March 2024 or later)

Despite the filtering and limitations on data availability, this list still provides a relatively diverse collection of components.

Fig 1. Example of an unfiltered view of Open Source projects on GitHub

The example highlights the limitations of the method, or at least the need for interpretation. For instance, the “open-vm-ware-tools” are included because this package is offered as open source by one company (VMWare). The commits for this project are updated by one GitHub account, but probably maintained and tested by a team of developers within VMWare. This fits the expectation of a VMWare product, whose source is open but the development process is not, having an Elephant factor of 1.

Nevertheless, the overview also provides an interesting insight into widely used packages where more than half of the development relies on two individuals and/or two organizations.

Get Started CHAOSS Metrics right away?

Our experiment shows that gaining insight into quality always strongly relates to subjective interpretation of the used characteristics:

What criteria do we set based on the risks to be mitigated?
Do we find it sufficient that, besides 1 or 2 developers, parties are also contractually liable if something goes wrong?
Is it desirable for one person to control an open-source package, and how do we deal with the resulting risks? Are there more expert parties we have under contract for this? And are those risks thus operationally and legally well covered?

As you read, the standards in the CHAOSS models and the value of an analysis are inextricably linked to the risk assessment and risk appetite of the purchasing organization. Therefore it is a valuable tool, specifically useful for covering realistic scenarios that are relevant to your own organization.

Actively applying and monitoring the CHAOSS metrics requires a relatively high ICT maturity level. After all, applying the metrics for consumers only makes sense if an SBOM is available for the software to be assessed using the method [12]. The SBOM remains the best place to start [13], your organization is able to respond to a High/High advisory [14] when asked, “Do we use this software?”

The step to proactive CHAOSS-metric research can be taken when reactive capabilities like SBOM are in place. It is unlikely that every consumer of a Linux distribution will scrutinize the software in a similar manner, but it raises questions to ask critical partners in your ICT supply chain.

Are you researching this security issue and have questions or findings you would like to share with us? Please contact NCSC-NL.

This expert blog was written by:

Daniël Sierat – CTI Specialist

Ruben Faber – Strategic Cybersecurity Advisor

Bas Schalbroeck – Senior CTI Specialist