SECURITY 101
Harnessing the Power of Secure Coding Practices for Effective CI/CD Security
February 13, 2023
This article will explore how development practices can compromise CI/CD security. Several kinds of problems will be explained, including practical resolutions for each one. Readers will learn what to look out for and how to protect themselves, both with their own vigilance and automated tools.
CI/CD pipelines increase the productivity of development teams by automating testing, quality control, and release procedures. They reduce your time to market, make it easier to measure progress, and improve visibility into the content of each change.
All this comes at a price: CI/CD carries security implications that can be challenging to address. In a worst-case scenario, hackers can abuse unsecured pipelines to ship malicious code to your customers.
Addressing the risks starts with the source code that moves through the pipeline. You need to understand who's committing, whether they're authorized to do so, and if they're making unsafe code revisions.
In this blog, we'll explain why your source code is so critical to CI/CD security by covering the common ways that pipelines can be compromised. Afterward, we'll look at how you can lessen the CI/CD security burden using "shift left" approaches and modern behavior-based anomaly detection tools.
Part of the problem around CI/CD security is the topic's relative lack of exposure. While most teams now understand the threat environment around their cloud resources, they often overlook CI/CD pipeline configurations.
It's tempting to assume that major providers like GitHub Actions, GitLab CI, Jenkins, and CircleCI are safe to use off the shelf. However, such implicit trust is misplaced because so many CI/CD threats come from your own source code. This means security must start with the developers who create the code.
What's stopping developers from committing malicious files to a project? If your answer is "nothing," then your CI/CD environment is at risk. Developer accounts are privileged assets that often have comprehensive read-write access to repositories. A compromised account, or one owned by a disgruntled employee, could be weaponized to push arbitrary code into your pipelines.
Besides deliberately causing harm, developers may also unintentionally introduce threats by neglecting security best practices or unknowingly installing unsafe third-party project dependencies. The outcome is the same in each case: Untrusted code could be executed in your environment and might be pushed through the CD pipeline to your customers.
Branch protection is one effective mitigation for these risks. Protecting source control branches, such as main or master, prevents non-whitelisted developers from merging into them without first obtaining independent approval. This prevents code from being shipped into production by a single account that might be rogue, compromised, or simply unaware of a new threat.
You can further guard against unsafe code by expanding your pipeline with automated vulnerability scanning. Run a tool like Trivy or Grype early in your pipeline, then abort subsequent stages if you find any vulnerabilities. This has two benefits: Unsafe dependencies are highlighted for you, and any risk of them compromising your CI/CD environment is solved by terminating the pipeline before they're executed.
Developers who either unknowingly or deliberately revise sensitive files undermine your source code’s integrity. Some files are supposed to be restricted to specific users so you can audit changes and ensure continual compliance.
One sensitive resource could be your CI pipeline's configuration: Files such as .gitlab-ci.yml or .github/workflows/deploy.yml are often configured by an operations team, with no developer input expected. What happens if a developer wants a change, edits the file themselves, and inadvertently introduces a security issue? Ordinarily, there's no way of being alerted to the problem.
Branch protection isn't sufficient here because the changes have immediate implications, irrespective of the branch they're committed to. You can address this risk by using the CODEOWNERS file to associate source files with the users responsible for them. A simple CODEOWNERS file looks like this:
The example states that changes to .gitlab-ci.yml should be approved by the @devops-lead user. Enabling the "Require code owner approval" setting in your source control platform (see here for GitLab and here for GitHub) will enforce that the pull requests (PRs) modifying the file can't be accepted until @devops-lead has reviewed them.
For even stronger protection, combine this mitigation with a rule that mandates PRs are also reviewed by someone other than their author. This guards against compromise of the @devops-lead account—despite being the code owner, they'll be unable to silently merge new threats into the main branch.
Git is not impervious to account takeover attacks. There are several ways in which an attacker could gain control of an account, such as a security lapse at your provider, social engineering and phishing techniques, or simple brute-force efforts against usernames without 2FA enabled.
Bad actors don't necessarily need to take over an existing account though: Impersonation and commit spoofing are equally significant problems, especially for public projects that anyone can join. Git's user.name and user.email settings can be set to anything you like, without verifying you're actually that individual. You can even go back and modify the authors on existing commits.
This means that bad actors can push changes while claiming to be a project maintainer. These malicious files are less likely to be spotted and questioned. They might even bypass tools in your pipeline that work based on the commit author, instead of the user account that made the push.
Fortunately, you can easily defend against this risk by using signed commits. This Git feature signs your commits using GPG or SSH keys. You upload the public key to your Git host, allowing it to verify that commits claiming to be made by you are authentic. Once you've set up signing for all your users, you can reject unsigned commits at the point they're pushed, or create a workflow that automatically triages them and quarantines the CI pipelines they create. Signing commits is not always possible though; for example, developers may work from untrusted machines to which they do not want to deploy their private key. It’s important to supplement or substitute signed commits with alternative impersonation detection mechanisms, such as anomalous developer behavior analysis.
Accidentally committed sensitive values such as API tokens, passwords, and certificates are risky because they could end up being written to CI/CD job logs. These logs are normally stored in plain text and may be exposed to users who shouldn't be able to access the secrets within them.
Using a dedicated secret detection CI job is the most effective way to identify these problems early on. Detectors analyze your source code using predefined and custom pattern matchers. The pipeline should be failed if a secret is discovered so the developer can remove it and rewrite the commit history. This prevents bad actors who manage to clone the repository, access the CI server, or retrieve job logs from authenticating to your third-party services.
With so many risks stemming from your source code, what can you do to protect your pipelines and secure your supply chain? Here are some effective methods that are easy to set up and maintain.
It all starts with the right mentality. Security's often a checkbox exercise at the end of a project. This is insufficient protection for all but the simplest of applications. If you're only thinking about security after code's been committed, merged, and deployed, then you're several steps behind where an attacker might be.
Shift-left security refers to tightly integrating security earlier in the development process. You should protect your pipeline at the point that code is committed. Several of the techniques we saw above are good examples: Configuring branch protection, CODEOWNERS, signed commits, and vulnerability analyzers can all stop unsafe code from ever reaching your CI/CD servers.
Software composition analysis (SCA) is another mechanism for discovering threats before they become irreversibly included in your project. SCA tools index your dependency graph, report vulnerabilities, and highlight compliance risks. With modern applications built on tens, hundreds, or even thousands of third-party packages, understanding your stack's composition is essential to accurately perceive its security posture.
We've clearly seen how developers are often responsible for new threats entering your codebase. Simply limiting what developers can do is often an effective defense, but it’s important not to disrupt engineering workflows. A modern permissions approach needs to be able to implement and maintain the principle of least privilege, without slowing down legitimate development activities.
Limit developers from pushing to restricted production environments, remove excessive access to sensitive job logs, and ensure they can't perform non-relevant administrative tasks such as modifying the CI/CD configuration or registering new job runners. Once you’ve implemented these protections, check that they’re not impeding the normal development workflow: Streamlined access to testing environments, staging servers, and development branch job logs will always be necessary to maintain good throughput.
Automation is another effective way to protect yourself. Beyond basic vulnerability scanners, there are advanced modern techniques that surface CI/CD threats in real time.
Arnica is a purpose-built solution for automating software supply chain security, from permissions management and anomaly detection to the contents of your source code. Arnica makes policy-based decisions in real time as new activity occurs. You'll get notified for each alert, such as when an unsigned commit is pushed or a secret is found in the repository. This reduces the burden on operators and security teams to manually find and fix these issues.
CI/CD is integral to modern software delivery workflows. It automates tedious tasks, increases consistency, and accelerates time to market. However, it also carries security risks because developers can introduce unsafe code that could be executed in the CI/CD environment. Even worse, compromising the repository allows attackers to abuse your server's resources or ship code straight to your customers.
We've looked at several ways to mitigate these risks by shifting security left, closer to the developer. Arnica uses a pipelineless approach to software supply chain security. This means security teams can easily establish and maintain 100% security scanning across the software supply chain from day one opening the door to run security workflows earlier and more often without requiring any code changes in the CI/CD pipeline. Additionally, pipelineless approach enables targeted alerting to the person/team with a personalized context and ability to easily fix an identified risk. Arnica also empowers the recipient of the alert to be able to fix the risk with a single click or automated policy.This increases the likelihood that threats will be recognized and mitigated before they enter the pipeline.
While manual reviews and basic security hygiene help, you should adopt a behavior-based real-time detection engine like Arnica to achieve the most robust defense. Arnica automates the security of your entire software supply chain. It will spot anomalies for you, provide visibility into threats, and vet new source code and dependencies before they cause a problem. This empowers you to confidently use CI/CD without constantly looking over your shoulder.
