SECURITY 101
Why secrets continue to be a massive problem in source code
May 30, 2023
Secrets in code have been exploited in a number of prominent data breaches in recent years. And yet, despite numerous secret scanning tools in the market, secrets are still often mishandled, causing them to be exposed in plain text wherever source code appears. To combat the exposure of hardcoded secrets, consider investing in secret scanning tools that allow developers to efficiently detect and remove problematic values before they get merged into your main branch and pushed to production.
In this article, we explore the many issues caused by exposed secrets in source repositories, as well as strategies for detecting and preventing commits that include secrets.
Leaked secrets pose a critical threat to an organization's software supply chain security, as secrets expose sensitive values such as API tokens, passwords, private keys, and certificates. Threat actors can then access your private data or perform privileged actions that target the services you depend on. Leveraging exposed hardcoded secrets is one of the most common methods for attackers to obtain sensitive data.
Hardcoded secrets have contributed to many prominent security incidents over just the past year.
In the widely publicized September 2022 Uber breach, attackers who had managed to enter the company’s network discovered a PowerShell script containing hardcoded credentials for its access management system.
A month later, Toyota announced they had inadvertently published code to GitHub that contained an access key capable of retrieving the data of over 296,000 customers. The incident demonstrated the risks of using industry-standard public tools, as incorrect GitHub security controls expose code to the open internet.
These incidents illustrate the importance of maintaining the strictest privileged access to secrets, even more so than to source code. Once a secret makes its way into your source code, anyone with view access to the repository—project managers, designers, contract developers, or team members—will be able to see it.
While source code is constant across different deployments, secrets are configurable values. You run the same code in development and production, but should use different secrets for databases, APIs, and other components that require authentication. When values are hardcoded, they can no longer be easily changed to suit different contexts.
Secrets should be highly restrictive and easy to change, but hardcoded secrets are widely accessible and difficult to modify, making it harder to take action when code is leaked containing secrets. Committing a new value is the only way to replace or rotate the secret, and even then, the new secret could still be exposed as it was initially.
Finally, it is challenging to audit access to hardcoded credentials. When secrets are visible in source code, it is challenging to identify who has accessed a value or what they used it for. Moreover, hardcoded values can easily spread across multiple files and repositories, creating a secrets sprawl that makes it difficult to get an accurate picture of your secrets inventory.
Secrets enter repositories when they are written into source code and committed by developers. For non-developers, this might sound crazy. But, hardcoded credentials have been used for configuration since the beginning of software development. This is because it is much quicker to hardcode a value when writing code than to take the time to establish a dedicated secrets management system, so developers are often compelled to take the easier route.
Developers may also lack training in proper secrets management and source code security, which increases the possibility of such mistakes. Yet even when developers are in fact aware of the risks and attempt to follow best practices, hardcoded secrets can still slip into repositories. Test values may be committed without the developer realizing it, or the developer may simply forget that a specific value should be treated as a secret.
There are often little to no controls to stop developers from committing secrets. Even with effective code review policies in place, secrets can be easily missed. Worse, a review might have just been rubber-stamped, and in any case the secret will have entered the repository by the time code is reviewed.
In addition to source code, secrets are just as likely to appear in your CI/CD and infrastructure as code (IaC) configurations, with equally damaging results. Consistently managing secrets across your organization is therefore key to keeping them out of your repositories.
Since secrets can easily slip their way into code, teams may be unaware it has happened until a leak or breach occurs. An effective secrets strategy must, therefore, address existing hardcoded secrets through historical secrets detection. It must also support real-time detection to prevent new secrets from being added to the repository. This can be achieved through the following methods.
Any secrets should be detected in real time, as code is pushed. Triaging secrets before your pipelines execute reduces the time they are visible in your repository. If your CI/CD infrastructure is compromised, malware can steal any secrets present in your code. Detection of hardcoded secrets in CI/CD offers an additional layer of security and provides developers with real-time notifications.
Because scanning tools like Gitleaks, GitHub secret scanning, and GitLab secret detection usually run as part of a CI/CD pipeline, they aren’t comprehensive enough to catch all hardcoded secrets. By the time they are executed, a secret may have already been pushed to the repository and exposed.
Once a secret has been found, it must be removed from the repository immediately. Simply pushing a new commit that deletes the secret isn't enough, since the file's previous versions will still exist in the repository's history.
However, rewriting Git history is a complicated and relatively uncommon task, making it challenging for developers to clean up after a secret has been committed. Moreover, changing a chain of previous commits often introduces conflicts that can be difficult to resolve. The Git manual even warns that branches you have pushed to others “should never be rewritten.” This further illustrates the importance of discovering and removing secrets as quickly as possible, since having fewer affected commits simplifies remediation.
Ideally, the process of detecting and removing committed secrets should be fully automated. Creating a simple, seamless process will facilitate timely and effective resolution.
To avoid ongoing secrets management issues, developers must be educated on what constitutes a secret, how they should be handled, and how to address hardcoded secrets as part of their regular workflows.
Even when following best practices, mistakes are inevitable. It is therefore critical that developers have access to the tools they need to detect and fix hardcoded secrets easily. If secrets are regularly being committed, however, this is an indication that additional support is needed to help developers recognize the risks associated with secrets and to adopt safer coding practices.
Implementing proper procedures for storing secrets after they have been removed from source code is also key to supply chain security. Adopting a secrets manager, such as AWS Secrets Manager, Google Cloud Secret Manager, or HashiCorp Vault, can solve secrets storage and access challenges, but there is often a significant learning curve involved. Integration can also be costly and time consuming if engineers are unfamiliar with the available solutions or are unable to recognize which values constitute a secret.
By providing clear documentation on how to use these tools, developers will be more likely to rely on them—even for tasks such as local testing, where they may have used hardcoded secrets many times before. Guidelines such as OWASP's Secrets Management Cheat Sheet can also serve as a helpful reference point.
Hardcoded secrets are a critical software supply chain risk, as they become accessible to everyone with repository access, anywhere the code runs. Moreover, secrets written into source code are difficult to change. You can't rotate the secret without pushing a new commit that affects every instance of your software. There is simply no justification for adding secrets to source code; yet many developers persist in doing so, because it is simple and convenient during testing.
Arnica, as part of its software supply chain security platform, provides real-time and historical secret detection tooling paired with a simple secret mitigation experience for developers. Any secrets that have been identified are then validated to ensure they are active, thus minimizing alert fatigue. Arnica provides instant developer notifications along with the required context to quickly address the issue.
For secrets detected within a code push, Arnica offers one-click automated mitigation on Slack and Microsoft Teams, to minimize disruption to developer workflows. It masks the detected secret and deletes its history, leaving no trace in the repository. All this is achieved in a single click, without requiring knowledge of the associated Git commands. This supports a “zero-hardcoded secrets” policy by enabling developers to respond to alerts and take immediate action.
