SECURITY 101
Why Secret Scanning Visibility Should Be Free & Understanding Where There is Value
July 11, 2023
If you're paying for a secrets scanning solution... make sure you're getting real value from it.
The exposure of sensitive credentials has been a problem since the inception of application security. And yet, despite consistent efforts – and a laundry list of tools – to address the problem, the infamous secret in code continues to wreak havoc. And it's unsurprising, considering that 80% of IT / DevOps team members believe that their organizations are not managing secrets effectively.
In 2022, within a month, Uber and Toyota both experienced very public breaches tied to exploited secret exposures and secrets continue to be a thorn in the side of AppSec teams as we move past the midpoint in 2023.
In the early 2000’s, the infosec community conversed principally through mailing lists like SecurityFocus and Full Disclosure, each list centering on a different topic with varying levels of technical depth. One of which was a DataLoss Mail List, dedicated to surfacing and discussing news stories about the public disclosure of sensitive data. A frequent source of stories came in the form of backup tapes either being forgotten about in storage or lost in transit. In those days, backup tape encryption was typically an expensive add-on device for your backup system. Most folks working in IT weren’t on the DataLoss Mailing List, so most IT folks didn’t think that exposure of unencrypted backup media was happening several times per week.
Seemingly, though, word had gotten out to the folks at the LTO Consortium, an organization of backup system manufacturers who maintains the industry standard, open-source Linear Tape Open backup tape format. Beginning with LTO-4 in 2007, all conforming tape drives would come with encryption capabilities standard. This was a year before the DataLoss Mailing List was turned over to the Open Security Foundation and turned into an excellent searchable database hosted at datalossdb.org. But over time as companies upgraded their backup systems, those stories of tapes falling off the truck on the way to the Iron Mountain dried up. And today the loss of backup tapes is such a non-issue that datalossdb.org doesn’t exist, its records have been expunged from archive.org’s Wayback Machine, and the Open Security Foundation is an Indonesian gambling site.
The backup tape industry came to realize that encryption technology is so cheap and the effects of not encrypting backups were so harmful that it just made sense to stop nickel-and-diming the marketplace. This story may ring a bell as secrets and secret scanning seem to be on a similar path.
Imagine if you had a contractor come by to check out that old deck you need fixed. As they walk out the door, they hand you a bill for the pleasure of telling you what you already knew for free: you need to fix your deck. You might view periodic secret scanning similarly.
Given the ease of scanning and the ample open-source tools available to regularly scan for secrets, we believe that the value is in the fix. Any paid secret scanning effort should include paths to mitigate secrets found in your code base. They should provide full context of the secret (where it is, who pushed it, how long it’s been exposed, etc.), identify the person best suited to easily fix the secret, and deliver to that person actionable recommendations for how to do so. Better yet, a paid secrets solution should actually be able to stop new secrets from being added to code before they ever reach feature branches or production – more on that momentarily.
Furthermore, there has, over the past half decade, been a proliferation of git secret scanning tools that provide a vital component in maintaining the integrity of codebases. As a result, periodic secret scanning has effectively become a widely available commodity. This is largely due to the successful integration of popular open-source initiatives like GitLeaks, Git-Secrets, and Detect-Secrets into the development lifecycle. These open-source tools have been seamlessly incorporated as mandatory Pull Request checks prior to the merging of code and initiating CI/CD pipelines. As such, any secret scanners that primarily offer visibility into code might logically be made freely available.
Just because you shouldn’t pay for periodic secret scanning doesn’t mean that you shouldn’t pay for any secret solution. In fact, given how pervasive the secrets problem is... a solution that can scan in real time, identify ownership, and solve the problem (as opposed to just telling you there is a problem) is invaluable.
As opposed to the majority of open-source scanners, which provide scanning at regular intervals (leaving a gap), real time secret scanning provides a tremendous amount of value. Real time scanning (paired with secret mitigations, which we will cover shortly) enables application security teams to enforce a ‘zero new hardcoded secrets’ policy. By scanning on every event (commit, push, pull request), you can detect and eliminate secrets before they ever enter your pipeline or production.
The impact of a ‘zero new hardcoded secret’ policy cannot be overstated. If you’re trying to bail water out of a sinking boat, plugging the leak first makes it dramatically easier to make progress toward your goal of keeping the boat afloat. In the same vein, if you can diminish (or eliminate) the number risks (here, secrets) that get added to your software development ecosystem, your team is in a much better position to reduce their security backlog (aka bail out the water that is in the boat).
Gone are the days where expensive enterprise security tooling can get away with just finding risks and calling it a day. And a tool’s “automated” approach is to automatically create tickets in Jira – where developers don’t want to be – well, that probably won't cut it.
Modern application security tools, like secret scanners, should make mitigating risks as easy as possible. This means striving for automated mitigations that keep developers in their workflows, leveraging the tools they use. Using automation in GitOps to tee-up a mitigation and in ChatOps (think direct messages in Slack) to push the mitigation across the line will keep developers focused without security negatively impacting their velocity.
In addition to the critical improvements to developer experience, this approach also makes for a far more secure product when developers are empowered to fix risks in real time. The downstream impact of fixing secrets in code in real time (or as early as possible) is that it reduces the likelihood that someone knowingly or unknowingly clones a repository that contains a secret, further expanding the sprawl of that secret exposure.
Identifying and eliminating secrets before they make it to the pipeline or production is critical. But newly introduced secrets are not the only problem. Nearly 2/3rd of DevOps and IT employees believe that their companies’ code repositories contain more than 500 secrets. That means, even if you stop introducing new secrets, there is still significant risk of breach or leaks due to existing secret exposure.
Eliminating existing secrets can be difficult. To make this process as easy as possible, secret scanning solutions should help you identify the best owner of the risk – aka the person best suited to easily fix the secret. For example, if the person who pushed the secret originally is still in the same role, they’d likely be the person best suited to rotate the secret given they have the best understanding of the code and context of what that secret is being used for.
If the developer that pushed the code is no longer at your company, then what? Establishing a clear understanding of product ownership will help identify the person or team best suited to understanding the code in question and fix the secret risk within that code.
The ability to scan code for secrets has become commoditized. And yet, secrets continue to be a major problem for application security teams and developers. As you decide how your organization is going to approach the secret problem, make sure that if you pay for a secret scanning solution, it is actually that: a solution.
To really start tackling the secrets problem, challenge the industry. Look for an ability to scan for secrets that already exist in code as well as scan in real time to stop the bleed and allow you to bail out the water from the boat. Look for an ability to mitigate secrets in ways that are non-disruptive (or even delightful) to the development process. And look for a tool that does the leg work for you – identifies the who and simplifies the how.
