N-able implemented a flexible, developer-friendly application security solution that improved efficiency, prioritization, and workflows, while enhancing collaboration between security and developers.
N-able builds software for Managed Service Providers (MSPs). N-able’s platform empowers service providers with secure backup tools, remote monitoring and management, endpoint security, DNS filtering, and more. In 2023, N-able set out to overhaul their Application Security program tooling. This effort was led by the Application Security team as well as a core group of volunteer security champions within the company and was focused on overcoming a number of friction points with existing tools. We sat down with the N-able Application Security team to hear first-hand how they are leveraging Arnica. Their thoughts are below.
The Challenges
Building an effective AppSec program with bandwidth & resource constraints
At N-able, our Application Security team supports a large number of developers relative to our team size. We also are tasked with implementing our application security program across a diverse set of products with thousands of repositories. Where our existing AppSec solutions would frequently take hours to scan a PR on our largest repositories, we needed a set of tools that would complete scans quickly, without interrupting developers, allowing our AppSec program to scale alongside our fast evolving and expanding development environment. As we set out to evaluate new tools, our primary focus was to find a tool that allowed AppSec to implement security while allowing our product teams to do their jobs, without friction.
Another motivation of this refresh was to consolidate what was previously a highly disparate in-tool developer experience across our various finding types (SAST, SCA, and everything else). We were navigating a large list of findings dropped into GitHub with instructions for developers to “log into the tool to see everything there.” Even within the tool, it wasn’t immediately clear what developers were expected to fix, except for whatever was “most critical.” It created a disjointed developer experience that didn’t give us the consistency we wanted and required significant individual training on how to use the tool. With Arnica, we’re communicating on the PR and in Teams what ‘must’ be fixed. Over time, more will be deemed ‘must fix’, but Arnica gives us the opportunity to walk, then run, before asking developers to sprint.
Beyond scalability and consolidation, a primary driver of our evaluation of Arnica was that our security team was not getting the level of needed support from our previous SAST & SCA vendors, which ended up slowing our efforts to make our desired improvements.
Too many AppSec findings and unclear prioritization
Another core problem we needed to overcome was ineffective prioritization of the findings coming from our existing application security scanners. Developers can’t solve every single issue that we find; even our most partnership-minded security champions were very vocal about the laundry list of findings they were getting with, in many cases, just a single metric of severity: CVSS. Our existing tools provided little ability to correlate severity across different industry severity scores and none of them provided any unique context based on our specific organization.
Poor developer experience inhibited security champion program success
Our security champions have been instrumental in evolving our security program. With such a high developer-to-security ratio, our security team relies on our champions to both inform and evangelize our efforts, but unless our program is worth evangelizing, even the greatest champions will have challenges.
With our previous Application Security solution, N-able was provided with few configuration options – we couldn’t block on a detected risk, let alone do so granularly. With limited ability to purposefully configure policies, we were not able to implement workflows for our developers that created measurable and consistent impact on application security outcomes. We needed to find a tool that allowed us to implement our Application Security controls in a flexible way, without causing headaches for the champions or locking us into a particular vendor's way of thinking.
To ensure the success of new security oriented workflows, we wanted to find a solution that could find and fix risks before the build. We established early in our evaluation process that we didn’t believe in post-build SAST and SCA scanning. At that point it’s too late.
The Solution
Massive efficiency and efficacy gains
We needed a highly configurable integration into GitHub that was able to elegantly accommodate our technologically diverse development environment. With Arnica, N-able was able to deploy our solution across dozens of GitHub organizations, containing thousands of repos, easily. Scan times have dramatically reduced and, because of the pipelineless deployment into our source code management (SCM) tool, we know that any new repository that gets added is automatically covered by Arnica. Not only was the deployment straightforward, we were also able to establish consistent workflows across our AppSec tools in order to create a far more cohesive experience for both developers and our security team.
Our previous solution made it difficult to track developer engagement and to correlate issue fixes to the tool or its workflows. Very quickly upon deploying Arnica we saw a measurable uptick in issues being resolved. We were able to take our boss aside and point to 5 different developers who, in a 3 day span, received an Arnica alert about a vulnerability and then immediately fixed it. This was a direct result of Arnica providing our application security team with the ability to match our workflows to our developers as well as visibility into the fact that these workflows were working.
In our evaluation of Arnica, the icing on the cake was how collaborative the Arnica team was in both positively impacting our roadmap and incorporating our feedback into their roadmap as well. The Arnica team was able to quickly understand our vision for our application security program and be a partner to us in pursuing that vision. Whether it’s on feature enhancements suggested by our team or a bug, Arnica is right there with us to work through any issues quickly. In our experience, Arnica has treated us like a partner, not just a customer.
Clear AppSec risk prioritization & actionable findings
With Arnica, we were able to establish policies that are much more acutely aligned to our desired definitions for severity and priority and build our program around that. How many critical severity vulnerabilities are there that have never been exploited compared to mediums that have been exploited? Arnica helps us reduce noise by providing metrics on the likelihood of exploitation and reprioritizing critical severity vulnerabilities based on Arnica’s logic, exposing the most important risks to deal with immediately. We set all of this up in the first month.
With Arnica up and running, we established our top priority SCA findings and we were able to get granular about what we were going to focus on. We established a highly refined definition of severity that fit our program and we established an explicit agreement with our security champions and the engineering teams that we were only going to surface findings that had a fix. For SAST, which typically has a fairly high false positive rate, we again defined a specific scope of focus and again came to an explicit agreement with engineering that we would only block on these agreed upon priorities, defined in Arnica’s policy engine by our team.
Now that we’ve found our rhythm with Arnica and established a baseline, we can communicate to engineering, “here’s what we’re focused on next.” This is one of the biggest wins we got from Arnica. Anyone who’s implemented a tool that could possibly impede developers from doing their jobs, knows how dangerous that is. We have very strong security standards, but we needed a solution that was flexible and powerful enough to implement those standards within our tools in a way that integrates with engineering teams’ workflows. Using Arnica to thoughtfully implement controls in adherence with our standards enables us to advance our program maturity.
The other tools we tried were not as effective in surfacing what was most important to us and injecting that into our developer workflows. Arnica is built that way out of the box.
Workflows that work with developers and drive security outcomes
Our most significant takeaway from moving to Arnica is on the flexibility of the policy engine and how granularly we can shape our deployment based on the mandates of the security program that we designed.
With our previous tool, it was often not clear to developers which findings should be prioritized first and at times could seem like they were on the receiving end of a security finding firehose. Now, our security team is able to build policies that direct engineering focus and prioritization in a targeted and purposeful way. And, within the confines of our policies, they’re able to respond in a way that is best suited to the moment and the finding. It’s not just block or don’t block. Arnica empowers our developers to comment, to ticket, to auto-fix, to be guided to a fix. And, yes, when needed Arnica can block the Pull Request when we deem it necessary.
Moving beyond the Security/Developer relationship, Arnica has also allowed us to mirror our written Legal policies within the Arnica platform. This has allowed us to block components with forbidden licenses from being included in our code. Again in this instance, Arnica allowed us to create workflows that model our internal security standards. In effect, this means our chosen AppSec tool has consistent alignment with our internal commitments. Each security control that we are able to implement directly in Arnica is a control we don’t have to build a separate process for– be it automated, custom script, or manual. This takes tedious process work off of the security team’s plate and frees us up to do more valuable work.
Arnica's flexible solution and policy engine gave us the opportunity to iteratively layer-in new workflows and controls to give developers time to acclimate to changes and provide feedback, rather than overwhelming them with an unfocused list of findings. As a result, the developer feedback has been positive and security impact has been measurable compared to our previous solution.
The Impact of Using Arnica to Build our Application Security Program
We originally planned to run Arnica in parallel with our other solutions for a period of time, to make sure we were comfortable with our new AppSec solution. But it didn’t take long before we abandoned that approach and moved from our previous stack to Arnica.
The efficiency gains, Arnica’s ability to fit the program we designed (especially via Arnica's flexible policy engine), and the positive impact on developers' ability to drive security outcomes all made the decision easy. The traditional application security tools we tested were overly oriented toward security engineers rather than developers, who actually remediate the issues. Additionally, we found the other tools to have rigid workflows, which would have required us to bend our program to meet their capabilities, rather than the other way around. Arnica distinguishes itself from the other application security tools we tested with their clear focus on the user experience for developers and providing flexibility to Application Security teams in order to implement workflows that conform to our program and its goals.
