Five Core Tenets Of Highly Effective DevSecOps Practices

One of the enduring challenges of building modern applications is to make them more secure without disrupting high-velocity DevOps processes or degrading the developer experience. Today’s cyber threat landscape is rife with sophisticated attacks aimed at all different parts of the software supply chain and the urgency for software-producing organizations to adopt DevSecOps practices that deeply integrate security throughout the software development life cycle has never been greater.

However, HOW organizations go about it is of critical importance. For example, locking down the development platform, instituting exhaustive code reviews, and enforcing heavyweight approval processes may improve the security posture of pipelines and code, but don’t count on applications teams to operate fluidly enough to innovate. The same goes for application security testing; uncovering a mountain of vulnerabilities does little good if developers have inadequate time or guidance to fix them.

At a high level, building and running a DevSecOps practice means that your organization is able to operate a secure delivery platform, test for software vulnerabilities, prioritize and remediate vulnerabilities, prevent the release of insecure code, and ensure the integrity of software and all of its artifacts.

But building and running a highly effective DevSecOps practice means achieving all of these objectives at the same (or higher) development velocity and overall level of developer satisfaction. The following five guiding principles are essential to getting there.

Tenet 1: Establish a collaborative, security-minded culture

A strong and productive culture is essential to the success of any team but it’s also the hardest element to get right. This is especially true of DevSecOps, as evidenced by a recent industry study revealing that “over half (51%) of IT decision makers report outright resistance to change among their teams whilst 47% say there is insufficient cross-team collaboration[1].”

The importance of culture for successful DevSecOps shouldn’t be underestimated, and it starts with accepting security as a priority for all stakeholders.

Make security a shared responsibility

If your organization builds, sells, or consumes software (which today is every conceivable organization on the planet), then every single employee has an impact on the overall security posture– not just those with ‘security’ in their titles. At its core, DevSecOps is a culture of shared responsibility, and operating with a common security-oriented mindset determines how well DevSecOps processes fit into place and can drive better decision-making when choosing DevOps platforms, tooling, and individual security solutions.

Mindsets don’t change overnight, but alignment and a sense of security accountability can be achieved through the following:

• Commitment to regular internal security training– tailored to DevSecOps– that includes developers, DevOps engineers, and security engineers. Skills gaps and needs shouldn’t be underestimated.
• Developer adoption of secure coding methodologies and resources
• Security engineering contributes to application and environment architecture, and design reviews. It’s always easier to identify and fix security issues early in the software development lifecycle.

Break down functional silos and collaborate continuously

Since DevSecOps is a result of the confluence of software development, IT operations, and security, breaking down silos and actively collaborating on a continuous basis is critical for success. Typically, DevOps-centric organizations operating without any formal DevSecOps framework see security entering the picture like an unwelcome party crasher. Process changes or tooling that are suddenly imposed (as opposed to collaboratively chosen and instantiated) invariably results in development pipeline friction and unnecessary toil for developers. A common scenario involves security mandating additional application security checks without consideration for their placement within the pipeline, or for how much workload is required to process scanner output and remediate vulnerabilities, which inevitably falls to developers.

Driving collaboration and operating as a cohesive DevSecOps team involves:

• Defining and agreeing upon a set of measurable security objectives, such as:
  • % decrease of application security incidents
  • % decrease time spent on audit
  • % increase in deployment frequency
  • % decrease in change failure rate
  • % decrease of vulnerabilities deployed to production
  • % of artifacts deployed to production with SBOM/SLSA
  • Decrease in lead time to zero-day vulnerability remediation

• Involvement from software developers and DevOps teams throughout the evaluation and procurement processes for new security tools
• Ensuring no DevSecOps process has a single functional gatekeeper
• Iteratively optimizing tooling choices and security practices for developer productivity and velocity

Tenet 2: Shift security information left, not security workload

Broach the subject of DevSecOps and it’s impossible not to mention ‘shift-left’. The shift-left security mantra is so prevalent in current DevSecOps-oriented articles, blogs, and marketing collateral, it’s easy to think that by simply moving security checks further upstream in the software development lifecycle you’ve achieved a working DevSecOps program. The reality is that WHAT you shift left is what makes or breaks your DevSecOps success.

Shift left security is predicated on the proven idea that performing application security tests earlier in software development pipelines (as opposed to just prior to production) results in a better overall chance of catching known code and artifact vulnerabilities and remediating them in a timely manner. However, if developers alone bear the entire burden of running tests, collecting scanner output, and prioritizing vulnerabilities on top of remediating them, the resulting mental load and toil is certain to impact the speed to production. Instead, the best approach lies in following these guidelines:

• Security should own the orchestration and automation of application security tests throughout CI and CD pipelines
• Remove the burden of deduplicating and prioritizing detected vulnerabilities from developers. Instead, security should ensure developers get a fully processed vulnerability list in a timely manner
• Accelerate remediation by generating actionable developer-oriented guidance for understanding and resolving each vulnerability

FIGURE 1: Orchestration of application security tests throughout the software development pipeline

Tenet 3: Maintain proper governance and guardrails

Because everything moves fast in the DevOps world, it’s easy to make mistakes. But even small mistakes or omissions, such as a missed CVE (Common Vulnerabilities and Exposures) or an unauthorized configuration change within a development pipeline, can come with hefty security and compliance risk. For this reason, the value of comprehensive governance and stringent guardrails throughout the entire development environment cannot be overestimated. If your DevSecOps practice is effective, you’ve made it easy for stakeholders to do the right things and hard for them to do the wrong things. This can be achieved with the following guidance:

• Enforce fine-grained Role-based Access Control (RBAC) throughout the development environment to ensure proper usage and operation. General RBAC is typically based on a single property (role), but fine-grained RBAC enables stronger security by taking into account multiple properties, such as time of day, user groups, organization hierarchy, etc.
• Overlay policies on top of pipelines to enable developers to control their pipelines and to give security and compliance teams the ability to require security checks. The Open Policy Agent (OPA) standard is an excellent policy-as-code approach for this.
• Use templates wherever possible to eliminate unforced errors that lead to security and compliance risk. Templates should contain security best practices, especially concerning the execution of security scans. Usage of templates should be enforced through policies that ensure security scans are performed.

Tenet 4: Focus on securing the software supply chain (and not just your own source code)

The challenge of securing modern applications has become increasingly complex, largely due to the vast array of open source software (OSS) components and other 3rd party artifacts that software producers use to build their applications. Each of these components introduces new potential vulnerabilities into the end product, which puts the software’s customers and consumers at risk. An application’s overall security and compliance risk is a function of all the code, people, systems, and processes that contribute to the development and delivery of that application’s software artifacts, both inside and outside of an organization.

As such, open source software artifacts are a desirable target for cyber attackers, as evidenced by the high-profile breaches that compromised Solarwinds, Log4j, and Codecov. Compromise one software building block, and there is potential to wreak havoc on the tens or hundreds of thousands of end consumers of that component. For this reason, the focus of DevSecOps must expand beyond the organization’s source code to the entire software supply chain, which is the SUM TOTAL of all the code, people, systems, and processes that contribute to the development and delivery of software artifacts, both inside and outside of an organization.

For the critical purpose of ensuring the integrity of any software produced by the organization, DevSecOps teams must adopt tools and practices in accordance with the SLSA framework and with Executive Order 14028.

Securing the software supply chain requires DevSecOps teams to:

• Govern the use of open source software components throughout CI and CD pipelines. This is best achieved through a policy-as-code approach (based on the OPA standard), which allows for authoring customized policies that factor in a broad range of OSS artifact attributes, such as version, license, PURL, and supplier, along with leading indicators of risk. Whether the goal is to ensure the proper use of open source libraries or block the use of specific OSS artifacts for security reasons, strong governance is essential.
• Adopt comprehensive capabilities for generating, managing, and analyzing software bills of materials (SBOMs) for software artifacts. An SBOM is essential for understanding the components and dependencies within an application, which in turn enables organizations to manage software risks effectively. More and more software-consuming organizations are requiring detailed SBOMs from vendors, consistent with Executive Order 14028 mandates.
• Generate and verify SLSA compliance beyond the minimum requirements of level 1. The SLSA framework is a highly effective means of protecting against artifact tampering. It allows for creating a verifiable record across the supply chain with information that associates identities and systems with the software. This information can be verified and signed throughout the software development lifecycle. The higher the level, the stronger the integrity guarantee.
• Establish a full chain of custody for all software artifacts. In the realm of software, chain of custody is detailed evidence of everything that happens to a software artifact throughout development pipelines, including who built or modified the artifact, which security tests it underwent, and what the test results were. Achieving a complete chain of custody is largely a function of the underlying CI/CD platform plus integrated pipeline tooling and it is crucial for maintaining the trustworthiness of software from development to deployment. Having a detailed software chain of custody also substantially accelerates vulnerability remediation, which is otherwise an exhaustive process of manually parsing logs and piecing together incomplete information in tracing the new vulnerability back to affected software components.

Tenet 5: Achieve ‘continuous security’ through automation and AI

DevOps has become synonymous with the practices of continuous integration and continuous deployment, so it stands to reason that DevSecOps should result in continuous security. A big part of DevSecOps success is being able to keep pace with (and even get ahead of) application development velocity. While it invariably takes time for a nascent DevSecOps program to build agility in addition to effectiveness, a key to accelerating DevSecOps maturity is the use of intelligent automation and AI. Here are several important recommendations for how and where to apply them:

• Orchestrate security scans throughout pipelines. This is easiest achieved with a platform approach, whereby the underlying DevOps platform integrates with a variety of SAST, SCA, Container, and DAST scanning tools and executes scans when the pipeline is run. Policy-as-code governance is another related form of automatic mitigation. For example, an OPA policy can be enforced to fail a pipeline if specific security criteria isn’t met.
• Automate vulnerability list deduplication and prioritization for developers. One of the biggest areas of toil for developers is having to deal with a mountain of unprocessed scanner output data. For the purpose of optimizing time-to-remediation for critical vulnerabilities (along with preserving developer productivity and experience), automating the process of deduplicating and prioritizing vulnerabilities is a must.
• Generate remediation guidance with AI. To further enhance the speed of remediation and minimize developer toil, providing AI-generated explanations for vulnerabilities and prescriptive remediation guidance is a huge benefit to developers.

Conclusion

While there is no doubt about the criticality of a highly effective DevSecOps practice to software-producing organizations, there are very few clear standards on how to build one that strengthens overall application security posture without adding toil or degrading the developer experience.

The five core DevSecOps tenets (along with their respective sets of guidelines) discussed in this paper enable DevSecOps teams to build and maintain a solid operational foundation. As modern DevOps technologies and practices continue to rapidly evolve, there will always be uncharted security issues to address. So long as developers, DevOps engineers, and security practitioners work together as a cohesive unit, the path to DevSecOps excellence is much clearer. If you’re interested in a further deep dive into these concepts, I encourage you to download the Definitive Guide to Secure Software Delivery.