Software Supply Chain Maturity
Software Supply Chain Maturity – Peer Insights
Consistent with our prior reports, this year's account once again draws from enterprise engineering professionals providing collective color about the software supply chain management approaches at work. The survey that was conducted this year to support this work asked 40 questions to measure the use of open source software (OSS) components, dependency management, governance, approval processes, and tooling. The survey also asked 17 questions regarding development practices, demographics, and job satisfaction.
Methodologies and Objectives
Two objectives of the survey continue with this year's edition:
- Provide a benchmark and maturity model that facilitates how organizations can evaluate themselves in comparison to their peers.
- Examine whether certain reported software supply chain practices correlate with desirable results.
Responses to all 57 questions were assessed against the following eight themes of software supply chain management practices:
Eight Themes of Software Supply Chain Management Practices
Remediation
How do you implement fixes to address identified OSS component risk?
Application inventory
Do you know all the applications your organization has in development/production, and who the stakeholders/owners are? Do you know the details about them, including how they are built, and the Software Bill of Materials (SBOM) for the OSS components they include?
Build & release
Do you understand how your software "parts" and processes come together to build and release applications into production?
Policy control
What is your tolerance for risk? Do you have automated policy enforcement?
Project consumption
Do you govern OSS component selection?
Giving back
Do you contribute to the OSS community?
Supplier hygiene
Do you know if your OSS components come from a trusted, quality supplier?
Digital transformation
What plans, resources, and training do you have to help institutionalize new processes and tools?
How Mature Are Today's Software Supply Chains?
We've summarized the responses from 662 individuals in Figure 4.2. We averaged each individual's responses on questions that fell into distinct themes. In each theme we scored the responses from 1 to 5, corresponding to the five stages of supply chain maturity. From Unmanaged (least mature) to Monitor & Measure (most mature), as noted in Figure 4.1.
Figure 4.1. Five Stages of Software Supply Chain Management Maturity
More Mature
This first stage is referred to as the Unmanaged stage because organizations are often operating with an "anything goes" mindset, are often reactive, and have minimal process/ oversight related to the themes.
A realization of some sort is usually the impetus for thrusting an organization into the Exploration stage. This is often triggered by an "event" that causes an "all hands on deck" reaction to uncover necessary information/solutions, or a champion of some sort leading an improvement effort. This stage is often focused on identifying the perceived problem/inefficiency, learning about current implementations, and starting to explore potential solutions.
In the midst of starting to define processes and implement tooling to improve the identified problem, Ad Hoc solutions reign as the teams work toward institutionalization and socialization of new tooling and processes.
In the Control stage, ad hoc solutions give way to more formalized governance processes across the enterprise. Socialization and institutionalization of these processes and tools is ongoing, but for the most part, stakeholders are bought in to the need for improvement measures and are working towards compliance.
The Monitor and Measure stage occurs once new processes and tools have been institutionalized, and organizations have reached a phase of being able to proactively address OSS component risk. In addition, a healthy amount of ROI is realized, and measurements to demonstrate success are available.
This first stage is referred to as the Unmanaged stage because organizations are often operating with an "anything goes" mindset, are often reactive, and have minimal process/ oversight related to the themes.
A realization of some sort is usually the impetus for thrusting an organization into the Exploration stage. This is often triggered by an "event" that causes an "all hands on deck" reaction to uncover necessary information/solutions, or a champion of some sort leading an improvement effort. This stage is often focused on identifying the perceived problem/inefficiency, learning about current implementations, and starting to explore potential solutions.
In the midst of starting to define processes and implement tooling to improve the identified problem, Ad Hoc solutions reign as the teams work toward institutionalization and socialization of new tooling and processes.
In the Control stage, ad hoc solutions give way to more formalized governance processes across the enterprise. Socialization and institutionalization of these processes and tools is ongoing, but for the most part, stakeholders are bought in to the need for improvement measures and are working towards compliance.
The Monitor and Measure stage occurs once new processes and tools have been institutionalized, and organizations have reached a phase of being able to proactively address OSS component risk. In addition, a healthy amount of ROI is realized, and measurements to demonstrate success are available.
Overall, respondents indicated the lowest levels of maturity in the Digital Transformation theme and the highest level of maturity in Remediation. Across the various themes, we see that the majority of respondents were graded less than the "4 - Control" level of maturity.
The "Control" level of maturity is the point at which an organization transitions from "figuring it out" to a minimal level of maturity that will enable high-quality outcomes. Note that the three levels of maturity (Unmanaged, Exploration, Ad Hoc) to the left of the "Control" level are suboptimal; this is where most of the survey responses were scored.
Figure 4.2. Software Supply Chain Maturity Score by Theme
Note: Dashed line represents overall survey median. Solid vertical lines represents theme median
Perception Still Disconnected From Reality on Software Supply Chain Maturity
In last year's survey, respondents reported a high level of maturity on the Remediation theme. However, our objective analysis showed that the remediation patterns in thousands of applications were suboptimal, suggesting a false sense of security. Nevertheless, this year, the self-reported level of maturity in Remediation has only grown higher.
Respondents indicate that they understand where the risk resides and are able to remediate affected components quickly. In fact, 68% of the respondents are confident that their applications are not using known vulnerable libraries. Furthermore, 84% of respondents reported scrutinizing the security history of an open source software component before deciding to use it in their software.
However, the data explored in other sections of this report shows a clear disconnect between what people think is happening and what is actually happening. Research conducted in a random sample of 55,000 enterprise applications reveals that 68% of the applications had known vulnerabilities in the underlying open source software components. This was not the only discrepancy we observed in this year's survey.
Rose-Tinted Glasses
We leveraged the demographic data collected during the survey and broke down the results by job title. The findings were illuminating. There is an ongoing bias towards seeing things in a better light, in which managers report higher stages of maturity compared to what is reported by other roles. Survey-wide, this discrepancy is statistically significant when comparing IT managers and those working in information security roles.
Compared to respondents working in
information security, the IT managers are:
1.8 times more likely to strongly agree to
"We know the Software Bill of Materials (SBOM) for every application."
2.4 times more likely to strongly agree to
"We address remediation of security issues as a regular part of development work (i.e., security issues treated as normal defects)."
3.5 times more likely to respond with "Less than 1 day" to
"When our team becomes aware of a vulnerability in an open source software component that we use, how long does it take (estimated) to mitigate this vulnerability across our application(s)?"
In an ideal world, management's perception should align with information security's experiences.
Finally, we broke down the results by the industry of the respondents. Those working in the technology sector assessed their maturity to be higher than most other industries across most of the themes of securing the software supply chain. This may be the case. However, our findings from the perception vs reality analysis in vulnerability data, and the rose-tinted glasses analysis of management's optimism painting a better picture than information security's reporting, suggest that there may be a false sense of confidence. This false sense of confidence might leave organizations vulnerable.
Finding 2 — The Gap Between Digital Transformation and Remediation
Another interesting finding that surfaced from this year's survey was the observed gap between Digital Transformation, which was the lowest self-scored maturity theme, vs. Remediation, the highest-scored maturity theme.
As shown in Figure 4.3, both roles closer to the actual work (engineers, etc., noted as "Practitioners") and managerial/executive roles (noted as "Managers") scored their organization as either very close to or above the Control stage of maturity when it comes to the Remediation of vulnerable open-source components. However, they both rated their organization's Digital Transformation efforts lower in the Ad Hoc stage of maturity.
Figure 4.3. Mean Stage of Maturity of Practitioners vs. Managers
This gap is especially interesting because of the scope of essential work captured within the Digital Transformation theme. The survey questions related to Digital Transformation are as follows:
We have a centralized committee/group/team that is responsible for monitoring and enforcing open-source component governance.
Which centralized committee/group/team is responsible for monitoring and enforcing open-source component governance?
How is your current open source risk management initiative resourced and supported?
These questions largely focus on whether an organization has matured sufficiently in their Digital Transformation efforts to invest the time, resources, and energy necessary to institutionalize open source governance measures, including:
It's also important to note the "monitoring and enforcing" language within the Digital Transformation questions outlined above. Without monitoring and enforcement of the entire software supply chain management process — in addition to the proper tools, training, and support — it is unlikely for an organization to actually remediate at a Control level of maturity. Indeed, the essential elements of the Digital Transformation theme are a prerequisite to ensuring the team has institutionalized the Risk Management and Remediation process within an organization.
Finding 3 — Supply Chain Maturity is Significantly Associated With Job Satisfaction
Fostering employee well-being, especially in light of the pandemic, has become an increasingly important topic. As a continuation of our research, this year's survey asked several questions to measure work satisfaction. Respondents rated their level of agreement with queries such as "I am satisfied with my job," "I would recommend this organization as a good place to work," and "I have the tools and resources to do my job well."
Figure 4.4. Job Satisfaction Correlations With Software Supply Chain Maturity
We wanted to assess whether there was a relationship between job satisfaction and the stages of maturity in the eight themes of software supply chain management. To test those associations, we combined the participants' responses to create an overall job satisfaction variable. We then correlated the job satisfaction responses to the participants' responses on the eight themes. The findings were striking.
Individuals from organizations with higher levels of maturity (Control and above) were:
Compared to individuals working at organizations with software supply chain practices at the Ad-hoc level and below.
2.7 times more likely to
strongly agree to "I am satisfied with my job."
2.8 times more likely to
strongly agree to "My job makes good use of my skills and abilities."
2 times more likely to
strongly agree to "I would recommend this organization as a good place to work."
3.6 times more likely to
strongly agree to "I have the tools and resources to do my job well."
Job satisfaction was associated with all eight themes of software supply chain management. The higher the level of maturity, the higher the reported employee satisfaction, and vice versa. The strongest relationship was between job satisfaction and Digital Transformation. In organizations mature in Digital Transformation, employees know who to ask for help and support regarding open source risk management. They appear to have the training, budget, and tools they need to address risk, and importantly, they have executive sponsorship.
The higher the level of maturity, the higher the reported employee satisfaction, and vice versa.
This year's findings indicate that improving practices tied to securing the software supply chain minimizes security and licensing risk. Moreover, these enhanced practices demonstrate an association with greater job satisfaction. Thus, improving the integrity of the software supply chain can potentially play a role in retaining talent.