Open Source Security Practices

 

In 2020, an OpenSSF project started collecting information on software security practices employed by open source projects. This substantially increased transparency of open source development by centrally tracking the usage of commonly accepted best practices.

In a previous year's State of the Software Supply Chain report, we analyzed the extent to which higher scores on these checks are associated with better security outcomes and found them to be highly predictive of vulnerability status. In this year's report, we delve into the state of adoption of Security Scorecard best practices overall and also by ecosystem (focusing on Java and JavaScript). We look at checks today as well as over time, looking back at the prior year to see both how community practices have evolved and how the checks themselves have evolved over time.

Our dataset consists of Scorecard checks from 2022-06-27 and 2023-07-03, retrieved from the Scorecard BigQuery database. Each check is scored from 0 to 10. If the check fails or there is lack of evidence for a check, the Scorecard project assigns it a score of -1. We mapped these -1 values to 0 for our analysis.

The Checks

 

The state of Scorecard check scores

In last year’s report, we conducted an analysis of the connection between Scorecard checks and known vulnerabilities. One outcome of this analysis was information on which checks are most closely associated with better vulnerability status. Figure 2.2 summarizes this information. Code Review (e.g. requiring review of pull requests before merging) was the most important practice, followed by Binaries (not checking binary data into the repository), Dependencies Pinned (pinning dependencies to specific versions), and Branch Protection (preventing direct pushes to the main branch).

The figure below shows how widely the Scorecard best practices have been adopted. The chart presents the average value for each check across our 2023 dataset. Note that Code Review, which we found last year to be the practice most highly associated with good security outcomes, is not widely practiced in general, with an average score of less than 1. Roughly 19% of projects score greater than 0 on Code Review, and only 2% receive the full score of 10 (indicating that all code changes have undergone review). Branch Protection, which supports code review by requiring changes to go through the review process, also scores low (just 0.24 on average). The Vulnerabilities check measures whether a project has open, unpatched vulnerabilities, either associated with the project itself or with its dependencies. The Vulnerabilities score starts at 10 and is decreased by 1 for each open vulnerability with a minimum score of 0 (e.g. a project with 10 open vulnerabilities and a project with 15 open vulnerabilities both score 0). This means that the average score of 8.9 indicates that projects on average have at least 1 open vulnerability. Of course not every project is vulnerable: just over 14% of projects have a Vulnerabilities score of less than 10 (indicating they have at least one open vulnerability). If we look at just vulnerable projects, we find that on the date our 2023 data was collected they had, on average, at least 7.7 open vulnerabilities per project.

Another important low-scoring check is Maintained, which checks whether a project is being actively maintained, either via regular code commits or timely responses to issues. Maintenance is generally a prerequisite for security patches. On average, projects score about 0.66 on the Maintained check. Out of 1,176,407 total projects, just 118,028 of them (~11%) have a Maintained score greater than 0.

Looking at the data by ecosystem (shown in the figure below), we see that Java projects score higher on average than JavaScript projects for a number of security-relevant checks.

Java projects also score 10x higher on Fuzzing, though very few projects in either ecosystem take advantage of fuzzing technology. Rates of vulnerabilities are comparable, with 15% of Java projects and 16% of JavaScript projects having unpatched vulnerabilities at the time of scan. Note that this is a measure of whether there are vulnerabilities that apply to the development branch of a project. This does not indicate whether the latest release of the project has a vulnerability, nor whether the recorded vulnerabilities will persist into the next release.

The importance of maintenance

The figure below shows how the average scores change when we focus on maintained projects only (projects with a Maintained score greater than zero). There are several notable differences.

Conclusion

Overall, these results provide guidance for open source users when choosing dependencies and important areas of focus for open source advocates and maintainers. The value of choosing well-maintained projects stands out, as these projects tend to be more diligent about a number of important software security best practices. The fact that 18.6% of projects stopped being maintained in the last year highlights the need to not only choose good dependencies, but monitor those dependencies for changes in their quality. The overall low rates of code review, even when considering just maintained projects, provide a clear area for improvement in open source development practices – especially given the importance of code review in predicting security status (established in last year’s report).

Based on our findings, enterprises looking to minimize their open source vulnerability risk should choose well-maintained projects that perform code review and monitor them to ensure they have not reached end-of-life.

While we’re only scratching the surface with this regional view of vulnerable downloads, you can see a deeper dive into open source consumption patterns within specific economic regions in Chapter 3 of this report where we further unravel the intricacies of dependency management on a global scale. We also summarize the role regulations are having on the industry in Chapter 5.