Do vulnerability counts from sources like the National Vulnerability Database (CVE data) and Open Source Vulnerability Database (OSVDB) really matter? A recent article by Robert Lamos at darkREADING, questioned the usefulness of the metrics generated by these reports since the counts don’t add up.
Looking at the trends, it’s been easy to see that vulnerabilities are increasing, but the real reason for this is hard to quantify. Unfortunately, some people make inaccurate decisions by drawing false conclusions when they assume correlation equals causation. What do I mean by this?
I recently listened to a podcast about the merits of the Paleo diet. One of the arguments used to support the diet misinforms people. The premise of the argument is that we should eat like our ancestors from the paleolithic period by avoiding dairy, grains, and legumes. This will lead to better health because these ancestors did not suffer from the myriad of diseases that inflict us today. This argument implies that the link between diet and health are both correlated and causal. The challenge with this inference is that people jump to improper conclusions since not all factors can be known.
Another example that was illustrated in the podcast relates to the argument that ties ice cream consumption to shark attacks. The claim is that the rise in shark attacks is due to people consuming more ice cream. So if you eat ice cream you are more likely to get attacked by a shark! Once again, this analysis is flawed since other factors, such as rising ocean temperatures, are not included, which leads to an improper relationship between cause and effect.
How does this relate to security? In some cases, this same phenomenon happens when people use statistics to measure the increase in risk. One could argue that the slope of the curve, whether positive or negative, does not truly measure "bad" or "better" because not all factors are considered in the analysis. There are many potential factors such as:
- The Gramm-Leach-Bliley Act forces companies to notify customers if their data has been breached.
- The number and attendance at security events such as RSA, Black Hat, etc., have risen over the years,increasing the number of security professionals, both good and bad, into the ecosystem.
I could also make the argument that the mainstream news coverage of application security events is the reason for the rise. For me these numbers and stats are just that, numbers and stats, while the real takeaway is: How does your organization measure up?
If you lack the ability to understand your own vulnerability growth curve, no external measurement or stat is going to help you!
