- BlogInfoSec.com - https://www.bloginfosec.com -

The Difference between Quantitative and Qualitative Risk Analysis and Why It Matters (Part 3)

As we saw in part 2 of this series [1], some of the traditional arguments used for distinguishing between quantitative and qualitative risk analysis (RA) are based upon dubious assumptions. Many writers assume that “quantitative” equals objective and numerical, while “qualitative” equals subjective and non-numerical. This is incorrect, however. Both quantitative and qualitative RA are compatible with objective and subjective approaches. Additionally, both types of RA must be numerical in order to be meaningful. So if the quantitative-qualitative distinction isn’t significant because one is objective and numerical while the other is subjective and non-numerical, then why does the distinction matter?

Reason #1: Rare But Catastrophic Threats

The expected value for events in general trends towards medium. High impact, low probability events end up having roughly the same expected utility (risk) as low impact, high probability events. This phenomenon affects both qualitative and quantitative RA. To see why, let’s examine a quantitative RA example first. Consider an ALE approach to comparing two risks. From an ALE perspective, a risk that involves a $5,000,000 loss expected once every 100 years is equivalent to a $50,000 loss expected once a year. As for qualitative RA, imagine a simple 3×3 risk matrix. The majority of the cells will fall into the middle category, 1 corner will fall into the lowest interval, and another corner will fall into the highest interval. In both the quantitative and qualitative examples just given, it is unclear, on the basis of RA, how to prioritize risks competing for the same limited security investment.

Nevertheless, it seems quite likely that this problem is more of a problem for qualitative RA than it is for quantitative RA, simply because qualitative RA tends to aggregate things into a very small number of categories, whereas it is unlikely that the expected utility (risk) of two different outcomes, as measured quantitatively, will be precisely equal. In other words, one reason the distinction between quantitative vs. qualitative RA matters is because, in some situations, quantitative RA makes it easier to prioritize risks.

Reason #2: Level of Granularity

There is a second reason why the distinction between quantitative and qualitative RA matters: the level of granularity required by the organization for the RA process. Since the purpose of RA is to optimize investments of limited resources in risk mitigation strategies, the output of the RA process needs to be granular enough to support the specific choices that have to be made, as well as the decision-making processes of the organization.

Let’s again consider a simple 3-tier risk classification scheme, where:

“Low” represents a risk with an expected loss of 0-$100,000 per year,
“Medium” represents a risk with an expected loss greater than $100,00 and less than $1M per year, and
“High” represents a risk with an expected loss of $1M or greater.

Now suppose we are considering a “Medium” risk. If all we know about the expected loss of this risk is what we get from the qualitative label “Medium,” then all we know is that the expected loss is somewhere between $100K and $1M, but we don’t know the expected loss with any degree of precision. For all we know, the expected loss could be $101K; it could also be $999K. Based upon the set of specific risks and proposed mitigating controls, this may or may not be a problem. If the proposed control costs only $10K (and does not itself create other risks to the organization), then the mitigating control would be a no-brainer (and a qualitative approach would be sufficient). But suppose instead that the proposed control costs $350K. In that case, the qualitative approach may not be sufficient. If we truly don’t know the expected loss for the risk in question, we have no reason for assuming that the proposed control costs less than the risk (expected loss) itself. In that case, a quantitative RA may be needed.

Conclusion

Because the qualitative vs. quantitative debate is so controversial, I want to emphasize that the preceding arguments are not intended to be arguments for the conclusion that quantitative RA is always (or even often) superior to qualitative RA. On the contrary, my opinion may be summed up as: “it depends.” The decision to choose a quantitative or a qualitative RA approach depends upon the number of risks to be evaluated, how closely the risks fit together, whether the output of a qualitative RA approach is granular enough to satisfy the needs of decision makers, organizational culture, and so forth. What I have argued is that the distinction between quantitative and qualitative RA matters, but not for the reasons that are often given.