Loading ...
Everyone seems to be measuring security. After all, the common mantra is “You can’t manage what you can’t measure.” And security cries out to be managed.
But which are useful security metrics? Are we investing appropriately in the measurement of various aspects of security? I think that it is clear from the literature and from experience that the search continues and that security breaches are increasing in number and strength despite literally hundreds of measures produced by security professionals and followed by management.
Some authors take the shotgun approach. Debra Herrmann, in her ambitious book Complete Guide to Security and Privacy Metrics (Auerbach, 2007) provides us with over 900 metrics (!) from which to choose. She essentially gathers together, in one enormous tome of 824 pages, virtually every metric that anyone has ever wanted to know (but were perhaps afraid to ask about). Certainly some of the 900 plus measures can lead to improved decision-making, but which of them actually do the job is anybody’s guess.
Andrew Jacquith offers a more realistic and manageable approach in his book Security Metrics: Replacing Fear, Doubt and Uncertainty (Addison Wesley, 2007). Jaquith’s book contains many sound arguments and helpful suggestions. But even his rational approach, based on much experience and considerable thought, does not ultimately make one believe that the problem has been solved. There remains an uncomfortable feeling that we are not quite there yet – wherever “there” may ultimately be. This is because, as, Douglas Hubbard notes his book How to Measure Anything: Finding the Value of Intangibles in Business (John Wiley, 2007), which I mentioned in my previous column, the greatest measurement effort is generally applied to the least useful measures. And the least effort is applied to those measures which yield the greatest value in terms of security risk reduction. Were Jacquith to introduce value loss estimates and to properly account for uncertainty along the lines that Hubbard suggests, I believe that we would then have something to crow about.
Of course there is good reason for not addressing intangibles and uncertainty. And the reason is that it can be quite difficult to measure those aspects of security which are thought by many to be immeasurable.
In support of using fuzzy and immeasurable security metrics, Gary Hinson lists seven areas where security management can occur even when aspects of security cannot be measured, as in his July 2006 article in the ISSA Journal on “Seven Myths about Information Security Metrics” (membership required to access link resource). The myths that Hinson points to are the following:
Myth 1: Metrics must be “objective” and “tangible”
Myth 2: Metrics must have discrete values
Myth 3: We need absolute measurements
Myth 4: Metrics are costly
Myth 5: You can’t manage what you can’t measure and you can’t improve what you can’t measure
Myth 6: It is essential to measure process outcomes
Myth 7: We need the numbers
After Hinson dismantles one myth after the other, one is left with a sense that we’ve been doing it incorrectly all along. This can be very discouraging. However, it can also be very liberating in that one can adopt a new perspective on security metrics and come up with measures that truly serve to help reduce risk.
The bottom line is that the most common and easily obtained security metrics tend to be the least useful, and those that might be the most useful, require much greater effort for them to be measured.
So where does that put us? It means that you should not develop security metrics for their own sake, but for how they might lead to greater control and higher value. Yes, the latter may be harder to come by and may not appear to be as precise or aesthetically pleasing as the simpler metrics, but what are we trying to achieve here – fancy charts or useful measures?
Popularity: 31%
