C. Warren Axelrod

Security Metrics and Tesla’s Safety Statistics

I have long railed against the inadequacy of popular easy-to-record security metrics. They usually lack critical information about the nature and severity of vulnerabilities and are therefore misleading in providing support for decision-making. I addressed this point in my article “Accounting for Value and Uncertainty in Security Metrics,” in the ISACA IS Control Journal (November 2008).

Tesla has just published its fourth quarter 2019 safety report , available at https://www.tesla.com/VehicleSafetyReport and showed that the number of crashes per mile driven is significantly lower than for other vehicles for Tesla cars without Autopilot engaged and lower still when Autopilot is engaged.

Further, Tesla’s Model 3 was tested by the NHTSA and found to have the lowest probability of injury of any vehicle ever tested. https://www.tesla.com/blog/model-3-lowest-probability-injury-any-vehicle-ever-tested-nhtsa

These are clearly great achievements by any standard and Tesla should be praised for them. However, there are some critical aspects omitted from these reports, recognizing that the additional information may be difficult and/or expensive to come by.

First is the relative severity of accidents that occur with regular everyday cars and Teslas, with Autopilot on or off. One might assume that if Teslas have fewer crashes and the probability of injury is lower, then the conditional probability of injury given a crash has taken place is less. Obvious? Well, no! If the accidents with Teslas were to be much more severe than average accidents and if injuries are greater, despite their lower probability, then we have an entirely different picture. Of course, the press broadcasts Tesla accidents that result in fatalities with inappropriate fanfare, but that does not imply that there are more of them.

The statistics need to be normalized for other differences too. The average Tesla is really new relative to the average age of all vehicles. Does this affect accidents? Probably. To what extent are failing components responsible for crashes? We should know this.

The average Tesla may not be driven under poor driving conditions. Does Autopilot work in heavy snow, for example? If not, you are missing road conditions in which many crashes occur.

Teslas have limited range. Does the length of journey have anything to do with rate of accidents? It is arguable that when drivers exceed a certain number of hours driving, they are more prone to accidents.

Do Teslas cause accidents that are not recorded or assigned to Tesla? I recall reading a report of an autonomous vehicle forcing another vehicle off the road. Do we know how common such situations might be? I don’t think so.

None of this is an attempt to diminish Teslas laudable achievements in safety and accident reduction. It’s just that the published statistics are not the whole story.

The same is true of security metrics. We often lack severity and context n these statistics. We also don’t know the indirect effects. If we are able to deter or avoid a cyberattack, then the hackers likely move on to more vulnerable targets. That’s good for the first organization, but not for the second. Are we interested in the impact of cyberattacks across the whole of cyberspace? Or are we just interested in our own wellbeing?

Yes, superficial statistics are easy together. But are they truly representative of the real world? Probably not. Whether that is Tesla’s world or that of cyberspace. Yes, we draw comfort from supportive statistics, but the frequency with which such statistics are challenged or debunked (especially in the medical field) should give us pause. Let’s petition for more useful statistics and metrics. If we don’t, we will continue to make decisions on statistics that are not particularly meaningful. I have been convinced that Teslas are safer than other vehicles on the road, but I cannot be certain, and I don’t know if there are other aspects that reduce the value of the statistics that are published. There is a lot of hype about autonomous and electric vehicles, but there are other things to consider that might detract from their long-term value. Talking about autonomous cars, you should read Todd Litman’s report, Autonomous Vehicle Implementation Predictions –Implications for Transport Planning, Victoria Transport Policy Institute, March 18, 2019. It is available at  https://www.vtpi.org/avip.pdf  Litman presents some very good ideas about some of the costs and disadvantages of driverless road vehicles that are omitted elsewhere. We need to consider similar factors for cybersecurity.

Post a Comment

Your email is never published nor shared. Required fields are marked *

*
*