In 2016, a Tesla Model S T-boned a tractor trailer at full speed, killing its lone passenger instantly. It was running in Autosteer mode at the time, and neither the driver nor the car's automatic braking system reacted before the crash. The US National Highway Traffic Safety Administration (NHTSA) investigated the incident, requested data from Tesla related to Autosteer safety, and eventually concluded that there wasn't a safety-related defect in the vehicle's design (PDF report).
But the NHTSA report went a step further. Based on the data that Tesla provided them, they noted that since the addition of Autosteer to Tesla's confusingly named "Autopilot" suite of functions, the rate of crashes severe enough to deploy airbags declined by 40%. That's a fantastic result.
Because it was so spectacular, a private company with a history of investigating automotive safety wanted to have a look at the data. The NHTSA refused because Tesla claimed that the data was a trade secret, so Quality Control Systems (QCS) filed a Freedom of Information Act lawsuit to get the data on which the report was based. Nearly two years later, QCS eventually won.
Looking into the data, QCS concluded that crashes may have actually increased by as much as 60% on the addition of Autosteer, or maybe not at all. Anyway, the data provided the NHTSA was not sufficient, and had bizarre omissions, and the NHTSA has since retracted their safety claim. How did this NHTSA one-eighty happen? Can we learn anything from the report? And how does this all align with Tesla's claim of better-than-average safety line up? We'll dig into the numbers below.
But if nothing else, Tesla's dramatic reversal of fortune should highlight the need for transparency in the safety numbers of self-driving and other advanced car technologies, something we've been calling for for years now.
How Many Crashes Per Million Miles?
How is it possible that the NHTSA would conclude that Autosteer reduces crashes by 40% when it looks like it actually increases the rate of crashes by up to 60%? And what's with this shady "up to" language? The short version of the QCS report could read like this: Tesla gave the NHTSA patchy, incomplete data, and the NHTSA made some assumptions about the missing data that were overwhelmingly favorable to Tesla, and fairly unlikely to reflect reality.
Crucially, none of the data about miles driven with Autosteer engaged have ever made it out of Tesla's hands. What Tesla gave the NHTSA, and which they claimed represented a trade secret, was the mileage on the odometer before and after Autosteer was installed, the mileage on the car at last check, and whether or not there was an airbag deployment. From this, the most you can conclude is about the accident rate per mile of cars with Autosteer installed, rather than the rate of accidents per mile with Autosteer active. Only Tesla knows what percentage of miles are driven with Autosteer on, and they're not telling, not even the government.
How You Get a Range of +40% to -60%
So did the NHTSA get data on how many miles were driven with Autosteer installed? Not quite. Tesla gave them data with the "Previous Mileage before Autosteer Install Reported" — an odometer value. They also gave "Next Mileage after Autosteer Install Reported" — another odometer value but there was no guarantee this would match up with the previous readout and the data varies quite a bite here.
For data samples where these two numbers did match, you can assume that's the mileage when Autosteer was installed. We'll call these cars "known mileage" — they correspond to Figure 1 in the QCS report. Known-mileage cars make up 13% of cars in the sample, and for these cars the accident rate increased by 60% after Autosteer was installed. There were 32 airbag deployments for 42 million miles driven before Autosteer was installed, and 64 deployments over 52 million miles afterwards. At first blush, this looks like Teslas with Autosteer installed appear to crash a lot more often than Teslas without. (Remember, we don't know anything about miles driven with Autosteer actually engaged.)
Another 34% of the vehicles in the data had Autosteer installed at odometer mileage zero, so we'll call these "factory Autosteer" cars. They correspond to QCS's Figure 2. These cars have a crash rate that's similar to the pre-Autosteer period from the first sample, reducing what looks like a negative safety effect from Autosteer. If you combine these two samples together, it looks like cars with Autosteer crash 37% more frequently than those without — almost the mirror image of the NHTSA's original result but better than a 60% increase in accidents.
The remaining 53% of the sample consists of cars with a pre-Autosteer mileage and a post-Autosteer mileage that don't line up. For example, maybe we know that the car didn't have Autosteer installed at 20,000 miles on the odometer, but it did by 60,000 miles, leaving a 40,000 mile murky middle zone. The car could have had Autosteer installed at the 20,001 mile mark, or at 59,999.
When adding up the airbag incidents and dividing by the mileage, which do you use? The NHTSA assumed 20,000 miles — the "Previous Mileage before Autosteer Install Reported". How you treat this incomplete data, and whether you use it at all, can make the difference between the NHTSA's +40% safety margin for Autosteer and the -37% to -60% margin you'd get otherwise. Yuck.
Doesn't Pass the Smell Test
My first instinct, as a former government statistician, is that there's something fishy with this data. Why does Autosteer seem to have such a huge effect in the group of cars where it was added aftermarket, but apparently no effect when it was added in the factory? I don't know what explains these differences, but I'd be wary to call it an Autosteer installation effect. Maybe these early adopters drove less cautiously? Maybe they used Autosteer more recklessly because they wanted to test out a new feature? Maybe early versions of Autosteer were buggy? And for the missing data, how can the conclusions reached be so sensitive to assumptions we have to make, swinging between a +40% and -40% effect?
To be fair, the NHTSA originally asked Tesla for incredibly detailed information (PDF) which would have made their work much more conclusive. Tesla replied that the odometer readings were a trade secret, and the NHTSA got a judge to rule that they weren't.
Still, in the end, Tesla gave them data with this "exposure mileage gap". Tesla apparently also informed the NHTSA why it couldn't give them the data they requested (PDF) but this reason is redacted in the FOIA report. All we know is that the NHTSA accepted their excuse.
Tesla probably knows the exact odometer readings. After all, Tesla prides themselves on the detailed data they collect about their cars in use. How can they not know the odometer mileage when their own Autosteer software was installed, for more than half of the cars in the study, more accurately than within 20,000 miles? Feel free to speculate wildly in the comments, because we'll probably never know the answer.
The bottom line in terms of data analysis is this: for the data with solid numbers, it looks like Autosteer being installed is associated with between a 35% and 60% increase in the number of crashes per million miles. The remaining half of the sample has imprecise data, and whether it is favorable toward Autosteer or not depends on what assumptions are made based on poor data.
Cold Hard Numbers Instead of PR Spin
When the NHTSA came out with their report citing the huge decrease in accidents attributed to Autosteer, Tesla was naturally ecstatic, and Musk was tweeting. Now that the results have gone the other way, he has declined to comment. Just prior to the release of the QSC paper, Tesla began releasing quarterly safety reports, but they're light on useful data. The note that the NHTSA has updated its assessment of their system's performance is a vague asterisk footnote in small type: "Note: Since we released our Q3 report, NHTSA has released new data, which we've referenced in our Q4 report." Right.
Tesla's safety report is very brief, short on numbers, and vague to boot. In particular, Tesla reports only miles driven using Autopilot, which is an overarching name for the sum of Tesla's driver-assist features including automatic emergency braking, adaptive cruise control, and lane-keeping or Autosteer. It's abundantly clear from other carmakers that automatic emergency braking systems help reduce collisions, but Tesla doesn't disaggregate the numbers for us, so it's hard to tell what impact Autosteer in particular has on vehicle safety.
Finally, the Tesla report includes both crashes and "crash-like events", whatever that means. Still, if you assume that Tesla is reporting more events than it otherwise would, their cars seem to have great crash statistics, even without Autopilot, when compared with the US national average. Why would that be? The Tesla advantage narrows significantly when you compare them with other luxury cars, driven by a demographic that's similarly free of new and inexperienced drivers. Tesla's report also shows that there are fewer accidents per mile with Autopilot engaged than without, but since you're only supposed to drive with Autopilot on the highway under clear conditions, this is hardly a surprise either.
It's simply hard to say what these numbers mean in terms of the safety of the underlying features. Their evasive behavior with regard to the NHTSA, coupled with the odd metrics that they're voluntarily reporting are hardly confidence inspiring.
In summary, the data Tesla supplied to the NHTSA was short of what was requested, and 50% of it was blurred for a reason that was redacted — that's not our idea of transparency. And we can't expect more from their voluntary safety reports either — no company releases information that could have a negative impact on their business operations unless they're forced to. So maybe rather than berating Tesla, instead we should be asking our government to take transparency in highway safety seriously, and supporting their efforts to do so whenever possible. After all, we're all co-guinea-pigs on the shared roadways.
via https://ift.tt/2GYlvPG
No comments:
Post a Comment