EV pioneer's Autopilot system might not have reduced serious accidents by as much as previously stated by US road safety agency, according to an investigation into the claims
Julian Rendell
20 February 2019

The regime for assessing the safety of self-driving car systems is under scrutiny after consultants discovered errors in an analysis of Tesla crashes by a US government agency, which then delayed release of the data for two years in an apparent attempt to divert attention from the issue.

Quality Control Systems Corp (QCSC) made a deep-dive into data that purported to back up an impressive claim by NHTSA, the powerful US road safety body, and Tesla that the Autosteer system in Autopilot had reduced serious accidents by 40% — a game-changing improvement.

Tesla boss Elon Musk tweeted the results as NHTSA’s findings were released in January 2017.

But QCSC found that the NHTSA analysis failed to take into account all the mileage driven by the 43,781 vehicles studied. In fact NHTSA only used mileage data for 14,791 vehicles. As a result, the crash rate before Autosteer was inflated, leading to the wrong conclusion, says QCSC.

Our Verdict

Tesla Model S 95D

In theory, this all-electric luxury car looks a hit. So is it in practice?

Find an Autocar car review

Driven this week

“The importance of this research goes well beyond the specific issues addressed in our statistical analyses,” said QCSC. “The larger question is whether the field experience of autonomous vehicles and advanced driver-assistance systems will be fairly and transparently assessed by public officials.”

When QCSC, which specialises in data analysis as a business tool, approached NHTSA for access to the publically-funded research, they were refused. The log-jam took two years to break, only after QCSC threatened court procedings to obtain the data via a freedom of information request.

In response to a request for a comment from Autocar, the Washington-based NHTSA released a limited statement: “The agency is reviewing the report released by Quality Control Systems Corp. with interest and will provide comment as appropriate.”

QCSC’s report is titled “NHTSA’s Implausible Safety Claim For Tesla’s Autosteer Driver Assistance System” and looked at data for airbag deployments of MY2014 – 2016 Model S and MY2016 Model X models equipped with Tesla’s Autopilot.

QCSC’s analysis of the data, contained in a 24-page report, found that airbag deployments actually increased from 0.76 million to 1.21 million, a 59% rise, rather than decreased.

“Before and after comparisons of the resulting crash rates are unbiased by missing data for exposure mileage because there are no missing data in this subset of the data,” said QCSC in its report.

When NHTSA analysed airbag deployment and mileage travelled data supplied by Tesla, NHTSA came to the conclusion that there were 1.3 deployments per million miles before Autosteer and 0.8 million after – a 40% reduction.

NHTSA’s findings were widely reported in January 2017, partly because of the fatal accident in Florida six months earlier when a Model S operating with Autopilot ploughed into a truck, having failed to detect the vehicle making a turn across the car’s path.

Autopilot is a Level 2 advanced cruise-control self-driving system that allows the driver to temporarily relax their grip on the steering wheel. Autosteer is the lane-keeping part of the Autopilot system, which also facilitates lane changes when the indicator is operated.

Introduced in 2014, Autosteer was enhanced in 2016 with an upgrade. Significantly, it means a lane change can be executed with less oversight by the driver. Ultimately Tesla aims for automatic lane-changing manoeuvres in Level 3 or 4 autonomy.

In response to the report, a Tesla statement read: "QCS’ analysis dismissed the data from all but 5714 vehicles of the total 43,781 vehicles in the data set we provided to NHTSA back in 2016. And given the dramatic increase in the number of Tesla vehicles on the road, their analysis today represents about 0.5% of the total mileage that Tesla vehicles have traveled to date, and about 1% of the total mileage that Tesla vehicles have traveled to date with Autopilot engaged.

"NHTSA’s original report not only indicated that “Tesla vehicles crash rate dropped by almost 40 percent after Autosteer installation”, the agency also concluded that it “did not identify any defects in the design or performance of the AEB or Autopilot systems,” nor did it find “any incidents in which the systems did not perform as designed”. They also found that “the potential for driver misuse was evaluated as part of Tesla’s design process and solutions were tested, validated, and incorporated into the wide release of the product.”

"Our own vehicle safety data for Q3 and Q4, which includes data from roughly two billion miles driven in Tesla vehicles, shows that drivers using Autopilot were significantly less likely to be involved in an accident than those driving without using Autopilot.”

READ MORE

Tesla Model 3 becomes EV with longest range in Europe

A day in the life of a Tesla supercharger

Tesla: will the fairytale end this year?

Join the debate

Comments
8

20 February 2019

...when there is a new technology and "new money" to be made like diesel and Co2 figures, books will tend to be cooked and the upshot is that the oppurtunity to make new money always wins.

20 February 2019

Not remotely shocked.

Tesla have a reputation for using their customers as their development vehicle drivers.

Other companies tend to use professional drivers for this, but you can't argue with only making a profit over a handfull of quarters over a decade...

Interesting products, to be sure - but first is rarely best.

20 February 2019
CarNut170 wrote:

Not remotely shocked.

Tesla have a reputation for using their customers as their development vehicle drivers.

Other companies tend to use professional drivers for this, but you can't argue with only making a profit over a handfull of quarters over a decade...

Interesting products, to be sure - but first is rarely best.

 

First is rarely best, well, Tesla were NOT first, not by any stretch of the imagination, and 99% of motoring fans know this, also, ALL brands beta test their cars with owners, thats how things come to light, or no new cars would ever be released.......

 

20 February 2019
CarNut170 wrote:

Tesla releasing a not-fully-developed system?!

Not remotely shocked.

That's not the indication of the report. The real thing to take away from this is that there are flaws in the NHTSA's original claim of Autopilot delivering a 40% safety improvement. But there's no solid new claim to take its place - the problem with the NHTSA's study was that they only reviewed 33% of the cars in the dataset, but the QCSC's study apparently only reviewed 13%.

Hopefully this will spur the NHTSA to release a more detailed report, and one way or the other, we'll know for sure.

Another thing I'd like to know is how it compares with other manufacturers. Virtually everybody offers an autosteer and autobrake function now, but you never hear about independent safety examinations of the technology for anyone except Tesla. I find it hard to believe that there's never been an accident in any other semi-autonomous vehicle.

20 February 2019

Lexus weren't the first to invent the thottle pedal but they still screwed up.

Vauxhall thought they'd sussed the heater controls, no AC control unit ever warmed a car so thoroughly!

typos1 - Just can’t respect opinion

20 February 2019

NHTSA is a well known reputable agency. QCSC? Never heard of them...is Autocar going after some fake conspiracy theory? Stay with reality and you will do better.

20 February 2019

If you simply do not like driving and can afford a Tesla. Just Uber everywhere instead.

Mesumguy

25 February 2019

There were major changes in the latest OS based on the appearance, services, ease of  access, download mozilla firefox latest version are available on this site.

Add your comment

Log in or register to post comments

Find an Autocar car review

Driven this week