Estimate time for Tesla to calculate its robotaxi safety

Stocks A to Z
,
1

We can try to estimate how long it will take Tesla to estimate the incident rate of their US robotaxi service.

We can borrow some ballpark stats from Waymo.

Waymo accumulates 1 million miles per week for about 1500 taxis that are L4 autonomous, or about 35,000 miles per taxi per year.

Waymo has about 1 incident per 1 million miles.

Tesla has 10 taxis testing L4 autonomy with onboard safety monitors, so assuming 35,000 miles per taxi per year, Tesla accumulates 350,000 miles of L4 testing data per year.

Assume Tesla needs to accumulate enough miles to observe 20 incidents (this would be near a minimum I would think to have some rigor to estimate something like confidence intervals on the incident rate - feel free to run the exact stats, if you need more detail, or another kind of stat).

Scenario 1
Assume Tesla has the same incident rate as Waymo: 1 incident per million miles.

Then 20 million miles are needed to observe 20 incidents.

At 350,000 miles per year (with 10 taxis), 57 years are needed to accumulate 20 incidents (on average).

Scenario 2
Assume Tesla has 3x worse the incident rate as Waymo: 3 incidents per million miles.

Then 6.7 million miles are needed to observe 20 incidents.

At 350,000 miles per year (with 10 taxis), 18.5 years are needed to accumulate 20 incidents (on average).

Conclusion
To move their robotaxi project faster, clearly Tesla gonna need more taxis and scale faster.

Or maybe Tesla can somehow leverage all of the data from their retail fleet (which maybe would be like having additional safety-monitored taxis), if that data can be equivalent in this way.

A lot of the Tesla retail mileage is highway, so probably much less useful. However, one would still expect Tesla to have a lot of city and suburban mileage from their retail fleet.

With safety monitors present, Tesla would actually have to use disengagement rates (when the safety monitor intervenes before a potential accident) and then somehow estimate what fraction of disengagements would have resulted in an accident of some kind to then estimate incident rate. Presumably Waymo does this as well.

I would be interested to learn of other estimates of this kind.

Tesla Austin “robotaxi”: 3 accidents in 7k miles
2

Calculating with 10 cars is just silly.

1 Like
3

I think we can already invalidate the comparison based on that assumption. Reports are that there have already been over a dozen incidents since the Tesla rollout - and with far less than 1 million miles.

1 Like
4

Why?

That’s current state, no? And hence their current run rate.

What is an appropriate number of test robotaxis to use for Tesla to estimate this, then?

Suppose Tesla 50x’s their test vehicles in the next few months, to 500 vehicles.

Then they will have a 20 incident data set in about 1 year, assuming an incident rate similar to Waymo.

But they might need to expand their geography and customer base accordingly to fill these 500 taxis and actually get rides to travel 35k miles per year per taxi.

Look, the above can all be wrong, but it’s based on what we know.

It’s an estimation/forecast exercise.

I’m open to alternatives.

I’m ok to be wrong.

Don’t forget to make your predictions here:

5

Fyi

I’m using event stats from Waymo, their definition of what I am calling incident:

But definition of incident can vary. City of Austin has an incident dashboard that is a much broader definition including much more minor things.

6

But it is obvious that it doesn’t represent a meaningful stage of development. Do it a couple weeks earlier and those 10s are zeros. Zero to 10 is a huge, nay infinite, rate of change. You need to wait until there is some meaningful track record before you can make a meaningful comparison.

2 Likes
7

Here is some food for thought: I can’t find the exact data, but it appears Waymo has something like 17,000 miles per disengagement.

One the earnings call, Musk said that Tesla now has 7,000 robotaxi miles. We’ve seen a number of disengagements in influencer videos. Approximately 11.

You can plug in your own assumptions, but it appears Tesla is three orders of magnitude worse than Waymo by that metric.

1 Like
8

To show, statistically, that an AI driver is 25% better than a human (incident rate is 75% of human rate), one power analysis (with 80% power) estimated that about 28 million driverless miles are needed when the incident is defined as “police reported accident.”

Rarer incidents like injuries and fatalities will require more miles. More common incident measures (less severe) will require fewer miles.

As an aside on 80% power:
This means if you were to do this measurement of 28 million miles and the AI driver incident rate is actually 75% of the human rate, 80 out of 100 times of doing this measurement (collecting 28 million AI driver miles 100 different times), you would observe a statistical difference between the AI rate and the human rate.

Waymo study (https://www.tandfonline.com/doi/full/10.1080/15389588.2024.2380522)

In one month in Austin, Tesla did 7,000 miles of supervised miles (not human driverless).

So, today, as far as we know, Tesla is not even getting the data needed to estimate true AI (no human supervision) safety.

It’s difficult to read the above and think Tesla is close to releasing an unsupervised driving product at scale for public driving scenarios.

(If they can’t build the product in any reasonable time, maybe Tesla should try to buy the product from someone else?

Are there any AI driving products for sale or license?)

This is one reason why one person cannot conclude from their 10s of thousands of miles of FSD driving (and likely much, much fewer miles) that an AI driving product is “good enough.”

Albaby has been saying the above over and over across many posts and threads.

But people keep citing single examples or experiences as evidence of product capability. Such examples show “possibility” not “performance generalizable to a large scale product.”

4 Likes
9

There is something about this which makes me think of starting to broadcast when the guitarist hasn’t yet tuned his guitar.

1 Like
10

Hasn’t the guitar player been telling us for years that the tuned guitar is just around the corner? Hard to fault the guy in the sound booth for believing him.

To continue to the analogy…

3 Likes
11

I don’t disagree.

Three questions for you.

  1. Do you think Tesla has broadcasted the AI driving capabilities (including unsupervised) of their products over the last 10ish years?

We all know all of the unrealized predictions.

There is also a growing list of legal proceedings that focus on the difference between capability claimed and capability observed.

  1. When do you expect the Tesla guitarist (AI driver) to begin playing (unsupervised driving with data collection and public reporting)?

(if I understand your analogy correctly)

  1. What date would you consider to be slow relative to the competition for the Tesla guitarist to begin playing unsupervised with data collection and public reporting of data?
1 Like
12

Wish I knew … might change my investing!

1 Like
13

Well, this sure is a tough crowd to ask question of, I’ll say that.

No one reveals their secrets, I guess.

1 Like
14

Serious question.

How do we know when the Tesla band starts playing?

Are they playing yet?