AI driver learns in simulated environments

Stocks A to Z
,
1

On the Q3 conference call Tesla said something that sounds very reasonable regarding autonomous driving (I know, newsworthy).

They said that they are using a simulation of real-world driving for training the model for their AI driver.

This means they can simulate any normal and very abnormal driving scenario and make small or large variations to scenarios and train over and over on a ton of data, weighting various scenarios more or less to tune their model behavior to the outcome that they desire.

Assuming the simulations sufficiently represent real-world images and driving scenarios, including enough quantity and variety of edge cases, this will allow them to train at a rapid pace and be much less dependent on collecting real world data (which they already have a ton of, but for weird and edge scenarios who knows).

I wonder if they are just now really embracing and scaling this simulation approach?

If yes, why weren’t they doing this much sooner?

Like years ago? (for example, I believe Waymo has done a ton of simulations)

It sounds like they are just now really embracing this simulation approach (hard to say, but appears that way, they say how they use it for reinforcement learning and that it’s not in one of their latest FSD versions, 14.1).

Obviously this is still a very difficult machine learning problem, but this gives me more confidence in their methodology to, eventually, solve AI driving.

Big challenges that remain are:

Is their camera-only approach robust enough to the wide variety of real-world perception scenarios? (night, glare, rain, weird angles)

How much data and compute does their end-to-end model need to train in order to reach sufficient safety levels?

Can they manage with the low interpretability and explainability of an end to end model?

Can they, without too much trouble, not overtrain to specific scenarios and weaken the ability of their model to generalize? (eg, overtrain in Austin, or any other scenario large or small).

Our world simulator for reinforcement learning is pretty incredible.
Our Tesla reality simulator, when you see it, the video that’s generated by the Tesla reality simulator and the actual video looks exactly the same. That allows us to have a very powerful reinforcement learning loop to further improve the Tesla AI. We’re going to be increasing the parameter count by an order of magnitude. That’s not in 14.1. There are also a number of other improvements to the AI that are quite radical. This car will feel like it is a living creature. That’s how good the AI will get with the AI four computer before AI five. AI five, like I said, is by some metrics forty times better. But just to say safely, it’s a 10x improvement.

1 Like
2

From 2021:

From 2022:

2 Likes
3

Tesla has been using simulation for years. While I cannot confirm this, I believe it’s based on the photorealistic game technology that Nvidia released years ago. Lets see what Google AI can come up with:

AI Overview

Yes, Tesla uses simulation for its self-driving development, and it has been incorporating photorealistic technology like Unreal Engine for this purpose, which is also used in many video games. This allows the company to generate synthetic training data for a wide range of driving scenarios, including rare and dangerous ones that are difficult to replicate in the real world, and to create more realistic visualizations for in-car displays.

  • Simulation for development:

Tesla uses simulations to train and test its self-driving software, combining real-world data with vast amounts of synthetic, simulated data to improve the system’s performance in diverse and unusual situations.

  • Use of game engine technology:

Tesla is using Unreal Engine, a powerful engine from the video game industry, to build high-fidelity simulations. This engine powers many popular games and provides the advanced graphics capabilities needed for realistic simulation.

  • Benefits of simulation:

This approach allows Tesla to test for rare “edge cases” that are difficult to encounter in normal driving, such as extreme weather or unexpected obstacles, leading to a safer and more robust AI system.

  • In-car visualization:

The use of this game-engine technology is also being integrated to create more realistic and detailed visualizations on the in-car display for owners, providing a “game-engine level” driving experience.

SOONER Like 2022?

Tesla is creating a simulation of San Francisco in Unreal Engine

Avatar for Fred LambertFred Lambert | Sep 20 2022 - 8:41 am PT

Tesla is creating a simulation of San Francisco in Unreal Engine.

The Captain

2 Likes
4

I wonder why the mention of simulation in the earnings call when discussing new stuff.

Makes simulation sound like something is new or different about it versus something they’ve been doing for years:

We’ve released 14.1, got a technology roadmap that’s, I think, pretty amazing. We’ll be adding reasoning to the car. Our world simulator for reinforcement learning is pretty incredible.

Our Tesla reality simulator, when you see it, the video that’s generated by the Tesla reality simulator and the actual video looks exactly the same. That allows us to have a very powerful reinforcement learning loop to further improve the Tesla AI. We’re going to be increasing the parameter count by an order of magnitude. That’s not in 14.1. There are also a number of other improvements to the AI that are quite radical.

5

I’m sure this is good, I’m surprised they make a big deal of it because I’m also sure it’s been done before.

My question (before) related to one of the few things Donald Rumsfeld said that I learned from: the “unknown unknown” problem. You can design a simulation for everything you know, and you can tweak it all over creation, but there are still going to be a fair number of edge cases: the “unknown unknowns” that you’re never going to plan for simply because they’ve never happened and you couldn’t imagine them happening.

That is not to say a human “paying attention” driver would do well in those situations either, just that the idea of “training on simulations” comes with inherent limitations, which should be acknowledged (and never seem to be.)

1 Like
6

It is, back in 2022 they were developing algorithmic FSD, now it’s AI FSD, different breed of fish.

The Captain

7

Note that Waymo - and almost everyone else - can’t fully utilize simulations because of their use of LiDAR and radar. They would not only have to render visual images for the cameras, but also generate realistic point clouds and whatever radar produces. AND make all three of them realistically synchronized.

So, for them, it’s not just realistically rendering a scene as a camera would see it, it’s creating point clouds as LiDAR would see it. And while as humans we can visually determine whether a rendering of a rainy day is realistic, even simulating camera flare from low sun angle, how do we know that the point cloud being produced is what a LiDAR would see on a rainy day?

I’m not saying it’s impossible to do - just really hard to not only do, but QA. Since Teslas only depend on cameras, they only need to simulate what cameras can see. And that’s something a lot of companies have been working on for decades. Who’s been doing that for LiDAR point clouds? Certainly no-where near the same scale/effort/resources/money, etc.

5 Likes
8

There were several companies doing this as far back as 2018. Even an open source project that had full Lidar simulation within games engines, such as Unity and Unreal.

Many(most?) of these companies were acquired or went out of business because all the self driving companies built their own in house

Mike

3 Likes
9

Looks like more of a recent development at Tesla, at least in terms of reaching a more mature simulator for training data.

For reference, Waymo blogged about their sensor simulator in 2021.

Waymo
waymo sensor simulator 2021

Tesla Simulator

  • February 2024: Following the public release of OpenAI’s text-to-video generative model Sora, Elon Musk showcased some of Tesla’s own generative video. He revealed that the company had been able to do real-world video generation for about a year, hinting at the technology that would eventually become the world simulator.
  • August 2025: The FSD V13 update introduced a “world model” concept, an internal simulation that predicts the future actions of other road users. This gave the AI system a greater ability to act proactively rather than reactively, significantly improving driving performance.
  • October 2025: Tesla’s AI Chief, Ashok Elluswamy, announced the company’s “neural world simulator”. Unlike previous simulations that required hand-built environments, this new system is trained on massive real-world video datasets to synthesize fully dynamic, realistic worlds from scratch.
10

For training (the data and compute intensive part), Tesla does not only use cameras.

Tesla also uses Lidar data during training (apparently).

Tesla uses Lidar to calibrate (or somehow validate) their camera data and Tesla drives extensively through each Robotaxi geofenced geography (and other areas, I believe) with Lidar-equipped vehicles to collect, presumably, Lidar data.

Perhaps if Tesla could also simulate Lidar (maybe they are but won’t disclose it?), this might help accelerate their training by not having to manually drive around so much collecting Lidar data and also then process the data for downstream training.

Ironically, Tesla was negative on Lidar and geofencing, but, here we are watching Tesla

  • implement geofences
  • collect extensive Lidar data within each geofence

Second ironically, there was so much discussion about Tesla’s data advantage, but here we are, so many years into the effort, watching Tesla

  • promote its simulated data.
1 Like
11

You guys are focused on “who said what and when”. It is irrelevant.
Tesla is focused on making the future happen at scale.

There is a reason why people talk about Tesla non stop. Continuous innovation in Energy, Autonomous, Optimus and EVs.

2 Likes
12

Well, Tesla hasn’t said what they use LiDAR for. Except for those who work(ed) there, it’s all speculation.

We know they ran LiDAR test vehicles in Austin and the SF Bay Area, but that’s, perhaps not-coincidentally, where Tesla’s autonomy engineers are. If we see Tesla running LiDAR in Phoenix, Miami or Tampa, for instance, that will tell us that they are indeed using LiDAR as some kind of geo-fencing precursor. But, if they don’t run LiDAR there, then it’s more of a non-location specific process. So, I think it’s too early to jump to conclusions on just what Tesla is using LiDAR for.

I don’t know of anyone that is actively simulating LiDAR to have data on which to train automony. Even Waymo’s simulations don’t involve LiDAR. There’s been a lot of commercial development for camera simulation of the real world because there is lots of money to be made for video games, movies, AI generated advertising, etc., but the commercial applications for synthesized LiDAR are very narrow and not very profitable even if you could find a few.

There’s nothing ironic about using all the tools at one’s disposal. Like hiring the tallest people for your basketball team AND then teaching them how to jump higher (ok bad analogy, but you get the point).

That said, the outstanding question is how much data do autonomous systems need to be trained on? Certainly the vast majority of Tesla’s data is boring driving in a lane on a highway. Everyone’s probably got way more of that data than they need/want. The question, maybe, is how much data do you have of people cutting you off, wandering into your lane, getting a blow-out in front of you, etc? And then, can you simulate those without having first seen a bunch of them? We’ve already heard from Tesla years ago that they have been curating the gathered data on which they train. It seems clear than quality of training data is more important than quantity, but without enough quantity can you have enough quality data?

I think applications of generative AI are interesting here. If you train an AI on a large number of “interesting” situations, how many more, with variations, could that AI generate? There are YouTube channels that examine special effects in movies, with physicists critiquing the realism, or lack thereof. So while one could generate all sorts of examples, for this case you’d want only realistic scenarios that follow the laws of physics for training. Just how does a car flip over in front of you? How high can it go, etc.? The more of these crazy things you see, the more realistic ones you can generate.

There are also location-specific cases to observe. From experience, I know that people in NY/NYC drive differently than those in LA, who drive differently than those in Chapel Hill, NC. Decades ago, it was apparently the custom in NC to drive to the end of a highway acceleration ramp, stop, and then look over your shoulder. He almost rear-ended the first instance he encountered, but then had that occur multiple times. I’d never seen that (still haven’t).

That Tesla is geofencing for now isn’t surprising. From just a implementation perspective, geofencing seems a necessity - getting tow trucks dispatched for flat tires or mechanical break-downs, limiting how far the limited number of vehicles travel without passengers, limiting demand for the limited number of vehicles, etc. There’s also probably some remote monitoring happening, and so they want to ensure cell service is available within the geofenced area.

But to speculate that Tesla is geofencing because they need to build local map data, and need to build that map data using LiDAR is just that- speculation. Let’s see Tesla LiDAR vehicles outside of Austin/SF Bay Area first.

2 Likes
13

Thank you! Somebody needed to say it!

I’ll add that there’s also nothing unusual about using “expensive techniques” in a controlled situation to validate and calibrate less expensive sensors that are then taken outside and used in “wild” (working) situations.

:cowboy_hat_face:
ralph

2 Likes
14

One has to be careful about the distinction between a geofence, which implies a current inability to serve outside the area, and a service area which is merely where one is going to provide service and one could change it tomorrow.

1 Like
15

I agree, but I’m not downplaying valuable tools and methods such as lidar, geofencing - you know where the irony originates.

Here’s some more research findings.

My speculation is lidar is used at the very least to calibrate against/with their camera data. Maybe they do it at several geographies just to increase the variety of data, which would be reasonable.

Teslas with Lidar driving around, from the internet:

Dallas

Tesla ground-truth validation vehicles equipped with LiDAR rigs were spotted in Plano, Texas, a smaller city located northeast of downtown Dallas, the state’s third-most populous city.

Arizona

Two Robotaxi units with LiDAR validation equipment were spotted in Gilbert, Arizona, recently, showing that Tesla is aiming to launch its ride-hailing service in the state soon

Waymo looks very capable at simulation, including lidar, for some time.

The very first image in the Waymo 2021 blog post from upthread is of a Lidar simulation.

Caption:

Waymo uses a robust suite of simulation tools to test and evaluate the Waymo Driver, from how it interacts with other road users to how its sensors see the world around it. The run above is a completely synthetic situation through the “lenses” of both our 360 and perimeter lidar.

From 2021, 3:16 shows a Waymo Lidar simulation

(https://m.youtube.com/watch?v=COgEQuqTAug)

16

Great finds on the LiDAR in Arizona. Interesting that they’re mounted up so high - does strike me as a mapping function. I’ve pinged my Tesla knowledgeable people and will report back given this new (to me) information.

On the Waymo LiDAR simulation, though, that’s not compelling. It’s a very short sequence, repeated over and over, and they don’t change the POV, which should be very easy to do with true 3D data. I suspect they tried this and shelved it. (Did you really watch 3 hours plus to get to that point?!)

17

Well, suit yourself.

A wide variety of data can be simulated.

I’m going to go out on a limb and say “I bet Waymo and Google can do rich camera and lidar simulations for AV driving, with appropriate sensor fusion.”

18

Oh, I believe they CAN.

I just don’t believe they do it on any sort of regular basis, and certainly not to simulate events such that they can train/test their system on those simulations.

They almost certainly do it for cameras/visual data. But, then how are they dealing with LiDAR when it exists in the real world and not in their simulations?

19

Not sure I 100% understand your question.

One answer is simulate it.

Another answer is model it with incomplete data - a very standard thing to have to do and I am certain in this application they handle incomplete data of a huge variety constantly.

Waymo’s caption from their 2021 blog - 4 years ago, which shows the simulated Lidar video - says:

Waymo uses a robust suite of simulation tools to test and evaluate the Waymo Driver, from how it interacts with other road users to how its sensors see the world around it. The run above is a completely synthetic situation through the “lenses” of both our 360 and perimeter lidar .

Not sure why you doubt that they would do this today and have been for a long time when

  • a great variety of data can be simulated
  • the analytics technology readily exists for Lidar simulation
  • Waymo is backed by Google (massively capable tech firm)
  • Waymo does sensor fusion
  • Waymo says they do it in 2021 with video demo

see also upthread:

1 Like
20

Yeah, like I said earlier, a long time ago and a very short sequence that repeats.

I asked Claude about Waymo and lidar simulation and the summary was:
“their actual lidar simulation techniques for testing the Waymo Driver aren’t detailed in the recent public research papers.”

It did find this recent paper from Waymo on simulation:

Which they claim is “the first end-to-end generative world model trained on a single loss function capable of point A-to-B simulation on a city scale.”

The actual paper is here:

But, guess what - the word “lidar” doesn’t appear anywhere in the text.

Lidar simulation remains a research topic, not an active, certainly not a large, part of simulation for autonomous driving training.