I’m not sure this is what Tesla is doing, but it makes sense:
You have a camera system that can figure out how far away things are. It does this by knowing how the Tesla is moving in space and seeing how things change in the camera. Karpathy, when he worked at Tesla, demonstrated this years ago, building a 3D map of the environment.
You want to check that the 3D map the Tesla is building is accurate. So you write some software to use LiDAR to determine object distance and you check that against the distances your camera software calculates.
If all is good with this check in multiple areas/scenarios, then your camera distance setup is good to go without the LiDAR in production vehicles.
Monocular depth estimation is a computer vision task where an AI model tries to predict the depth information of a scene from a single image. In this process, the model estimates the distance of objects in a scene from one camera viewpoint. Monocular depth estimation has many applications and has been widely used in autonomous driving, robotics, and more. Depth estimation is considered one of the hardest computer vision tasks, as it requires the
Actually, this is the process to which I was referring:
Not using a single image, but a series of images from a moving vehicle. This is 5 years ago and in it Karpathy also talks about using radar depth data to annotate the camera images, but as Tesla vehicles no longer have radar, this isn’t being done with production vehicles any more.
The video briefly describes 3 methods for determining depth. The first is the vehicle moving in space and seeing how the images change. The second uses radar to annotate depth. The third uses its own neural net feedback to refine its depth predictions.
And, I think the test cars with LiDAR were done as a check of whatever algorithm Tesla is using today, 5 years after this video was made.
OK, I get that. My question is why they need to do this in Austin. Couldn’t they do this sort of mapping/comparison somewhere else, like privately? Or are they going to try to map every intersection in Austin (and presumably therefore every intersection in each city they open up?)
I understand the need to have accuracy in the measurements, but what they see in Austin today won’t be the same thing a non-lidar car sees tomorrow. There will be traffic, potholes, construction, signals moved, etc. Does that mean the lidar information becomes degraded over time? And doesn’t that imply a lot of retraining over time?
Just a comment about starting with fewer or more sensor types.
In machine learning, the goal is to find the data set (sensor configuration and its data) and an associated model architecture (neural network, etc) that is sufficiently predictive to do AI driving.
The universe of data and models is enormous.
How does one search this vast collection of data and models to find the best ones?
This is fundamentally an optimization problem.
This is the challenge of feature selection and model selection.
There is no single approach on how to search the vast collection of features and models, but certain approaches have been developed in machine learning, when applied to any problem (not just AI driving).
One approach is to start with fewer features (eg, fewer sensors and associated data, such as camera only) and then add more to check if there is AI model improvement.
A second approach is to start with more features (eg, more sensor types and associated data, such as camera plus other sensors) and then remove to check if removal doesn’t materially degrade AI model performance.
Tesla says they have already progressed through the step of reducing sensor types, except for much more limited data collection of camera plus lidar.
Waymo and others remain multi-sensor.
Yet, Waymo is way ahead of Tesla when the benchmark is miles of robotaxi fares.
Two of Tesla’s “supposed” scale advantages are vast data collection and manufacturing efficiency, right?
But as of today, these advantages have not yet realized a lead in actual robotaxi fare mileage. (I think we can all agree on this statement and I am talking robotaxi, not driver assist.)
The current total (deployed, not TAM) US robotaxi footprint is small, so I think we can agree that a manufacturing advantage in robotaxi-ready vehicles is not very relevant today.
That leaves the data advantage of Tesla collecting sensor data from its large fleet of retail vehicles.
Yet, Tesla is behind Waymo on robotaxi miles.
Waymo has 25+ million miles and Tesla just started collecting robotaxi miles a few days ago.
Tesla will not catch Waymo in robotaxi miles, and not dominate Waymo by this metric, unless it grows its robotaxi fleet to catch and surpass Waymo’s fleet.
If Tesla has a data advantage, what exactly is the advantage?
(besides, “we have a ton of miles of camera data”)
When and how will this advantage manifest itself into a faster robotaxi rollout?
Because, through today, the Tesla robotaxi rollout is slower than Waymo’s.
And, so far, Tesla’s robotaxi rollout has two key items in common with Waymo: geofence and safety monitor/driver.
If that’s your benchmark, congrats. Waymo will continue to be ahead of Tesla for another year at least, but 24 months from now, my money is on Tesla pulling ahead.
Tesla’s approach is different because:
Tesla’s approach was to work on autonomy world-wide first; Waymo’s was to first work with a specific type of special vehicle in a specific geo-fenced area on specific types of roads.
Tesla uses cars that are ¼ the cost of Waymos.
Tesla already mass produces those cars, heck they’re the world’s most popular car. Not EV: car.
Tesla claims they don’t need to create nor maintain HD maps everywhere they want to go.
It’s been a long time coming for Waymos to operate on highways. Do you think Tesla will take years to have that ability in its robotaxis?
Tesla can more easily move from a robotaxi business to a autonomy for personally owned car business than Waymo.
The last item could be huge for profits.
As for who’s ahead, one has to understand what their end-games are and what their approaches were. If Waymo were to now try to operate without LiDAR, how many years would that send them back? And if they don’t, that limits their ability to move their tech into the market for personally owned vehicles. Heck, how does one mount a LiDAR unit on top of a convertible car?
And it’s early days (literally) for Tesla. The coming months will show us whether Tesla’s approach enables faster scaling than Waymo or not.
That’s a good question to which I don’t have the answer.
It could be anything from the team being in Austin perparing for the RT rollout to needing to get some kind of map information before doing the service in that area.
If the latter, it hurts Musk’s/Tesla’s credibility as to the need for high definition pre-mapping. If the former, then it’s just more testing with the latest software. I do believe most of the autonomy software engineers are in California, but with Tesla HQ in Austin, the team might be distributed.
I think the most significant factor is that Texas doesn’t (or at least didn’t) have any licensing or permitting requirements for establishing an AV service, unlike California. So if you want to launch (or at least closed beta) an AV service, Texas allows you to just go out and do it, rather than deal with any regulatory hurdles or disclosing any information you might prefer to keep confidential.
Or at least, they did. There are some new changes that take effect in a few months.
There’s no such thing as private for Tesla anymore. People follow their test cars around with drones EVERYWHERE. There’s a guy who lives near an intersection they’ve been using to test unprotected left turns with each subsequent version of software. He has setup a video camera on his house aimed at that intersection and posts videos of their testing.
Tesla does a lot of their testing, probably most of it, in locations where they have a facility nearby. After finding and informing them about a HUGE bug* in their FSD version a few years ago, I offered to do some testing for them locally, but they didn’t take me up on the offer. Mostly because they don’t do that kind of testing here, because the nearest Tesla design facility is very far from here.
* The bug was how they handled diverging diamond interchanges, and it was a catastrophic bug. Turns out that the bug wasn’t really in the self driving software stack, but rather in the way Google presented the map data to the car.
This is true. However, I not sure what the salient difference is between:
Me hailing a robotaxi by sending it my destination, it stopping in front of my house, entering the vehicle, and pushing “GO” on the display, and it taking me to my destination
Me sending my destination to my Tesla parked in the driveway or on the street, entering the vehicle, pushing the blue “FSD GO” button, and the car taking me to my destination.
There are obviously some back-end differences, choosing which car, optimizing paths across network, determining fare, processing payment, etc. But as far as technology and/or useful data collected, there isn’t all that much difference.
It will be interesting to find out, but I don’t think so, for the reasons I mentioned above. Except in Texas, AV ride hailing requires permits, which are time consuming to get. Tesla needs to start that process fairly soon if they have a chance of meeting your time frame.
Tesla’s goal is a universal solution. Tesla’s approach is to start with geofencing on specific types of roads.
From the point of view of running a business with all of the associated risks, for the two cases above,
who is liable for the outcomes of the driving decisions?
I agree, in the sense that going from Level 2 which is you are doing to Level 4 is probably only a 0.1% improvement or something. I made up that number. So not a massive absolute improvement. But getting that last bit is really hard. Tesla doesn’t seem to quite be there yet.
There are two main differences, from the perspective of data collection/driving (obviously it’s a huge difference for the passenger experience):
The former gives the company data about how the car responds to “post-disengagement” scenarios. In FSD, if the car encounters something that it can’t immediately handle, it gets handed off to a driver. So it doesn’t get data on how the AI responds to the events that occur after that hand-off. In robotaxi mode, the AI has to deal with everything, so scenarios that would normally result in a hand-off instead get addressed by the AI.
Robotaxi gives you data on the all important PU-DO - “pick up and drop off.” All FSD rides start with the passenger already in the car, and almost always with the car parked; and they end with the passenger choosing to get out of the car in a location they have chosen for themselves, again almost always with the car parked. Taxi rides are different. The car has to pick a place where it can accommodate passengers to enter, usually without being in a parking space, and it has to do the same at trip end. PU-DO is apparently a pretty tricky part of the rideshare/Uber/robotaxi business, and FSD usage doesn’t give Tesla much (if any) data on those events.
One quibble. I specifically said “ride hailing.” That permit took Waymo six years to get in California. Presumably, regulators are now more familiar with the tech, there now exists a framework, the tech has improved, data have been validated, etc. and now adding new cities in CA appears to take on the order of months to a year. So much faster, but still not fast. Tesla will have to start from scratch. Your guess is as good as mine, but it won’t be fast.
And setting aside the AV component for a minute, it took Uber and Lyft years to get ride hailing permits nationwide. So even just that part will take some doing. It is a jurisdiction by jurisdiction thing.
Upthread it was posited that once Elon snaps his fingers there will be 35K robotaxis rolling off the line every week. It won’t be like that.
While the highlighted today might make the statement technically accurate today surely you can see a difference in the strategic position of a company which can … no matter how good testing is going … expand at 1-2K per year versus one that has the potential to expand at 50 to 100X that per month. Doesn’t mean that is the only barrier, but it is certainly one less barrier.
Underlying some of these discussions is just a difference in perspective on what factor is going to be most relevant in limiting growth at scale. There are a number of candidates:
Vehicle production - making cars at speed and cost to get them on the road.
AI/“Brain” - getting the driver capable of the most autonomy in the most places.
Servicing the vehicles and customers - supporting the cars through teleops and cleaning, getting customers onboarded, getting your app downloaded, attending to things like lost items and complaints and so on.
Regulatory - getting licensing and approval in jurisdictions, both in the U.S. and abroad.
These are interrelated, of course. The better your AI, the more independent your cars are, which reduces your service/support costs for them (fewer teleoperators, fewer messed-up trips, etc.).
So, if the functional bottleneck to expanding at scale is vehicle production, Tesla would have an almost insuperable advantage over Waymo. They use their “stock” vehicles and there’s a massive number of them already on the road. If the functional bottleneck to expanding at scale is regulatory, though, then they don’t have much of an advantage at all.
We have almost no information about the actual cost structure of these businesses yet. I suspect that in these early days, the cars are actually a relatively small part of the cost of providing the service and not really the limiting factor - it’s all the other stuff (teleoperators, response teams, customer service, regulatory approval and compliance, and costs of customer acquisition) that really limit the scope and speed of build out. That’s because neither Waymo nor Tesla’s AI is yet good enough to functionally replace a human driver, and so requires a lot of “stuff” outside the car for it to work (well, except Tesla is still in the phase of putting the monitor inside the car in the front seat).
Again, we have to highlight today. Waymo has partnered with Toyota and Hyundai (and probably some others) to equip Driver on their vehicles. Both of those companies are much larger than Tesla. And is currently partnered with Magna Steyr, who can build whatever brand you want. So if we look into the future a bit it isn’t hard to see how Waymo could scale rapidly with its manufacturing partners.
