Waymo’s co-ceo gives a long presentation, with lots of good information, including discussion on multi-sensor versus single sensor (camera) and vanilla versus structured end-to-end models (and the exciting Bitter Lesson).
Highlights:
- Multi-sensor may be more likely to achieve higher safety than single sensor.
- Single sensor may never achieve super-human safety (or take a very long time to achieve it).
- Naive, pure black-box neural network models have weaknesses relative to models with structure informed by human knowledge of subjects like physics, traffic rules and human behavior (even when considering the bitter lesson).
Excerpts
the information from all of those sensors gets fused into a single view of the world around us that is much more precise and generally vastly superior to what you get with any one sensor.
Redundancy, of course, does not necessarily mean multiple sensing modalities, but if you need redundancy anyway, you might as well benefit from the complementary physics of the different sensing modalities
the Waymo Foundation model is a multimodal world action language model. It’s kind of a mouthful, so let me unpack the ingredients. It’s a multimodal model because it is able to process these multimodal sensor inputs: cameras, LiDARs, and radar. It’s a world model because it inherently understands how the world works—the physics, the dynamics, as well as the social and semantic aspect of it. It’s an action model because we are not just passively observing how the world evolves; we’re an active participant.
It’s an end-to-end model, which has a couple of nice properties. It allows us to effectively backpropagate the gradient from the task that we actually care about all the way to the early layers of the model. It allows the encoder to learn the right rich representations for what the generative part needs to solve the task.
a well-known lesson in the AI community that general methods that leverage massive compute and massive data will always beat methods that rely on handcrafted, engineered human knowledge. That’s the so-called bitter lesson that Richard Sutton published and formulated in 2019
one nuanced area where this lesson shows up is the use of structure in your models. Depending on how you use your structure, you can end up on either side of the bitter lesson. Essentially, structure that fights scale will always lose, and structure that channels scale always wins.
But if you need to reach superhuman levels of performance in a fully autonomous agent in a safety-critical environment, just doing that basic vanilla end-to-end is not enough.
But in the physical world, there does exist structure. You have laws of physics, you have rules of the road, you have objects that behave in reasonably predictable ways. You can use that structure in addition to the learned representations to boost your performance, simplify validation, and at the end of the day, just get better scaling laws.
If all you have is a black box end-to-end system, you are forced to do all of your evaluation and all of your training in the end-to-end setup, all the way from sensors to decisions to actuation. Having that intermediate structured representation allows you to mix and match. You can do some training at larger scale and some evaluation in the space of those compact structured representations, and some in the full space of end-to-end from sensors to decisions. Finally, we get strong, verifiable feedback signals for both evaluation and for training, training recipes to support things like reinforcement learning.
To build and safely deploy an agent in the physical world, it is absolutely critical to have a good large-scale, realistic, high-fidelity simulator.
Some related points made in various discussions:
