Just to be clear (“this…, but that…” phrasing suggested to me a difference in concepts), these are different pairs of labels for the same error concepts.
Shashua introduces bias error and variance error first, and then explains them using the ideas of approximation error and generalization error, respectively.
I found it compelling because it applies a foundational concept in machine learning to AI driving.
Shashua is focused on the bias-variance tradeoff, per the title of the post, because the bias-variance tradeoff is one of the foundational theoretical results in machine learning - and model selection in particular.
How well models like Tesla’s and Waymo’s AI driving models perform will depend directly on their bias and variance error.
Total model prediction error = model bias error + model variance error
Model bias means, that under hypothetical repeated estimations (trainings) of the same underlying process, the mean model prediction will differ from reality by some fixed amount. The “bias” measures the deviation between the mean model prediction and reality.
The model variance is, under hypothetical repeated estimations (trainings) of the same underlying process, the mean squared deviation between the model prediction and reality. The variance measures random uncertainty in model predictions: noise/error around the mean model prediction.
If you believe that E2E (end-to-end neural network) is the more general model, then, based on the theory of the bias-variance tradeoff, this claim has a sound theoretical justification.
Why would the general model have lower bias, higher variance and the more specialized model higher bias, lower variance?
The general model is more flexible, it can fit a more diverse set of underlying processes, so it has lower bias.
But flexibility comes at a cost, the flexibility to fit a variety of scenarios also means the model has more inherent variability, so it has higher variance.
The more specialized model is the opposite. Less flexible to fit different processes, so higher bias, but also lower variance because it is a more constrained model with less inherent flexibility.
If you know the underlying system process very well, then the specialized model is the way to go: the embedded human knowledge is doing a lot of the work, you don’t need as much data to “discover” how it works because you already know how the system works. You won’t need as much compute because the specialized model will be smaller than a general black box E2E model.
If you do not know the underlying system well, it’s a black box, then the general model is the way to go. But you’ll need a lot more data to tell the model how the system works and then compute to go with the big model and the big data.
Regarding Tesla and Waymo, we could say that Tesla has the more general E2E model while Waymo has a more structured CAIS (compound AI system) model.
Then the question is, where does each approach live on the bias-variance tradeoff curve?
Which model has lower total prediction error currently and in the future?
For Tesla, how big of a model and thus how much data and compute will they need?
For Waymo, they still will need plenty of data, but if they are including some knowledge of how the system works (eg, objects like stop signs have special meaning, let’s make sure we always perceive them), or they think specialization is helpful (eg, a submodel that specializes on perception), is that knowledge really helpful in improving prediction?
Currently Waymo is much further along in unsupervised miles, with publicly reported safety outcomes on a 50+ million mile data set and accumulating 4 million unsupervised miles per month and accelerating.
Tesla reported only 7k miles in one month in Austin for supervised driving. They’ll need 10s of millions of miles of unsupervised driving (or some equivalent data set) to estimate accident rate if it is near human levels.
7k miles is such a small amount of data, especially for a general model, it’s not clear to me at all what value this data has. It’s small for training and it’s small for safety validation.
Tesla also announced a new model with 10x more parameters, suggesting they still need a bigger model (and with each bigger model, more data and more compute).
“Autonomous decisions: The bias-variance tradeoff in self-driving technology” (Autonomous decisions: The bias-variance tradeoff in self-driving technology | Mobileye Blog)