Info from Reuters that is potentially very damaging to Tesla’s current case for autonomy.
This is the most consequential evidence I have seen about Tesla’s modeling and model training approach per se.
Regarding overtraining,
From this Reuters article,
Inside Tesla, as these events approached, staffers worked long hours mapping routes and training the software on specific hazards to make the company’s self-driving technology appear more capable than it really is, four of the former Tesla employees told Reuters. The staffers said these labor-intensive safeguards are impossible to deploy on a broad scale.
The Utah data-labeling staff, three of the employees said, doubled in the half-year before the Austin launch to about 300 workers. The department, they said, worked primarily on projects to make the carefully controlled Austin test go smoothly.
As Tesla data labelers prepared for the rollout, the software was still erratic, two of the employees said. With each FSD update, some driving behaviors improved. Others worsened. In the Utah office, two large screens displayed statistics on miles between driver interventions for FSD – a key autonomous-driving safety metric.
“It would go up and down like the stock market” with no consistent improvement, one of the former employees said.
“It was like, ‘OK, we trained a car’” to operate in a restricted zone, the person said.
If true, the above is evidence of overtraining on small scenarios (eg, small geographies, hyper-specific areas, specific demos).
To the extent overtraining is needed to produce the degree of autonomy that Tesla has so far shown in Austin and Texas, this is a damaging design error in their overall machine learning approach.
It could mean their data, model and/or training approach do not generalize sufficiently well with sufficient safety to scale an L4 fleet.
In another thread I asked how much data might be needed for a more “black box” neural network model: