Last year at /r/RealTesla, a Chinese video of a car rocketing at full speed for 1+ minutes before crashing / killing a pedestrian made the rounds. We all recognized it as one of the weirder cases of "Sudden Unintended Acceleration", and I think that particular video really changed some minds.
While a lot of SUA events are from driver-error, it began a search into why Teslas seemed to be getting more SUA above-and-beyond the industry normal. This investigation (now filed under NHTSA) suggests that the ADC could be miscalibrated during a load-dump (or other electrical surge-like) scenario.
If the ADC associated with the accelerator pedal is off, then the Tesla will have the pedal at the wrong level of acceleration until the next calibration event, which is not going to happen until over a minute later.
This is extremely similar to that Chinese runaway Tesla, and perfectly seems to explain it. I'm glad that someone seems to have gotten to the bottom of this.
[–]dragontamer5 points1 year ago* (last edited 1 year ago)
(2 children)
I need a followup comment post. I cannot believe how huge this PDF is. The alleged evidence is comprehensive and solid.
Page 16 shows the voltage levels of the 12V line (eventually feeding the sensors / microcontrollers ADC) getting wrecked by just... turning left and right with the 1.2kW steering wheel. To be fair, this seems to happen in all cars, but its important to note that the following 12V line is "normal" case of cars.
Given this "changing 12V" situation (as you steer left-to-right and right-to-left), how does this affect the other parts of the Model 3? There's a lot of analog circuitry going on here, but there's a few notes.
The accelerator pedal is controlled by two voltages. Top voltage is how far down the pedal has been pushed, and the bottom voltage is how far up the pedal has been pushed. (so its a bit redundant for safety reasons, good design here). It takes over 10 pages to fully explain, but just know that there is a safety check, but its not sufficient for #2.
The 1.65V reference voltage is affected by the 12V supply voltage. It seems like the 1.65V is inadequately isolated. This means that the 1.65V somehow drops to 0.3V. This has a major effect on the ADC. When the voltage-reference changes voltage, the analog-circuitry in #1 goes haywire... especially if it happens during a calibration event.
Page 19 shows how to replicate this problem and prove the issue with the physical hardware on the Model 3. Pin44 is key on the circuit board.
Messing with the voltage-reference consistently causes the car to think the accelerator is pushed. This is proven in the following table when they miss with that voltage-reference.
So we can see the accelerator pedal at 0% (not pushed at all), but the software of the car pushing the pedal harder-and-harder as this voltage droop over the 12V line gets worse and worse. This is affected at the voltage level, before the signal is even digital or in memory. The voltage conversion itself (aka: the ADC, the analog-digital converter, is busted), so the Tesla logs in the final computer / logs will "prove" that the accelerator was pushed.
But that's false. The physical pedal was at 0%. Its just a voltage glitch that confused the computer.
...but its important to note that the following 12V line is “normal” case of cars.
This is actually more damning of Tesla, though. The fact this is normal behavior means anybody working in this space should know and would therefore compensate for high and low voltage scenarios. Then consider the fact that the new LiPo 12v battery is only 6.9Ah and you're basically discharging it at 14.5C to get 100A. The relationship of voltage sag to current draw in a LiPo battery isn't exactly unknown science. The Model 3 flooded battery is 45 Ah, and claims to have ~ 400 CCA rating. I'm not sure what the sag would look like there, but based on these crashes I bet it's just as bad.
how does this affect the other parts of the Model 3?
As I mentioned on Discord, if you exposed the FSD computer or even infotainment computer to this kind of huge voltage range, the best outcome would be software crashes because of spurious values in logic circuits. So they clearly knew to compensate for this in the power supplies for these computers. Using simple voltage dividers and op-amps as the reference for the ADCs and then skipping the gain compensation in the DSP is unbelievably lazy.
Worse, IMO, is the fact that TI calls out that you should either design or use a voltage reference generator, of which they offer several compatible with the DSP. Two seconds on the site and I found REF34-Q1 which is an automotive grade component with a cut off voltage of Vout + 50mV. And of course the design guide, data sheet, and other documentation describe how to best use this component in an automotive environment. In other words, this is lazy software and lazy hardware design when there are countless reference designs available.
Of course, the fact that so many systems run from this same 12v line on the inverter boards calls into question how they are properly isolated and protected from such dramatic voltage drops. You've got CAN, LIN, the FET drivers, etc. all running off this same rail. When the inverter boards started blowing up, I assumed it was because Tesla wrote a unified firmware for controlling new and old FETs and they were perhaps overdriving some of them and causing them to blow. But now I'm wondering if there isn't something rooted in hardware causing the issue. If the AC compressor or PAS cause large voltage swings, is it possible that this is causing another reference somewhere to be incorrect which in turn causes FETs to switch improperly and blow each other up? There's a lot of field and position sensors in the drive units, so I could see this happening more now than ever before.
I need a followup comment post. I cannot believe how huge this PDF is. The alleged evidence is comprehensive and solid.
Page 16 shows the voltage levels of the 12V line (eventually feeding the sensors / microcontrollers ADC) getting wrecked by just... turning left and right with the 1.2kW steering wheel. To be fair, this seems to happen in all cars, but its important to note that the following 12V line is "normal" case of cars.
Given this "changing 12V" situation (as you steer left-to-right and right-to-left), how does this affect the other parts of the Model 3? There's a lot of analog circuitry going on here, but there's a few notes.
The accelerator pedal is controlled by two voltages. Top voltage is how far down the pedal has been pushed, and the bottom voltage is how far up the pedal has been pushed. (so its a bit redundant for safety reasons, good design here). It takes over 10 pages to fully explain, but just know that there is a safety check, but its not sufficient for #2.
The 1.65V reference voltage is affected by the 12V supply voltage. It seems like the 1.65V is inadequately isolated. This means that the 1.65V somehow drops to 0.3V. This has a major effect on the ADC. When the voltage-reference changes voltage, the analog-circuitry in #1 goes haywire... especially if it happens during a calibration event.
Page 19 shows how to replicate this problem and prove the issue with the physical hardware on the Model 3. Pin44 is key on the circuit board.
Messing with the voltage-reference consistently causes the car to think the accelerator is pushed. This is proven in the following table when they miss with that voltage-reference.
So we can see the accelerator pedal at 0% (not pushed at all), but the software of the car pushing the pedal harder-and-harder as this voltage droop over the 12V line gets worse and worse. This is affected at the voltage level, before the signal is even digital or in memory. The voltage conversion itself (aka: the ADC, the analog-digital converter, is busted), so the Tesla logs in the final computer / logs will "prove" that the accelerator was pushed.
But that's false. The physical pedal was at 0%. Its just a voltage glitch that confused the computer.
This is actually more damning of Tesla, though. The fact this is normal behavior means anybody working in this space should know and would therefore compensate for high and low voltage scenarios. Then consider the fact that the new LiPo 12v battery is only 6.9Ah and you're basically discharging it at 14.5C to get 100A. The relationship of voltage sag to current draw in a LiPo battery isn't exactly unknown science. The Model 3 flooded battery is 45 Ah, and claims to have ~ 400 CCA rating. I'm not sure what the sag would look like there, but based on these crashes I bet it's just as bad.
As I mentioned on Discord, if you exposed the FSD computer or even infotainment computer to this kind of huge voltage range, the best outcome would be software crashes because of spurious values in logic circuits. So they clearly knew to compensate for this in the power supplies for these computers. Using simple voltage dividers and op-amps as the reference for the ADCs and then skipping the gain compensation in the DSP is unbelievably lazy.
Worse, IMO, is the fact that TI calls out that you should either design or use a voltage reference generator, of which they offer several compatible with the DSP. Two seconds on the site and I found REF34-Q1 which is an automotive grade component with a cut off voltage of Vout + 50mV. And of course the design guide, data sheet, and other documentation describe how to best use this component in an automotive environment. In other words, this is lazy software and lazy hardware design when there are countless reference designs available.
Of course, the fact that so many systems run from this same 12v line on the inverter boards calls into question how they are properly isolated and protected from such dramatic voltage drops. You've got CAN, LIN, the FET drivers, etc. all running off this same rail. When the inverter boards started blowing up, I assumed it was because Tesla wrote a unified firmware for controlling new and old FETs and they were perhaps overdriving some of them and causing them to blow. But now I'm wondering if there isn't something rooted in hardware causing the issue. If the AC compressor or PAS cause large voltage swings, is it possible that this is causing another reference somewhere to be incorrect which in turn causes FETs to switch improperly and blow each other up? There's a lot of field and position sensors in the drive units, so I could see this happening more now than ever before.
It's totally crazy how tesla uses the ADC on a safety critical component!
The question if the ADC's reference voltage is stable (enough) is a pretty basic one in any design.
I hope this goes public enough that tesla is forced to change/recall the affected components (at least in europe).