Last year at /r/RealTesla, a Chinese video of a car rocketing at full speed for 1+ minutes before crashing / killing a pedestrian made the rounds. We all recognized it as one of the weirder cases of "Sudden Unintended Acceleration", and I think that particular video really changed some minds.
While a lot of SUA events are from driver-error, it began a search into why Teslas seemed to be getting more SUA above-and-beyond the industry normal. This investigation (now filed under NHTSA) suggests that the ADC could be miscalibrated during a load-dump (or other electrical surge-like) scenario.
If the ADC associated with the accelerator pedal is off, then the Tesla will have the pedal at the wrong level of acceleration until the next calibration event, which is not going to happen until over a minute later.
This is extremely similar to that Chinese runaway Tesla, and perfectly seems to explain it. I'm glad that someone seems to have gotten to the bottom of this.
As I mentioned on discord, this is completely nuts and I have some thoughts. First and foremost, when these SUA reports started coming to light, I was entirely convinced that they were cases of pedal misapplication. The angle of the cabin design and position of the drive relative to the pedal box puts you physically in a position where it's conceivable to me that an inattentive operator would simply press the wrong pedal more often than other brands I've driven. This is apparently not the case, and I'll apologize to every single person I made this argument to. I could have never conceived of such a slapdash design making its way to production vehicles.
Nest, IANAL but I feel like this leaves Tesla open to quite a lot of legal liability. First and foremost the lawsuits Tesla filed against customers for "defamation" can all be called into question at this point. Second, the property damage, injuries, and fatalities all seem like they'd be ripe for any lawyer willing to take on the case. The design is extremely poor in my non-EE opinion and if they could get an expert to testify in court that using an unfiltered 12v reference is a mistake that no engineer should have made, then they seem like they wouldn't stand a chance. Additionally, any lawyer engaging in discovery over this issue just might find communications from engineering staff to management warning of this issue either during design or testing, if any testing was actually done. If such documents exist, it would demonstrate that Tesla knew of the deficient design and still charged ahead claiming customers were at fault. These all seem like likely scenarios to my non-expert mind.
Finally, the claim that NHTSA was told there wasn't enough evidence for an investigation and to stop their inquiry is a major misstep by any government agency. Once the crashes started adding up, it seems to me that any inquiry into a deficient electrical or mechanical design was warranted. Especially with some of the speeds measured in these crashes and their locations at such public spots. We've seen pharmacies, grocery stores, small shops, large event spaces, arenas, major US intersections, and tiny european streets be the scene of so many of these crashes that I simply can not imagine dismissing an investigation. I don't know what liability an agency like NHTSA or ODI could face, but this is a pretty serious screw-up on their behalf. It also calls into question whether a larger systems review or analysis will be done against Tesla's vehicles. It seems like we're relying on the private sector too much for this work, and I'm concerned there are larger systemic failures lurking under the covers here.
[–]dragontamer4 points1 year ago* (last edited 1 year ago)
(2 children)
I know we talked about this on Discord but I forgot one key fact.
Tesla has an isolated 12V battery pack already. If the 12V battery pack remained isolated (for cabin / windows / sensors / etc. etc.), then all of this could have been easily avoided.
Tesla vehicles aren't an ICE vehicle. Tesla has innately separate power supplies for power / steering (aka: the main battery pack), and a separate 12V battery pack for other purposes. This isn't like ICE cars where the 12V line cannot be physically separated from the alternator or other aspects of the vehicle.
There's certainly two links from BC Front to the EPAS (I keep calling it IPAS for no good reason). And they're absolutely labeled BATT1 and BATT2. BATT1 is labeled elsewhere in the diagrams as 12v, which I presume means BATT2 means HVDC. But all of the other HV components are called out as HV in some manner, whereas the EPAS just says BATT2. I'm assuming this is tapping the HVDC power in the BC front, though.
Now, having looked at the wiring diagram, I'm wondering something about all the EPAS warning posts we've seen on forums over the years. It almost always seems to be related to a faulty ground or a faulty 12v battery connection somewhere, or on rare occasion a dying 12v battery. So now I'm wondering whether the EPAS controller is actually experiencing a fault because of low control circuit voltage or if there's something else going on here. And if it's the latter, could it manifest as low supply voltage to the DSP / ADC for the APP sensors?
Ultimately, most of these systems are fused with a FET and current sensors. So you would imagine that Tesla would have detailed logs somewhere in the system about supply voltages and current to all these peripherals. I'd love to see what that data looks like in a scenario where the EPAS warning is triggered, or on a vehicle that's stationary but receives steering input. Certainly 1.2kW is absolutely nothing to the HV pack, but if it's going through the same DC/DC that charges the 12v battery, it seems possible that the current could be too high and it could trigger some voltage sag. But the fact that these are all such common circuit designs, I'm just not sure I understand how it went so wrong.
A lot of the reverse-engineering / hard work seems to have come from the Youtube video, rather than the .pdf / complaint. I think I was confusing who did which work and who was making the arguments.
Fair enough, yeah. I still think the only way this gets any real traction is for a lawyer to hire an engineering team to reproduce this work on the bench and on a vehicle. I'm not sure how you do the latter safely without modifying the boards or modifying the 12v subsystem. Obviously monitoring the +12v bus is easy enough, but the impact of calibration during a low power event would have to be orchestrated carefully.
At 2:27 he pans across the EPAS module and the BATT1 and BATT2 connectors look exactly the same to me. They are not HVDC cables with the typical bright orange sheath, but rather just red and black. I think this is just two 12v sources. One is likely the DC/DC converter and the other is likely the battery. But for sure there's no HV connection here and there's no secondary battery source, it would actually just be that DC/DC converter being called "batt2".
Odd design, but here we are. I'm positive there's no secondary 12v source here and that it's just feeding from "redundant" controller boards and everything feeds through the 12v battery. But I'm just open to the possibility there might be two sources on the body controllers. Again, all would be sourced from the battery and the DC/DC converter would feed through it, but that battery is in the way of all of this and is not actually redundant at all.
As I mentioned on discord, this is completely nuts and I have some thoughts. First and foremost, when these SUA reports started coming to light, I was entirely convinced that they were cases of pedal misapplication. The angle of the cabin design and position of the drive relative to the pedal box puts you physically in a position where it's conceivable to me that an inattentive operator would simply press the wrong pedal more often than other brands I've driven. This is apparently not the case, and I'll apologize to every single person I made this argument to. I could have never conceived of such a slapdash design making its way to production vehicles.
Nest, IANAL but I feel like this leaves Tesla open to quite a lot of legal liability. First and foremost the lawsuits Tesla filed against customers for "defamation" can all be called into question at this point. Second, the property damage, injuries, and fatalities all seem like they'd be ripe for any lawyer willing to take on the case. The design is extremely poor in my non-EE opinion and if they could get an expert to testify in court that using an unfiltered 12v reference is a mistake that no engineer should have made, then they seem like they wouldn't stand a chance. Additionally, any lawyer engaging in discovery over this issue just might find communications from engineering staff to management warning of this issue either during design or testing, if any testing was actually done. If such documents exist, it would demonstrate that Tesla knew of the deficient design and still charged ahead claiming customers were at fault. These all seem like likely scenarios to my non-expert mind.
Finally, the claim that NHTSA was told there wasn't enough evidence for an investigation and to stop their inquiry is a major misstep by any government agency. Once the crashes started adding up, it seems to me that any inquiry into a deficient electrical or mechanical design was warranted. Especially with some of the speeds measured in these crashes and their locations at such public spots. We've seen pharmacies, grocery stores, small shops, large event spaces, arenas, major US intersections, and tiny european streets be the scene of so many of these crashes that I simply can not imagine dismissing an investigation. I don't know what liability an agency like NHTSA or ODI could face, but this is a pretty serious screw-up on their behalf. It also calls into question whether a larger systems review or analysis will be done against Tesla's vehicles. It seems like we're relying on the private sector too much for this work, and I'm concerned there are larger systemic failures lurking under the covers here.
I know we talked about this on Discord but I forgot one key fact.
Tesla has an isolated 12V battery pack already. If the 12V battery pack remained isolated (for cabin / windows / sensors / etc. etc.), then all of this could have been easily avoided.
Tesla vehicles aren't an ICE vehicle. Tesla has innately separate power supplies for power / steering (aka: the main battery pack), and a separate 12V battery pack for other purposes. This isn't like ICE cars where the 12V line cannot be physically separated from the alternator or other aspects of the vehicle.
There's certainly two links from BC Front to the EPAS (I keep calling it IPAS for no good reason). And they're absolutely labeled BATT1 and BATT2. BATT1 is labeled elsewhere in the diagrams as 12v, which I presume means BATT2 means HVDC. But all of the other HV components are called out as HV in some manner, whereas the EPAS just says BATT2. I'm assuming this is tapping the HVDC power in the BC front, though.
Now, having looked at the wiring diagram, I'm wondering something about all the EPAS warning posts we've seen on forums over the years. It almost always seems to be related to a faulty ground or a faulty 12v battery connection somewhere, or on rare occasion a dying 12v battery. So now I'm wondering whether the EPAS controller is actually experiencing a fault because of low control circuit voltage or if there's something else going on here. And if it's the latter, could it manifest as low supply voltage to the DSP / ADC for the APP sensors?
Ultimately, most of these systems are fused with a FET and current sensors. So you would imagine that Tesla would have detailed logs somewhere in the system about supply voltages and current to all these peripherals. I'd love to see what that data looks like in a scenario where the EPAS warning is triggered, or on a vehicle that's stationary but receives steering input. Certainly 1.2kW is absolutely nothing to the HV pack, but if it's going through the same DC/DC that charges the 12v battery, it seems possible that the current could be too high and it could trigger some voltage sag. But the fact that these are all such common circuit designs, I'm just not sure I understand how it went so wrong.
The weasel words are getting to me, honestly.
A lot of the reverse-engineering / hard work seems to have come from the Youtube video, rather than the .pdf / complaint. I think I was confusing who did which work and who was making the arguments.
Fair enough, yeah. I still think the only way this gets any real traction is for a lawyer to hire an engineering team to reproduce this work on the bench and on a vehicle. I'm not sure how you do the latter safely without modifying the boards or modifying the 12v subsystem. Obviously monitoring the +12v bus is easy enough, but the impact of calibration during a low power event would have to be orchestrated carefully.
Ok, double comment time. So, reviewing this video: https://www.youtube.com/watch?v=rDYbvI32OBE
At 2:27 he pans across the EPAS module and the BATT1 and BATT2 connectors look exactly the same to me. They are not HVDC cables with the typical bright orange sheath, but rather just red and black. I think this is just two 12v sources. One is likely the DC/DC converter and the other is likely the battery. But for sure there's no HV connection here and there's no secondary battery source, it would actually just be that DC/DC converter being called "batt2".
Odd design, but here we are. I'm positive there's no secondary 12v source here and that it's just feeding from "redundant" controller boards and everything feeds through the 12v battery. But I'm just open to the possibility there might be two sources on the body controllers. Again, all would be sourced from the battery and the DC/DC converter would feed through it, but that battery is in the way of all of this and is not actually redundant at all.