There's a protection diode for reverse current on the esp and your computers usb has quite a bit of protection against this kind of stuff.
I wouldn't worry about it. Even if there are errors in your cabling, it will very rarely break things if that is what you are asking. I catch myself all the time modifying things while it's on which you aren't really suppose to do.
Faulty parts will happen but you will either be able to physically see the bad solder or you will have broken it yourself and will have seen the smoke coming out (mostly when using an external power sources). I also know you can break individual pins with negative voltage which is harder to notice. ESD can happen as well and will be much more subtle, but there should be some protection on the board for that.
Its rare, but it's always a good idea to have multiple esps, so you can so you can switch them out to verify it isn't the chip.