Alt text: picture of a Costco mechanical fish (cat) toy taken apart.
I had one of these fail last night from the battery lead coming off as pictured. That is something I expected. Upon opening the mechanism, both sides of the lithium cell have very concerning damage in a piercing type of abrasion. The cell is relatively small.
Speculatively, I expect the stuffed toy is flammable, making this an ideal house fire starter if the cell gets bridged layers in the battery from a puncture and goes into a runaway state.
All it needs is a way to secure the battery with some double sided tape and extra insurance like some kind of soft filler fabric or material to stop it from rattling around. Maybe add a bit of hot glue to the battery leads at the circuit board to insure they do not fail from a lack of strain relief.
I haven't retraced the circuit, but the battery looks fine otherwise. It appears to have a DW01 like protection board at the cell, and a 4054 batman according to the silkscreen.
This is a serious issue that should not be ignored.
By "regular" batteries I assume you mean alkaline ones like your standard AA and AAA. Lithium Ion batteries aren't really more "powerful", more likely the circuit was simply designed for constant 5V because that is cheap and easy nowadays. Small lithium batteries like the one used here and a simple charging circuit and port are dirt cheap when purchased in bulk.
Not to mention there's a size difference both for the battery itself and mounting area for it. For something like a AA battery there needs to be a user accessible battery compartment to replace them. It's cheaper and easier to just not do that.
Other versions of this use coin cell batteries, not AA or AAA.