I kept joking with my kids that if they were naughty we'd have squidmas instead.
I used ChatGPT to create squidmas songs. It managed to perfectly understand the idea and generate squid based Christmas songs that made total sense.
"All I want for squidmas" was a mistake!
It seems unlikely this was in its training data.
There's a good Asimov story where they're trying out a new type of brain and they discover it has a subconscious layer that they didn't explicitly create. It can dream and when it does it doesn't follow the 3 rules.