The problem is breathless AI news stories have made people misunderstand LLMs. The capabilities tend to get a lot of attention, but not so much for the limitations.

And one important limitation of LLM's: they're really bad at being exactly right, while being really good at looking right. So if you ask it to do an arithmetic problem you can't do in your head, it'll give you an answer that looks right. But if you check it with a calculator, you find the only thing right about the answer is how it sounds.

So if you use it to find cases, it's gonna be really good at finding cases that look exactly like what you need. The only problem is, they're not exactly what you need, because they're not real cases.