The judge wrote that he “does not aim to suggest that AI is inherently bad or that its use by lawyers should be forbidden,” and noted that he’s a vocal advocate for the use of technology in the legal profession. “Nevertheless, much like a chain saw or other useful [but] potentially dangerous tools, one must understand the tools they are using and use those tools with caution,” he wrote. “It should go without saying that any use of artificial intelligence must be consistent with counsel's ethical and professional obligations. In other words, the use of artificial intelligence must be accompanied by the application of actual intelligence in its execution.”
I won't even go that far. I can very much believe that you can build an AI capable of doing perfectly-reasonable legal arguments. Might be using technology that looks a lot different from what we have today, but whatever.
The problem is that the lawyer just started using a new technology to produce material that he didn't even validate, without determining whether-or-not it actually worked for what he wanted to do in its current state, and where there was clearly available material showing that it was not in that state.
It's as if a shipbuilder started using random new substance in its ship hull without actually conducting serious tests on it or even looking at consensus in the shipbuilding industry as to whether the material could fill that role. Meanwhile, the substance is slowly dissolving in water. Just slapped it in the hull and sold it to the customer.
EDIT: Hmm. Actually, I thought that the judge was saying that the lawyer needed to use AI-generated stuff in a human-guided role, but upon consideration, I may in fact be violently agreeing with the judge. "Actual intelligence" may simply refer to what I'm saying that the lawyer should have done.