AI

3909 readers

31 users here now

Artificial intelligence (AI) is intelligence demonstrated by machines, unlike the natural intelligence displayed by humans and animals, which involves consciousness and emotionality. The distinction between the former and the latter categories is often revealed by the acronym chosen.

founded 3 years ago

147

‘Embarrassingly simple’ probe finds AI in medical image diagnosis ‘worse than random’ (venturebeat.com)

submitted 2 weeks ago by [email protected] to c/[email protected]

23 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 56 points 2 weeks ago (10 children)

We have models that are specifically made to be good at these kinds of tasks. Why would you choose the ones that aren't and then make generalizing claims about how AI sucks in this domain?

[–] [email protected] 13 points 2 weeks ago* (last edited 2 weeks ago) (8 children)

Yeah this is probably just straight up misinformation. By no means is a diagnosis going to be made by a generalist multimodal LLM. Diagnosis is a literally a binary classification (although that is an oversimplification) and on medical CV you are optimizing on that directly.

[–] [email protected] -3 points 2 weeks ago* (last edited 2 weeks ago) (7 children)

They did not use a LLM.

In a recent experiment, they set out to determine how reliable LMMs are in medical diagnosis — asking both general and more specific diagnostic questions — as well as whether models were even being evaluated correctly for medical purposes.

Curating a new dataset and asking state-of-the-art models questions about X-rays, MRIs and CT scans of human abdomens, brain, spine and chests, they discovered “alarming” drops in performance.