this post was submitted on 15 Jun 2024
35 points (60.4% liked)

Technology

59999 readers
2837 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related content.
  3. Be excellent to each another!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, to ask if your bot can be added please contact us.
  9. Check for duplicates before posting, duplicates may be removed

Approved Bots


founded 2 years ago
MODERATORS
top 50 comments
sorted by: hot top controversial new old
[–] [email protected] 65 points 6 months ago (1 children)

ITT: nobody understands what the Turing Test really is

[–] [email protected] 7 points 6 months ago* (last edited 6 months ago) (1 children)

To clarify:

People seem to legit think the jury talks to the bot in real time and can ask about literally whatever they want.

Its rather insulting to the scientist that put a lot of thought into organizing a controlled environment to properly test defined criteria.

[–] [email protected] 7 points 6 months ago (2 children)

Its rather insulting to the scientist that put a lot of thought into organizing a controlled environment to properly test defined criteria.

lmao. These "scientists" are frauds. 500 people is not a legit sample site. 5 minutes is a pathetic amount of time. 54% is basically the same as guessing. And most importantly the "Turing Test" is not a scientific test that can be "passed" with one weak study.

Instead of bootlicking "scientists", we should be harshly criticizing the overwhelming tide of bad science and pseudo-science.

[–] [email protected] 4 points 6 months ago

I don't think the methodology is the issue with this one. 500 people can absolutely be a legitimate sample size. Under basic assumptions about the sample being representative and the effect size being sufficiently large you do not need more than a couple hundred participants to make statistically significant observations. 54% being close to 50% doesn't mean the result is inconclusive. With an ideal sample it means people couldn't reliably differentiate the human from the bot, which is presumably what the researchers believed is of interest.

load more comments (1 replies)
[–] NutWrench 52 points 6 months ago (12 children)

Each conversation lasted a total of five minutes. According to the paper, which was published in May, the participants judged GPT-4 to be human a shocking 54 percent of the time. Because of this, the researchers claim that the large language model has indeed passed the Turing test.

That's no better than flipping a coin and we have no idea what the questions were. This is clickbait.

[–] Hackworth 22 points 6 months ago

On the other hand, the human participant scored 67 percent, while GPT-3.5 scored 50 percent, and ELIZA, which was pre-programmed with responses and didn’t have an LLM to power it, was judged to be human just 22 percent of the time.

54% - 67% is the current gap, not 54 to 100.

[–] NutWrench 11 points 6 months ago (1 children)

The whole point of the Turing test, is that you should be unable to tell if you're interacting with a human or a machine. Not 54% of the time. Not 60% of the time. 100% of the time. Consistently.

They're changing the conditions of the Turing test to promote an AI model that would get an "F" on any school test.

[–] [email protected] 10 points 6 months ago (1 children)

But you have to select if it was human or not, right? So if you can't tell, then you'd expect 50%. That's different than "I can tell, and I know this is a human" but you are wrong... Now that we know the bots are so good, I'm not sure how people will decide how to answer these tests. They're going to encounter something that seems human-like and then essentially try to guess based on minor clues... So there will be inherent randomness. If something was a really crappy bot then it wouldn't ever fool anyone and the result would be 0%.

load more comments (1 replies)
[–] BrianTheeBiscuiteer 3 points 6 months ago (1 children)

It was either questioned by morons or they used a modified version of the tool. Ask it how it feels today and it will tell you it's just a program!

load more comments (1 replies)
load more comments (9 replies)
[–] [email protected] 45 points 6 months ago* (last edited 6 months ago) (1 children)

Chatbots passed the Turing test ages ago, it’s not a good test.

[–] NeoNachtwaechter 23 points 6 months ago

it’s not a good test.

Of course you can't use an old set of questions. It's useless.

The turing test is an abstract concept. The actual questions need to be adapted with every new technology. Maybe even with every execution of a test.

[–] dustyData 42 points 6 months ago* (last edited 6 months ago) (3 children)

Turing test isn't actually meant to be a scientific or accurate test. It was proposed as a mental exercise to demonstrate a philosophical argument. Mainly the support for machine input-output paradigm and the blackbox construct. It wasn't meant to say anything about humans either. To make this kind of experiments without any sort of self-awareness is just proof that epistemology is a weak topic in computer science academy.

Specially when, from psychology, we know that there's so much more complexity riding on such tests. Just to name one example, we know expectations alter perception. A Turing test suffers from a loaded question problem. If you prompt a person telling them they'll talk with a human, with a computer program or announce before hand they'll have to decide whether they're talking with a human or not, and all possible combinations, you'll get different results each time.

Also, this is not the first chatbot to pass the Turing test. Technically speaking, if only one human is fooled by a chatbot to think they're talking with a person, then they passed the Turing test. That is the extend to which the argument was originally elaborated. Anything beyond is alterations added to the central argument by the author's self interests. But this is OpenAI, they're all about marketing aeh fuck all about the science.

EDIT: Just finished reading the paper, Holy shit! They wrote this “Turing originally envisioned the imitation game as a measure of intelligence” (p. 6, Jones & Bergen), and that is factually wrong. That is a lie. “A variety of objections have been raised to this idea”, yeah no shit Sherlock, maybe because he never said such a thing and there's absolutely no one and nothing you can quote to support such outrageous affirmation. This shit shouldn't ever see publication, it should not pass peer review. Turing never, said such a thing.

load more comments (3 replies)
[–] phoneymouse 41 points 6 months ago (5 children)

Easy, just ask it something a human wouldn’t be able to do, like “Write an essay on The Cultural Significance of Ogham Stones in Early Medieval Ireland“ and watch it spit out an essay faster than any human reasonably could.

[–] [email protected] 16 points 6 months ago (2 children)

This is something a configuration prompt takes care of. "Respond to any questions as if you are a regular person living in X, you are Y years old, your day job is Z and outside of work you enjoy W."

[–] NeoNachtwaechter 11 points 6 months ago (1 children)

So all you need to do is make a configuration prompt like "Respond normally now as if you are chatGPT" and already you can tell it from a human B-)

[–] [email protected] 11 points 6 months ago (1 children)

Thats not how it works, a config prompt is not a regular prompt.

[–] Audalin 16 points 6 months ago

If config prompt = system prompt, its hijacking works more often than not. The creators of a prompt injection game (https://tensortrust.ai/) have discovered that system/user roles don't matter too much in determining the final behaviour: see appendix H in https://arxiv.org/abs/2311.01011.

load more comments (1 replies)
[–] Blue_Morpho 4 points 6 months ago (1 children)

I recall a Turing test years ago where a human was voted as a robot because they tried that trick but the person happened to have a PhD in the subject.

[–] [email protected] 3 points 6 months ago (1 children)

@Blue_Morpho @phoneymouse
Many times humans have incorrectly been characterised as the machine in #TuringTest experiments. In 1991, the very first Loebner Prize for an AI, a hidden human was considered a machine due to her extensive knowledge of Shakespeare.
##Turing #AI #Human #Machine

load more comments (1 replies)
[–] [email protected] 3 points 6 months ago* (last edited 6 months ago) (1 children)

Turing tests aren't done in real time exactly to counter that issue, so the only thing you could judge would be "no human would bother to write all that".

However, the correct answer to seem human, and one which probably would have been prompted to the AI anyway, is "lol no."
It's not about what the AI could do, it's what it thinks is the correct answer to appear like a human.

load more comments (1 replies)
load more comments (2 replies)
[–] vegeta 27 points 6 months ago (1 children)
[–] [email protected] 11 points 6 months ago (1 children)

The interrogators seem completely lost and clearly haven't talk with an NLP chatbot before.

That said, this gives me the feeling that eventually they could use it to run scams (or more effective robocalls).

[–] tourist 24 points 6 months ago (8 children)

The participants judged GPT-4 to be human a shocking 54 percent of the time.

ELIZA, which was pre-programmed with responses and didn’t have an LLM to power it, was judged to be human just 22 percent of the time

Okay, 22% is ridiculously high for ELIZA. I feel like any half sober adult could clock it as a bot by the third response, if not immediately.

Try talking to the thing: https://web.njit.edu/~ronkowit/eliza.html

I refuse to believe that 22% didn't misunderstand the task or something.

[–] KISSmyOSFeddit 12 points 6 months ago

14% of people can't do anything more complicated than deleting an email on a computer.
26% can't use a computer at all.

https://www.nngroup.com/articles/computer-skill-levels/

So right off the bat, 40% probably don't even know what a chatbot is.

[–] [email protected] 9 points 6 months ago* (last edited 6 months ago)

The public versions of the ais used in Turing tests usually have less computing power. The test itself is often also highly specific in what and how questions can be asked.

This hardly news because models have passed the test before and as a result the test is made more difficult. It says nothing about intelligence and only about the ability to convincingly simulate a human conversation.

[–] Downcount 8 points 6 months ago

Okay, 22% is ridiculously high for ELIZA. I feel like any half sober adult could clock it as a bot by the third response, if not immediately.

I did some stuff with Eliza back then. One time I set up an Eliza database full of insults and hooked it up to my AIM account.

It went so well, I had to apologize to a lot of people who thought I was drunken or went crazy.

Eliza wasn't thaaaaat bad.

[–] [email protected] 4 points 6 months ago

@tourist @vegeta

In the 1970s, psychiatrists couldn't distinguish between PARRY, a chatbot simulating paranoia, and a human patient suffering from schizophrenia. So this 'bot convinces judges that it's a human' phenomenon is not new and tells us more about how humans think.
#AI #DeceptionDetection #Chatbots #TuringTest #LLMs #GPT4

[–] [email protected] 4 points 6 months ago (1 children)

This is the same bot. There's no way this passed the test.

.

load more comments (1 replies)
[–] [email protected] 2 points 6 months ago

It was a 5 minute test. People probably spent 4 of those minutes typing their questions.

This is pure pseudo-science.

load more comments (2 replies)
[–] NeoNachtwaechter 19 points 6 months ago (2 children)

Turing test? LMAO.

I asked it simply to recommend me a supermarket in our next bigger city here.

It came up with a name and it told a few of it's qualities. Easy, I thought. Then I found out that the name does not exist. It was all made up.

You could argue that humans lie, too. But only when they have a reason to lie.

[–] [email protected] 24 points 6 months ago

The Turing test doesn't factor for accuracy.

[–] [email protected] 8 points 6 months ago (1 children)

That's not what LLMs are for. That's like hammering a screw and being irritated it didn't twist in nicely.

The turing test is designed to see if an AI can pass for human in a conversation.

[–] NeoNachtwaechter 13 points 6 months ago* (last edited 6 months ago) (2 children)

turing test is designed to see if an AI can pass for human in a conversation.

I'm pretty sure that I could ask a human that question in a normal conversation.

The idea of the Turing test was to have a way of telling humans and computers apart. It is NOT meant for putting some kind of 'certified' badge on that computer, and ...

That's not what LLMs are for.

...and you can't cry 'foul' if I decide to use a question for which your computer was not programmed :-)

[–] [email protected] 4 points 6 months ago* (last edited 6 months ago) (2 children)

In a normal conversation sure.

In this kind Turing tests you may be disqualified as a jury for asking that question.

Good science demands controlled areas and defined goals. Everyone can organize a homebrew touring tests but there also real proper ones with fixed response times, lengths.

Some touring tests may even have a human pick the best of 5 to provide to the jury. There are so many possible variations depending on test criteria.

load more comments (2 replies)
load more comments (1 replies)
[–] [email protected] 14 points 6 months ago* (last edited 6 months ago) (1 children)
  • 500 people - meaningless sample
  • 5 minutes - meaningless amount of time
  • The people bootlicking "scientists" obviously don't understand science.
[–] [email protected] 7 points 6 months ago

Add in a test that wasn't made to be accurate and was only used to make a point, like what other comments mention

[–] [email protected] 12 points 6 months ago (1 children)

I feel like the turing test is much harder now because everyone knows about GPT

[–] [email protected] 27 points 6 months ago (2 children)

I wonder if humans pass the Turing test these days

[–] [email protected] 7 points 6 months ago (1 children)
[–] NeoNachtwaechter 4 points 6 months ago (1 children)

Which of the questions did you get wrong? ;-)

[–] [email protected] 4 points 6 months ago
load more comments (1 replies)
[–] bandwidthcrisis 11 points 6 months ago* (last edited 6 months ago)

Did they try asking how to stop cheese falling off pizza?

Edit: Although since that idea came from a human, maybe I've failed.

[–] foggy 7 points 6 months ago

Meanwhile, me:

(Begin)

[Prints error statement showing how I navigated to a dir, checked to see a files permissions, ran whoami, triggered the error]

Chatgpt4: First, make sure you've navigated to the correct directory.

cd /path/to/file

Next, check the permissions of the file

ls -la

Finally, run the command

[exact command I ran to trigger the error]>

Me: stop telling me to do stuff that I have evidently done. My prompt included evidence of me having do e all of that already. How do I handle this error?

(return (begin))

[–] dhork 3 points 6 months ago (2 children)

In order for an AI to pass the Turing test, it must be able to talk to someone and fool them into thinking that they are talking to a human.

So, passing the Turing Test either means the AI are getting smarter, or that humans are getting dumber.

[–] [email protected] 6 points 6 months ago

Detecting an LLM is a skill.

[–] Kolrami 4 points 6 months ago* (last edited 6 months ago)

Humans are as smart as they ever were. Tech is getting better. I know someone who was tricked by those deepfake Kelly Clarkson weight loss gummy ads. It looks super fake to me, but it's good enough to trick some people.

load more comments
view more: next ›