Technology

59709 readers

5673 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each another!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 1 year ago

MODERATORS

360

OpenAI, Google, Anthropic admit they can’t scale up their chatbots any further (pivot-to-ai.com)

submitted 1 week ago* (last edited 1 week ago) by [email protected] to c/technology

87 comments fedilink hide all child comments

I'm usually the one saying "AI is already as good as it's gonna get, for a long while."

This article, in contrast, is quotes from folks making the next AI generation - saying the same.

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 6 points 5 days ago* (last edited 5 days ago) (3 children)

Larger models train faster (need less compute), for reasons not fully understood. These large models can then be used as teachers to train smaller models more efficiently. I've used Qwen 14B (14 billion parameters, quantized to 6-bit integers), and it's not too much worse than these very large models.

Lately, I've been thinking of LLMs as lossy text/idea compression with content-addressable memory. And 10.5GB is pretty good compression for all the "knowledge" they seem to retain.

[–] brucethemoose 1 points 4 days ago (2 children)

I don't think Qwen was trained with distillation, was it?

It would be awesome if it was.

Also you should try Supernova Medius, which is Qwen 14B with some "distillation" from some other models.

[–] [email protected] 1 points 4 days ago (1 children)

Hmm. I just assumed 14B was distilled from 72B, because that's what I thought llama was doing, and that would just make sense. On further research it's not clear if llama did the traditional teacher method or just trained the smaller models on synthetic data generated from a large model. I suppose training smaller models on a larger amount of data generated by larger models is similar though. It does seem like Qwen was also trained on synthetic data, because it sometimes thinks it's Claude, lol.

Thanks for the tip on Medius. Just tried it out, and it does seem better than Qwen 14B.

[–] brucethemoose 1 points 4 days ago* (last edited 4 days ago)

Llama 3.1 is not even a "true" distillation either, but its kinda complicated, like you said.

Yeah Qwen undoubtedly has synthetic data lol. It's even in the base model, which isn't really their "fault" as its presumably part of the web scrape.