this post was submitted on 26 Jul 2023
868 points (96.5% liked)
Technology
59476 readers
3525 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Setting aside the obvious answer of "because capitalism", there are a lot of obstacles towards democratizing this technology. Training of these models is done on clusters of A100 GPU's, which are priced at $10,000USD each. Then there's also the fact that a lot of the progress being made is being done by highly specialized academics, often with the resources of large corporations like Microsoft.
Additionally the curation of datasets is another massive obstacle. We've mostly reached the point of diminishing returns of just throwing all the data at the training of models, it's quickly becoming apparent that the quality of data is far more important than the quantity of the data (see TinyStories as an example). This means a lot of work and research needs to go into qualitative analysis when preparing a dataset. You need a large corpus of input, each of which are above a quality threshold, but then also as a whole they need to represent a wide enough variety of circumstances for you to reach emergence in the domain(s) you're trying to train for.
There is a large and growing body of open source model development, but even that only exists because of Meta "leaking" the original Llama models, and now more recently releasing Llama 2 with a commercial license. Practically overnight an entire ecosystem was born creating higher quality fine-tunes and specialized datasets, but all of that was only possible because Meta invested the resources and made it available to the public.
Actually in hindsight it looks like the answer is still "because capitalism" despite everything I've just said.
I know the answer to pretty much all of our “why the hell don’t we solve this already?” questions is: capitalism.
But I mean, as Lrrr would say “why does the working class, as the biggest of the classes, doesn’t just eat the other one?”.
The short answer is friction. The friction of overcoming the forces of violence the larger class has at its disposal and utilizes at the smallest hint of uprising is greater than the friction of accepting the status quo.
The friction of accepting the status quo only seems to grow stronger though.
One would hope
Most people don’t even think that’s an option though.
The end of history, with the fall of USSR and capitalism winning the propaganda wars, means most people don’t even see a different future.
Why would you fight a future that looks the same?
People need to wake up and have hope for a different, better future. That’s the only way they’ll more against this.
But for that 100+ years of propaganda have to be overcome…