this post was submitted on 17 Nov 2023
534 points (99.1% liked)
Technology
59665 readers
3602 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Be careful not to be caught up in the application of Goodhart's Law going on in the field right now.
There's plenty of things GPT-4 trounces everything else on, they just tend to be things outside the now standardized body of tests, which suggests the tests have become the target and are no longer effective measurements.
This is perhaps most apparent in things like Orca, where we directly use the tests as the target, have GPT-4 generate synthetic data that improves Llama performance on the target, and then see large gains in smaller models on the tests.
But those new models don't necessarily have the same capabilities on more abstract capabilities, such as the recent approach of using analogy to solve problems.
We are arguably becoming too myopic in how we are measuring the success of new models.