Technology

Seriously. People like to project forward based on how quickly this technological breakthrough came on the scene, but they don't realize that, barring a few tweaks and improvements here and there, this is it for LLMs. It's the limit of the technology.

It's not to say AI can't improve further, and I'm sure that when it does, it will skillfully integrate LLMs. And I also think artists are right to worry about the impact of AI on their fields. But I think it's a total misunderstanding of the technology to think the current technology will soon become flawless. I'm willing to bet we're currently seeing it at 95% of its ultimate capacity, and that we don't need to worry about AI writing a Hollywood blockbuster any time soon.

In other words, the next step of evolution in the field of AI will require a revolution, not further improvements to existing systems.

It is just that everyone now refers to LLMs when talking about AI even though it has sonmany different aspects to it. Maybe at some point there is an AI that actually understands the concepts and meanings of things. But that is not learned by unsupervised web crawling.

It is possible to get coherent output from them though. I’ve been using the ChatGPT API to successfully write ~20 page proposals. Basically give it a prior proposal, the new scope of work, and a paragraph with other info it should incorporate. It then goes through a section at a time.

The numbers and graphics need to be put in after… but the result is better than I’d get from my interns.

I’ve also been using it (google Bard mostly actually) to successfully solve coding problems.

I either need to increase the credit I giver LLM or admit that interns are mostly just LLMs.

So is your brain.

Relative complexity matters a lot, even if the underlying mechanisms are similar.

Idk, an ai delivering low quality results for free is a lot more cash money than paying someone an almost living wage to perform a job with better results. I think corporations won't care and the only barrier will be whether or not the job in question involves enough physical labor to be performed by an ai or not.

The problem is that these things never hit a point of competition with humans, they're either worse than us, or they blow way past us. Humans might drive better than a computer right now, but as soon as the computer is better than us it will always be better than us. People doubted that computers would ever beat the best humans at chess, or go, but within a lifetime of computers being invented they blew past us in both. Now they can write articles and paint pictures, sure we're better at it for now, but they're a million times faster than us, and they're making massive improvements month over month. you and I can disagree on how long it'll take for them to pass us, but once they do they'll replace us completely, and the world will never be the same.

That's called context. For chatgpt it is a bit less than 4k words. Using api it goes up to a bit less of 32k. Alternative models goes up to a bit less than 64k.

Model wouldn't know anything you said before that

That is one of the biggest limitations of current generation of LLMs.

Yeah but humans can use critical thinking, even on themselves when they make shit up. I've definitely said something and then thought to myself "wait that doesn't make sense for x reason, that can't be right" and then I research and correct myself.

AI is incapable of this.

It seems like Altman is a PR man first and techie second. I wouldn't take anything he actually says at face value. If it's 'unfixable' then he probably means that in a very narrow way. Ie. I'm sure they are working on what you proposed, it's just different enough that he can claim that the way it is now is 'unfixable'.

Standard Diffusion really how people get the different-model-different-application idea.

I disagree, with models such as llama it has become clear that there are interesting advantages on increasing (even more) the ratio of parameters/data. I don't think next iterations of models from big-corp will 10x the param count until nvidia has really pushed hardware, models are getting better over time. ChatGPT's deterioration is mostly coming from openAI's ensuring safety and is not a fair assessment of progress on LLMs in general, the leaderboard of open source models has been steadily improving over time: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

But bigger models have new "emergent" capabilities. I heard that from a certain size they start to know what they know and hallucinate less.

Hallucinations is common for humans as well. It's just people who believe they know stuff they really don't know.

We have alternative safeguards in place. It's true however that current llm generation has its limitations

Well to be honest it is the best way, I mean, I'm pretty sure their purpose was a tool to aid people, and not to replace us... Right?

You can potentially solve this problem outside of the network, even if you can't solve it within the network. I consider accuracy to be outside the scope of LLMs, and that's fine since accuracy is not part of language in the first place. (You may have noticed that humans lie with language rather often, too.)

Most of what we've seen so far are bare-bones implementations of LLMs. ChatGPT doesn't integrate with any kind of knowledge database at all (only what it has internalized from its training set, which is almost accidental). Bing will feed in a couple web search results, but a few minutes of playing with it is enough to prove how minimal that integration is. Bard is no better.

The real potential of LLMs is not as a complete product; it is as a foundational part of more advanced programs, akin to regular expressions or SQL queries. Many LLM projects explicitly state that they are "foundational".

All the effort is spent training the network because that's what's new and sexy. Very little effort has been spent on the ho-hum task of building useful tools with those networks. The out-of-network parts of Bing and Bard could've been slapped together by anyone with a little shell scripting experience. They are primitive. The only impressive part is the LLM.

The words feel strange coming off my keyboard, but....Microsoft has the right idea with the AI integrations they're rolling into Office.

The potential for LLMs is so much greater than what is currently available for use, even if they can't solve any of the existing problems in the networks themselves. You could build an automated fact-checker using LLMs, but the LLM itself is not a fact-checker. It's coming, no doubt about it.

I've already seen many commercials using what is clearly AI generated art and voices (so not specifically ChatGPT). That is a job lost for a designer and an actor somewhere.

You mean the free version from a website.

Think about the powerful ones. Government ones. Wall Street ones. Etc.

Not ChatGPT, but other new AI stuff is likely to take a few jobs. Actors and voice-actors among other.

I have experience in creating supervised learning networks. (not large language models) I don't know what tokens are, I assume they are output nodes. In that case I think increasing the output nodes don't make the Ai a lot more intelligent. You could measure confidence with the output nodes if they are designed accordingly (1 node corresponds to 1 word, confidence can be measured with the output strength). Ai-s are popular because they can overcome unknown circumstances (most of the cases), like when you input a question slightly different way.

I agree with you on that Ai has a problem understanding the meaning of the words. The Ai's correct answers happened to be correct because the order of the words (output) happened to match with the order of the correct answer's words. I think "hallucinations" happen when there is no sufficient answers to the given problem, the Ai gives an answer from a few random contexts pieced together in the most likely order. I think you have mostly good understanding on how Ai-s work.

And then they can use a third model to detect falsehoods in the second model and a fourth model to detect falsehoods in the third model and... well, it's LLMs all the way down.

Ai models are already computationally intensive. This would instantly double the overhead. Also being able to detect problems does not mean you're able to fix them.