this post was submitted on 18 Jun 2023
359 points (99.4% liked)
Programmer Humor
32571 readers
316 users here now
Post funny things about programming here! (Or just rant about your favourite programming language.)
Rules:
- Posts must be relevant to programming, programmers, or computer science.
- No NSFW content.
- Jokes must be in good taste. No hate speech, bigotry, etc.
founded 5 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Sure it can create new words. It can't create new tokens, would be more correct, I think. But a token is just a text fragment, and, as far as I know, they can range from being several words to being single characters.
I got it it say vilumplox. It doesn't return any Google search results.
If it was Windows 95 it could generate them
https://www.youtube.com/watch?v=cwyH59nACzQ&t=306s
Tokens are usually never multiple words. Think of them like "information units". If you have a plural word "hats", there are two tokens: the "hat" one and the "s' that adds more information about it being plural. Combinations of words only really occur for proper names.