8
this post was submitted on 30 Sep 2023
8 points (90.0% liked)
Free Open-Source Artificial Intelligence
2936 readers
3 users here now
Welcome to Free Open-Source Artificial Intelligence!
We are a community dedicated to forwarding the availability and access to:
Free Open Source Artificial Intelligence (F.O.S.A.I.)
More AI Communities
LLM Leaderboards
Developer Resources
GitHub Projects
FOSAI Time Capsule
- The Internet is Healing
- General Resources
- FOSAI Welcome Message
- FOSAI Crash Course
- FOSAI Nexus Resource Hub
- FOSAI LLM Guide
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I have come to believe Moore's law is finite, and we're starting to see the exponential end of it. This leads me to believe (or want to believe) there are other looming breakthroughs for compute, optimization, and/or hardware on the horizon. That, or crazy powerful GPUs are about to be a common household investment.
I keep thinking about what George Hotz is doing in regards to this. He explained on his podcast with Lex Fridman that there is much to be explored in optimization, both with quantization of software and acceleration of hardware.
His idea of 'commoditize the petabyte' is really cool. I think it's worth bringing up here, especially given the fact it appears one of his biggest goals right now is solving the at-home compute problem. But in a way that you could actually run something like a 180B model in-house no problem.
George Hotz' tinybox
($15,000)
738 FP16 TFLOPS
144 GB GPU RAM
5.76 TB/s RAM bandwidth
30 GB/s model load bandwidth (big llama loads in around 4 seconds)
AMD EPYC CPU
1600W (one 120V outlet)
Runs 65B FP16 LLaMA out of the box (using tinygrad, subject to software development risks)
You can pre-order one now. You have $15k laying around, right? Lol.
It's definitely not easy (or cheap) now, but I think it's going to get significantly easier to build and deploy large models for all kinds of personal use cases in our near and distant futures.
If you're serving/hosting models, it's also worth checking out vLLM if you haven't already: https://github.com/vllm-project/vllm
Hardware moves notoriously slow, so I imagine we still have several years before a good solution manifests in the market.
Somebody needs to build a good Asimov character roleplay and coax the secret for the positronic brain out of him. I'd like to buy the new AMD R-Daneel Olivaw 5000 please. Hell, I'll settle for a RB-34 Herbie model right now.