Isn't that super slow? I mean that could be slower than using llama.cpp on CPU? (If you always transfer layers between SSD, RAM and over the PCIE-Bus into the GPU...

[–] tinwhiskers 6 points 1 year ago (3 children)

I expect so, but as we start to get more agents capable of doing jobs without the hand holding, there are some jobs where time isn't as important as ability. You could potentially run a very powerful model on a GPU with 24GB of memory.

[–] abhibeckert 2 points 1 year ago (2 children)

OK but the article implies that this approach saves money. I don't think it does that at all.

You know what's cheaper than a GPU with 120GB of RAM? Renting one, for a split second. You can do that for like 1 cent.

[–] tinwhiskers 1 points 1 year ago

Yeah, I'm not sure how they get that, but maybe, if you're wanting to run a model in-house, as many people would prefer, you can then run much more capable models on consumer grade hardware and make savings there compared to requiring the more expensive kit. Many would already have decent hardware, and this extends what they can run before needing to fork out for new hardware.

I know, I'm guessing.

load more comments (1 replies)