this post was submitted on 18 Nov 2024
19 points (100.0% liked)

Hardware

691 readers
124 users here now

All things related to technology hardware, with a focus on computing hardware.

Rules (Click to Expand):

  1. Follow the Lemmy.world Rules - https://mastodon.world/about

  2. Be kind. No bullying, harassment, racism, sexism etc. against other users.

  3. No Spam, illegal content, or NSFW content.

  4. Please stay on topic, adjacent topics (e.g. software) are fine if they are strongly relevant to technology hardware. Another example would be business news for hardware-focused companies.

  5. Please try and post original sources when possible (as opposed to summaries).

  6. If posting an archived version of the article, please include a URL link to the original article in the body of the post.

Some other hardware communities across Lemmy:

Icon by "icon lauk" under CC BY 3.0

founded 1 year ago
MODERATORS
Alphane_Moon
CSFFlame
19
Lenovo's enterprise hardware biz still struggles for profits (www.theregister.com)
submitted 3 days ago by Alphane_Moon to c/hardware
11 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] Alphane_Moon 9 points 3 days ago (10 children)

Lenovo claimed it won 24 percent of the PC market and saw AI PCs account for over ten percent of notebook sales. Group president Luca Rossi said PC sales should improve as buyers have two good reasons to upgrade: end of life for devices bought to run Windows 10 which are now ripe for replacement, and a desire to adopt AI PCs and offer users new experiences.

Windows 10 EOL is a fair argument. I have yet to see any research showing consumers looking to buy "AI PCs" specifically. I suspect most people just get a new laptop and it happens to be an "AI PC".

[–] Brkdncr 5 points 3 days ago (1 children)

Enterprise is different. Lots of business decision makers are prepping their workforce for AI, and do t want to put their data on someone’s cloud. Local AI will be a big deal.

[–] Alphane_Moon 3 points 3 days ago (2 children)

Are the current crop of NPUs really suitable for this?

I play around with video upscaling and local LLMs. I have a 3080 which is supposed to be 238 TOPS. It takes about 25 min to upscale a ~5 min SD video to HD (sometime longer depending on the source content). The "AI PC" NPUs are rated at around ~50 TOPs, so that would be a massive increase in upscale time (closer to 2 hours for ~5 min SD source).

I also have a local LLM that I've been comparing against ChatGPT. For my limited use case (elaborate spelling/typo/style checking), the local LLM (llama) works comparable to ChatGPT, but I run it on a 3080. Is this true for local LLMs that run on NPUs? I would speculate that more complex use cases (programming support?), you would need even more throughput from your NPU.

I have much more experience with upscaling though and my experiments/usage of local LLMs is somewhat limited compared to ChatGPT usage.

[–] brucethemoose 1 points 2 days ago (1 children)

NPUs are basically useless for LLMs because no software supports them. They also can't allocate much memory, and they don't support the exotic quantization schemes modern runtimes use very well.

And speed wise, they are rather limited by their slow memory busses they're attached to.

Even on Apple, where there is a little support for running LLMs on NPUs, everyone just does the compute on the GPU anyway because its so much faster and more flexible.

This MIGHT change if bitnet llms take off, or if Inte/AMD start regularly shipping quad channel designs.

[–] Alphane_Moon 1 points 2 days ago (1 children)

Yes, I was reading through the documentation of some of the tools I use and I noticed minimal info about NPU support.

Will take a look at bitnet (if my tools support it), curious how it would compare to Llama which seems decent for my use cases.

[–] brucethemoose 2 points 2 days ago* (last edited 1 day ago) (1 children)

Bitnet is theoretical now and unsupported by NPUs anyway.

Basically they are useless for large models :P

The IGPs on the newest AMD/Intel IGPs are OK for hosting models up to like 14B though. Maybe 32B if with the right BIOS, if you don't mind very slow output.

If I were you, on a 3080, if you keep desktop vram usage VERY minimal, I would run TabbyAPI and a 4bpw exl2 quantization of Qwen 2.5 14B coder, instruct, and RP finetune... pick your flavor. I'd recommend this one in particular.

https://huggingface.co/bartowski/SuperNova-Medius-exl2/tree/4_25

Run it with Q6 cache and set the context to like 16K, or whatever you can fit in your vram.

I guarantee this will blow away whatever llama (8b) setup you have.

[–] Alphane_Moon 1 points 1 day ago (1 children)

Cheers, will give it a go. I want to move away from cloud LLMs.

[–] brucethemoose 2 points 1 day ago* (last edited 1 day ago) (1 children)

Pick up a 3090 if you can!

Then you can combine it with your 3080 and squeeze Qwen 72B in, and straight up beat GPT-4 in some use cases.

Also, TabbyAPI can be tricky to set up, ping me if you need help.

[–] Alphane_Moon 1 points 1 day ago (1 children)

Not planning on getting a new/additional GPU at this point. My local LLM project is more of curiosity, I am more knowledgeable on the AI upscaling side. :)

Thanks for the offer, will consider it!

[–] brucethemoose 2 points 1 day ago* (last edited 1 day ago)

Last time bothering you! I used to be really into the GAN space myself, but the newer diffusion models really blow them away. Check this out: https://github.com/mit-han-lab/nunchaku

This can squeeze Flux 1D onto your 3080, and (with the right pipeline/settings) it should blow anything else away at "enhancing" a low res image with img2img. It should also work with batching and torch.compile so you can get quite a lot of throughput from your 3080. Of course, there's no temporal consistency yet (or it may be, it's hard to keep up with all the adapter releases), but I'm sure its coming... And you can kinda hack some in with 2D models anyway.

[–] Brkdncr 5 points 3 days ago

No one said it was a smart decision. It’s just one that’s being made.

load more comments (8 replies)