Natural Language Programming | Prompting (chatGPT)

57 readers

1 users here now

Welcome ton !nlprog where anything Natural Language Programming related is game, prompts, ideas, projects and more for any model are welcome.

We follow Lemmy’s code of conduct.

Communities

Useful links

founded 1 year ago

MODERATORS

[email protected]

Is there some resource to benchmark LLM performance on a hardware (self.nlprog)

submitted 1 year ago by circle to c/[email protected]

4 comments fedilink hide all child comments

As the title suggests, basically i have a few LLM models and wanted to see how they perform with different hardware (Cpus only instances, gpus - t4, v100, a100). Ideally it's to get an idea on the performance and overall price(vm hourly rate/ efficiency)

Currently I've written a script to calculate ms per token, ram usage(memory profiler), total time taken.

Wanted to check if there are better methods or tools. Thanks!

top 4 comments

sorted by: hot top controversial new old

[–] [email protected] 1 points 1 year ago (1 children)

I've seen a couple around, grabbed this one off my list:

lm-evaluation-harness

[–] circle 2 points 1 year ago (1 children)

Thanks. Does this also conduct compute benchmarks too? Looks like this is more focused on model accuracy (if I'm not wrong)

[–] [email protected] 1 points 1 year ago (1 children)

seems like, keep an eye, when i run across one I will post it, usually to the model's community.

[–] circle 2 points 1 year ago

sure, thank you!