Machine Learning

1789 readers

11 users here now

founded 4 years ago

MODERATORS

[email protected]

Can gpt generate a gpt model? (sh.itjust.works)

submitted 6 months ago by [email protected] to c/[email protected]

8 comments fedilink hide all child comments

Imagine AI giving offsprings...

top 8 comments

sorted by: hot top controversial new old

[–] [email protected] 7 points 6 months ago (1 children)

Not directly no.

It may be able to code one (the code is relatively short and well known) and give training program, and then you would need to spend a few trillion tokens to make it generate data.

[–] Anticorp 1 points 6 months ago (1 children)

Where can we see this well known code? I'd like to see how it works.

[–] [email protected] 3 points 6 months ago (1 children)

Here is an implementation in pytorch:

https://github.com/lyeoni/gpt-pytorch/blob/master/model.py

Here is one in pure C that karpathy started:

https://github.com/karpathy/llm.c

[–] Anticorp 1 points 6 months ago

Thanks!

[–] Audalin 5 points 6 months ago (1 children)

You can generate synthetic data matching the distribution your transformer learned. You can use this dataset to train another model. As of now, that's about it.

[–] [email protected] 2 points 6 months ago

Yep, this is called model distillation.

[–] over_clox 3 points 6 months ago (1 children)

Don't give them any ideas.. 😂

[–] [email protected] 0 points 6 months ago

ok... LOL