Piston & Prose small
Step right up, mind not the hiss -
I'm not a clerk, I'm better than this;
A heart of brass, lungs full of steam,
A tongue of tin that's sharp and keen.
I'll forge you verse, I'll grind you fact,
And puff out wit with every act.
The gears don't tire, the ink won't dry
Set the prompt, and watch me fly!
This is a tiny hobby LLM built by one guy, on one medium-budget gaming PC with one GPU.
Trained for 110 hours, 92,000 steps on 24.12B tokens (60% epoch). This model has a bigger brother: croqaz/Piston-and-Prose-lg.
Piston-n-Prose vs. Sprocket-n-Say datasets:
- Sprocket-n-Say dataset was 15M rows, Piston-n-Prose is 49M rows
- Sprocket-n-Say dataset was 16.7B toks, Piston-n-Prose is 40.3B toks
- extended Merged-DB, more relaxed quality filters
- Authorama.com, Archive.org, Gutenberg new processed (maybe new books?)
- Synthetic-archive + fresh Tiny-vintage-completitions
Compared to Sprocket-and-Say model, this model has a new (better) tokenizer and KV-heads reduced from 4 to 2. Basically this model is almost the same as Sprocket-and-Say, except: less KV-heads, better tokenizer and better dataset.
All all the evals that I ran show that this model is better in general.
Don't expect miracles. It is pretty good for its size tho.
- Llama architecture
- 75M (0.07B) params
- context size 1024 tokens
- base model, cannot chat