Update README.md
Browse files
README.md
CHANGED
|
@@ -118,7 +118,7 @@ The later grounded-chat checkpoint has a separate assistant-token evaluation: he
|
|
| 118 |
|
| 119 |
Sabaki scores **20.33** on the Open SLM Leaderboard's Intelligence Index (INT), using the full benchmark evaluation of the 200M token SFT stage. The index adjusts each benchmark for chance performance, averages ARC-Easy and ARC-Challenge into a single ARC component, and weights HellaSwag, ARC, and PIQA at 1.0 each and ArithMark-3 at 0.65.
|
| 120 |
|
| 121 |
-
From our own testing of models on the leaderboard we have found that many of the models do very poorly on real world questions and tasks when post-trained. This observation has been seen on different model designs, ruling out the possibility of design choices producing this outcome. The most likely conclusion is that the other models were not built for real use and were trained on data that would help the benchmark rather than produce a overall rounded pre/post trained model. We are not accusing other groups of benchmark inflation or contaminated training data, just pointing out the diffrences in our
|
| 122 |
|
| 123 |
The following comparison uses the [Open SLM Leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard) scores and formula checked on 12 October 2026. Higher is better.
|
| 124 |
|
|
@@ -144,6 +144,10 @@ By score alone, Sabaki would sit 21st among the 210 existing entries with all fi
|
|
| 144 |
|
| 145 |
This preview can produce incorrect facts, invented citations, repetitive text, and unreliable instructions. Languages other than English have not been evaluated separately. Protocol tokens for tools and other structured outputs do not establish reliable tool-use capabilities. The model has not received comprehensive safety evaluation or preference optimization.
|
| 146 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 147 |
## License
|
| 148 |
|
| 149 |
The model weights are available under the **Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)** license. Attribution is required, and commercial use is not permitted. Software retains its own license.
|
|
|
|
| 118 |
|
| 119 |
Sabaki scores **20.33** on the Open SLM Leaderboard's Intelligence Index (INT), using the full benchmark evaluation of the 200M token SFT stage. The index adjusts each benchmark for chance performance, averages ARC-Easy and ARC-Challenge into a single ARC component, and weights HellaSwag, ARC, and PIQA at 1.0 each and ArithMark-3 at 0.65.
|
| 120 |
|
| 121 |
+
From our own testing of models on the leaderboard we have found that many of the models do very poorly on real world questions and tasks when post-trained. This observation has been seen on different model designs, ruling out the possibility of design choices producing this outcome. The most likely conclusion is that the other models were not built for real use and were trained on data that would help the benchmark rather than produce a overall rounded pre/post trained model. We are not accusing other groups of benchmark inflation or contaminated training data, just pointing out the diffrences in our approaches.
|
| 122 |
|
| 123 |
The following comparison uses the [Open SLM Leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard) scores and formula checked on 12 October 2026. Higher is better.
|
| 124 |
|
|
|
|
| 144 |
|
| 145 |
This preview can produce incorrect facts, invented citations, repetitive text, and unreliable instructions. Languages other than English have not been evaluated separately. Protocol tokens for tools and other structured outputs do not establish reliable tool-use capabilities. The model has not received comprehensive safety evaluation or preference optimization.
|
| 146 |
|
| 147 |
+
## Future
|
| 148 |
+
|
| 149 |
+
Our approch to this group of models is to create a strong pretrained/postrained model that can be easily adapted for uses that require long context and precise recalling of information, this requires more of the model to be dedicated to attention. A variant of this model is currently being trained with less of the parameters being alocated to attention and vocab, however this model will see less tokens during the training so comparison may not be representative.
|
| 150 |
+
|
| 151 |
## License
|
| 152 |
|
| 153 |
The model weights are available under the **Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)** license. Attribution is required, and commercial use is not permitted. Software retains its own license.
|