Local-Axiom-AI commited on
Commit
91197f1
·
verified ·
1 Parent(s): b01492b

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +5 -1
README.md CHANGED
@@ -118,7 +118,7 @@ The later grounded-chat checkpoint has a separate assistant-token evaluation: he
118
 
119
  Sabaki scores **20.33** on the Open SLM Leaderboard's Intelligence Index (INT), using the full benchmark evaluation of the 200M token SFT stage. The index adjusts each benchmark for chance performance, averages ARC-Easy and ARC-Challenge into a single ARC component, and weights HellaSwag, ARC, and PIQA at 1.0 each and ArithMark-3 at 0.65.
120
 
121
- From our own testing of models on the leaderboard we have found that many of the models do very poorly on real world questions and tasks when post-trained. This observation has been seen on different model designs, ruling out the possibility of design choices producing this outcome. The most likely conclusion is that the other models were not built for real use and were trained on data that would help the benchmark rather than produce a overall rounded pre/post trained model. We are not accusing other groups of benchmark inflation or contaminated training data, just pointing out the diffrences in our aproches.
122
 
123
  The following comparison uses the [Open SLM Leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard) scores and formula checked on 12 October 2026. Higher is better.
124
 
@@ -144,6 +144,10 @@ By score alone, Sabaki would sit 21st among the 210 existing entries with all fi
144
 
145
  This preview can produce incorrect facts, invented citations, repetitive text, and unreliable instructions. Languages other than English have not been evaluated separately. Protocol tokens for tools and other structured outputs do not establish reliable tool-use capabilities. The model has not received comprehensive safety evaluation or preference optimization.
146
 
 
 
 
 
147
  ## License
148
 
149
  The model weights are available under the **Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)** license. Attribution is required, and commercial use is not permitted. Software retains its own license.
 
118
 
119
  Sabaki scores **20.33** on the Open SLM Leaderboard's Intelligence Index (INT), using the full benchmark evaluation of the 200M token SFT stage. The index adjusts each benchmark for chance performance, averages ARC-Easy and ARC-Challenge into a single ARC component, and weights HellaSwag, ARC, and PIQA at 1.0 each and ArithMark-3 at 0.65.
120
 
121
+ From our own testing of models on the leaderboard we have found that many of the models do very poorly on real world questions and tasks when post-trained. This observation has been seen on different model designs, ruling out the possibility of design choices producing this outcome. The most likely conclusion is that the other models were not built for real use and were trained on data that would help the benchmark rather than produce a overall rounded pre/post trained model. We are not accusing other groups of benchmark inflation or contaminated training data, just pointing out the diffrences in our approaches.
122
 
123
  The following comparison uses the [Open SLM Leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard) scores and formula checked on 12 October 2026. Higher is better.
124
 
 
144
 
145
  This preview can produce incorrect facts, invented citations, repetitive text, and unreliable instructions. Languages other than English have not been evaluated separately. Protocol tokens for tools and other structured outputs do not establish reliable tool-use capabilities. The model has not received comprehensive safety evaluation or preference optimization.
146
 
147
+ ## Future
148
+
149
+ Our approch to this group of models is to create a strong pretrained/postrained model that can be easily adapted for uses that require long context and precise recalling of information, this requires more of the model to be dedicated to attention. A variant of this model is currently being trained with less of the parameters being alocated to attention and vocab, however this model will see less tokens during the training so comparison may not be representative.
150
+
151
  ## License
152
 
153
  The model weights are available under the **Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)** license. Attribution is required, and commercial use is not permitted. Software retains its own license.