Post
16
SO-Arena scored 21 System One models, and the most accurate one ranks only seventh on reliability.
System One models pick from fixed options in one forward pass, so teams use them for routing, tool choice and verification. A leaderboard score does not say whether the pick survives shuffled options or a missing right answer. SO-Arena (2,919 tasks, 11 domains) measures that separately.
š What's new
ā¢ Reliability R = sqrt(S*N). S is stability over 12 option reorderings. N = 2AUC-1 on spotting that the correct option was removed
ā¢ JevAny-27B is the most reliable System One model yet ranks eleventh in capability
ā¢ 14 of 18 models reach S above 0.8, but N runs from near chance to about 0.8
š The numbers
ā¢ GPT-6 Sol (82.3) and Luna (79.7) lead. The best System One models, GLiDE and KnowLine-31B, score 70.2
ā¢ Open-weight System One models answer within 0.3 s per task on one H100
ā¢ Knowledge, sentiment, retrieval: comparable to Luna, with little gain from a bigger backbone or thinking
ā¢ Math: thinking lifts StartLux Auto from 50.8 to 92.5. In intent, safety, tools and web, the larger backbone helps far more
ā ļø Caveats
ā¢ The backbone gain mixes size with Qwen3.5 vs Qwen3.8. Thinking is tested on one model
ā¢ Two closed System Two references only, and no independent replication yet
Before adopting a model, check per-domain scores, then reorder options and drop the correct one on your own task. The MIT repo ships the scoring and reliability tooling.
SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models (2611.15777)
https://github.com/holdrain/SO-Arena
Do you test decision models for order sensitivity before putting them in an agent?
š Read it elsewhere
ā¢ The deep dive on Medium: https://medium.com/p/e923d2f58e85
ā¢ The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/so-arena-finds-the-most-accurate
ā¢ The thread on X: https://x.com/saavoji/status/2121665100981444943
ā¢ Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#so-arena-finds
System One models pick from fixed options in one forward pass, so teams use them for routing, tool choice and verification. A leaderboard score does not say whether the pick survives shuffled options or a missing right answer. SO-Arena (2,919 tasks, 11 domains) measures that separately.
š What's new
ā¢ Reliability R = sqrt(S*N). S is stability over 12 option reorderings. N = 2AUC-1 on spotting that the correct option was removed
ā¢ JevAny-27B is the most reliable System One model yet ranks eleventh in capability
ā¢ 14 of 18 models reach S above 0.8, but N runs from near chance to about 0.8
š The numbers
ā¢ GPT-6 Sol (82.3) and Luna (79.7) lead. The best System One models, GLiDE and KnowLine-31B, score 70.2
ā¢ Open-weight System One models answer within 0.3 s per task on one H100
ā¢ Knowledge, sentiment, retrieval: comparable to Luna, with little gain from a bigger backbone or thinking
ā¢ Math: thinking lifts StartLux Auto from 50.8 to 92.5. In intent, safety, tools and web, the larger backbone helps far more
ā ļø Caveats
ā¢ The backbone gain mixes size with Qwen3.5 vs Qwen3.8. Thinking is tested on one model
ā¢ Two closed System Two references only, and no independent replication yet
Before adopting a model, check per-domain scores, then reorder options and drop the correct one on your own task. The MIT repo ships the scoring and reliability tooling.
SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models (2611.15777)
https://github.com/holdrain/SO-Arena
Do you test decision models for order sensitivity before putting them in an agent?
š Read it elsewhere
ā¢ The deep dive on Medium: https://medium.com/p/e923d2f58e85
ā¢ The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/so-arena-finds-the-most-accurate
ā¢ The thread on X: https://x.com/saavoji/status/2121665100981444943
ā¢ Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#so-arena-finds
