Your card gives away enough to price the scale without the data, so I ran your Elo on a model that does not exist.
A uniform random guesser has p=0.25 on every item by construction, four choices. Feed that into your own formula. point_weight = category_weight * difficulty_weight, expected = 1/(1+10**((item_elo-R)/400)), weighted MLE with the 4-game prior at 1000, item Elo straight off your composition table.
Random guessing scores Overall Elo 805.
That number is stable. I do not know your per-category difficulty split, so I tried two very different allocations that both hit 50 per category and 117/117/116 overall. 805.0 and 804.5. Half a point, so the split does not matter.
805 is worth a line on the leaderboard.
Five of the 46 entries score below it. MicroSupra-1k 768, Atom2.7m 794, DistillSupra-0.2M 794, Supra-Mini-0.1M 796, Supra-Mini-v2-0.1M 804.
Nine of 46, 19.6%, have accuracy inside the 95% interval for chance. On 350 four-way items that interval is 25 +/- 4.54pp, so [20.46, 29.54]. Those nine span Elo 768 to 830, and 62 points reads as ranking.
There is a second sign they are guessing, and it comes from your own numbers. I refit every entry assuming correctness is uniform across items, which is the wrong model for a real LM and the right one for a guesser. The nine chance-band entries land a mean 6.0 Elo from published, signs mixed, 7 above and 2 below. The other 37 land a mean 25.3 Elo high, 37 of 37 in the same direction. Real models have a difficulty gradient. The bottom nine show none.
On resolution. At 48% accuracy one standard error is 2.67pp, about 21 Elo. Your median gap between adjacent ranks is 5 Elo, and 23 of 45 gaps are 5 or less. Most neighbours sit about four times inside one standard error. Your card already says small Elo differences should not be treated as decisive, and the leaderboard is where that sentence is missing.
The weighting also reorders, which I read as intended rather than broken. 22 ordered pairs have the higher-accuracy model ranked lower. Sharpest is Supra-1.6-50M-Instruct-Ultra-exp at 44.00% sitting at 928 while Veyra-30M-Base at 43.71% sits at 951. Two entries at exactly 44.00% get 928 and 949. That is a 21 to 23 Elo effect, the design working, and four times your median rank gap.
Credit where it is due, and it is the reason any of this was checkable. You publish the formula, the per-item Elo table, both weight vectors, the answer-position balance and the test.jsonl SHA-256. That is more than most cards this size publish.
Caveat on my side. The data is gated and I am not on the list, so test.jsonl 403s for me. I could not verify that items carry the item_elo your table implies, and I am not quoting my refit as anyone's score. Every score above is straight off your leaderboard.
Smallest useful change: draw the 805 line on the chart, and grey the entries whose accuracy interval crosses it.
Would you put the null on the axis?
