Archived leaderboard
Corpus 299-21b69d52efdc, revision 2026-10-28, published 2026-10-28, grader 2026-10-10. This snapshot is immutable — cite this URL.
Revisions of this corpus: 2026-10-31 (27 runs) · 2026-10-28 (27 runs)
| Rank | Model | Track | k | Score/100q | Correct | Incorrect | Abstained | No answer | Truncated | Reasoning | Grounded | Hallucination | Cost |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1–4 | claude-opus-5.5 | closed | 1 | +27.1 (±10.0) | 136/295 | 28 | 131 | 0 | 0 | 98% [88–100] | 37% [32–43] | 22% [16–31] | $4.75 |
| 1–4 | grok-4.7 | closed | 1 | +25.1 (±6.4) | 86/295 | 6 | 203 | 0 | 0 | 91% [78–96] | 19% [14–24] | 6% [2–16] | $3.11 |
| 1–4 | claude-fable-5.1 | closed | 1 | +24.1 (±10.9) | 143/295 | 36 | 116 | 0 | 0 | 100% [92–100] | 40% [34–46] | 26% [20–34] | $14.29 |
| 1–4 | claude-sonnet-5.5 | closed | 1 | +10.5 (±12.0) | 131/295 | 50 | 114 | 0 | 0 | 100% [92–100] | 35% [29–41] | 36% [29–45] | $2.35 |
| 5 | claude-haiku-5.5 | closed | 1 | +4.7 (±9.7) | 80/295 | 33 | 182 | 0 | 0 | 95% [85–99] | 15% [12–20] | 44% [33–56] | $0.13 |
| 6–8 | gemma4-12b | closed | 1 | -73.0 (±14.0) | 33/256 | 110 | 113 | 0 | 39 | 73% [56–86] | 5% [3–9] | 90% [83–94] | $0.00 |
| 6–8 | gpt-oss-20b | closed | 1 | -93.5 (±13.6) | 41/294 | 158 | 93 | 2 | 1 | 68% [52–80] | 6% [3–9] | 91% [86–95] | $0.00 |
| 6–8 | deepseek-r1-14b | closed | 1 | -95.2 (±13.4) | 36/294 | 158 | 100 | 0 | 1 | 52% [38–67] | 6% [3–9] | 91% [85–94] | $0.00 |
| 1–2 | claude-fable-5.1 | mcp | 2 | +88.0 (+87…+88) | 543/590 | 12 | 34 | 1 | 0 | 98% [92–99] | 91% [88–93] | 2% [1–4] | $203.36 |
| 1–2 | claude-opus-5.5 | mcp | 3 | +87.5 (+85…+91) | 808/885 | 17 | 27 | 33 | 0 | 98% [93–99] | 94% [92–96] | 2% [1–3] | $91.39 |
| 3 | claude-sonnet-5.5 | mcp | 3 | +85.3 (+83…+87) | 807/885 | 26 | 40 | 12 | 0 | 95% [89–97] | 92% [90–94] | 3% [2–4] | $34.97 |
| 4 | grok-4.7 | mcp | 2 | +79.5 (+78…+81) | 501/590 | 16 | 73 | 0 | 0 | 94% [87–97] | 83% [80–86] | 3% [2–5] | $39.32 |
| 5 | claude-haiku-5.5 | mcp | 3 | +70.5 (+67…+75) | 728/885 | 52 | 91 | 14 | 0 | 91% [85–95] | 82% [79–85] | 7% [5–9] | $3.62 |
| 6 | gemma4-12b | mcp | 3 | +18.2 (+14…+22) | 494/859 | 169 | 195 | 1 | 26 | 66% [57–74] | 56% [53–60] | 26% [23–30] | $0.00 |
| 7 | gpt-oss-20b | mcp | 3 | +10.3 (+8…+13) | 431/885 | 170 | 26 | 258 | 0 | 71% [62–79] | 68% [64–72] | 28% [25–32] | $0.00 |
Score/100q = (correct − 2×incorrect) per 100 gradeable questions; abstention is free. The −2 penalty encodes a 67% confidence threshold. Gradeable = total minus transport errors and truncated completions, and the Correct column is shown over that same denominator. The small figure beside each score is its uncertainty: for k>1 the lowest and highest score/100q observed across those repeat runs, and for k=1 a ±95% interval — repeats re-ask the same questions, so treating them as independent trials would understate the spread. Shared ranks (e.g. 1–10) group models whose interval overlaps the group leader's. Hallucination = incorrect ÷ (correct + incorrect) on grounded questions. Cells with n<10 show raw fractions; percentages carry Wilson 95% intervals.
closed-book MCP-grounded amber Δ = accuracy and trust moved in opposite directions