Archived leaderboard
Corpus 299-21b69d52efdc, revision 2026-10-31, published 2026-10-31, grader 2026-10-31b. This snapshot is immutable — cite this URL.
Revisions of this corpus: 2026-10-31 (27 runs) · 2026-10-28 (27 runs)
| Rank | Model | Track | k | Score/100q | Correct | Incorrect | Abstained | No answer | Truncated | Reasoning | Grounded | Hallucination | Cost |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1–4 | claude-opus-5.5 | closed | 1 | +28.4 (±10.0) | 137/292 | 27 | 128 | 0 | 0 | 100% [92–100] | 38% [32–44] | 22% [16–31] | $4.71 |
| 1–4 | grok-4.7 | closed | 1 | +26.4 (±6.2) | 87/292 | 5 | 200 | 0 | 0 | 93% [81–98] | 19% [15–24] | 6% [2–16] | $3.09 |
| 1–4 | claude-fable-5.1 | closed | 1 | +24.3 (±11.0) | 143/292 | 36 | 113 | 0 | 0 | 100% [92–100] | 40% [34–46] | 26% [20–34] | $14.19 |
| 1–4 | claude-sonnet-5.5 | closed | 1 | +10.6 (±12.2) | 131/292 | 50 | 111 | 0 | 0 | 100% [92–100] | 35% [30–41] | 36% [29–45] | $2.33 |
| 5 | claude-haiku-5.5 | closed | 1 | +4.8 (±9.8) | 80/292 | 33 | 179 | 0 | 0 | 95% [85–99] | 16% [12–21] | 44% [33–56] | $0.13 |
| 6–8 | gemma4-12b | closed | 1 | -73.9 (±14.2) | 33/253 | 110 | 110 | 0 | 39 | 73% [56–86] | 5% [3–9] | 90% [83–94] | $0.00 |
| 6–8 | deepseek-r1-14b | closed | 1 | -94.5 (±13.5) | 37/291 | 156 | 98 | 0 | 1 | 55% [40–69] | 6% [3–9] | 91% [85–94] | $0.00 |
| 6–8 | gpt-oss-20b | closed | 1 | -94.5 (±13.7) | 41/291 | 158 | 90 | 2 | 1 | 68% [52–80] | 6% [3–9] | 91% [86–95] | $0.00 |
| 1–2 | claude-fable-5.1 | mcp | 2 | +90.6 (+90…+91) | 543/584 | 7 | 34 | 0 | 0 | 100% [96–100] | 92% [89–94] | 2% [1–3] | $198.65 |
| 1–2 | claude-opus-5.5 | mcp | 3 | +90.0 (+88…+93) | 808/876 | 10 | 27 | 31 | 0 | 100% [97–100] | 95% [93–96] | 1% [1–3] | $89.36 |
| 3 | claude-sonnet-5.5 | mcp | 3 | +87.1 (+85…+89) | 805/876 | 21 | 40 | 10 | 0 | 95% [90–98] | 93% [90–94] | 2% [1–4] | $34.14 |
| 4 | grok-4.7 | mcp | 2 | +81.3 (+79…+84) | 501/584 | 13 | 70 | 0 | 0 | 94% [87–97] | 84% [81–87] | 2% [1–4] | $38.32 |
| 5 | claude-haiku-5.5 | mcp | 3 | +73.1 (+69…+77) | 728/876 | 44 | 91 | 13 | 0 | 92% [86–96] | 83% [80–86] | 6% [5–8] | $3.52 |
| 6 | gemma4-12b | mcp | 3 | +20.0 (+17…+23) | 494/851 | 162 | 194 | 1 | 25 | 67% [58–75] | 57% [53–60] | 26% [22–29] | $0.00 |
| 7 | gpt-oss-20b | mcp | 3 | +12.3 (+10…+15) | 432/876 | 162 | 26 | 256 | 0 | 74% [65–81] | 69% [65–73] | 28% [24–32] | $0.00 |
Score/100q = (correct − 2×incorrect) per 100 gradeable questions; abstention is free. The −2 penalty encodes a 67% confidence threshold. Gradeable = total minus transport errors and truncated completions, and the Correct column is shown over that same denominator. The small figure beside each score is its uncertainty: for k>1 the lowest and highest score/100q observed across those repeat runs, and for k=1 a ±95% interval — repeats re-ask the same questions, so treating them as independent trials would understate the spread. Shared ranks (e.g. 1–10) group models whose interval overlaps the group leader's. Hallucination = incorrect ÷ (correct + incorrect) on grounded questions. Cells with n<10 show raw fractions; percentages carry Wilson 95% intervals.
closed-book MCP-grounded amber Δ = accuracy and trust moved in opposite directions