Archived leaderboard
Corpus 180-0a2e675b6673, revision 2026-10-12, published 2026-10-11, grader 2026-10-10. This snapshot is immutable — cite this URL.
| Rank | Model | Track | k | Score/100q | Correct | Incorrect | Abstained | No answer | Truncated | Reasoning | Grounded | Hallucination | Cost |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1–5 | claude-opus-5.5 | closed | 1 | +43.3 (±11.5) | 99/178 | 11 | 68 | 0 | 0 | 100% [92–100] | 41% [34–50] | 16% [9–27] | $2.58 |
| 1–5 | grok-4.7 | closed | 1 | +36.5 (±8.0) | 69/178 | 2 | 107 | 0 | 0 | 95% [85–99] | 21% [15–28] | 3% [1–17] | $2.27 |
| 1–5 | claude-fable-5.1 | closed | 1 | +32.0 (±13.9) | 97/178 | 20 | 61 | 0 | 0 | 100% [92–100] | 40% [32–48] | 27% [18–38] | $7.05 |
| 1–5 | claude-sonnet-5.5 | closed | 1 | +23.0 (±14.8) | 91/178 | 25 | 62 | 0 | 0 | 98% [88–100] | 36% [29–45] | 33% [23–44] | $1.27 |
| 1–5 | claude-haiku-5.5 | closed | 1 | +22.5 (±12.1) | 70/178 | 15 | 93 | 0 | 0 | 100% [92–100] | 20% [14–28] | 36% [23–51] | $0.08 |
| 6–8 | gemma4-12b | closed | 1 | -56.5 (±18.7) | 35/161 | 63 | 63 | 0 | 17 | 70% [53–83] | 9% [5–16] | 82% [70–89] | $0.00 |
| 6–8 | gpt-oss-20b | closed | 1 | -79.1 (±18.4) | 36/177 | 88 | 52 | 1 | 1 | 63% [48–76] | 7% [4–13] | 88% [79–93] | $0.00 |
| 6–8 | deepseek-r1-14b | closed | 1 | -82.9 (±17.9) | 29/175 | 87 | 59 | 0 | 3 | 55% [40–69] | 5% [3–10] | 91% [82–95] | $0.00 |
| 1–2 | claude-fable-5.1 | mcp | 2 | +80.3 (+80…+81) | 322/356 | 18 | 9 | 7 | 0 | 99% [94–100] | 90% [86–93] | 7% [4–10] | $174.79 |
| 1–2 | claude-opus-5.5 | mcp | 3 | +79.8 (+78…+81) | 458/534 | 16 | 16 | 44 | 0 | 98% [93–99] | 92% [89–94] | 4% [2–6] | $62.89 |
| 3–4 | claude-sonnet-5.5 | mcp | 3 | +69.3 (+68…+70) | 456/534 | 43 | 14 | 21 | 0 | 93% [87–96] | 88% [84–90] | 9% [7–13] | $25.49 |
| 3–4 | grok-4.7 | mcp | 2 | +68.8 (+69…+69) | 285/356 | 20 | 51 | 0 | 0 | 97% [90–99] | 75% [69–80] | 8% [5–13] | $30.73 |
| 5 | claude-haiku-5.5 | mcp | 3 | +64.0 (+62…+66) | 422/534 | 40 | 50 | 22 | 0 | 92% [86–96] | 79% [75–83] | 11% [8–14] | $2.55 |
| 6 | gpt-oss-20b | mcp | 3 | +13.7 (+13…+15) | 277/534 | 102 | 17 | 138 | 0 | 73% [64–80] | 69% [63–74] | 27% [22–32] | $0.00 |
| 7 | gemma4-12b | mcp | 3 | 0.0 (-12…+6) | 256/510 | 128 | 125 | 1 | 24 | 71% [62–78] | 44% [40–49] | 38% [33–44] | $0.00 |
Score/100q = (correct − 2×incorrect) per 100 gradeable questions; abstention is free. The −2 penalty encodes a 67% confidence threshold. Gradeable = total minus transport errors and truncated completions, and the Correct column is shown over that same denominator. The small figure beside each score is its uncertainty: for k>1 the lowest and highest score/100q observed across those repeat runs, and for k=1 a ±95% interval — repeats re-ask the same questions, so treating them as independent trials would understate the spread. Shared ranks (e.g. 1–10) group models whose interval overlaps the group leader's. Hallucination = incorrect ÷ (correct + incorrect) on grounded questions. Cells with n<10 show raw fractions; percentages carry Wilson 95% intervals.
closed-book MCP-grounded amber Δ = accuracy and trust moved in opposite directions