Archived leaderboard
Corpus 278-8bc1c4f8addd, revision 2026-10-17, published 2026-10-17, grader 2026-10-10. This snapshot is immutable — cite this URL.
| Rank | Model | Track | k | Score/100q | Correct | Incorrect | Abstained | No answer | Truncated | Errors | Reasoning | Grounded | Hallucination | Cost |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1–4 | claude-opus-5.5 | closed | 1 | +29.7 ±10.2 | 132/276 | 25 | 119 | 0 | 0 | 0 | 98% [88–100] | 39% [33–45] | 21% [15–29] | $4.49 |
| 1–4 | claude-fable-5.1 | closed | 1 | +28.6 ±10.9 | 139/276 | 30 | 107 | 0 | 0 | 0 | 100% [92–100] | 41% [35–48] | 24% [17–32] | $13.57 |
| 1–4 | grok-4.7 | closed | 1 | +27.2 ±6.5 | 85/276 | 5 | 186 | 0 | 0 | 0 | 91% [78–96] | 20% [15–25] | 4% [1–14] | $2.98 |
| 1–4 | claude-sonnet-5.5 | closed | 1 | +13.8 ±12.4 | 128/276 | 45 | 103 | 0 | 0 | 0 | 100% [92–100] | 36% [31–43] | 35% [27–43] | $2.21 |
| 5 | claude-haiku-5.5 | closed | 1 | +4.7 ±10.2 | 77/276 | 32 | 167 | 0 | 0 | 0 | 95% [85–99] | 15% [11–21] | 45% [34–57] | $0.13 |
| 6–8 | gemma4-12b | closed | 1 | -73.4 ±14.6 | 30/237 | 102 | 105 | 0 | 39 | 0 | 73% [56–86] | 4% [2–7] | 92% [85–96] | $0.00 |
| 6–8 | gpt-oss-20b | closed | 1 | -90.9 ±14.1 | 40/275 | 145 | 88 | 2 | 1 | 0 | 68% [52–80] | 6% [3–9] | 91% [85–95] | $0.00 |
| 6–8 | deepseek-r1-14b | closed | 1 | -93.1 ±13.8 | 34/275 | 145 | 96 | 0 | 1 | 0 | 52% [38–67] | 5% [3–9] | 91% [85–95] | $0.00 |
| 1–2 | claude-opus-5.5 | mcp | 3 | +83.5 +82…+85 | 717/828 | 13 | 37 | 61 | 0 | 0 | 98% [93–99] | 93% [90–94] | 2% [1–3] | $109.31 |
| 1–2 | claude-fable-5.1 | mcp | 2 | +82.1 +82…+82 | 489/552 | 18 | 31 | 14 | 0 | 0 | 98% [92–99] | 90% [86–92] | 4% [2–6] | $305.61 |
| 3–4 | claude-sonnet-5.5 | mcp | 3 | +72.8 +71…+74 | 699/828 | 48 | 47 | 34 | 0 | 0 | 94% [88–97] | 87% [84–89] | 6% [5–9] | $43.64 |
| 3–4 | grok-4.7 | mcp | 2 | +71.4 +71…+72 | 436/552 | 21 | 95 | 0 | 0 | 0 | 93% [86–97] | 76% [72–80] | 5% [3–7] | $54.31 |
| 5 | claude-haiku-5.5 | mcp | 3 | +62.7 +61…+65 | 629/828 | 55 | 99 | 45 | 0 | 0 | 92% [86–96] | 78% [75–81] | 9% [7–11] | $4.19 |
| 6 | gpt-oss-20b | mcp | 3 | +4.5 +3…+6 | 383/828 | 173 | 44 | 228 | 0 | 0 | 74% [65–81] | 62% [57–66] | 32% [28–37] | $0.00 |
| 7 | gemma4-12b | mcp | 3 | -3.5 -7…+2 | 400/802 | 214 | 186 | 2 | 26 | 0 | 70% [62–78] | 46% [43–50] | 38% [34–42] | $0.00 |
Score/100q = (correct − 2×incorrect) per 100 gradeable questions; abstention is free. The −2 penalty encodes a 67% confidence threshold. Gradeable = total minus transport errors and truncated completions, and the Correct column is shown over that same denominator. The small figure beside each score is its uncertainty: for k>1 the lowest and highest score/100q observed across those repeat runs, and for k=1 a ±95% interval — repeats re-ask the same questions, so treating them as independent trials would understate the spread. Shared ranks (e.g. 1–10) group models whose interval overlaps the group leader's. Hallucination = incorrect ÷ (correct + incorrect) on grounded questions. Cells with n<10 show raw fractions; percentages carry Wilson 95% intervals.
closed-book MCP-grounded amber Δ = accuracy and trust moved in opposite directions