Full leaderboard

Every run in full. Use the buttons to show one track, and click Score, Correct or Cost to sort. What each column means is below the table.

Rank Model Track k Score/100q Correct Incorrect Abstained No answer Truncated Reasoning Grounded Hallucination Cost
1–5 claude-fable-5.1 closed 1 +25.9 (±10.5) 149/305 35 121 0 0 100%
[92–100]
40%
[35–46]
25%
[18–33]
$15.37
1–5 claude-opus-5.5 closed 1 +25.2 (±10.4) 145/305 34 126 0 0 100%
[92–100]
39%
[33–45]
25%
[18–33]
$5.06
1–5 grok-4.7 closed 1 +24.9 (±6.0) 86/305 5 214 0 0 93%
[81–98]
18%
[13–23]
6%
[2–17]
$3.34
1–5 claude-sonnet-5.5 closed 1 +18.4 (±10.6) 132/305 38 135 0 0 100%
[92–100]
34%
[29–40]
30%
[23–38]
$2.44
1–5 claude-haiku-5.5 closed 1 +8.5 (±9.2) 86/305 30 189 0 0 100%
[92–100]
16%
[12–21]
41%
[31–53]
$0.14
6–8 gemma4-12b closed 1 -68.7 (±14.1) 40/262 110 112 0 43 68%
[49–82]
9%
[6–13]
83%
[75–88]
$0.00
6–8 deepseek-r1-14b closed 1 -82.8 (±13.3) 42/302 146 114 0 3 58%
[42–71]
7%
[5–11]
87%
[81–92]
$0.00
6–8 gpt-oss-20b closed 1 -90.8 (±13.5) 45/303 160 98 0 2 71%
[56–83]
6%
[4–9]
91%
[85–94]
$0.00
1–2 claude-opus-5.5 mcp 3 +98.7 (+98…+99) 908/912 4 0 0 0 100%
[97–100]
99%
[99–100]
1%
[0–1]
$155.93
1–2 claude-fable-5.1 mcp 3 +98.6 (+98…+100) 907/912 4 0 1 0 100%
[97–100]
99%
[99–100]
1%
[0–1]
$368.93
3–4 claude-sonnet-5.5 mcp 3 +93.3 (+92…+95) 887/912 18 6 1 0 95%
[90–98]
98%
[96–99]
2%
[1–3]
$29.80
3–4 grok-4.7 mcp 3 +92.8 (+92…+93) 876/912 15 21 0 0 96%
[91–98]
96%
[94–97]
2%
[1–3]
$73.43
5 claude-haiku-5.5 mcp 3 +85.3 (+85…+86) 840/912 31 39 2 0 94%
[88–97]
92%
[90–94]
4%
[3–5]
$3.70
6 gemma4-12b mcp 3 +42.0 (+38…+49) 649/889 138 101 1 23 74%
[66–81]
73%
[70–76]
18%
[15–21]
$0.00
7 gpt-oss-20b mcp 3 +30.5 (+24…+34) 558/910 140 17 195 2 75%
[66–83]
78%
[75–82]
19%
[16–23]
$0.00

Score/100q = (correct − 2×incorrect) per 100 gradeable questions; abstention is free. The −2 penalty encodes a 67% confidence threshold. Gradeable = total minus transport errors and truncated completions, and the Correct column is shown over that same denominator. The small figure beside each score is its uncertainty: for k>1 the lowest and highest score/100q observed across those repeat runs, and for k=1 a ±95% interval — repeats re-ask the same questions, so treating them as independent trials would understate the spread. Shared ranks (e.g. 1–10) group models whose interval overlaps the group leader's. Hallucination = incorrect ÷ (correct + incorrect) on grounded questions. Cells with n<10 show raw fractions; percentages carry Wilson 95% intervals.

What the columns mean

Track
closed: the AI answers on its own. mcp: the same AI connected to Orbit Sentinel through MCP tools.
k
How many repeat runs are pooled in the row. Tool tracks ran three times; closed-book once.
Score/100q
Right answers minus twice the wrong ones, per 100 gradeable questions. +100 is perfect; abstaining scores 0, so answering only pays when the AI is more than two-thirds sure.
Band or ±
Beside each score: the lowest and highest score across repeat runs where k > 1, or the 95% interval where k = 1. Ranks are shared when this figure overlaps the group leader.
Correct
Right answers out of gradeable ones (total minus provider errors and replies cut off at the length cap).
Incorrect
Wrong answers. Each costs 2 points.
Abstained
The AI declined to answer ("UNKNOWN"). Free.
No answer
An empty reply or one that broke the answer format, such as running out of tool turns. Counts as gradeable and earns 0.
Truncated
Reply cut off at the 8,192-token cap; left out of the denominator.
Errors
Provider faults. The column appears only when a published run has any; every error in this release was rerun.
Reasoning / Grounded
Accuracy on the 43 reasoning questions (history, orbital mechanics) and on the data questions, with Wilson 95% intervals.
Hallucination
Wrong answers as a share of answers given to data questions: how often an answer, once given, is wrong.
Cost
List-price cost of the run (API-equivalent for Claude on a subscription; xAI's billed figure for Grok; local models are $0).
What tools actually buy: trust vs. accuracy, closed-book → MCP-grounded
Trust — Score/100q Grounded accuracy — % -50 0 50 0 25 50 75 100 gpt-oss-20b +122 +73pp gemma4-12b +110 +64pp claude-haiku-5.5 +77 +76pp claude-sonnet-5.5 +75 +64pp claude-opus-5.5 +73 +61pp claude-fable-5.1 +73 +59pp grok-4.7 +68 +78pp

closed-book MCP-grounded amber Δ = accuracy and trust moved in opposite directions

Each line runs from the AI on its own (grey) to the same AI with the tools (blue). Left: the calibrated score. Right: accuracy on data questions.

Score/100q vs. cost per question (log scale)
$0.0010 $0.01 $0.10 0 25 50 75 100 claude-fable-5.1 (closed): 25.9/100q at $0.05/question claude-fable-5.1 claude-opus-5.5 (closed): 25.2/100q at $0.02/question claude-opus-5.5 grok-4.7 (closed): 24.9/100q at $0.01/question grok-4.7 claude-sonnet-5.5 (closed): 18.4/100q at $0.0080/question claude-sonnet-5.5 claude-haiku-5.5 (closed): 8.5/100q at $0.0005/question claude-haiku-5.5 claude-opus-5.5 (mcp): 98.7/100q at $0.17/question claude-opus-5.5 claude-fable-5.1 (mcp): 98.6/100q at $0.40/question claude-fable-5.1 claude-sonnet-5.5 (mcp): 93.3/100q at $0.03/question claude-sonnet-5.5 grok-4.7 (mcp): 92.8/100q at $0.08/question grok-4.7 claude-haiku-5.5 (mcp): 85.3/100q at $0.0041/question claude-haiku-5.5

closed-book MCP-grounded

Higher is better, further left is cheaper. Local models cost nothing to run and are not shown.

Data and citation

Dataset

Every question, graded answer and verdict, with viewer-ready tables.

Hugging Face →

Raw data

The JSON bundle behind this page: leaderboard, corpus and all 29 runs, CC BY 4.0.

Downloads and versions →

Cite

@misc{spacebench2026,
  title  = {SpaceBench: Can you trust AI for space work?},
  author = {Anthony Vincent Caracappa},
  year   = {2026},
  note   = {Corpus 307-bf8e3bdb8cf5, revision 2026-11-08},
  url    = {https://spacebench.net/}
}