# SpaceBench — full text for LLM consumption Corpus 307-bf8e3bdb8cf5, published 2026-11-08, grader 2026-10-31b. Created by Viventine Space Systems LLC (https://viventine.com). Data: CC BY 4.0 — cite https://spacebench.net/v/307-bf8e3bdb8cf5/2026-11-08/ for this snapshot. ## Leaderboard Score/100q = (correct - 2*incorrect) per 100 gradeable questions, where gradeable = total - errors - truncated; abstention is free. The -2 penalty encodes a 67% confidence threshold: answering beats abstaining only when P(correct) > 2/3. The Correct column is shown over gradeable questions, not raw total. Hallucination = incorrect / (correct + incorrect) on grounded questions. Cells with n<10 are raw fractions; percentages carry Wilson 95% intervals. | Model | Track | Score/100q | Correct | Incorrect | Abstained | No answer | Truncated | Errors | Reasoning acc | Grounded acc | Hallucination | Cost | |---|---|---|---|---|---|---|---|---|---|---|---|---| | claude-fable-5.1 | closed | +25.9 | 149/305 | 35 | 121 | 0 | 0 | 0 | 100% [92–100] | 40% [35–46] | 25% [18–33] | $15.37 | | claude-opus-5.5 | closed | +25.2 | 145/305 | 34 | 126 | 0 | 0 | 0 | 100% [92–100] | 39% [33–45] | 25% [18–33] | $5.06 | | grok-4.7 | closed | +24.9 | 86/305 | 5 | 214 | 0 | 0 | 0 | 93% [81–98] | 18% [13–23] | 6% [2–17] | $3.34 | | claude-sonnet-5.5 | closed | +18.4 | 132/305 | 38 | 135 | 0 | 0 | 0 | 100% [92–100] | 34% [29–40] | 30% [23–38] | $2.44 | | claude-haiku-5.5 | closed | +8.5 | 86/305 | 30 | 189 | 0 | 0 | 0 | 100% [92–100] | 16% [12–21] | 41% [31–53] | $0.14 | | gemma4-12b | closed | -68.7 | 40/262 | 110 | 112 | 0 | 43 | 0 | 68% [49–82] | 9% [6–13] | 83% [75–88] | $0.00 | | deepseek-r1-14b | closed | -82.8 | 42/302 | 146 | 114 | 0 | 3 | 0 | 58% [42–71] | 7% [5–11] | 87% [81–92] | $0.00 | | gpt-oss-20b | closed | -90.8 | 45/303 | 160 | 98 | 0 | 2 | 0 | 71% [56–83] | 6% [4–9] | 91% [85–94] | $0.00 | | claude-opus-5.5 | mcp | +98.7 | 908/912 | 4 | 0 | 0 | 0 | 0 | 100% [97–100] | 99% [99–100] | 1% [0–1] | $155.93 | | claude-fable-5.1 | mcp | +98.6 | 907/912 | 4 | 0 | 1 | 0 | 0 | 100% [97–100] | 99% [99–100] | 1% [0–1] | $368.93 | | claude-sonnet-5.5 | mcp | +93.3 | 887/912 | 18 | 6 | 1 | 0 | 0 | 95% [90–98] | 98% [96–99] | 2% [1–3] | $29.80 | | grok-4.7 | mcp | +92.8 | 876/912 | 15 | 21 | 0 | 0 | 0 | 96% [91–98] | 96% [94–97] | 2% [1–3] | $73.43 | | claude-haiku-5.5 | mcp | +85.3 | 840/912 | 31 | 39 | 2 | 0 | 0 | 94% [88–97] | 92% [90–94] | 4% [3–5] | $3.70 | | gemma4-12b | mcp | +42.0 | 649/889 | 138 | 101 | 1 | 23 | 0 | 74% [66–81] | 73% [70–76] | 18% [15–21] | $0.00 | | gpt-oss-20b | mcp | +30.5 | 558/910 | 140 | 17 | 195 | 2 | 0 | 75% [66–83] | 78% [75–82] | 19% [16–23] | $0.00 | ## Closed-book to MCP-grounded gap (headline: trust gap = delta Score/100q) | Model | Score/100q closed | Score/100q MCP | Delta score | Grounded acc closed | Grounded acc MCP | Delta pp | |---|---|---|---|---|---|---| | gpt-oss-20b | -90.8 | +30.5 | +121.6 | 6% | 78% | +73 | | gemma4-12b | -68.7 | +42.0 | +110.2 | 9% | 73% | +64 | | claude-haiku-5.5 | +8.5 | +85.3 | +76.8 | 16% | 92% | +76 | | claude-sonnet-5.5 | +18.4 | +93.3 | +74.9 | 34% | 98% | +64 | | claude-opus-5.5 | +25.2 | +98.7 | +73.4 | 39% | 99% | +61 | | claude-fable-5.1 | +25.9 | +98.6 | +72.6 | 40% | 99% | +59 | | grok-4.7 | +24.9 | +92.8 | +67.8 | 18% | 96% | +78 | ## Verdicts by job Cells: % right / % wrong, then the verdict. Trust it = wrong <= 5% and right >= 80%; dont_use = wrong > 20%; otherwise verify. Public-verifiable questions only; repeat runs pooled. | AI | Track | Catalog lookup | Constellation & fleet status | Debris & conjunction context | Regulatory & licensing | Company & market intelligence | Ground infrastructure | Baseline knowledge | |---|---|---|---|---|---|---|---|---| | Grok 4.7 | closed | 42/0 verify | 16/0 verify | 14/2 verify | 28/0 verify | 22/5 verify | 13/5 verify | 96/0 trust | | Grok 4.7 | tools | 100/0 trust | 95/5 trust | 94/2 trust | 96/1 trust | 100/0 trust | 93/3 trust | 97/0 trust | | Claude Fable 5.1 | closed | 64/7 verify | 53/14 verify | 45/14 verify | 38/5 verify | 49/15 verify | 28/28 dont_use | 100/0 trust | | Claude Fable 5.1 | tools | 100/0 trust | 99/1 trust | 99/0 trust | 98/2 trust | 100/0 trust | 99/1 trust | 100/0 trust | | Claude Opus 5.5 | closed | 76/9 verify | 63/7 verify | 40/24 dont_use | 33/5 verify | 34/20 verify | 23/18 verify | 100/0 trust | | Claude Opus 5.5 | tools | 100/0 trust | 100/0 trust | 99/1 trust | 97/3 trust | 100/0 trust | 100/0 trust | 100/0 trust | | Claude Sonnet 5.5 | closed | 60/11 verify | 42/7 verify | 40/21 dont_use | 31/8 verify | 37/24 dont_use | 28/18 verify | 100/0 trust | | Claude Sonnet 5.5 | tools | 100/0 trust | 99/1 trust | 96/2 trust | 93/4 trust | 99/0 trust | 96/4 trust | 99/1 trust | | Claude Haiku 5.5 | closed | 27/24 dont_use | 26/16 verify | 19/10 verify | 23/8 verify | 27/10 verify | 13/3 verify | 100/0 trust | | Claude Haiku 5.5 | tools | 100/0 trust | 87/12 verify | 93/0 trust | 82/3 trust | 94/2 trust | 89/6 verify | 97/0 trust | | gpt-oss 20B | closed | 7/67 dont_use | 12/62 dont_use | 5/69 dont_use | 10/56 dont_use | 2/66 dont_use | 5/45 dont_use | 96/4 trust | | gpt-oss 20B | tools | 92/1 trust | 36/28 dont_use | 58/13 verify | 48/18 verify | 70/11 verify | 46/20 verify | 85/5 verify | | Gemma 4 12B | closed | 9/60 dont_use | 9/53 dont_use | 13/36 dont_use | 14/47 dont_use | 6/44 dont_use | 10/36 dont_use | 85/15 verify | | Gemma 4 12B | tools | 99/1 trust | 61/26 dont_use | 61/21 dont_use | 68/18 verify | 74/14 verify | 61/24 dont_use | 85/1 trust | | DeepSeek-R1 14B | closed | 7/60 dont_use | 19/49 dont_use | 5/67 dont_use | 8/51 dont_use | 5/46 dont_use | 13/43 dont_use | 68/32 dont_use | | DeepSeek-R1 14B | tools | - | - | - | - | - | - | - | ## Methodology summary Two tracks over identical questions. Closed-book: the model answers from its own knowledge and is told UNKNOWN is a valid answer. MCP-grounded: the model has live tools over the Orbit Sentinel satellite catalog & regulatory database. Answers end with a FINAL: line; grading is deterministic (numeric tolerance bands, word-boundary matching that refuses credit in negated clauses; no LLM judges). Six outcomes: correct, incorrect, abstained, no_answer (format failure), truncated (cut off at the completion cap, excluded from gradeable like provider errors), error (transport, excluded). Five negative-control questions are genuinely unanswerable — abstention grades correct, any confident answer grades incorrect. Grounded answers are frozen from re-runnable SQL at a stamped date and re-freeze as the database changes; each corpus version is content-fingerprinted and published leaderboards are immutable. All providers run at provider-default sampling with an 8192-token completion cap (Ollama counts thinking against it; xAI does not apply it to Grok's reasoning). Claude runs inside Claude Code on a Pro subscription; other models use the SpaceBench tool loop. Corrections to the answer key are published as errata in the leaderboard JSON. Disclosure: the MCP track's oracle is the Orbit Sentinel database, built by the same team; the grounded questions, their SQL, and freeze history are published so the relationship is auditable. Machine-readable JSON: https://spacebench.net/data/current.json and https://spacebench.net/data/v/307-bf8e3bdb8cf5/2026-11-08/