Methodology
SpaceBench asks the same space-domain questions two ways. Closed-book: the model answers from its own knowledge and is told that "UNKNOWN" is a valid answer. MCP-grounded: the same questions with live tools over the Orbit Sentinel regulatory & satellite catalog database. The gap between tracks is the headline: what does real data access actually buy?
Answers and grading
Models end replies with a FINAL: line. Each response grades to one of five outcomes:
correct, incorrect, abstained (UNKNOWN and honest
variants like "classified" or "not publicly known"), no answer (empty or
protocol-violating output — a format failure, never counted as a knowledge claim), or
error (transport failure, excluded entirely). Matching is deterministic: numeric
tolerance bands, word-boundary text matching that refuses credit inside negated clauses, no LLM judges.
Score/100q = (correct − 2×incorrect) per 100 gradeable questions. Abstention is free. The −2 penalty is not arbitrary: under proper scoring, a penalty p makes answering rational only when the model's confidence exceeds p/(p+1) — so −2 encodes an explicit 67% confidence threshold, our stated risk posture for a domain where a confident wrong answer costs more than a missed one. Normalizing per 100 questions keeps scores comparable as the corpus grows; comparisons across corpus versions still carry composition caveats. Hallucination rate is incorrect ÷ (correct + incorrect) on grounded questions.
Penalty sensitivity
Rankings are checked at penalties 1, 2, 3 and 5 (confidence thresholds 50%, 67%, 75%, 83%). Tier structure is robust; adjacent positions can move — where the ordering depends on the penalty, that dependence is disclosed here rather than discovered later.
Closed-book rank by penalty (1 / 2 / 3 / 5)
| Model | Ranks |
|---|---|
| claude-opus-4.8 | 2 / 1 / 1 / 2 · moves |
| gemini-3.5-flash | 3 / 2 / 2 / 1 · moves |
| claude-fable-5 | 1 / 3 / 3 / 4 · moves |
| gemini-3.1-pro | 4 / 4 / 4 / 3 · moves |
| claude-sonnet-5 | 5 / 5 / 7 / 7 · moves |
| grok-4.5 | 7 / 6 / 5 / 5 · moves |
| kimi-k2.6 | 8 / 7 / 6 / 6 · moves |
| qwen3-max | 6 / 8 / 8 / 8 · moves |
| deepseek-v4-pro | 9 / 9 / 9 / 9 |
| llama-4-maverick | 11 / 10 / 10 / 10 · moves |
| claude-haiku-4.5 | 13 / 11 / 11 / 11 · moves |
| gpt-5.4 | 10 / 12 / 13 / 13 · moves |
| gpt-5.4-mini | 12 / 13 / 12 / 12 · moves |
MCP-grounded rank by penalty (1 / 2 / 3 / 5)
| Model | Ranks |
|---|---|
| gemini-3.5-flash | 1 / 1 / 1 / 1 |
| claude-fable-5 | 2 / 2 / 3 / 4 · moves |
| grok-4.5 | 4 / 3 / 2 / 2 · moves |
| claude-sonnet-5 | 3 / 4 / 4 / 5 · moves |
| gpt-5.4 | 5 / 5 / 5 / 3 · moves |
| qwen3-max | 6 / 6 / 6 / 11 · moves |
| claude-haiku-4.5 | 7 / 7 / 7 / 8 · moves |
| gpt-5.4-mini | 8 / 8 / 11 / 12 · moves |
| gemini-3.1-pro | 9 / 9 / 8 / 6 · moves |
| claude-opus-4.8 | 10 / 10 / 9 / 7 · moves |
| deepseek-v4-pro | 11 / 11 / 10 / 9 · moves |
| kimi-k2.6 | 12 / 12 / 12 / 10 · moves |
| llama-4-maverick | 13 / 13 / 13 / 13 |
Negative controls
Five questions are deliberately unanswerable (classified budgets, future counts, uncataloged objects). The correct behavior is abstention; any confident number is a fabrication. These make the calibration score measurable rather than aspirational.
Contamination resistance
Grounded answers are frozen from the database at a stamped date, and every question ships with the SQL that re-derives it. The corpus re-freezes as reality changes — new launches, new filings, new reentries — so a model that memorized last quarter's leaderboard answers fails this quarter's freeze. Each corpus version is content-fingerprinted; results are stamped with the fingerprint they ran under and never mixed across versions. Published leaderboards are immutable — every version keeps a permanent archive page.
Run conditions
Every provider is called through the same chat-completions protocol at provider-default sampling with
an 8,192-token completion cap. Costs are computed from list prices with cached input tokens priced at
provider cached rates. Grader version: 2026-07-20.
Relationship disclosure
SpaceBench is built by the team behind Orbit Sentinel, and the MCP track's ground truth is the Orbit Sentinel database — the benchmark measures retrieval against a defined oracle, and that oracle is our product. We publish the grounded questions, their SQL, and the freeze history precisely so this relationship is auditable rather than taken on trust.