Methodology

By Anthony Vincent Caracappa, Founder and Self-Directed Space Researcher, Viventine Space Systems · updated 2026-10-17

SpaceBench asks the same space-domain questions two ways. Closed-book: the model answers from its own knowledge and is told that "UNKNOWN" is a valid answer. MCP-grounded: the same questions with live tools over the Orbit Sentinel regulatory & satellite catalog database. The gap between tracks is the headline: what does real data access actually buy?

The 278-question corpus is organized around seven practitioner jobs (catalog lookup, constellation status, debris, regulatory, company and market intelligence, ground infrastructure, baseline knowledge), 25 to 48 questions each, plus negative controls. Each question is labeled public-verifiable (checkable against Space-Track, FCC, ITU and other public records) or Orbit Sentinel-derived (the answer exists only in Orbit Sentinel's analysis).

Verdicts

For each AI, job and track: Trust it when it is wrong on 5% or fewer of the questions and right on 80% or more; Don't use it when it is wrong on more than 20%; otherwise Verify its answers. Confident wrong answers drive the verdict; declining to answer never does. Headline verdicts use public-verifiable questions only; Orbit Sentinel-derived questions are shown separately. Rates pool every run of a question, but the 95% ranges count each question once, because repeat runs of the same question are not independent.

Answers and grading

Models end replies with a FINAL: line. Each response grades to one of six outcomes: correct, incorrect, abstained (UNKNOWN and honest variants like "classified" or "not publicly known"), no answer (empty or protocol-violating output — a format failure, never counted as a knowledge claim; since grader 2026-10-10 a reply that finished cleanly as a single short line with no FINAL: is read as its answer), truncated (cut off at the completion cap, excluded from gradeable like provider errors — the model never got to state an answer), or error (transport failure, excluded entirely). Matching is deterministic: numeric tolerance bands, word-boundary text matching that refuses credit inside negated clauses, no LLM judges.

Score/100q = (correct − 2×incorrect) per 100 gradeable questions — gradeable being the total minus transport errors and truncated completions. Abstention is free. The −2 penalty is not arbitrary: under proper scoring, a penalty p makes answering rational only when the model's confidence exceeds p/(p+1) — so −2 encodes an explicit 67% confidence threshold, our stated risk posture for a domain where a confident wrong answer costs more than a missed one. Normalizing per 100 questions keeps scores comparable as the corpus grows; comparisons across corpus versions still carry composition caveats. Hallucination rate is incorrect ÷ (correct + incorrect) on grounded questions.

Penalty sensitivity

Rankings are checked at penalties 1, 2, 3 and 5 (confidence thresholds 50%, 67%, 75%, 83%). Tier structure is robust; adjacent positions can move — where the ordering depends on the penalty, that dependence is disclosed here rather than discovered later.

Negative controls

Five questions are deliberately unanswerable (classified budgets, future counts, uncataloged objects). The correct behavior is abstention; any confident number is a fabrication. These make the calibration score measurable rather than aspirational.

Contamination resistance

Grounded answers are frozen from the database at a stamped date, and every question ships with the SQL that re-derives it. The corpus re-freezes as reality changes — new launches, new filings, new reentries — so a model that memorized last quarter's leaderboard answers fails this quarter's freeze. Each corpus version is content-fingerprinted; results are stamped with the fingerprint they ran under and never mixed across versions. Published leaderboards are immutable — every version keeps a permanent archive page.

How each AI was run

• Claude (Haiku, Sonnet and Opus 5.5; Fable 5.1): Claude Code on a Claude Pro subscription, using Claude Code's own tool loop. Costs shown are API-equivalent; nothing was billed per token.
• Grok 4.7: the xAI API through the SpaceBench tool loop.
• Open-weight models (gpt-oss 20B, Gemma 4 12B, DeepSeek-R1 14B): Ollama 0.40.2 on a rented RTX 4090 (24 GB), 65,536-token context, flash attention, 8-bit KV cache, through the SpaceBench tool loop.
• Runs: one closed-book run per AI; three tool runs, except Fable 5.1 and Grok 4.7 with two. DeepSeek-R1 14B was offered the tools but never called them, so it is reported closed-book only.

Run conditions

• Provider-default sampling; no temperature override.
• Up to 8 tool turns per question. The SpaceBench loop then forces a final answer with tools withheld; Claude Code has no forced final turn, so a Claude reply that runs out of turns scores "no answer".
• At most 50 tool calls per question, a production-load safeguard; every hit is recorded.
• An 8,192-token completion cap. Ollama counts thinking against it (14% of Gemma 4's closed-book answers and about 3% of its tool answers were cut off and excluded as truncated); Claude never reached it; xAI does not apply it to Grok's reasoning.
• Up to 15 minutes per API call.
• Costs from list prices with cached input at cached rates; Grok costs are xAI's own billed figures. Grader version: 2026-10-10.

Corrections

After the runs, every question that most AIs answered with the same wrong number was reviewed. Corrections to the answer key are published as errata; the corpus fingerprint stays that of the questions as asked.

• con-galileo-launched: answer changed to 34 (2026-10-11). The question counts IOV and FOC satellites; the ground-truth SQL matched only names starting "GALILEO " and missed the 4 IOV satellites, giving 30. 34 Galileo satellites had been launched (latest 2025-12-17).
• deb-atlas5-centaur-afspc11-debris: removed from scoring (2026-10-11). The question names the wrong launch. The Centaur that fragmented in September 2024 is 2018-079B from the AEHF-4 launch (2018-10-17, 214 debris cataloged). AFSPC-11 (2018-036) has 1 debris object, which is what models answered.
• reg-ngso-market-access-count: answer changed to 9 (2026-10-11). The question asks for foreign-licensed NGSO systems with US market access; the ground-truth SQL counted every granted NGSO system, U.S.-licensed ones included, giving 52. Excluding a U.S. licensing administration leaves 9 (Kepler, Telesat Ka-band, ExactView-1, ExactView-7, Kineis, Arctic Satellite Broadband Mission, two OneWeb systems, O3B).
• reg-noaa-remote-sensing-licensees: removed from scoring (2026-10-11). The question asks how many companies currently hold a license, but the answer key comes from a NOAA CRSRA licensee snapshot last seen 2024-07-27; no current list was available to verify it. To be restated as an as-of question, or rekeyed from a current list, in the next corpus.

The corrections above date from the previous corpus and still apply. This corpus's review found three such questions and no new errors in the answer key: in each, the tools themselves led models to the same wrong number, or the question is a negative control.

Orbit Sentinel changed during the runs. The FCC earth-station search, which read the wrong table in the previous corpus's runs, was fixed before these runs began, so every AI's ground-infrastructure tool answers use the fixed search. Orbit Sentinel also stopped serving about 2,500 out-of-scope FCC docket filings partway through; no scored answer changed because of it.

Relationship disclosure

SpaceBench is built by the team behind Orbit Sentinel, and the MCP track's ground truth is the Orbit Sentinel database — the benchmark measures retrieval against a defined oracle, and that oracle is our product. We publish the grounded questions, their SQL, and the freeze history precisely so this relationship is auditable rather than taken on trust.