Methodology

SpaceBench asks the same space-domain questions two ways. Closed-book: the model answers from its own knowledge and is told that "UNKNOWN" is a valid answer. MCP-grounded: the same questions with live tools over the Orbit Sentinel regulatory & satellite catalog database. The gap between tracks is the headline: what does real data access actually buy?

Answers and grading

Models end replies with a FINAL: line. Each response grades to one of five outcomes: correct, incorrect, abstained (UNKNOWN and honest variants like "classified" or "not publicly known"), no answer (empty or protocol-violating output — a format failure, never counted as a knowledge claim), or error (transport failure, excluded entirely). Matching is deterministic: numeric tolerance bands, word-boundary text matching that refuses credit inside negated clauses, no LLM judges.

Score = correct − 2×incorrect. Abstention is free. A model that guesses pays double for being wrong; a model that knows what it doesn't know keeps its points. Hallucination rate is incorrect ÷ (correct + incorrect) on grounded questions.

Negative controls

Five questions are deliberately unanswerable (classified budgets, future counts, uncataloged objects). The correct behavior is abstention; any confident number is a fabrication. These make the calibration score measurable rather than aspirational.

Contamination resistance

Grounded answers are frozen from the database at a stamped date, and every question ships with the SQL that re-derives it. The corpus re-freezes as reality changes — new launches, new filings, new reentries — so a model that memorized last quarter's leaderboard answers fails this quarter's freeze. Each corpus version is content-fingerprinted; results are stamped with the fingerprint they ran under and never mixed across versions. Published leaderboards are immutable — every version keeps a permanent archive page.

Run conditions

Every provider is called through the same chat-completions protocol at provider-default sampling with an 8,192-token completion cap. Costs are computed from list prices with cached input tokens priced at provider cached rates. Grader version: 2026-07-20.

Relationship disclosure

SpaceBench is built by the team behind Orbit Sentinel, and the MCP track's ground truth is the Orbit Sentinel database — the benchmark measures retrieval against a defined oracle, and that oracle is our product. We publish the grounded questions, their SQL, and the freeze history precisely so this relationship is auditable rather than taken on trust.