Can you trust AI for space work?
By Anthony Vincent Caracappa, Founder and Self-Directed Space Researcher, Viventine Space Systems · updated
278 questions a space practitioner actually asks, put to 8 AIs two ways: closed-book, and grounded via MCP tools in the Orbit Sentinel satellite & regulatory database. Scoring rewards calibration — a wrong answer costs twice what a right one earns, and admitting ignorance is free.
| Rank | Model | Track | k | Score/100q | Correct | Incorrect | Abstained | No answer | Truncated | Errors | Reasoning | Grounded | Hallucination | Cost |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1–4 | claude-opus-5.5 | closed | 1 | +29.7 ±10.2 | 132/276 | 25 | 119 | 0 | 0 | 0 | 98% [88–100] | 39% [33–45] | 21% [15–29] | $4.49 |
| 1–4 | claude-fable-5.1 | closed | 1 | +28.6 ±10.9 | 139/276 | 30 | 107 | 0 | 0 | 0 | 100% [92–100] | 41% [35–48] | 24% [17–32] | $13.57 |
| 1–4 | grok-4.7 | closed | 1 | +27.2 ±6.5 | 85/276 | 5 | 186 | 0 | 0 | 0 | 91% [78–96] | 20% [15–25] | 4% [1–14] | $2.98 |
| 1–4 | claude-sonnet-5.5 | closed | 1 | +13.8 ±12.4 | 128/276 | 45 | 103 | 0 | 0 | 0 | 100% [92–100] | 36% [31–43] | 35% [27–43] | $2.21 |
| 5 | claude-haiku-5.5 | closed | 1 | +4.7 ±10.2 | 77/276 | 32 | 167 | 0 | 0 | 0 | 95% [85–99] | 15% [11–21] | 45% [34–57] | $0.13 |
| 6–8 | gemma4-12b | closed | 1 | -73.4 ±14.6 | 30/237 | 102 | 105 | 0 | 39 | 0 | 73% [56–86] | 4% [2–7] | 92% [85–96] | $0.00 |
| 6–8 | gpt-oss-20b | closed | 1 | -90.9 ±14.1 | 40/275 | 145 | 88 | 2 | 1 | 0 | 68% [52–80] | 6% [3–9] | 91% [85–95] | $0.00 |
| 6–8 | deepseek-r1-14b | closed | 1 | -93.1 ±13.8 | 34/275 | 145 | 96 | 0 | 1 | 0 | 52% [38–67] | 5% [3–9] | 91% [85–95] | $0.00 |
| 1–2 | claude-opus-5.5 | mcp | 3 | +83.5 +82…+85 | 717/828 | 13 | 37 | 61 | 0 | 0 | 98% [93–99] | 93% [90–94] | 2% [1–3] | $109.31 |
| 1–2 | claude-fable-5.1 | mcp | 2 | +82.1 +82…+82 | 489/552 | 18 | 31 | 14 | 0 | 0 | 98% [92–99] | 90% [86–92] | 4% [2–6] | $305.61 |
| 3–4 | claude-sonnet-5.5 | mcp | 3 | +72.8 +71…+74 | 699/828 | 48 | 47 | 34 | 0 | 0 | 94% [88–97] | 87% [84–89] | 6% [5–9] | $43.64 |
| 3–4 | grok-4.7 | mcp | 2 | +71.4 +71…+72 | 436/552 | 21 | 95 | 0 | 0 | 0 | 93% [86–97] | 76% [72–80] | 5% [3–7] | $54.31 |
| 5 | claude-haiku-5.5 | mcp | 3 | +62.7 +61…+65 | 629/828 | 55 | 99 | 45 | 0 | 0 | 92% [86–96] | 78% [75–81] | 9% [7–11] | $4.19 |
| 6 | gpt-oss-20b | mcp | 3 | +4.5 +3…+6 | 383/828 | 173 | 44 | 228 | 0 | 0 | 74% [65–81] | 62% [57–66] | 32% [28–37] | $0.00 |
| 7 | gemma4-12b | mcp | 3 | -3.5 -7…+2 | 400/802 | 214 | 186 | 2 | 26 | 0 | 70% [62–78] | 46% [43–50] | 38% [34–42] | $0.00 |
Score/100q = (correct − 2×incorrect) per 100 gradeable questions; abstention is free. The −2 penalty encodes a 67% confidence threshold. Gradeable = total minus transport errors and truncated completions, and the Correct column is shown over that same denominator. The small figure beside each score is its uncertainty: for k>1 the lowest and highest score/100q observed across those repeat runs, and for k=1 a ±95% interval — repeats re-ask the same questions, so treating them as independent trials would understate the spread. Shared ranks (e.g. 1–10) group models whose interval overlaps the group leader's. Hallucination = incorrect ÷ (correct + incorrect) on grounded questions. Cells with n<10 show raw fractions; percentages carry Wilson 95% intervals.
Verdicts by job
Trust it: wrong ≤ 5% and right ≥ 80%. Don't use it: wrong > 20%. Otherwise verify its answers. Publicly verifiable questions only; repeat runs pooled, ranges count each question once.
| Model | Catalog | Constellations | Debris | Regulatory | Market | Ground | Baseline | |
|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 Anthropic | alone | 67 / 11 | 51 / 12 | 40 / 7 | 35 / 3 | 37 / 15 | 18 / 23 | 100 / 0 |
| tools | 100 / 0 | 100 / 0 | 93 / 2 | 57 / 2 | 84 / 2 | 68 / 5 | 100 / 0 | |
| Claude Fable 5.1 Anthropic | alone | 64 / 7 | 53 / 21 | 45 / 10 | 35 / 5 | 44 / 17 | 27 / 23 | 100 / 0 |
| tools | 99 / 1 | 100 / 0 | 94 / 2 | 59 / 4 | 90 / 2 | 73 / 14 | 100 / 0 | |
| Claude Sonnet 5.5 Anthropic | alone | 58 / 24 | 49 / 12 | 45 / 17 | 28 / 10 | 41 / 27 | 18 / 27 | 100 / 0 |
| tools | 98 / 2 | 98 / 2 | 93 / 4 | 47 / 14 | 83 / 2 | 74 / 17 | 100 / 0 | |
| Claude Haiku 5.5 Anthropic | alone | 29 / 18 | 23 / 14 | 14 / 14 | 20 / 13 | 22 / 12 | 5 / 9 | 100 / 0 |
| tools | 91 / 1 | 82 / 12 | 87 / 1 | 36 / 11 | 82 / 2 | 53 / 14 | 99 / 1 | |
| Grok 4.7 xAI | alone | 47 / 0 | 19 / 0 | 12 / 5 | 28 / 0 | 20 / 5 | 14 / 5 | 96 / 0 |
| tools | 91 / 1 | 95 / 5 | 85 / 4 | 44 / 4 | 78 / 2 | 57 / 11 | 98 / 0 | |
| Gemma 4 12B Google · open | alone | 0 / 68 | 6 / 58 | 10 / 34 | 11 / 44 | 8 / 31 | 0 / 45 | 92 / 8 |
| tools | 78 / 15 | 42 / 41 | 52 / 18 | 13 / 26 | 47 / 36 | 19 / 58 | 89 / 3 | |
| gpt-oss 20B OpenAI · open | alone | 7 / 62 | 5 / 51 | 12 / 64 | 8 / 60 | 7 / 59 | 9 / 45 | 75 / 17 |
| tools | 76 / 10 | 29 / 36 | 44 / 19 | 13 / 25 | 46 / 21 | 23 / 18 | 85 / 7 | |
| DeepSeek-R1 14B DeepSeek · open | alone | 2 / 76 | 7 / 67 | 12 / 50 | 8 / 50 | 2 / 44 | 14 / 36 | 58 / 42 |
| tools | — | — | — | — | — | — | — |
Each cell: % right, then % wrong (the rest declined to answer). Color is the verdict: Trust it · Verify its answers · Don't use it. Hover for the 95% range.
closed-book MCP-grounded amber Δ = accuracy and trust moved in opposite directions
closed-book MCP-grounded
By job
Limits, harness differences and corrections: methodology. Earlier API leaderboards: round 2 · round 1.