← All verdicts

Can you trust Gemma 4 12B for space work?

By , Founder and Self-Directed Space Researcher, Viventine Space Systems · updated

On its own, it earns "Trust it" on none of the jobs. Connected to Orbit Sentinel, trust it for baseline knowledge.

Job Closed-book With Orbit Sentinel tools n
right / wrong range declined right / wrong range declined
Catalog lookup
0 / 68
0–9 32
78 / 15
64–87 8 45×3
Constellation & fleet status
6 / 58
2–20 36
42 / 41
29–57 17 43×3
Debris & conjunction context
10 / 34
4–23 56
52 / 18
37–66 30 42×3
Regulatory & licensing
11 / 44
4–25 44
13 / 26
6–27 61 40×3
Company & market intelligence
8 / 31
3–22 61
47 / 36
32–62 18 41×3
Ground infrastructure
0 / 45
0–15 55
19 / 58
8–39 23 22×3
Baseline knowledge
92 / 8
67–99 0
89 / 3
70–97 8 23×3
Orbit Sentinel-only questions
0 / 38
0–23 62
75 / 14
48–91 11 13×3

% right / % wrong per cell; color is the verdict (green trust it, amber verify, red don't use). Range is the 95% interval for % right, counting each question once; n is questions × runs for the tools track.

How it was run
Open weights, run locally (Ollama on one RTX 4090)
Tool loop
SpaceBench tool loop
Runs
1 on its own, 3 with tools; rates pool all runs, ranges count each question once
Verdict rule
Trust it: wrong ≤ 5% and right ≥ 80%. Don't use it: wrong > 20%. Otherwise verify its answers.

Limits, harness differences and corrections: methodology.