- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SatelliteOpsGym is an interactive benchmark for evaluating AI agents in satellite fault management. It combines documented fault cases, real nominal telemetry, a spacecraft knowledge graph, and operational commands. An LLM-assisted simulator generates command-dependent changes, allowing agents to investigate faults, intervene, and observe subsequent telemetry. The benchmark contains 300 tasks, with eight LLMs evaluated on 15 selected scenarios. The authors report a gap between fault recognition, command validity, and recovery, supported by replay tests and an expert-validation study.
Critical strengths
Moving from offline telemetry analysis to interactive intervention is a useful contribution. The environment requires agents to inspect command consequences through subsequent observations, which addresses an important limitation of static fault-detection datasets.
The implementation includes thoughtful controls: private fault states, frozen recovery targets, typed commands, state-dependent validation, and auditable trajectories. The distinction between command acceptance and actual improvement is particularly valuable.
The involvement of experienced satellite engineers strengthens the work. Assessing operational plausibility and recovery judgments is a meaningful step toward validating a benchmark in this domain.
Critical weaknesses
Several metrics measure narrower behaviors than their names suggest. Appendix B.2 awards full Observation credit for a committed assessment even when it is wrong. Diagnosis only checks whether an actionable fault is recognized; on the fault-only main evaluation, always declaring a fault would receive full credit. Verification gives full credit whenever an unsupported resolution claim is avoided, even without substantive verification. Planning measures pre-action root-cause localization rather than intervention planning. These definitions make the capability profiles and Overall Score difficult to interpret. They also weaken the diagnosis-to-recovery comparison, since the stages measure substantially different levels of difficulty.
The main evaluation is too limited to establish a reliable benchmark ranking. Only 15 of the 300 tasks are evaluated, covering four subsystems, with at most one valid trajectory per model–scenario pair. The selection criteria and coverage of the three task families are unclear. Appendix B.2 also allows metric-specific exclusions for missing evaluation artifacts, but the actual denominators and exclusion counts are not reported. With such a small sample, these differences could materially affect the rankings. Repeated runs, uncertainty estimates, and results on a larger shared task set are needed. Quantitative Safety results are also missing despite safety being a central motivation.
The validation does not yet establish the reliability of action-dependent simulation. The reported 70/70 expert agreement is encouraging, but agreement about the outcome of a generated trace does not establish that alternative commands would produce realistic consequences. Appendix C.2 describes operational-plausibility ratings without reporting their numerical results or inter-rater agreement. In addition, GPT-6-Astra supplies command transitions and RCA judgments while also being an evaluated agent. This makes sensitivity to simulator and judge choice important. The replay experiment should also clarify whether Command Inference is rerun or recorded proposals are reused; these test different forms of reproducibility.
Questions for authors
- How do scores and rankings change when Command Inference and the RCA judge are replaced independently with other models?
- Can you provide per-scenario scores, the denominator and exclusion reason for each metric, and rankings computed on a fully shared evaluable subset?
- Does the 70/70 expert agreement concern the exact three-level recovery grades or a binary outcome? Please also report the operational-plausibility ratings and agreement among individual raters.
We thank the reviewer for these questions. We clarify the model-dependence analysis, metric comparability, and expert validation below.
Q1. How do scores and rankings change when Command Inference and the RCA judge are replaced independently with other models?
We investigated potential model-family coupling through a crossed evaluation with Fable and Astra, each serving as both an agent and the Gym backend for scenario generation and state-transition simulation. The four Agent/Gym combinations were tested on two fault cases. We used the trajectory-grounded Recovery Score, computed from fault-state transitions and predefined recovery criteria rather than an LLM judge, to avoid introducing another judge-model confound.
Family-matched settings did not outperform cross-family settings on average. Of four paired comparisons, one favored the matched backend, two were tied, and one favored the cross-family backend. Rankings were also backend-sensitive: under Fable-Gym, Astra averaged 0.50 versus Fable's 0.25; under Astra-Gym, both averaged 0.00. These results do not support a consistent same-family advantage.
We also compared one-step transitions under teacher forcing: both simulators received the same frozen scenario, audited pre-action state, and command, without propagating either model's predictions. They agreed on most high-level actions, including POWER_CYCLE, COMMUNICATION_RESET, and SET_SUN_SENSOR OFF. For RESELECT_ATTITUDE_SOURCE, Fable predicted recovery whereas Astra predicted no effect; the same discrepancy appeared in Fable- and Kimi-originated traces. This points to action-specific transition semantics rather than exclusive preference for Fable agents. Because the study covers only two fault cases and changes the Gym backend rather than substituting Command Inference and the RCA judge independently, it offers preliminary evidence on backend coupling, not a complete isolation of either module's effect. We will clarify this scope in the revision.
Q2. Can you provide per-scenario scores, the denominator and exclusion reason for each metric, and rankings computed on a fully shared evaluable subset?
All eight agents were evaluated on the same 15 scenarios under identical frozen initial conditions. Effective denominators can differ because some evaluation artifacts are unavailable and certain metrics apply only to specific trajectories. In particular, Execution applies only when a trajectory contains a command attempt. Incorrect or missing agent assessments receive the prescribed score rather than being excluded for poor performance. We will report per-scenario scores, each metric's effective denominator and exclusion reasons, and rankings recomputed on a common subset with valid records for every compared agent. We will compare these rankings with the originally reported results to make the impact of exclusions explicit.
Q3. Does the 70/70 expert agreement concern exact three-level recovery grades or a binary outcome? Please also report operational-plausibility ratings and agreement among individual raters.
The reported 70/70 refers to exact agreement on the three-level Recovery grades—0 (no recovery), 0.5 (partial recovery), and 1 (predefined recovery target achieved)—not merely binary recovery status. In the double-blind validation, six certified satellite operations engineers, each with at least five years of experience, evaluated 70 anonymized agent trajectories across ten fault scenarios. Each trajectory was independently reviewed by at least three experts. For all 70 trajectories, the experts' majority grade exactly matched the SatelliteOpsGym outcome. This is 100% agreement between the majority judgment and Gym, not unanimous agreement among individual raters. We will state this distinction and describe the validation protocol more precisely in the revised manuscript.