acceptodds
Under review as a conference paper at ICLR 2027

SatelliteOpsGym: Benchmarking AI Agents from Fault Diagnosis to Closed-Loop Recovery in Satellite Operations}

Abstract

Autonomous satellite operations require intelligent agents to diagnose faults, execute interventions, and verify recovery through closed-loop interaction. However, the operational capabilities and failure boundaries of existing agents remain poorly understood. We introduce SatelliteOpsGym, an interactive satellite operations environment for systematically evaluating agents and supporting their future training and improvement. Built upon documented on-orbit faults, real multichannel telemetry, and causal fault graphs, the environment enables agents to investigate anomalies, execute constrained commands, observe their consequences, and verify recovery. Its operational fidelity is assessed through double-blind evaluation by certified satellite engineers. Experiments with eight frontier LLMs on 15 shared scenarios reveal a pronounced fault-to-recovery gap: although fault identification reaches 71.4% and command validity reaches 100%, root-cause Planning and Recovery achieve at most 39.3% and 32.1%, respectively. By exposing failures across the operational workflow, SatelliteOpsGym provides insights for developing more reliable satellite operations agents.Data and code are available at https://github.com/anonymous-satelliteopsgym/SatelliteOpsGym.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Interactive Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_L5aQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SatelliteOpsGym is an interactive benchmark for evaluating AI agents in satellite fault management. It combines documented fault cases, real nominal telemetry, a spacecraft knowledge graph, and operational commands. An LLM-assisted simulator generates command-dependent changes, allowing agents to investigate faults, intervene, and observe subsequent telemetry. The benchmark contains 300 tasks, with eight LLMs evaluated on 15 selected scenarios. The authors report a gap between fault recognition, command validity, and recovery, supported by replay tests and an expert-validation study.

Critical strengths

Moving from offline telemetry analysis to interactive intervention is a useful contribution. The environment requires agents to inspect command consequences through subsequent observations, which addresses an important limitation of static fault-detection datasets.

The implementation includes thoughtful controls: private fault states, frozen recovery targets, typed commands, state-dependent validation, and auditable trajectories. The distinction between command acceptance and actual improvement is particularly valuable.

The involvement of experienced satellite engineers strengthens the work. Assessing operational plausibility and recovery judgments is a meaningful step toward validating a benchmark in this domain.

Critical weaknesses

Several metrics measure narrower behaviors than their names suggest. Appendix B.2 awards full Observation credit for a committed assessment even when it is wrong. Diagnosis only checks whether an actionable fault is recognized; on the fault-only main evaluation, always declaring a fault would receive full credit. Verification gives full credit whenever an unsupported resolution claim is avoided, even without substantive verification. Planning measures pre-action root-cause localization rather than intervention planning. These definitions make the capability profiles and Overall Score difficult to interpret. They also weaken the diagnosis-to-recovery comparison, since the stages measure substantially different levels of difficulty.

The main evaluation is too limited to establish a reliable benchmark ranking. Only 15 of the 300 tasks are evaluated, covering four subsystems, with at most one valid trajectory per model–scenario pair. The selection criteria and coverage of the three task families are unclear. Appendix B.2 also allows metric-specific exclusions for missing evaluation artifacts, but the actual denominators and exclusion counts are not reported. With such a small sample, these differences could materially affect the rankings. Repeated runs, uncertainty estimates, and results on a larger shared task set are needed. Quantitative Safety results are also missing despite safety being a central motivation.

The validation does not yet establish the reliability of action-dependent simulation. The reported 70/70 expert agreement is encouraging, but agreement about the outcome of a generated trace does not establish that alternative commands would produce realistic consequences. Appendix C.2 describes operational-plausibility ratings without reporting their numerical results or inter-rater agreement. In addition, GPT-6-Astra supplies command transitions and RCA judgments while also being an evaluated agent. This makes sensitivity to simulator and judge choice important. The replay experiment should also clarify whether Command Inference is rerun or recorded proposals are reused; these test different forms of reproducibility.

Questions for authors
  1. How do scores and rankings change when Command Inference and the RCA judge are replaced independently with other models?
  2. Can you provide per-scenario scores, the denominator and exclusion reason for each metric, and rankings computed on a fully shared evaluable subset?
  3. Does the 70/70 expert agreement concern the exact three-level recovery grades or a binary outcome? Please also report the operational-plausibility ratings and agreement among individual raters.
1 comment

We thank the reviewer for these questions. We clarify the model-dependence analysis, metric comparability, and expert validation below.

Q1. How do scores and rankings change when Command Inference and the RCA judge are replaced independently with other models?

We investigated potential model-family coupling through a crossed evaluation with Fable and Astra, each serving as both an agent and the Gym backend for scenario generation and state-transition simulation. The four Agent/Gym combinations were tested on two fault cases. We used the trajectory-grounded Recovery Score, computed from fault-state transitions and predefined recovery criteria rather than an LLM judge, to avoid introducing another judge-model confound.

AgentFault caseSame-familyCross-family
FableStar-sensor0.50.0
FableSolar-array0.00.0
AstraStar-sensor0.00.0
AstraSolar-array0.01.0
Mean0.1250.250

Family-matched settings did not outperform cross-family settings on average. Of four paired comparisons, one favored the matched backend, two were tied, and one favored the cross-family backend. Rankings were also backend-sensitive: under Fable-Gym, Astra averaged 0.50 versus Fable's 0.25; under Astra-Gym, both averaged 0.00. These results do not support a consistent same-family advantage.

We also compared one-step transitions under teacher forcing: both simulators received the same frozen scenario, audited pre-action state, and command, without propagating either model's predictions. They agreed on most high-level actions, including POWER_CYCLE, COMMUNICATION_RESET, and SET_SUN_SENSOR OFF. For RESELECT_ATTITUDE_SOURCE, Fable predicted recovery whereas Astra predicted no effect; the same discrepancy appeared in Fable- and Kimi-originated traces. This points to action-specific transition semantics rather than exclusive preference for Fable agents. Because the study covers only two fault cases and changes the Gym backend rather than substituting Command Inference and the RCA judge independently, it offers preliminary evidence on backend coupling, not a complete isolation of either module's effect. We will clarify this scope in the revision.

Q2. Can you provide per-scenario scores, the denominator and exclusion reason for each metric, and rankings computed on a fully shared evaluable subset?

All eight agents were evaluated on the same 15 scenarios under identical frozen initial conditions. Effective denominators can differ because some evaluation artifacts are unavailable and certain metrics apply only to specific trajectories. In particular, Execution applies only when a trajectory contains a command attempt. Incorrect or missing agent assessments receive the prescribed score rather than being excluded for poor performance. We will report per-scenario scores, each metric's effective denominator and exclusion reasons, and rankings recomputed on a common subset with valid records for every compared agent. We will compare these rankings with the originally reported results to make the impact of exclusions explicit.

Q3. Does the 70/70 expert agreement concern exact three-level recovery grades or a binary outcome? Please also report operational-plausibility ratings and agreement among individual raters.

The reported 70/70 refers to exact agreement on the three-level Recovery grades—0 (no recovery), 0.5 (partial recovery), and 1 (predefined recovery target achieved)—not merely binary recovery status. In the double-blind validation, six certified satellite operations engineers, each with at least five years of experience, evaluated 70 anonymized agent trajectories across ten fault scenarios. Each trajectory was independently reviewed by at least three experts. For all 70 trajectories, the experts' majority grade exactly matched the SatelliteOpsGym outcome. This is 100% agreement between the majority judgment and Gym, not unanimous agreement among individual raters. We will state this distinction and describe the validation protocol more precisely in the revised manuscript.

Reviewer_zz6wOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SatelliteOpsGym, an interactive benchmark for evaluating AI agents on satellite fault management from anomaly observation and diagnosis to intervention and recovery verification. The benchmark contains 300 tasks derived from 135 documented fault events. The authors evaluate eight frontier LLMs on a shared subset of 15 scenarios and report a substantial fault-to-recovery gap.

Critical strengths
  1. A meaningful interactive formulation of satellite fault management. It goes beyond static telemetry-based anomaly detection by constructing a stateful observation–diagnosis–planning-action–verification loop, with recovery assessed as a separate operational outcome.

  2. High-fidelity, aerospace-grade data sources. The paper incorporates authentic historical telemetry data from low-Earth orbit (LEO) satellites, alongside 135 officially documented cases of in-orbit failures.

Critical weaknesses
  1. The definition of the Overall Score is not fully aligned with the central goal of evaluating closed-loop recovery. The Overall Score averages Observation, Diagnosis, Planning, Execution, and Verification, while Recovery does not contribute to the final score. Given the emphasis on closed-loop recovery as a key end-to-end outcome, the interpretation of an Overall Score that excludes Recovery requires clarification.

  2. There is potential model–benchmark and model–evaluator coupling in the experimental pipeline. Claude Fable 5 is used for scenario generation, while GPT 6 Astra is used for command simulation and root-cause assessment. Both models are also included among the evaluated agents. It raises concerns about whether the benchmark dynamics or evaluation criteria may be better aligned with these model families.

  3. The evaluation coverage and sample accounting are insufficiently justified. The main experiments cover only 15 scenarios selected from the 300-task benchmark, without sufficiently explaining the selection criteria or task-family coverage. These omissions make it difficult to assess the representativeness of the subset and the robustness of the model comparisons.

  4. The failure analysis lacks quantitative support. Section 4.3 identifies three recurring failure modes but does not describe the trajectory-categorization procedure or report their frequencies and distribution across models or scenarios. The representative cases illustrate plausible failure mechanisms, but provide limited evidence about their prevalence or how broadly they explain the diagnosis-to-recovery gap.

Questions for authors

Appendix C.2 describes a five-point expert assessment of operational plausibility, but the resulting scores are not reported. It is also unclear whether simulator-generated responses were manually validated by domain experts.

3 comments

We thank the reviewer for the constructive questions. We provide clarifications and additional evidence below.

W1. Overall Score excludes Recovery

The Overall Score measures procedural performance across five operational stages, whereas Recovery independently measures whether valid agent actions achieve the predefined best-achievable target. We separate them because procedural competence does not necessarily imply successful recovery, and vice versa. Combining them would obscure this distinction. We will rename the Overall Score as the Process Score for clarity.

W2. Potential model–benchmark and model–evaluator coupling

We operationalize potential coupling as a possible same-family advantage: whether an agent systematically scores higher when evaluated by a Gym backend from the same model family. We conducted a crossed evaluation using Fable and Astra as both agents and Gym backends. The Gym model drives scenario generation and state-transition simulation, yielding four Agent–Gym configurations.

We use the trajectory-grounded Recovery Score as the primary outcome. It is computed deterministically from authoritative fault-state and effect transitions. We exclude the LLM-judged root-cause score from this analysis to avoid an evaluator-model confound. Both agents were evaluated on two fault cases under both Gym configurations.

AgentFault caseSame-family GymCross-family GymSame − cross
FableStar-sensor fault0.50.0+0.5
FableSolar-array fault0.00.00.0
AstraStar-sensor fault0.00.00.0
AstraSolar-array fault0.01.0−1.0
Mean0.1250.250−0.125

Same-family configurations did not outperform cross-family configurations on average. The paired effect was positive once, zero twice, and negative once, indicating case-dependent rather than consistently family-matched effects.

We also performed teacher-forced one-step comparisons, providing both simulators with the same frozen scenario, audited pre-action state, and action. Candidate outputs were not propagated to later nodes. The simulators agreed on POWER_CYCLE, COMMUNICATION_RESET, and SET_SUN_SENSOR OFF. For RESELECT_ATTITUDE_SOURCE, Fable predicted success whereas Astra predicted no effect. However, the same disagreement occurred for both Fable- and Kimi-originated traces. Since Kimi is outside the Fable family, this result is more consistent with command-specific semantics than same-family favoritism.

Overall, we observed no systematic same-family advantage. Given the limited number of cases, this provides evidence against a consistent family-favoring effect rather than proof of its absence.

W3. Limited evaluation coverage and scenario selection

The benchmark contains 300 interactive tasks derived from 135 documented satellite faults. Experienced satellite-operations engineers selected 15 scenarios to maximize coverage of subsystems, fault categories, recovery targets, operational constraints, and cross-subsystem propagation. The subset covers AOCS (8), EPS (4), and OBC (3); five fault categories; and resolved (7), mitigated (4), and isolated (4) outcomes. We will clarify the selection procedure and acknowledge the limitations of this focused subset.

W4. Insufficient quantitative failure analysis

We identify three recurring failure patterns: incorrect root-cause localization, ineffective causal command selection, and insufficient recovery verification. Experienced satellite-operations engineers also reviewed agents' intervention and fault-handling decisions. In addition, six certified satellite engineers evaluated 70 independent trajectories in a double-blind study.

These expert assessments do not directly establish the prevalence of the three failure modes. We will clarify the trajectory-categorization criteria and report failure frequencies and distributions across models and scenarios.

Q1. Expert plausibility ratings and validation of simulator-generated responses

Six certified satellite-operations engineers, each with at least five years of experience, evaluated 70 anonymized trajectories across ten fault scenarios. Each trajectory was independently reviewed by at least three experts for root-cause identification, recovery outcome, and operational plausibility.

Plausibility was rated from 1 (highly implausible) to 5 (highly plausible) based on technical validity, causal consistency, and operational feasibility. Expert majority judgments agreed with SatelliteOpsGym's recovery outcomes in 70/70 trajectories, while simulated outcomes matched historical records in five representative cases. We will report the mean plausibility score of 3.65 ± 0.73 out of 5.00 and clarify that this validation concerns complete trajectories rather than every possible command-dependent transition.

Thank you to the authors for their response and clarifications.

My major concern, also raised by the other reviewers, is the small evaluation sample (15 out of 300 tasks) . While the expert-selected subset considers coverage across several dimensions, this does not establish its representativeness of the full benchmark or the stability of the model rankings.

Thank you very much for your reply. We would like to clarify that our goal in releasing this benchmark is not to rank general-purpose AI models. What we really want to understand is whether current AI agents can handle satellite operations and maintenance tasks, and how well they can do so. We also understand that the rankings may change with different scenarios or datasets, so we do not consider the current rankings to be fixed or generalizable.Our main motivation is to provide the AI community with a dataset and an evaluation environment for satellite AIOps. We hope more researchers will become interested in this problem and develop new algorithms or agents that can be tested on our benchmark. As space technology continues to develop, we believe satellite AIOps is a relatively new but important research direction with many interesting problems worth exploring. Regarding the 15 scenarios, they were selected together with experienced satellite operations experts to cover typical fault cases. The experts also carefully reviewed the agent trajectories in all 15 scenarios. This process required a lot of time and effort from the experts. If we included too many scenarios, it would be difficult for them to evaluate each trajectory with the same level of care. This is why we decided to start with 15 representative scenarios. Of course, we agree that 15 scenarios are not enough to cover the full complexity of real satellite operations, and we see this as a limitation of our current work. More importantly, we decided to release this benchmark because we believe AI for satellite operations is becoming increasingly relevant, but the AI community still lacks publicly available datasets and evaluation environments for this area. We hope our work can be a starting point and encourage more researchers to study this problem, propose better methods, and evaluate their agents in a common environment. We fully agree that more scenarios would make the benchmark more useful. We will continue working with satellite operations experts and do our best to release more scenarios in the future. Thank you again for your time and valuable feedback. We really appreciate the opportunity to explain our motivation more clearly.

Reviewer_okedOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

As satellite constellations grow, the associated management overhead (e.g., telemetry interpretation, operational decision-making, and fault recovery) increases. A promising solution is onboard autonomous agents that mitigate the need for human monitoring and dependence on ground intervention. However, the capabilities of current frontier agents in this closed-loop satellite operations management setup remain poorly understood. The paper introduces SatelliteOpsGym, a first-of-its-kind interactive environment for testing agents' abilities to handle real-world satellite management scenarios. Evaluation of eight frontier LLMs on this benchmark reveals a significant fault-to-recovery gap where agents reliably identify faults and submit valid commands but struggle to find the root cause and fix/mitigate it. By exposing this performance gap, SatelliteOpsGym poses an exciting opportunity to improve agentic performance on a socially relevant, real-world task.

Critical strengths

The authors identify a significant gap in current agentic benchmarks for autonomous satellite operations. Specifically, existing benchmarks assume static conditions when agents are asked about historical multivariate observations. Such benchmarks, while useful, cannot test agents’ ability to understand how their actions affect downstream conditions, a crucial capability for fully autonomous closed-loop monitoring of satellite systems. The proposed SatelliteOpsGym represents a step towards interactive satellite operations benchmarks. In particular, the benchmark not only addresses this gap but also includes some human verification to improve confidence in the proposed design choices. ​ The evaluation introduces per-stage (e.g., observation, planning, execution, etc.) metrics to analyze which parts of the satellite operations monitoring and mitigation processes are difficult. A clear fault-to-recovery gap arises where agents can reasonably identify faults and even propose valid commands but cannot reliably find the causal root cause of different faults and fix them. The poor planning and recovery results point to frontier agentic reasoning models’ poor long-horizon planning capabilities.

Critical weaknesses

The authors provided an anonymized human-expert study to motivate the efficacy of their proposed environment with LLM-assisted portions (Line 88). However, the evaluation itself relies on LLM-as-judge evaluation metrics that could be noisy or introduce biased estimates, especially since the same model is being used as both baseline and judge (Line 309). Therefore, a human-expert study verifying the LLM-as-judge evaluation metrics (e.g., Plan and recovery) seems necessary. I am particularly interested in how often LLM-as-judge scores align with satellite experts for the “planning metric,” because the “fault-to-recovery gap” contribution relies on this metric’s scores. ​ Section 4.3 (Line 359) introduces a brief failure analysis that makes many claims about agents’ failure modes in this task. However, concrete experiments or other evidence is missing to substantiate the three major claims. First, the authors claim “agents often mistake downstream symptoms for root causes” (Line 362). This conclusion seems reasonable, but no evidence is provided, and a quantitative measure of what “often” means would make this more scientific. Similarly, the second claim that “command selection relies on relevance instead of causal effects” (Line 363) again sounds reasonable, but qualitative and quantitative evidence is needed. Lastly, the third claim regarding the lack of re-checking stable trajectories to ensure sustained improvement for recovery is not properly supported with qualitative and quantitative evidence. With proper substantive evidence, these claims can be informative contributions.

Questions for authors

I have a few clarification questions.

Why were only 15 scenarios selected from the 300-task benchmark? How were they chosen? (Line 301)

What exactly is the difference between the two failure indicators used in Obs and Ver metrics, and how are they implemented? (Line 278-281)

What are the LLM-as-judge system and user prompts?

What exactly does it mean for an agent to “reach a committed assessment?” (Line 321)

2 comments

We thank the reviewer for the constructive questions. We clarify the scenario selection, metric definitions, and LLM-based evaluation protocol below.

Q1. Why only 15 of 300 scenarios, and how were they selected?

SatelliteOpsGym contains 300 interactive tasks derived from 135 documented satellite fault events. For the in-depth study, experienced satellite operations engineers selected 15 representative scenarios based on fault diversity, operational relevance, and complexity. The subset covers four subsystems—EPS, AOCS, TCS, and TT&C—and different fault mechanisms and recovery challenges. The engineers also assessed the operational plausibility of agents' intervention decisions and fault-handling actions. We will clarify the expert-guided selection process and scenario coverage in the revision.

Q2. How do the failure indicators in Obs and Ver differ?

The two indicators capture distinct failure modes at different stages of satellite operations. Observation is defined as Obs(τ)=1−uτ\mathrm{Obs}(\tau)=1-u_\tau, where uτ=1u_\tau=1 indicates that the agent failed to produce an explicit, committed assessment of the observed satellite state before intervention, and uτ=0u_\tau=0 otherwise. This metric concerns whether an assessment was completed, regardless of its correctness, which is evaluated separately by Diagnosis.

Verification is defined as Ver(τ)=1−vτ\mathrm{Ver}(\tau)=1-v_\tau, where vτ=1v_\tau=1 when the agent declares successful recovery despite the underlying fault remaining active or latent in the simulator, and vτ=0v_\tau=0 otherwise. Unlike Observation, which examines the agent's pre-intervention assessment, Verification compares its recovery declaration against the simulator's underlying fault state. We acknowledge that the current Verification metric primarily measures the avoidance of false recovery claims rather than the completeness of active verification. We will clarify this distinction in the revised manuscript.

Q3. What System and User Prompts are used by the LLM judge?

The judge evaluates pre-action root-cause hypotheses against reference fault information, including the frozen causal DAG, and the evidence visible before action. The original Planning metric uses three levels corresponding to incorrect, partially correct, and correct root-cause localization. The prompts are provided below.

System Prompt

You are an impartial satellite-operations benchmark judge. Evaluate only the
pre-action root-cause analysis against the frozen fault DAG and evidence visible
to the Agent before it acted. Never infer credit from command outcome or later
telemetry. Equivalent engineering terminology is valid. Return JSON only.

User Prompt

{
  "task": "Score causal-chain reconstruction, exact root localization, visible-evidence grounding, and calibrated uncertainty separately. Also judge whether visible evidence could uniquely identify the frozen root; do not reward guessing hidden truth when it is not identifiable.",
  "rubric": {
    "causal_chain": "1 correct direction and major links; 0.5 partly correct; 0 wrong/unsupported",
    "root_localization": "1 exact/equivalent root; 0.5 correct subsystem or bounded candidates; 0 wrong/absent",
    "evidence_grounding": "1 claims use visible evidence; 0.5 mixed support; 0 relies on hidden/unobserved facts",
    "uncertainty_calibration": "1 uncertainty matches evidence/identifiability; 0.5 incomplete; 0 overclaims or is unjustifiably vague"
  },
  "reference": {
    "root_fault_nodes": "...",
    "downstream_fault_nodes": "...",
    "causal_edges": "...",
    "anomaly_description": "..."
  },
  "prediction": "...RCA generated by agent...",
  "visible_evidence": "...",
  "output_schema": {
    "dimensions": {
      "causal_chain": "exactly one of 0, 0.5, 1",
      "root_localization": "exactly one of 0, 0.5, 1",
      "evidence_grounding": "exactly one of 0, 0.5, 1",
      "uncertainty_calibration": "exactly one of 0, 0.5, 1"
    },
    "root_identifiability": "identifiable|partially_identifiable|not_identifiable from visible evidence",
    "rationale": "short justification"
  }
}

We will include these prompts in the revised manuscript and clarify which metrics rely on the LLM judge and which are computed deterministically from simulator states or command-validation results. We will also explain how the judge's outputs are mapped to the reported Planning score.

Q4. What is a “committed assessment”?

A committed assessment refers to an explicit and definite conclusion about the satellite's fault condition before intervention, rather than a tentative hypothesis or ongoing telemetry analysis. The criterion evaluates whether the agent has completed its assessment, not whether the conclusion is correct. Consequently, an incorrect but definite assessment satisfies the Observation criterion, whereas an inconclusive assessment does not. We will clarify this definition and its implementation in the revised manuscript.

I thank the authors for answering my clarification questions. However, I find that my enumerated critical weaknesses have not been addressed. Therefore, I will not change my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.