acceptodds
Under review as a conference paper at ICLR 2027

Diagnosing Faults in Reinforcement Learning Simulators with Canonical Polynomial Invariants

Abstract

A large literature incorporates physical structure into learned dynamics on the premise that models respecting the underlying physics should predict better. We test this premise using exact polynomial invariants recovered from trajectories and represented canonically as reduced Gröbner bases over ℚ. We find that exact polynomial invariants provide little additional benefit for residual-based downstream uses. On Gymnasium's Acrobot, consistency regularisation reduces algebraic residuals without improving long-horizon rollout fidelity, while one-step validation error is more strongly associated with long-horizon accuracy; similarly, shaping potentials recovered from systems with mass errors of up to 100% achieve similar learning performance to the physically correct potential. The main benefit observed here instead appears in diagnosis, where canonical representations permit exact ideal-equality and membership decisions, and where a reference generating set supplies a physical vocabulary that generic two-sample tests do not have: handed the same per-generator residuals, a Kolmogorov–Smirnov baseline localises almost as well, so the advantage comes from the generating set and not from the algebra that reads it. We develop a two-level diagnostic comprising screening, which identifies the violated physical constraint, and attribution, which recovers the faulty invariant and identifies the responsible physical parameter. Normal-form deflation removes algebraically trivial multiples, while quotient-space recovery avoids tolerance-based nullspace selection. Across fifteen injected faults, screening localises all fifteen when no observation noise is added, with no observed false alarms. Attribution identifies the responsible parameter in all seven single-parameter faults. Paired difference tests detect all faults but provide no constraint-level localisation, showing that the advantage is localisation and not detection. Finally, applying the diagnostic to 350 release pairs across eleven RL environments finds no evidence of changed simulator dynamics, but exposes a stale forward-kinematics read in Reacher's observation function, an integrator-drift ceiling on Acrobot, and a chaotic-amplification limit on comparisons in InvertedDoublePendulum. In conclusion, exactness is more useful for diagnosing violations of physical constraints than for improving prediction.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, simulator diagnostics, world models, polynomial invariants, Gröbner bases, fault localisation, system identification, physics-informed learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_SJLZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

My comments are outlined below.

  1. Overall, the paper is likely to be AI-generated and is more like an experimental report, without a contribution to ML frameworks or practical implementation.

  2. The experiment focuses on simulation/simulator. Even if the conclusion is right, it will face a challenging Sim2Real gap. This has been a significant effort to make RL safe for real physical systems, with the capability of tolerating learning faults, Sim2Real gaps, and distribution shift. The paper lacks such comparisons.

  3. The 9-page formal content does not include formal statistical results.

  4. What is the formal definition of fault? Is it the same as a safety violation in the studies of safe RL?

Critical strengths

The paper is likely to be an AI-generated experimental report. I did not find any innovations from the perspective of ML frameworks or practical implementation in real physical engineering systems.

Critical weaknesses
  1. Overall, the paper is likely to be an AI-generated experimental report, without a contribution to ML frameworks or practical implementation.

  2. The experiment focuses on simulation/simulator. Even if the conclusion is right, it will face a challenging Sim2Real gap. This has been a significant effort to make RL safe for real physical systems, with the capability of tolerating learning faults, Sim2Real gaps, and distribution shift. The paper lacks such comparisons.

  3. The 9-page formal content does not include formal statistical results.

  4. What is the formal definition of fault? Is it the same as a safety violation in the studies of safe RL?

Questions for authors

See Critical Weaknesses.

Reviewer_Ds3kOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors consider learning canonical forms that respect conservation laws as a form of physics-informed AI for detecting faults in, e.g., simulation models. They further test the claim that models incorporating physics knowledge can provide better predictions, but find that this doesn't hold in their experiments. The diagnosis tool further finds issues in commonly used simulators.

Critical strengths

Using canonical forms for fault diagnosis seems like an interesting use case for physics-informed AI. It seems intuitive that imposing such a form simplifies diagnosis. The issues found in widely used simulation environments are potentially interesting.

Critical weaknesses

The paper is not very accessible. It lacks clearly defined problem settings; many claims are made without references or justification, and the Gröbner basis, the paper's key ingredient, is never formally defined. After reading the paper, it is largely unclear to me how the algorithm works and what is actually happening in the experiments.

Questions for authors
  1. Can you provide a clear step-by-step explanation of how the algorithm works, including definitions of the Gröbner basis and other required tools?

  2. What exactly is happening in the prediction performance evaluation? Are the 48 models trained with a fixed set of parameters and then evaluated on the same set?

  3. What is the potential impact of the issues you found in the simulation environments?

Reviewer_XwwKOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates whether integrating exact physical structure into learned dynamics improves prediction accuracy, and finds that exact polynomial invariants provide little benefit for residual-based downstream uses such as consistency regularisation of world models and potential-based reward shaping. Instead, the authors propose a two-level algebraic diagnostic system for RL simulators based on polynomial invariants. In a screening step, violations of physical constraints are detected and the violated constraint is localised; a subsequent attribution step recovers the invariant of the faulty system, fits a parametric template to it, and identifies the responsible physical parameter. The diagnostic is evaluated on fifteen injected faults in Acrobot and Reacher and applied to 350 release pairs of Gym/Gymnasium/MuJoCo across eleven environments, where it finds no changed dynamics but exposes an observation inconsistency in Reacher, an integrator-drift floor in Acrobot, and a chaotic-amplification limit in InvertedDoublePendulum.

Critical strengths

None

Critical weaknesses
  • Clarity: the paper is hard to follow. It would help to structure the paper along the stated contributions; currently the order of Sections 4–6 is roughly the reverse of the contribution list. The motivation contains a lot of repetition (the point that canonicity is needed only for equality decisions is restated in the abstract, introduction, Section 5 and Section 6.3), and many sentences place their key information at the very end. In addition, terminology is often used before it is introduced (e.g., "the screen" in Section 2), and much of the evidence required to follow the main text is deferred to the 30-page appendix.

  • The paper combines two narratives: a negative result on physics-informed prediction (Sections 7.2–7.3) and a diagnostic tool (Sections 6, 7.1, 7.4). The general conclusion that "enforcing physical structure" does not improve learned dynamics rests on a single environment (Acrobot), a single architecture (MLPs with a consistency penalty) and, for shaping, five seeds. Architectural approaches such as HNNs/LNNs, which are cited as motivation, are not tested. It is unclear whether this conclusion generalises beyond the studied setting.

Questions for authors
  • Could the authors clarify how the paper's two claims relate? Is the main contribution the negative result on prediction or the diagnostic, and would the paper be stronger if it focused on one of them?

  • Do the negative results in Section 7.2 also hold for structural approaches (e.g., Hamiltonian/Lagrangian networks) or for environments other than Acrobot?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.