acceptodds
Withdrawn from ICLR 2027

Consequence Audits for Regression Diagnosis in Learned Dynamics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A learned dynamics update can reduce prediction error while introducing errors that have larger effects on a downstream controller. We evaluate these effects by inserting a model prediction into a reference simulator and comparing paired finite-horizon continuations under a fixed controller. Each violation yields a reproducible consequence witness; comparing labels before and after an update identifies regressions. We also study how to obtain these witnesses from a finite model bank under a simulation budget, using the signed responses of queried predictions to prioritize subsequent queries. In a post-hoc analysis of a six-task repair study, 3,931 of 4,221 regressions under uniform repair have lower one-step mean squared error (MSE). A separate fresh-source experiment finds 7,286 failure sources versus 6,569 for MSE round-robin at the same simulation budget. This discovery advantage is not consistent across three new tasks, and witness weighting does not establish an incremental repair benefit. In a repeated-use test, consequence-based selection yields lower return than cost-matched selection rules. The audit thus exposes model-update regressions that prediction error alone can miss, while its selection performance depends on how the model is used.

Primary area:
learning on time series and dynamical systems
Keywords:
learned dynamics, model evaluation, regression diagnosis, model-based reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_LS7mOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the discrepancy between prediction accuracy and downstream control performance in learned dynamics models. The authors argue that reducing mean squared error does not necessarily improve control outcomes, as changes in prediction error directions can lead to more harmful consequences. To evaluate this issue, they introduce a consequence-based audit that compares finite horizon returns from true and predicted successor states under a fixed controller. They further propose a budgeted search method that uses Ridge Regression to approximate normalized consequence loss and prioritize subsequent simulation queries, supported by a local Taylor approximation bound. Experiments across several continuous control tasks demonstrate that model updates can reduce MSE while introducing new violations. The proposed method improves failure discovery in the initial task suite, although its advantages are less consistent on new tasks and do not translate into clear improvements in model repair or repeated use selection.

Critical strengths

The paper seemingly addresses an important limitation of evaluating learned dynamics models solely through prediction error magnitude. Even when MSE decreases, changes in the direction of prediction errors can lead to worse downstream control outcomes. I find this observation meaningful, as prediction accuracy alone does not fully capture the consequences of model errors in decision-making systems. The experiments provide concrete examples that support this concern and motivate evaluating learned dynamics beyond MSE.

Critical weaknesses
  1. Connection between the proposed consequence loss and actual control performance.

The proposed consequence loss compares returns from true and predicted successor states under the same fixed controller. However, these returns can differ even when the controller takes identical actions in both continuations. Therefore, the metric does not directly measure the performance loss caused by using an incorrect model prediction in actual control. Furthermore, the one sided violation criterion allows large negative consequence losses to pass, even though such values may indicate substantial discrepancies between predicted and true states. Although the authors explicitly define their objective as conditional deterioration, its relationship to practical model reliability remains unclear.

  1. Applicability of the theoretical approximation.

The theoretical analysis relies on a twice differentiable consequence response with bounded curvature. However, some of the most consequential model errors may occur near action switching regions, termination boundaries, or other discontinuities. These cases are not covered by the proposed Taylor approximation bound. Moreover, the required gradient and curvature bounds are not estimated in practice, and the analysis does not establish a guarantee on violation discovery. Consequently, the theoretical results provide limited justification for the algorithm in the settings where consequence auditing may be most important.

  1. Methodological novelty.

The proposed acquisition method combines residual magnitude, geometric exploration, and local Ridge Regression to prioritize simulation queries. These are relatively standard techniques, and the approximation bound follows from Taylor expansion and basic Ridge Regression properties. While their application to consequence auditing is reasonable, the additional methodological contribution is not entirely clear, especially given that simpler response based alternatives outperform the proposed method on the new tasks.

  1. Evidence of downstream benefits

The experiments provide convincing examples of model updates that reduce MSE while introducing new consequence violations. However, the proposed acquisition method does not demonstrate consistent advantages on new tasks. Furthermore, consequence based repair fails to establish a significant incremental benefit over simpler repair strategies, while consequence based model selection produces lower returns than the baselines in the repeated use evaluation. These results substantially limit the practical implications of the proposed method beyond regression diagnosis.

Questions for authors
  1. What happens if the consequence loss is evaluated using its absolute value?

The current violation criterion considers only Lsm>τsL_{sm}>\tau_s, allowing arbitrarily large negative consequence losses to be treated as acceptable. However, a large negative loss also indicates a substantial discrepancy between the reference and predicted continuations. Although Appendix I.5 provides a post-hoc sensitivity analysis using a two-sided criterion, how would replacing LsmL_{sm} with ∣Lsm∣|L_{sm}| affect the proposed acquisition strategy and its failure discovery performance?

  1. How does the proposed approximation behave when the consequence function is not twice differentiable?

The theoretical approximation assumes that gg is twice differentiable with bounded curvature. However, consequential prediction errors may arise near action switching thresholds, termination boundaries, or other discontinuities, where these assumptions may not hold. How does the proposed acquisition method behave in such cases? In particular, what theoretical or empirical evidence supports the reliability of the approximation when the smoothness assumptions are violated?

Reviewer_Qm4GOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates a problem in dynamics learning: an update to a dynamics model may reduce prediction MSE while nevertheless leading to a lower final return. The authors also propose a model repair approach to address this issue.

Critical strengths

Prediction MSE does not necessarily align with downstream policy performance, so the underlying concern is valid and worth investigating.

Critical weaknesses
  • Problem formulation. Although the mismatch between prediction MSE and policy performance is a legitimate concern, I have serious reservations about the specific formulation studied in this paper.

    First, the paper assesses performance degradation as follows: starting from a real state and a given action, it obtains a predicted next state from the world model and a reference next state from the simulator. It then compares the subsequent returns obtained by rolling out from these two states under the true simulator dynamics. In my view, this formulation does not adequately capture the relationship between dynamics model quality and policy value. Instead, it primarily measures the value difference between two candidate next states. Under this criterion, a model could simply predict a high-value state, regardless of whether that state is dynamically plausible, and obtain a positive value gain. Such a gain would say little about the quality of the learned dynamics.

    More fundamentally, the effect of dynamics errors on policy performance depends on how the model is used. For example, some model-based reinforcement learning methods use the model to generate additional training samples. In this case, its effect on the learned policy cannot be directly established through the proposed simulator-based comparison. In planning-based methods, the model may instead guide the selection of actions that move the agent closer to a goal, requiring a different assessment of how prediction errors affect decisions. Different ways of combining models and policies therefore induce different relationships between dynamics errors and policy performance. I do not believe the proposed formulation is justified as a general measure of this relationship.

    Second, if a simulator is already available, what necessitates learning a separate dynamics model in this setting? Why not train the policy directly using the simulator? Even if the simulator is imperfect, approaches such as domain adaptation and simulator calibration could be considered. The paper therefore needs a clearer justification for the role of the learned model and the practical value of simulator-based model repair.

  • Methodology and integration into policy learning. Even setting aside the formulation concerns, a learned predictor of the performance impact of model errors may be effective in a restricted setting, but its generalizability remains unclear. As discussed above, this relationship depends on the environment and on how the dynamics model interacts with the policy. A predictor may overfit data collected at a particular training epoch, without generalizing to other model checkpoints, model–policy integration schemes, or environments.

    It is also unclear how the proposed repair mechanism should be integrated into dynamics model and policy learning. For example, how many dynamics updates should be performed before invoking model repair? Applying repair after every update would require justification, particularly regarding computational cost. A complete treatment should specify a coherent training procedure that combines model learning, policy learning, and model repair. Given a simulator and a dataset, this procedure should ultimately produce a policy trained or operated using the learned dynamics model, whose performance can be directly evaluated.

  • Experimental evaluation. Following the methodological concerns above, the central evaluation should establish whether the proposed approach improves the performance of a complete model-based reinforcement learning or planning system. However, most of the reported experiments are diagnostic analyses and do not directly evaluate the resulting policy’s performance. Consequently, the experiments do not adequately establish whether the proposed repair method delivers meaningful improvements in downstream control.

Questions for authors

See Weaknesses.

Reviewer_CXAiOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors note that in optimal control problems, reducing the prediction error of a transition model may not necessary lead to improved rewards in the actual environment. They propose alternate metrics and methods for measuring predictor improvements with respect to the reward gained by the controller.

Critical strengths

The problem under study is important: it is true that improved state predictions do not necessarily result in better control, and methods for improving control or evaluating transition predictor models with respect to the controller are valuable.

Critical weaknesses

The paper is very hard to parse. For instance, the introduction is essentially presenting the technical framework of the paper in prose rather than with equations. General background on the problem being studied is not given. The term "regressions" is used repeatedly before being defined in Section 2 (as far as I know this is not standard terminology in optimal control). By way of example, consider the first few sentences from Section 2:

"State and intervention. A source ss specifies a physical state xsx_s, a fixed policy πs\pi_s, and action asa_s. Let ff denote the ground-truth (GT) simulator transition with its auxiliary source context suppressed, and f^m\hat{f}_m the mmth predictor. Here a model prediction is its output under the stated task interface. After the reference transition, ηs\eta_s records the integration snapshot, policy clock and retained context common to both branches. The model branch replaces only the physical successor coordinates. Both branches then follow reference dynamics under πs\pi_s. For a valid initial state, JHπ(z;η)=∑h=0H−1rhJ^\pi_H(z;\eta) = \sum_{h=0}^{H-1} r_h is the undiscounted HH-step continuation return, with zero rewards after absorption.

In this partial paragraph alone, the reader must infer the following:

  • What is a simulator transition, what is its auxiliary source context, and what does it mean to suppress the source context?
  • What are the predictors f^m\hat{f}_m predicting?
  • What is a task interface?
  • What is an integration snapshot?
  • What is a policy clock?
  • What is the context retained by "both branches"? What are the branches themselves?
  • What does it mean for the model branch to replace the physical successor coordinates?
  • What is does the input zz represent in the definition of JHπ(z;η)J^\pi_H(z;\eta)? How do the inputs zz and η\eta impact the value of this function? What is the precise definition of rhr_h?
  • What is absorption?

For many of these questions, one could make a reasonable guess, but it is not the reader's job to be guessing what the authors meant and especially not at such a high frequency. Similar issues persist throughout the paper.

Questions for authors

Lines 214-215: "The prespecified interval levels for Studies 1–3 are 99.9609375%, 99.990234375% and 99.9951171875%." I am curious how you arrived at these percentages.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.