acceptodds
Withdrawn from ICLR 2027

Retain Knowledge, Adapt to Change: When Does Fitting an Environment Code Help?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fitting a small environment code can improve a frozen world model after conditions change. The same update can also improve prediction when the dynamics have not changed, because the initial code was estimated imperfectly. We study how to distinguish these effects. Our evaluation combines provably inert physical interventions with matched code replay: codes updated on unchanged and changed probe trajectories are both scored on the same held-out changed trajectories. This defines a signed,change-specific prediction gain without subtracting errors from different test distributions. In a controlled comparison of discrete-time CAVIA and CoDA implementations and a jointly trained context model, ten training seeds and three shift amplitudes expose judgments that aggregate gains hide. On the rolling ball, all three obtain positive change-specific gains; a 6.5-percentage-point total-gain advantage of the joint context model over CoDA becomes an unresolved -0.8-point difference in change-specific gain. On the pendulum, the joint context model's change-informed update is harmful even after control transfer is accounted for. Complementary experiments across nine systems show when code fitting helps prediction and test whether modular structure is needed. Dynamically equivalent interventions also show why counting changed physical parameters cannot by itself explain adaptation difficulty. The contribution is a controlled way to measure what real-change probe observations are worth to an update under a specified intervention, code parameterisation, loss and update schedule, alongside the prediction accuracy that matters at deployment.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
world models, model-based reinforcement learning, online adaptation, context variables, system identification, evaluation protocols, change-specific gain, calibration versus adaptation, meta-learning, contact dynamics, negative results

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_5EiSOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to evaluate environment-code adaptation in world models when physical conditions change. The authors point out that prediction gains after an update can come either from actually learning about the changed dynamics or simply from ordinary recalibration of an imperfectly fitted initial code. To separate these two effects, they propose a matched code replay protocol that uses provably inert physical interventions as a control branch and scores both control and changed updates on the same real-change query trajectories. Using controlled experiments with CAVIA, CoDA, and a joint context model on simulated physical systems such as the rolling ball and pendulum, the paper shows that decomposing total gain into recalibration and change-specific components can change the comparison across adaptation methods and reveal harmful updates hidden by aggregate metrics.

Critical strengths

The idea of using physical laws to construct inert interventions (where nominal parameters are modified while the underlying dynamics remain unchanged, such as scaling g and L by the same factor in a pendulum) is clever and intuitive. Building on this control, the matched replay comparison cleanly separates ordinary recalibration from gains that actually come from changed observations. This decomposition is interesting and useful, as it reveals clear differences between adaptation methods on the pendulum that looking at aggregate gains alone would hide.

Critical weaknesses

My main concern is that the empirical validation is limited to simple toy settings. The primary experiments only test two 2D physical systems (a pendulum and a rolling ball) using small two-layer MLPs with simplified implementations of CAVIA and CoDA. Consequently, it is unclear whether the proposed measurement protocol changes any practical conclusions on standard benchmarks or larger world models.

Questions for authors

The paper states that the inert and unchanged branches are numerically identical and that the inert arm reproduces the unchanged arm bitwise. If so, why not directly use an unchanged branch as the control experiment, and what does the inert intervention add beyond that?

Reviewer_AMpqOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper appears to evaluate different methods for learning environmental representations in simulation by training two models (and a joint version) on Ball and Pendulum tasks to determine which methods can better learn environmental simulation parameters to the models.

Critical strengths

None noted.

Critical weaknesses
  • The overall goal and purpose of the work is quite unclear. It mentions learning codes however it does not appear there is any actual code generation in the work, as CoDA and CAVIA (and their joint-context implementation) appear to just be neural network architectures.

  • The work evaluates on two different tasks, Ball and Pendulum, without much discussion of their environment(s) (if that is important? the paper is unclear). These seem quite simple for the task being presented.

  • The paper introduces many variables (and general content) without previous or any description, e.g.:

    • equation 1 doesn't describe what H_t is, equation 2 has mostly undefined variables

    • G C and S are not defined for table 1

    • The paper introduces GRU on page 6 without describing what it is or citing it. Is this the GRU recurrent neural network model?

    • figure 1 does not really add anything beyond the text, and also includes more undefined variables

Questions for authors
  • See the first critical weakness -- it was very unclear what the actual purpose of the paper was. Are predictive neural networks being retrained online in a simulated system? When and why is this occurring?
Reviewer_TRmCOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose a novel method for evaluating world model adaptation. Their main goal is in disentangling model updates based adaptations to what changed in the environment vs general model improvements that would occur with a static environment. They apply this to 3 scenarios across rolling ball and pendulum systems.

Critical strengths

The paper isolates a genuine confound in the adaptation evaluation literature. The observation that a method with a weak initial state can appear to "adapt well" simply by recalibrating is not always obvious.

The general approach makes sense and addresses the motivating problem.

I appreciate the inclusion of confidence intervals as uncertainty quantification is rare in many papers.

Critical weaknesses

In general I feel that this paper is presenting a complicated solution to a simple problem. If the goal is in identifying if model updates are coming from responses to environment changes vs simply improving from observations then there are simpler ways to achieve this. Examples would be in monitoring to track data/gradient attribution and concept drift. There is existing work in this space but the authors do not engage with it. Even the initial premise of having trouble comparing the quality of updates between models because one starts from a better initial state is problematic to me as that simply points to either a limitation in experiment design not controlling for variables properly, or not accounting for initial state in the post-update analysis.

Questions for authors

Continual/online learning is a mature field, do you have comparisons to prior work in this space or justifications for why comparative baselines were not included?

Reviewer_fC6eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a new evaluation method for determining if improvement after a distribution shift means the model learned to respond to the shift properly. They do this by comparing the actual gain to the gain if the code was updated on unchanged dynamics instead of changed dynamics, and it shows that gains might be misleading and sometimes show ineffective adaptations.

Critical strengths

The paper addresses a problem involving true adaptation to distribution shifts or just recalibration. The experiments are structured properly, have a couple of adaptation methods and different environmental changes.

Critical weaknesses

I feel the experiments are limited to controlled scenarios and might not generalize to uncontrolled environments. The paper also shows that changing the adaptation schedule greatly affects performance. This is observed with CAVIA on the pendulum. I am concerned as to whether the observed differences are due to the adaptation methods or simply the chosen optimization settings.

Questions for authors

How can the proposed method be extended to realistic situations where the matched trajectories and inert interventions cannot be constructed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.