acceptodds
Under review as a conference paper at ICLR 2027

Before the Next Outcome: History-Aware Test-Time Adaptation for Latent World Models

Abstract

Test-time adaptation of latent world models typically relies on real transitions observed after the agent acts, leaving the model unadapted at the beginning of deployment. We study pre-outcome test-time adaptation: adapting a pretrained world model using only the target-domain history that is already available, be- fore the next real state is observed. This setting is challenging because historical variation mixes appearance nuisance with genuine dynamics changes, while repre- sentation and dynamics adaptation are coupled through a shared latent space. We propose HATTA (History-Aware Test-Time Adaptation), which exploits the two forms of evidence already present in the observed history. HATTA uses historical transition compatibility to select counterfactual visual hypotheses and construct a dynamics-aware representation correction, while observed local motion is used to calibrate the action-conditioned dynamics predictor. A latent-space coordina- tion step controls the accepted representation update and re-encodes the original history before dynamics adaptation. HATTA requires no new future outcome, source-domain data, reward, goal, or shift label. Experiments on PushT visual shifts and PointMaze dynamics shifts show that short histories provide useful pre- outcome signals for both visual and dynamics adaptation. The unified method retains most of the gains of the two specialized branches and achieves overall point estimates close to a feedback-based AdaJEPA reference without using new real transition feedback. These results suggest that target-domain history can sup- port meaningful world-model adaptation before additional interaction feedback becomes available.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Test-Time Adaptation, World Models, JEPA

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cc2GOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a method for test time adaption of latent world models that works even before "the next real state becomes available". Three mechanisms are presented: 1. The encoder is updated by sending the transition history through a filter bank, encoding the modified observations and choosing the the filter that the world model is best at predicting next latents. Then the encoder is trained toward predicting this latent from the original observation. 2. They limit the drift of the encoder through this mechanism by interpolating between the original parameters and the updated ones. 3. The target-local dynamics adaption aims at only updating the model such that it changes the prediction in the dimensions that the history actual informs (only the subspace defined through the transitions in the history). The method is evaluated on the PushT task and PointMaze and compared against an AdaJEPA variant.

Critical strengths
  • Limiting the dynamics adaption to what is supported by the history makes sense to me. I think using the subspace defined by the latent transitions is the right move.
Critical weaknesses

I want to preface this by saying I am not familiar with test time adaption, but with world models in general. For me the paper was hard to read and I think a clearer writing could greatly improve it.

Here are my main concerns:

  1. The method seems very hand-crafted/engineered: Especially two mechanisms rely on assumptions that may not generally hold: 1. The encoder adaption mechanism uses a set of visual hypotheses. I do not see any justification, why specifically these hypotheses. 2. The latent space coordination is mainly a hyper parameter on adaption strength and the method seems to be rather sensitive to its value. Because of this I have doubts how generally applicable the method is and under which circumstances it can actually work.

  2. The experiments are not significant enough: Given that the mechanisms seems rather engineered to me, I would expect more convincing results. There is only a baselines for comparison on only two tasks. For PushT the improvement is less than 1% and for PointMaze the AdaJEPA is a tiny bit better even (also considering it is only three seeds). Given that the work overall does not give much new insight into test time adaption, but presents a method, that is not very well supported by the empirical results, this become a major blockage to recommend acceptance.

Questions for authors
  1. Can you give some additional details on the experiments? How long are the episodes? How often is the world model updated (also in comparison to the baseline V0-X1)?
  2. At several places the paper emphasizes that HATTA adapts before the next outcome arrives. Isn't this always true? Don't the existing approaches for TTA also do this, just in a different way? I really struggle to understand how this is a contribution or a differentiating property of HATTA.
  3. Is there any way to decide what the latent space coordination parameter ρ\rho should be before running the agent?
  4. Did you also try larger K? Isn't the history much larger in reality with many tasks or set of tasks spanning hundreds to thousands of transitions and thus, providing a much larger history to learn from?
Reviewer_sD2DOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper proposes a paradigm, referred to as HATTA, that uses existing observations to perform test-time training (TTT) for world models.
  • In a first of its kind, the authors propose to use TTT on historical observations before predicting the next states. The authors demonstrate its effectiveness on two benchmarks, previously set up by a similar work, AdaJEPA (Wang et al. 2026)
Critical strengths

Making use of historical observations: A very practical idea which takes into account past observations so the predicted future states can be better, and is also supported by the evaluation protocol, albeit not major performance gains.

Critical weaknesses

No time analysis: despite discussing pretrained world models that are ready for deployment in the paper, the authors fail to provide the additional time complexity that HATTA introduces. In my opinion, such an analysis for TTT is very important and is also analysed by baselines such as AdaJEPA.

Questions for authors

Could you please help clarify the computational cost and time cost to justify the gains shown in the quantitative results? Since the performance gain is not significantly larger, I expect the time and computational complexity to also be similar to the other methods, or only marginally higher for HATTA to be effective.

Reviewer_uALrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates test-time adaptation of latent world models using already observed target-domain history before the next real outcome becomes available. The authors propose HATTA, which uses historical transition compatibility to guide visual representation adaptation and observed local motions to calibrate the dynamics predictor. A latent-space coordination mechanism is further introduced to balance the interaction between these two adaptation processes. Experiments on PushT visual shifts and PointMaze dynamics shifts show modest improvements in planning performance, suggesting that historical information can provide useful adaptation signals without requiring additional environment interactions.

Critical strengths
  1. The authors recognize the interaction between representation changes and dynamics prediction in a shared latent space. The proposed framework attempts to coordinate both adaptation processes, which I think is a meaningful problem worth exploring.

  2. The paper evaluates visual-only, dynamics-only, and unified adaptation under different distribution shifts. Although the overall improvements are limited, I think the factor-specific results and the analysis of latent coordination provide some useful insights into the trade-offs between representation correction and dynamics calibration.

Critical weaknesses
  1. One question is that the proposed setting may not be fundamentally different from existing history-based world model adaptation. The observed history already contains real state-action transitions, which can be directly used to compute prediction errors and update the model. Therefore, I am not well convinced that adapting before the next outcome introduces a substantially new learning problem.

  2. In Section 3.4, the authors use observed movements from different historical states to calibrate the action-conditioned responses at the current state. However, the difference between two historical movements depends on both the applied actions and their initial states. I think this introduces a potential inconsistency in the supervision, since the method implicitly assumes that action-response relationships remain similar across these states. This assumption may not hold for nonlinear dynamics or contact transitions, where even a small state difference can lead to very different outcomes.

  3. One important motivation of this paper is that updating the visual encoder changes the latent space used by the dynamics predictor. However, the proposed coordination mechanism mainly interpolates between the original and visually adapted encoder parameters using a fixed coefficient ρ\rho. I wonder whether this actually addresses the underlying representation-dynamics inconsistency or simply reduces the strength of the visual update. Figure 4 shows that improving visual-shift performance generally comes at the cost of dynamics-shift performance, while the unified HATTA does not outperform either specialized branch in its corresponding setting. Furthermore, visual and dynamics shifts are evaluated on different environments, leaving it unclear whether the method can actually coordinate both adaptations when these shifts occur simultaneously.

Questions for authors
  1. Could the authors include a comparison with standard prediction-error adaptation using exactly the same historical transitions and computational budget? I think this would help better isolate the contribution of the proposed adaptation objectives.

  2. The proposed dynamics adaptation combines motion-scale matching, relative action-response matching, and orthogonal preservation. Could the authors provide an ablation of these three components? I would be interested in understanding which objective contributes most to the improvement and whether all three are necessary.

  3. In Section 3.2, the authors select visual transformations based on historical prediction compatibility. Could the authors discuss how sensitive the method is to the composition of the candidate transformation bank?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.