- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies test-time adaptation of pretrained latent world models for goal-conditioned planning under distribution shift. It proposes Optimistic Test-Time Adaptation (OpTTA), which augments prediction loss on observed transitions with the goal-reaching cost of imagined candidate trajectories, differentiated with respect to the predictor parameters.
Critical strengths
-
The problem is relevant and the intervention is simple. Prediction accuracy on the agent’s own transitions need not imply effective planning. Incorporating the downstream planning objective into model adaptation is a useful direction.
-
The evaluation includes useful ablations and transparent limitations. The paper examines adaptation layers, optimism strength and scheduling, and candidate populations. The appendix provides results by corruption and severity, and the paper explicitly reports negative results on TwoRoom and the larger computational overhead for DINO-WM.
Critical weaknesses
The justification for optimizing the world model toward the goal under fixed candidate actions is insufficient. In Equation (7), candidate action sequences are held fixed while the predictor is updated to reduce their imagined distance to the goal. However, these candidates are not known to reach the goal in the real environment. An accurate prediction of an ineffective action sequence can therefore be penalized by the planning term, encouraging the model to make that sequence appear more successful without changing its actual consequences. I recognize that this bias is intentional and motivated by optimistic model estimation. Nevertheless, the paper does not sufficiently establish why this particular surrogate produces useful exploration or better action selection, rather than misleading predictions of action effectiveness. The prediction loss on a short buffer provides an anchor, but does not directly constrain the outcomes of unexecuted candidates. Figure 4 provides useful diagnostics, yet does not directly test this concern.
Questions for authors
Justification of the goal-directed model update. Candidate action sequences in Equation (7) are not guaranteed to reach the goal in the real environment. Why should their predicted outcomes be pushed toward the goal by updating the world model?
4 comments
Response to Reviewer mvpR (2/2)
3. An accurate prediction of an ineffective sequence can be penalized.
Yes, by design: this is the optimistic bonus of point 2, applied to effective and ineffective candidates alike. Table R1 measures it: OpTTA's update lowers the imagined cost of 78% of the elites that do not reach the goal, as it does for 70% of those that do (rows 1a–1b). CEM uses only the ordering of candidates, so a bonus that lowers both alike can mislead it only by changing that ordering, and row 2 shows no such harm on average: a goal-reaching candidate is ranked cheaper than a non-reaching one in 64.3% of pairs after OpTTA's update, against 62.5% after AdaJEPA's. Two mechanisms keep it in check:
- Eq. 8: it bounds the planning gradient on the adapted predictor block relative to the prediction gradient, $\Vert \alpha_k g_{\rm plan}\Vert \le \rho\sqrt{k/K_{\max}}\Vert g_{\rm pred}\Vert $; with in every result outside the ablation, the planning gradient never exceeds the prediction gradient, so the prediction loss stays the anchor of every update.
- Execution: once the planner selects an ineffective candidate that the update makes look effective, executing it adds new transition evidence to the prediction loss, which OpTTA's update still fits (Table R1, row 5); the better the model fits a transition, the less optimistic its prediction of it. The planner executes one step per replan, so this evidence accumulates over successive replans.
A planning term that lowers the imagined cost of effective candidates more than ineffective ones (row 1b) would be a better choice still; our design is a reasonable one that improves planning, and we do not claim it is optimal.
4. Figure 4 (Figure 5 in the revised manuscript) does not directly test this concern.
Agreed. Figure 4 measures prediction error and imagined cost, both inside the model; Table R1 tests the concern against real outcomes:
- Same starting point: at every update along AdaJEPA's trajectory, we apply both updates to the same predictor and buffer, one Adam step each as in the paper: AdaJEPA's with the prediction loss alone, OpTTA's with the planning term added, so the comparison isolates the objective from the trajectory.
- Real outcomes: every scored candidate is run in the simulator from the state it was planned from; the probe leaves the agent unchanged, so its episodes are identical to those of the run without it.
- The direct test (rows 1a, 2 and 5): prediction-only updates penalize good plans that OpTTA's update leaves alone, goal-reaching candidates are ranked cheaper at least as often, and its update still fits the executed transitions.
5. The short buffer does not supervise unexecuted candidates.
Agreed: neither AdaJEPA's nor OpTTA's objective directly supervises unexecuted continuations using the current buffer. This is where the two objectives differ:
- Prediction-only adaptation: when several models fit the buffer equally well, the prediction loss cannot choose between them, so which one prediction-only adaptation picks has nothing to do with the task.
- OpTTA: Eq. 7 breaks the tie toward the model under which the goal is cheaper to reach, the RBMLE principle; executing that model's plan then produces the evidence to confirm it or, through the prediction loss, rule it out (point 2).
- Guarantee and test: Appendix B proves that this limits repeated errors when the evidence is retained and plans are executed in full. OpTTA's one-step execution falls outside those conditions, and for a five-transition buffer the appendix's finite-buffer bound is weak, so we test it empirically: OpTTA's update still fits the executed transitions, and the next plan's success differs only within noise (rows 4–5).
The main results of this response are added to the revised manuscript. We also request you to take a look at our general response, which lists the new results (TS on further tasks, the Sandwich-Residuals comparison, persistent runs without resets and the equal wall-clock comparison) and the improvements made in the manuscript.
Follow-up on our response
Dear Reviewer mvpR,
Thank you again for your review. Our response and revised manuscript give a detailed justification for optimism together with a direct test of your concern (Table R1) and a supplementary formal analysis (Appendix B). The revised manuscript highlights these additions in blue. We would appreciate it if you could read the response and consider updating your assessment, and we are happy to answer any remaining questions.
Regards, Authors
Dear Reviewer mvpR,
As the discussion period closes today, we would like to note that our response and revision include a detailed justification for Eq. 7 along with background on optimism (Appendix B), and a matched single-update probe that tests the concern directly (Table 13). These additions further strengthen the paper. We hope that the revision provides the requested justification of the algorithm based on the optimism principle and would be grateful if you would revisit your assessment. Thank you!
Regards, Authors
Response to Reviewer mvpR (1/2)
Thank you for your review. We are glad you found the problem relevant, the algorithm simple and the ablations useful. In short: execution can correct an optimistic model error, but a pessimistic error can keep a good plan from ever being tested; Eq. 7 trades the second kind for the first. The revised manuscript adds a paragraph to Section 4 (Why add an optimistic term?) that justifies this design, and background on RBMLE in Appendix B.
1. Eq. 7 and CEM are the two coordinate steps of one RBMLE-inspired joint objective.
At fixed αk, the joint objective over the model parameters θ and CEM's sampling distribution π is
minθ,πLpred(θ;Bk)+αkEa∼πJ(a,θ).
Its two coordinate problems are:
minθLpred(θ;Bk)+Mαk∑m=1MJ(a(m),θ).
Minimizing over π first gives minθLpred(θ;Bk)+αkminaJ(a,θ), the RBMLE objective that Theorem B.2 analyzes. The candidates in Eq. 7 are therefore not arbitrary: they are CEM's current solution to the same objective.
2. Execution can correct optimistic errors, but not pessimistic errors on plans it never tries.
A plan whose imagined cost is too low tends to be selected and executed, and its observed transitions then enter the prediction loss. A plan whose imagined cost is too high may never be selected, and then nothing corrects it. Prediction-only updates can create this second kind of error on good plans; the planning term trades it for the first kind, which execution can correct.
This is easy to see in the two-armed bandit case. Suppose the prediction-only estimates of the two arms are μ^1 and μ^2. Then the optimistic estimates of UCB (Auer et al., 2002) are μ1UCB>μ^1 and μ2UCB>μ^2, so the estimates are optimistic for both the good and the bad arm. A bad arm that is played is eventually corrected under either algorithm, but a good arm that is never played, as can happen with prediction-only learning, is never corrected. In OpTTA, the planning term plays the role of the optimistic bonus.
We test this argument in a controlled experiment that compares a single OpTTA update with a single AdaJEPA update on the same data (Table R1). The results support both halves:
Table R1. One AdaJEPA update vs one OpTTA update, applied to the same predictor and buffer at each of the 1,884 updates along AdaJEPA's trajectory (LeWM PushT, 3 seeds); every plan is run in the simulator. Rows 1a–1b: the 30 best candidates of the executed plan's CEM population, split by whether they reach the goal. Row 2: pairs of a goal-reaching and a non-reaching candidate in which the goal-reaching one has the lower imagined cost. Rows 3–4: CEM re-solved at the next replan. Row 5: prediction loss on the segment just executed. ±: s.e. over seeds; p: Wilcoxon signed-rank test over episodes.
(continued in the next comment)