acceptodds
Withdrawn from ICLR 2027

ErrataWM: Correcting Frozen World Models with a Non-Parametric Memory of Their Own Errors

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Latent world models enable strong planning, but once trained they are static: deployed in new environments with shifted dynamics they keep making the same systematic prediction errors, and those errors flow straight into the planner. The usual remedy, gradient-based test-time training, is expensive, rewrites the whole model, and tends to forget the environments it was already good at. We first establish an empirical premise: a frozen world model's errors are not noise but a function of the starting context, so that similar states yield similar mistakes. This holds broadly, across navigation, manipulation, and driving, five world models, and four architectures including regression, autoregressive, and video diffusion models, with context-specific predictability as high as 0.81. Building on this, we propose ErrataWM, a frozen world model paired with a non-parametric memory that turns its recurring errors into scope-gated correction prototypes and applies them at test time with no gradient updates. Because the memory operates in a shared frozen feature space, one model corrects heterogeneous world models across tasks. ErrataWM reduces one-step prediction error by a context-specific margin of 0.12 to 0.15 over a context-blind control, matches gradient-based adaptation without any weight update, and improves downstream success by 9 to 13 points on navigation and manipulation. A single shared memory corrects a navigation and a manipulation world model at once and transfers zero-shot to an unseen driving diffusion model, reaching success competitive with recent test-time world-model methods at a fraction of their adaptation cost.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
world models, non-parametric memory, test-time correction, retrieval-augmented prediction, prediction-error modeling, model-based reinforcement learning, self-correcting models, foundation world models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9t7VOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ErrataWM augments frozen world models with a non-parametric memory of prediction errors. Residuals are stored as context-indexed prototypes and retrieved through applicability gates and reliability estimates to correct future predictions. Small correction modules are trained offline, while deployment adaptation uses gradient-free memory updates. Experiments evaluate prediction accuracy across navigation, manipulation, and driving, alongside navigation and manipulation success rates.

Critical strengths
  1. Context-indexed prototypes, positive/negative applicability scopes, and reliability updates provide a concrete mechanism for reusing historical errors while suppressing harmful corrections.
  2. Table 2 includes kNN, random-context, TENT, and TTT controls, with ErrataWM achieving the best prediction improvement in seven of eight settings.
Critical weaknesses
  1. The prior-work comparison is unreliable. Table 4 incorrectly labels Feedback-WM as updating model weights, attributes 81.6% LIBERO success to AdaPower despite its original Table 1 reporting 41.5%, and lists τ₀-WM results that I could not locate in the cited version. These issues, together with cost comparisons limited to TENT/TTT, undermine the broader competitiveness and efficiency claims.
  2. The control pipeline and mechanism lack sufficient validation. Multi-step prediction using corrected DINO features, corrected-image generation, and the RECON/CALVIN control protocols are insufficiently specified. Removing the context-contrastive loss barely changes prediction gains, yet no downstream ablation establishes the benefit over simpler correction. No physical closed-loop robot evaluation supports the deployment claims.
Questions for authors
  1. Please provide the exact source versions, tables, and evaluation settings for Table 4, and correct the weight-update classifications. Which competitiveness conclusions remain valid?
  2. Can you specify the implemented multi-step correction/MPC pipeline, corrected-image generation, RECON execution environment, and LIBERO/CALVIN evaluation protocols.
  3. What is the downstream control performance without Lcontrol, and how much does context-specific memory improve over kNN or context-independent correction under the same control settings?
Reviewer_fCZLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether the prediction errors of a frozen world model depend on the starting context, so that similar states lead to similar mistakes, and shows this across navigation, manipulation and driving with five world models. Based on this, the authors propose ErrataWM, a test-time correction method that does not need gradient updates to the world model. It stores correction prototypes in a shared frozen feature space and uses positive and negative context scopes during inference to decide when a correction should be applied, although the components of the memory system do require meta-training. The method is compared with a shuffled-context control, a kNN baseline and gradient-based test-time adaptation, and is also evaluated on downstream planning. A single shared memory is also evaluated across different world models.

Critical strengths
  1. The paper shows that prediction errors contain useful information across five world models (Table 1, Page 4). The learned memory also outperforms the kNN baseline in all eight rows of Table 2, by 0.033 to 0.056 (Page 7), which suggests that learning how to use past errors is more useful than simply retrieving similar examples.

  2. The method also learns when not to apply corrections, which helps prevent harmful changes without updating the model's weights. Removing this scope supervision roughly doubles the rate of harmful corrections on navigation and manipulation (Hurt rises from 0.226 to 0.447 on navigation and from 0.195 to 0.432 on manipulation; Table 11, Page 17). The shared memory also improves predictions across different world models, including the driving model, which is outside its training tasks (0.113, against 0.227 for a per-task memory; Table 6, Page 9). This suggests that the method can generalize beyond the models it was trained on, even though this generalization is modest.

Critical weaknesses
  1. The paper's main claim about context-specific corrections is not fully convincing. The context-specific margin is ∆read = G − G_rand: the gain of ErrataWM minus the gain of the same read with shuffled query contexts (Appendix C, Page 15; Appendix A.2, Page 14). Table 11 (Page 17) shows that removing the context-control loss L_control barely changes the gain G (0.196 vs 0.191 on navigation, 0.174 vs 0.171 on manipulation, 0.227 vs 0.223 on driving), even though it greatly reduces the margin (0.121 to 0.012, 0.108 to 0.009, and 0.083 to 0.007). On navigation, this means that the shuffled-context gain falls from 0.179 without L_control to 0.075 with it. This suggests that the larger margin comes mainly from making the shuffled-context baseline worse, rather than from improving the actual corrections. Since the paper presents context-specific correction as a key contribution (“a context-specific margin of 0.12 to 0.15 over a context-blind control”, Page 1), it should show more clearly that using the correct context gives a real benefit over a general correction.
Questions for authors
  1. (Clarification) Could you show whether using the correct context actually improves planning? Table 3 (Page 8) and Table 12 (Page 18) do not include results without the context-control loss, and Table 3 has no shuffled-context row. Reporting planning success and retention for ErrataWM without L_control, and planning success for the shuffled-context read, would help clarify whether this loss and the correct context provide a real benefit.

  2. (Clarification) Why is Feedback-WM listed as updating model weights in Table 4 (Page 8), when its paper says that “it does not update the world model parameters at test time” (Feedback-WM v1, Page 18)? It uses gradients with respect to the candidate actions to guide sampling (Feedback-WM v1, Eq. 7, Page 4), not to update model parameters, and this paper itself describes it as feeding prediction errors back “through an online observer” (Page 3). Also, where does AdaPower's LIBERO success rate in Table 4 (0.816) come from? These points matter for understanding the comparison with gradient-based methods.

  3. (Suggestion) It would be interesting to compare this approach with selecting from a bank of specialized LoRAs using similar feature matching. Could the authors discuss whether this would work as well, and how its training and update costs would compare with the meta-training and online memory updates of the proposed approach?

Reviewer_B6UVOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes ErrataWM, a memory-based framework for correcting prediction errors in frozen world models without updating their parameters at test time. It stores recurring residual patterns as memory prototypes and uses context-based retrieval, positive and negative context scopes, and reliability gating to determine when and how to apply corrections. The memory is updated online as new prediction errors are observed. Experiments across navigation, manipulation, and driving prediction tasks show improvements over frozen world models and kNN-based residual correction.

Critical strengths
  1. The paper is motivated by the observation that frozen world models exhibit recurring, context-dependent errors, providing a practical direction for test-time adaptation without updating model parameters.
  2. Experiments across multiple tasks and world models show improvements over kNN-based correction and suggest potential for cross-task and cross-model transfer.
Critical weaknesses
  1. The presentation of the core idea could be improved. The notion of context-dependent prediction errors and its connection to the proposed memory mechanism are not sufficiently explained. In addition, the overview figure is information-dense and does not clearly highlight the key mechanism, making the overall framework difficult to follow.
  2. The generalization claims could be better substantiated. In particular, the zero-shot transfer experiment uses residual whitening statistics computed from the nuScenes meta-training split, and the role of this target-domain information should be clarified.
Questions for authors
  1. Tables 2 and 6 and Appendix Table 11 report different gains for RECON, which may reflect different memory configurations or evaluation protocols. For example, the gain on RECON in Table 2 is 0.168, whereas Table 6 reports gains of 0.196 and 0.168 for task-specific and shared memory, respectively, and Appendix Table 11 reports 0.196 for the full method. What accounts for these discrepancies, and can the authors explicitly specify the configurations used in each table?
  2. The zero-shot transfer experiment on nuScenes uses residual whitening statistics computed from its meta-training split (Appendix A.1). How are these target-domain statistics used during transfer, and does their use require access to target-domain information beyond the unseen world model itself?
  3. ErrataWM consistently outperforms kNN, although the additional prediction error reduction is relatively modest (3.3–5.6 percentage points). Could the authors further clarify the contributions of the proposed components and whether their additional complexity is justified by the gains in prediction and downstream control?
  4. The sequential adaptation experiments support continual improvement and retention of previous performance, but the claims of continual improvement and no forgetting appear broader than the evaluated settings. To what extent do these findings generalize to substantial environmental changes, outdated prototypes, or limited memory capacity?
  5. Section 3 shows that prediction residuals are context-predictable, but this does not necessarily establish reliable correction in unseen contexts. How are query samples separated from the residual retrieval pool, particularly across trajectories and environments, and what evidence supports the generalization of the proposed correction framework?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.