- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Fast Residual Memory (FRM), a test-time adaptation scheme for frozen latent world models. A trajectory-local fast-weight matrix M_t (128×128) is reset to zero at the start of each trajectory. After each transition it is updated with a delta rule that associates a key from the predictive context with a value derived from the realized latent residual r_t = z*{t+1} − ẑ^s{t+1}. Corrections are retrieved through a learned query and decoder and added to the frozen prediction. The projection parameters and the update rate are meta-trained with a support–query objective: 4 support transitions are written, 1 held-out query transition is evaluated, and the loss is backpropagated through the delta-rule updates. At deployment, only M_t evolves, in a causal predict–observe–update loop.
Experiments use LeWM (PushT damping and gravity, TwoRooms wind, Reacher density) and DINO-WM (PushT, Wall, PointMaze action gain) under seen, interpolated, and extrapolated dynamics. The method reduces one-step latent MSE by roughly 7–36% over the frozen model. The paper also includes analyses of mid-trajectory dynamics switches (Keep vs. Reset), cross-trajectory transfer of fast states (largely unsuccessful), and simple residual-reuse baselines (Last, Mean, EMA, kNN).
Critical strengths
Clean problem framing. World models receive a free, causal supervision signal at every step, and this signal is usually discarded. Using it without touching the pretrained predictor is a sensible and practical design. The predict–observe–update protocol is clearly specified, and it is causal: a residual can never influence its own prediction. Appropriate training objective. Support–query meta-training separates memory formation from the evaluation targets. Backpropagating through the inner delta-rule updates means the projections are optimized for predictive usefulness on future transitions, not reconstruction of stored residuals. Breadth of evaluation. The paper uses two architecturally different backbones (LeWM and DINO-WM) and several physically distinct shifts (damping, external force, additive disturbance, inertia, actuator gain), with explicit interpolation and extrapolation splits. Per-condition curves are given in Appendix D. Honest analysis sections. The Keep/Reset experiment and the negative transfer result are informative, and the authors do not oversell them. The conclusion that the fast state acts as a local, context-dependent error memory rather than a reusable dynamics representation is well supported. The residual-reuse baselines in Appendix E (especially EMA and kNN) are the right first sanity checks.
Critical weaknesses
- Novelty relative to prior work is limited and partly unacknowledged.
The memory mechanism itself is standard. The delta-rule fast weights come from Schlag et al. (2021), and the setup closely follows TTT layers (Sun et al., 2025) and Titans, with normalized keys and a learned step size. Normalized keys and a scalar step make the update essentially normalized LMS on learned features. The idea of “learned features offline, fast linear adaptation of residual dynamics online” has substantial precedent in model-based RL and adaptive control that is not discussed. Examples include: ALPaCA (Harrison et al., 2018): a meta-learned Bayesian last layer with online updates. Neural-Fly (O’Connell et al., 2022): learned basis functions with online-adapted linear coefficients for residual aerodynamics. GrBAL/ReBAL (Nagabandi et al., 2019): meta-learning online adaptation of dynamics models to changed dynamics such as payloads and terrain. Context-aware dynamics models (CaDM; Lee et al., 2020) and RMA (Kumar et al., 2021): recent transition history is encoded to infer latent dynamics parameters. The contribution is therefore mainly the transfer of this paradigm to frozen JEPA-style latent world models, plus the empirical analysis. That is useful, but the paper should position itself explicitly against this line of work. Several directly related latent-world-model adaptation methods are cited but not compared against: AdaJEPA, SPREAD, WorldAgen, and WAM-TTT.
- Missing baselines that would isolate the claimed mechanism.
Adapter-0 is a weak control. With M_t = 0, the retrieved correction is R_ψ(0), which is at most a constant bias. The near-identical Base and Adapter-0 numbers confirm this. It does not test whether the gains come from online adaptation or from training additional parameters on shifted-dynamics data. The key missing control is a static residual predictor. This would be, for example, an MLP or small transformer mapping the same context c_t to r_t, trained on the same multi-dynamics trajectories. Because c_t contains H steps of latent history and actions, a static model may already be able to infer the dynamics regime implicitly. Related controls would be a recurrent or history encoder in the spirit of CaDM/RMA, or simply a longer history window. Stronger online adaptation baselines are also missing, under matched compute and memory budgets. Candidates include gradient-based online fine-tuning of a small adapter or LoRA on the predictor, recursive least squares (RLS) on the same learned features (the natural second-order counterpart to the LMS-style delta rule), and a gated or decaying delta rule such as Gated DeltaNet. The last would also speak directly to the staleness problem the paper itself identifies in Section 4.4. Gains may reflect generic error correction rather than shift adaptation. Please report FRM on the original, unshifted pretraining dynamics. If the gains there are comparable, the method is mostly correcting the backbone’s generic, locally autocorrelated errors. The failure of even same-dynamics transfer (Fig. 5) points somewhat in this direction. A linear probe decoding the dynamics parameter from M_t would also help clarify what the memory encodes.
- Train/test mismatch in memory dynamics.
Meta-training uses exactly K = 4 support writes followed by 1 query with frozen memory. Deployment interleaves writes and reads for 15–40 steps. The learned η and projections are therefore never optimized for the long-horizon accumulation or interference behavior they face at test time, and the method has no forgetting or gating mechanism. An ablation over K and Q, or training with interleaved online updates, would show whether the reported gains are near what the mechanism can achieve, and whether the Keep vs. Reset gap is an artifact of this mismatch.
- Evaluation scope does not match the motivation.
The introduction motivates the work with planning and control, but all results are one-step latent MSE. Latent-space corrections may improve MSE while moving predictions off the representation manifold, or may not help (or may hurt) multi-step imagined rollouts. During planning, imagined contexts never yield residuals, so M_t is applied to contexts it has not seen. Multi-step rollout error, and ideally MPC/CEM planning success under shift (both backbones support this), are needed to support the claimed practical value. The shifts are also low-dimensional and largely stationary within a trajectory. The TwoRooms and Wall wind is literally a constant additive displacement (Eq. 12), which is close to the ideal case for an EMA or bias-like correction. This likely explains why EMA is competitive in Table E.1. More structured shifts, such as contact-rich changes, object property changes, or partial observability, would better test context-dependent retrieval.
- Metric and reporting inconsistencies.
Table 1 vs. Table E.1: the two tables report different LeWM Reacher results, with loss 0.053 vs. 0.056 and relative gain 36.45% vs. 32.65% (pp. 7 and 18). TwoRooms also differs slightly (31.25% vs. 31.15%). The evaluation difference should be explained. The reported relative gains do not follow from the reported aggregate losses via Eq. 11. TwoRooms: (0.350 − 0.271)/0.350 ≈ 22.6%, but 31.25% is reported. PushT-Damping: (0.101 − 0.093)/0.101 ≈ 7.9%, but 8.71% is reported. In Table E.1, EMA on PushT-Damping has higher loss than Base but a positive relative gain, which the authors acknowledge. This suggests gains are averaged per step or per trajectory (a mean of ratios), which can be dominated by low-loss transitions. The aggregation should be stated precisely. Relative gain computed on aggregate losses should be reported alongside, or used as the headline metric. Uncertainty reporting covers only variation across 50 test trajectories, from what appears to be a single trained memory module per setting. Results across several independent meta-training seeds (mean ± std) are needed to assess robustness, particularly for the smaller gains (e.g., DINO-WM Wall, 7.25%). Unequal trajectory counts. In PushT, n(t) drops to 1–2 trajectories at late indices, so the late parts of the online curves in Figs. 3 and 7 are not reliable. The claim of a “growing separation” between TTT and Adapter-0 should be quantified, for example as gain vs. t averaged only over indices with sufficient support.
- Smaller issues.
Oracle switch time. The Reset strategy in Section 4.4 assumes the dynamics switch time is known, so it is an oracle. This should be stated, and a simple residual-based change detector would make the result practically meaningful. Narrow transfer study. The transfer analysis (Section 4.5) covers only PushT gravity. Its conclusion is reasonable but rests on a single setting. Missing DINO-WM details. Implementation details (Appendix B) cover only LeWM. DINO-WM predicts patch-level features, so it is unclear how the context is constructed, what the key/value dimensions are, and whether the correction is per-patch or global. Missing overhead numbers. Compute and latency overhead per step is not reported, although lightweight cost is a stated motivation. Terminology. The method is called “FRM” in the appendix and “TTT” in the tables. It would also help to clarify that no parameters are trained at test time, which makes this in-context or fast-weight adaptation, distinct from TTT in the sense of Sun et al. (2020).
Questions for authors
Were Tables 1 and E.1 produced with the same checkpoints, trajectories, and aggregation rules? Please reconcile the Reacher (and TwoRooms) values. How exactly is relative gain aggregated: per step, per trajectory, or per condition? Why do the reported gains differ from Eq. 11 applied to the reported losses? How does FRM compare to a static residual predictor (no fast state) trained on the same multi-dynamics data with the same context window, and to a history or context-encoder baseline (CaDM/RMA-style)? How does FRM compare to online adaptation methods at similar compute and memory cost, for example online gradient updates of a small adapter, or RLS on the same learned features? How variable are the results across independently trained memory modules (multiple seeds)? What gain does FRM achieve on the original, unshifted dynamics? Can the dynamics parameter be decoded from M_t? How sensitive are the results to the number of support steps K used in meta-training? Does training with interleaved online updates, or adding a decay or gating term, reduce the Keep vs. Reset gap? Does the correction improve multi-step rollout accuracy and downstream planning success under shift, given that M_t receives no new residuals during imagined rollouts? For DINO-WM, how are context, keys, and residual values constructed from patch features? What is the per-step compute and latency overhead relative to the frozen model?