acceptodds
Under review as a conference paper at ICLR 2027

LaST: Latent State Tracking for Persistent Object Memory in Video World Models

Abstract

Video world models must preserve object identity across interruptions in visibility. Yet the observations establishing that identity can leave the local attention window long before an object reappears. Persistent memory must retain relevant visual information and make it available to later generation. We introduce LaST, Latent State Tracking, which separates persistent object memory from local video generation. A recurrent writer maintains scene context, object geometry, and appearance across context windows. A learned reader uses geometry to retrieve appearance at the requested locations and combines decoded scene and object fields to correct a frozen generator's denoising predictions. We pretrain memory through future reconstruction, then jointly optimize state construction and reader corrections over multiple generation windows. Parameter-free rescaling regulates spatial contrast. On the full WRBench evaluation set, LaST improves all five metrics over both KV baselines at both 1.3B and 5B. A capacity study on MOSEv2 relates reconstruction quality and object coverage to the number of retained slots. These results support retaining and reusing object history beyond the generator's local attention window.

Primary area:
generative models
Keywords:
video world models, video diffusion, persistent memory, latent state representations, occlusion

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aBB2OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

LaST decouples persistent object memory from local video generation. A recurrent writer maintains separate scene, geometry, and appearance records in fixed-size object slots; a learned reader uses stored geometry to retrieve appearance at queried space–time locations and corrects the denoising predictions of a frozen generator (Wan2.1/2.2), with parameter-free rescaling regulating spatial contrast. Training proceeds in stages: state pretraining via future reconstruction with teacher-supplied geometry, reader warm-up via appearance replacement (residual ranking + keep consistency), and joint optimization over multi-window generated histories with a detached long-history support mask. On the full WRBench set, LaST improves all five metrics over both Short-KV and Full-KV baselines at 1.3B and 5B scales; a capacity study on MOSEv2 relates reconstruction error and object coverage to slot count, and efficiency measurements show 70.0% lower history storage and 21.5% lower continuation latency (1.3B, seven scenes).

Critical strengths

A well-motivated and cleanly designed decomposition. Separating scene/geometry/appearance records and interfacing the memory to a frozen generator purely through denoising corrections is an appealing design; the staged training pipeline (future reconstruction → teacher-guided reads → reader warm-up → joint optimization on generated histories) is careful and principled.

Diagnostics go beyond average scores. History-content interventions, state-update comparisons, and repeated-continuation experiments provide evidence that the retained history is actually being used (e.g., object similarity 0.840 vs. 0.331 for Full-KV on MOSEv2; 4th-round continuation similarity 0.700 vs. 0.176/0.133).

Critical weaknesses

The headline gains on the main benchmark are modest and possibly within noise. At 1.3B, improvements over Short-KV are ≈0.02–0.05 on most metrics (e.g., re-observed spatial 0.613→0.659; visible state 0.565→0.604), and WRBench is an automated LLM-judge metric without a human study; the large 5B integrity jump (0.364→0.741) is unexplained relative to the modest gains elsewhere.

High system complexity relative to demonstrated benefit, supported by very small samples. The pipeline is intricate (staged pretraining, teacher-guided reconstruction, replacement-based warm-up, joint training with support masks), yet much of the supporting evidence rests on 4 sequences per dataset (interventions) and 2 sequences per dataset (repeated continuation). Stronger evidence — harder baselines (e.g., retrieval-based memory such as WorldMem/Context-as-Memory rather than KV windows) and human evaluation would be needed to establish that the complexity pays off.

Questions for authors

How robust are the WRBench gains? e.g., which prompts or scenes drive the improvements, and do the margins hold consistently or concentrate on a subset of the benchmark?

How do the five WRBench metrics compare against stronger memory baselines such as retrieval-based history (WorldMem / Context-as-Memory) rather than only local-KV windows? What explains the large 5B integrity gain (0.364→0.741) while other metrics improve modestly is integrity dominated by a subset of prompts or scenes?

The reader corrects the generator's clean prediction; what happens when memory content is stale or wrong (e.g., appearance changed off-screen)? Are there failure analyses beyond the mismatch-state experiments?

8 comments
AuthorsOpenReview ↗

LaST improves identity recovery after reappearance and preserves updated states across revisits.

Thank you for the thoughtful review and for recognizing our memory decomposition and training design. Your questions motivate a closer look at where the gains arise, how they vary across tasks, and whether the same object is recovered after reappearance or a memory update.

Q1. How robust are the gains, especially 5B integrity?

A1. Integrity gains persist after excluding the largest contributors, with trained correction contributing substantially.

At 5B, zeroing persistent records while retaining the reader (S=0) gives integrity 0.718 and revisit spatial 0.667. Training without persistent history gives 0.714 and 0.674, respectively.

5B integrity differences. Columns exclude the top one or five contributors, ranked by LaST minus Short-KV; all rows use the same exclusions.

ComparisonAllExclude 1Exclude 5
LaST − Short-KV0.3770.3650.351
No-history − Short-KV0.3500.3400.329
LaST − no-history0.0270.0250.022

This sample set differs from the eight-task CI analysis below. Excluding scene groups gives LaST-minus-S=0 revisit spatial differences of 0.021–0.039 (1.3B) and 0.058–0.091 (5B).

Paired differences and 95% CIs. Differences are LaST minus the baseline. We use 20,000 source resamples with seeds 20260924/20260925 for 1.3B/5B. The first three metrics use eight tasks; revisits use three/six.

1.3B

Metricvs. Short-KVvs. Full-KV
Integrity0.043 [-0.033, 0.122]0.088 [-0.039, 0.220]
Visible spatial0.042 [0.013, 0.065]0.076 [0.036, 0.114]
Visible state0.039 [0.011, 0.067]0.043 [0.018, 0.074]
Revisit spatial0.046 [0.009, 0.076]0.151 [0.099, 0.210]
Revisit state0.028 [-0.010, 0.061]0.136 [0.123, 0.151]

5B

Metricvs. Short-KVvs. Full-KV
Integrity0.377 [0.234, 0.518]0.400 [0.294, 0.500]
Visible spatial0.047 [-0.006, 0.089]0.042 [0.002, 0.076]
Visible state0.027 [-0.029, 0.084]0.002 [-0.057, 0.057]
Revisit spatial0.124 [0.044, 0.208]0.110 [0.033, 0.182]
Revisit state0.033 [-0.035, 0.099]0.035 [-0.038, 0.104]

The paired 5B LaST revisit means are 0.732099 (spatial) and 0.621734 (state) on six tasks; the main table uses eight. Intervals describe task variation. Against Short-KV at 5B, integrity and revisit spatial intervals exclude zero; the other three include zero.

Q2. How does LaST compare with stronger memory baselines?

A2. LaST leads in revisit scores; Flat is slightly higher in integrity and visible-state consistency.

An asterisk marks paper results; other rows are new measurements. I: integrity (paper crop); VS/VT: visible spatial/state; RS/RT: revisit spatial/state.

1.3B

MethodIVSVTRSRT
Learned Flat.496.780.605.643.631
Retrieval.479.763.588.638.628
LaST*.494.783.604.659.639

5B

MethodIVSVTRSRT
Learned Flat.744.807.578.711.629
Retrieval.701.790.563.699.622
LaST*.741.812.576.745.637

Flat matches persistent-state size, trainable parameters, training data and budget. Retrieval uses the same frozen generator. WorldMem and Context as Memory have their own trained interfaces and different backbones. Context as Memory's official code is not yet public. WorldMem results will follow with backbone details.

Q3. Does LaST recover the same instance after disappearance?

A3. LaST's identity advantage over Flat increases after reappearance at 5B.

MethodOrdinary identityReappearance identity
Short-KV0.620.43
S=00.730.51
Trained no-history0.740.54
Retrieval0.750.60
Learned Flat0.800.63
LaST0.810.69

The advantage is six percentage points after reappearance versus one in ordinary cases. In blinded, two-choice comparisons without ties, preference for LaST is 0.71 against Short-KV, 0.65 against S=0 and 0.56 against Flat. Visual preference and instance identity are evaluated separately. Human identity judgments accompany automated scores, with agreement of 0.81 in ordinary cases and 0.74 after reappearance.

Q4. Can LaST recover when appearance changes off-screen or memory is wrong?

A4. At 5B, event correctness remains 0.72 at the second revisit after a reliable update.

MethodAfter updateSecond revisit
S=00.460.42
Learned Flat0.680.59
LaST0.780.72

Unchanged-object identity retention is 0.79 for LaST and 0.75 for Flat. Removing keep or local-preservation reduces it to 0.65 or 0.62, testing preservation during updates.

With incorrect history and a reliable new observation, event correctness is 0.72 for LaST and 0.64 for Flat. Without reliable new evidence, incorrect history is adopted in 0.12 and 0.19 of cases.

AuthorsOpenReview ↗

Additional WorldMem results with the public Minecraft checkpoint

Thank you for highlighting WorldMem as a relevant memory-based approach. Following our earlier responses, we provide the additional evaluation below, together with the checkpoint and evaluation settings.

Q. What does the released WorldMem checkpoint achieve on our natural-scene inputs?

A. We evaluated its public Minecraft checkpoint without fine-tuning on 25 fixed WRBench T0 camera-revisit cases. The authors clarify that their RealEstate10K experiments use a separate model (author clarification). As of November 16, 2026, we could not locate an official public release of that model. We therefore used the released Minecraft checkpoint.

Each case uses one initial image and a 60-degree turn-and-return camera trajectory: 81 frames at 640 × 360, 16 fps, and seed 42. We retain WorldMem's history retrieval and memory attention, with 20 sampling steps, an eight-frame generation window, and eight memory conditions. No future reference images are supplied; task text is used for evaluation rather than as an input to WorldMem.

Automatic scores on all 25 videos

MetricScore ↑Evaluated videos
Visual integrity (D2)0.055825/25
Visible spatial fidelity (D3)0.610225/25
Visible state fidelity (D4)0.460425/25

These scores use the current WRBench evaluator. Its D2 uses global/local DINOv2 features, whereas the LaST main table uses crop-based integrity; the two integrity columns should not be directly compared.

Return evidence across all 25 outputs

AI-assisted frame inspection covered all 81 frames of every video, with ambiguous late frames additionally examined at their original resolution.

Observed return evidenceVideosFraction of all cases
Clearly identifiable return to the original region00%
Late scene fragments; region return remains ambiguous520%
No identifiable late return evidence2080%

These are qualitative inspection counts, not blinded human ratings or official D5/D6 scores. The generated videos showed substantial loss of subjects and scene structure. The automatic evaluator found no eligible videos for conditional D5/D6 scoring; the table above retains all 25 cases and describes the affected outputs explicitly.

This experiment characterizes zero-shot transfer of the released Minecraft checkpoint to natural scenes. We report it separately from the matched-backbone comparisons in our earlier responses, since the training domain, backbone, and input interface differ.

Could you provide a comparison with existing open-source memory-based methods for video world models, such as LSM-World or GIM-World? My primary concern is the lack of a comprehensive comparison against methods for similar directions; most of your baselines consist of foundational models or methods developed by the authors themselves. While such experiments are sufficient for an ablation or analysis study, they fail to demonstrate that your proposed method offers a significant advantage over existing memory-based techniques. I would like to see a comparison with LSM-World and GIM-World included in the main tables. If you could incorporate both quantitative and qualitative results, including visualizations in the main text, the paper will be better. The current version is not yet convincing enough.

Reviewer_qpWhOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LaST, a persistent object memory framework for improving long-horizon consistency in video world models. The method maintains compact scene, geometry, and appearance representations across generation windows and uses a geometry-aware memory reader to correct the denoising predictions of a frozen Wan video generator.

The authors introduce a staged training strategy combining future reconstruction, reader warm-up, and joint optimization over generated histories. Experiments demonstrate consistent improvements over Short-KV and Full-KV baselines on WRBench, together with substantial reductions in history storage.

Critical strengths

S1. Effective and well-motivated persistent memory architecture.

LaST addresses an important limitation of long-horizon video generation by maintaining object information beyond the local attention window. The separation of scene, geometry, and appearance records, together with geometry-aware retrieval, provides an appealing approach to persistent memory.

The staged training strategy is particularly thoughtful, as it separately addresses memory retention and effective memory utilization. The ability to augment a frozen pretrained generator further enhances the practical value of the method.

S2. Strong empirical performance and memory efficiency.

LaST consistently outperforms both KV-based baselines across all five WRBench metrics at the evaluated 1.3B and 5B scales. In particular, re-observed spatial consistency improves from 0.608 to 0.745 at 5B.

The additional memory intervention and multi-round generation experiments provide useful evidence of history dependence. The compact 12 KiB persistent state and substantial reduction in history storage are also attractive practical advantages.

Critical weaknesses

W1. Limited component-level ablations.

While the paper provides useful memory intervention and capacity studies, the individual contributions of the proposed architectural components are not fully isolated.

In particular, it remains unclear whether the improvements primarily arise from introducing persistent memory or from the specific scene/geometry/appearance decomposition. Similarly, the necessity of geometry-aware retrieval and parameter-free rescaling is not sufficiently demonstrated.

Additional controlled ablations would strengthen the architectural contribution. Nevertheless, the existing experiments provide encouraging evidence for the effectiveness of the overall framework.

Questions for authors

Q1. Component-level ablations.

Could the authors compare LaST's structured memory against a simpler unstructured latent memory baseline under matched memory capacity, trainable parameter count, and training budget? Additionally, ablations independently removing scene, geometry, or appearance components, as well as geometry-aware retrieval and parameter-free rescaling, would help establish which design choices are essential to the observed improvements.

Q2. Input parameterization of the memory correction.

One appealing interpretation of the proposed approach is that it enables conditioning a pretrained Flow Matching model on additional memory (S) without modifying the base generator. From this perspective, a general correction function Rϕ(xσ,σ,D,S)R_\phi(x_\sigma,\sigma,D,S), where (D) denotes the original generation conditions, could in principle capture the information needed to recover the desired memory-conditioned velocity field.

However, the paper adopts the more compact formulation Rϕ(b,S)R_\phi(b,S), where b=xσ−σvθ(xσ,σ,D)b=x_\sigma-\sigma v_\theta(x_\sigma,\sigma,D). I would be interested to understand the motivation behind this design choice. Is the clean-latent prediction (b) expected to provide sufficient information about the noisy input, denoising timestep, and original conditions for effective memory correction? Alternatively, does this simpler parameterization offer practical advantages in optimization, stability, or generalization?

I would appreciate the authors' perspective on this trade-off. A comparison with a reader explicitly conditioned on the denoising timestep could also be informative.

4 comments
AuthorsOpenReview ↗

Structured memory improves identity after reappearance, and clean-latent correction provides an effective reader interface.

Thank you for the thoughtful review and for recognizing the value of our persistent memory architecture, staged training, and frozen-generator interface. We examine what the structured records contribute under matched budgets and explain the choice of inputs to the memory reader.

Q1. What does structured memory add over an unstructured learned memory?

A1. At 5B, LaST achieves 0.69 reappearance identity accuracy versus 0.63 for learned Flat under matched budgets.

Ordinary identity accuracy is 0.81 for LaST and 0.80 for Flat. The larger difference after disappearance indicates a benefit in recovering the same instance when it returns. Flat matches LaST's persistent-state size, trainable parameter count, training data, and training budget.

The WRBench results show higher revisit spatial and state scores for LaST at both scales, while Flat scores slightly higher on integrity. An asterisk marks paper results; the remaining rows are new measurements.

WRBench at 1.3B

MemoryIntegrityRevisit spatialRevisit state
Learned Flat0.4960.6430.631
LaST*0.4940.6590.639

WRBench at 5B

MemoryIntegrityRevisit spatialRevisit state
Learned Flat0.7440.7110.629
LaST*0.7410.7450.637

We also evaluate each component through generation quality after reappearance. The first four variants in each table are retrained; reader correction and rescaling are disabled at inference.

Component ablations at 1.3B

VariantRevisit spatialRevisit state
Without scene records0.6350.632
Without geometry records0.6420.630
Without appearance records0.6500.624
Without geometry-aware retrieval0.6540.637
Reader correction off (KV subset)0.6130.611
Rescaling off0.6590.639

Component ablations at 5B

VariantRevisit spatialRevisit state
Without scene records0.7010.628
Without geometry records0.7140.624
Without appearance records0.7310.617
Without geometry-aware retrieval0.7380.636
Reader correction off (six tasks)0.6080.589
Full LaST (same six tasks)0.7320.622
Rescaling off0.7450.637

Among the record ablations, scene removal produces the largest drop in revisit spatial consistency, and appearance removal produces the largest drop in revisit state consistency at both scales. Geometry removal reduces both scores. Geometry-aware retrieval has a smaller numerical effect, while disabling rescaling leaves the rounded means unchanged. The reader comparison uses the same six tasks for both conditions at 5B.

Q2. Why correct the clean-latent prediction rather than condition the reader on every diffusion input?

A2. The clean-latent prediction aligns memory correction with the scene content being generated.

The compact reader Rϕ(b,S)R_\phi(b,S) receives the predicted clean latent b=xσ−σvθ(xσ,σ,D)b=x_\sigma-\sigma v_\theta(x_\sigma,\sigma,D) and the retained memory SS. The generator has already processed the noisy input, noise level, and original conditions. Their effect on the predicted scene is therefore available through bb, allowing the reader to correct that scene using stored information.

This motivates the input choice without requiring bb to be a sufficient statistic for all diffusion inputs. We test it empirically against a timestep-conditioned reader and the general interface Rϕ(xσ,σ,D,S)R_\phi(x_\sigma,\sigma,D,S), which exposes the noisy input, timestep, and original conditions directly. All variants use matched trainable parameter counts and training budgets, and are evaluated on generation quality.

Reader inputs at 5B

ReaderIntegrityRevisit spatialRevisit state
Compact*0.7410.7450.637
Compact + timestep0.7430.7480.639
Full-input reader0.7420.7470.638

Adding the timestep raises revisit spatial and state scores by 0.003 and 0.002, respectively. Exposing all diffusion inputs yields no further increase over timestep conditioning in these means. These results support the compact reader as an effective interface, with a small improvement from explicit timestep conditioning.

Thank you for the detailed response and the additional controlled experiments. Overall, the rebuttal has substantially strengthened the experimental justification for the proposed architecture and resolved most of my original concerns. I remain the score and appreciate the authors' efforts in providing these targeted evaluations.

AuthorsOpenReview ↗

Thank You and Manuscript Updates

Dear Reviewer qpWh,

Thank you for taking another look at our work and for your helpful feedback. We are glad the additional experiments helped address your concerns.

We will include the matched-budget comparisons, component ablations, and reader-input analysis in the revised PDF, with the additions marked in blue for easy reference.

Thank you again for your time and thoughtful suggestions.

Best regards,
The Authors

AuthorsOpenReview ↗

Revised PDF Uploaded

Dear Reviewer qpWh,

We have uploaded the revised PDF. The matched-budget comparisons, component ablations, and reader-input analysis are now included in Appendix A.9 (Tables S5–S7, pp. 25–26), with the additions marked in blue.

Thank you again for your helpful feedback and for taking the time to discuss our work.

Best regards, The Authors

Reviewer_HtE4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a persistent object-centric memory framework for video world models. A recurrent writer maintains fixed-size scene, geometry, and appearance records, and a learned reader uses the decoded fields to correct the clean-latent predictions of a frozen Wan generator, followed by parameter-free rescaling. Memory is pretrained via teacher-guided future reconstruction on MOSEv2, VOST, and YouTube-VOS, then jointly optimized with the reader over multi-window generated histories. The method is evaluated against Short-KV and Full-KV on WRBench, with additional state-intervention and multi-round diagnostics on MOSEv2 and VOST.

Critical strengths
  • Separating scene, geometry, and appearance records is a meaningful design for keeping object information available after it leaves the generator’s local attention window. The memory has a fixed size regardless of video length, so it is an efficient alternative to storing historical KV features. It also lasts beyond the generator’s context without changing the generator’s weights.
  • Teacher-guided reconstruction separates remembering an object’s appearance from locating it. Appearance-replacement supervision teaches the reader to rely on relevant memory, and joint training over multiple generation windows aligns what the memory keeps with how it is used during generation.
  • The quantitative results are promising. LaST improves every WRBench metric over both short- and long-history KV baselines at both model scales. The largest gains are in consistency after an object comes back into view, which suggests that persistent object memory can help preserve scene coherence across interruptions in visibility.
  • The approach is memory-efficient. The persistent state is only a few kilobytes and noticeably reduces history storage and continuation latency compared with keeping the full KV history. The memory intervention and multi-round continuation experiments are also a welcome effort to test whether the retained history actually influences generation.
Critical weaknesses

Major Concerns:

  • The baselines do not isolate what the proposed memory contributes. LaST is compared only with training-free KV variants of the same frozen generator, while LaST itself adds modules trained on extra video data. There is no matched comparison with recent memory methods or a learned-memory baseline of similar size, and the external WRBench entries come from different settings.
  • A no-memory variant of LaST already beats the short-history baseline and on one dataset even exceeds full LaST. This suggests that much of the gain comes from the learned correction pathway rather than from stored history. The KV baselines also look unusually weak, and LaST’s large jump in visual integrity is not explained by memory alone.
  • The only ablation varies slot capacity and measures reconstruction rather than generation quality. With many interacting losses and several training stages, nothing isolates the separate geometry and appearance records, the reader correction versus rescaling, or the individual training objectives, so it is unclear which components are actually necessary.
  • The evidence for object-specific memory and identity preservation is weak. Feature similarity and pixel error cannot confirm that the same instance persists through occlusion. On VOST, removing target-object records has about the same effect as removing other objects, and on both datasets removing target records is worse than having no memory at all. This points to sensitivity to unusual states rather than true object-level recall.
  • Statistical support and the efficiency analysis are limited. The main results have no confidence intervals or significance tests, and the diagnostic experiments rely on very few sequences and frames. The reported speed-up also leaves out the cost of building and updating memory, so the true end-to-end efficiency advantage remains uncertain.

Minor Concerns:

  • Object geometry uses simple 2D motion with no camera pose, so it is unclear how objects are relocated after the camera moves away and back.
  • How the fixed slots handle new objects, reappearance, and identity switches when the slot budget runs out should be explained.
  • Robustness in crowded scenes, under long occlusions, fast motion, or large viewpoint changes is not characterized.
  • Some gains are very small, and per-scene results with uncertainty estimates would show whether they are meaningful.
Questions for authors
  1. Can the authors report WRBench results for LaST without memory, a matched learned-memory baseline, and a recent memory method on the same backbone?
  2. Can the authors provide ablations that separate the geometry and appearance records, the reader correction versus rescaling, and the main training objectives, evaluated on generation quality rather than only on reconstruction?
  3. Can the authors provide object-level identity evidence across reappearance and explain why removing target records hurts more than removing all memory?
4 comments
AuthorsOpenReview ↗

Structured memory improves identity after reappearance under matched budgets, while trained correction contributes to visual quality.

Thank you for the thoughtful review and for recognizing the value of our structured memory design. Your questions help clarify the roles of memory and trained correction; we address these with matched controls, ablations, and expanded analyses.

Q1. What does memory add beyond trained correction?

A1. LaST improves reappearance identity over Flat; no-history controls retain high integrity.

S=0 clears persistent records but retains the reader. No-history training uses three local latent frames under matched budgets. Long-KV (Full-KV) uses 10/15 history latent frames at 1.3B/5B; Short-KV uses three. * marks paper results; other rows are new measurements. I: integrity (paper crop); VS/VT: visible spatial/state; RS/RT: revisit spatial/state.

1.3B

MethodIVSVTRSRT
Short-KV*.450.741.565.613.611
Long-KV*.405.706.561.509.503
S=0.486.766.594.628.620
No-history.488.770.596.634.624
Learned Flat.496.780.605.643.631
Retrieval.479.763.588.638.628
LaST*.494.783.604.659.639

5B

MethodIVSVTRSRT
Short-KV*.364.765.549.608.589
Long-KV*.342.769.574.622.587
S=0.718.793.567.667.611
No-history.714.796.569.674.614
Learned Flat.744.807.578.711.629
Retrieval.701.790.563.699.622
LaST*.741.812.576.745.637

At 5B, reappearance identity accuracy is .69 for LaST and .63 for Flat. Flat matches state size, trainable parameters, data, and budget. Retrieval uses our frozen generator. Context as Memory's official code is not yet public; WorldMem results will follow with backbone details.

Q2. Which components and objectives matter?

A2. Among record ablations, removing scene records gives the largest spatial drop; removing appearance gives the largest state drop.

5B variantRSRT
Remove scene.701.628
Remove geometry.714.624
Remove appearance.731.617
Remove geometry-aware retrieval.738.636
Reader off (6 tasks).608.589
LaST (same 6 tasks).732.622
Rescaling off.745.637

Retrieval has a small effect; rescaling leaves rounded means unchanged. We retrain the first four models; reader and rescaling are disabled at inference.

With matched training budgets, removing state pretraining, joint training, or generated-history training gives RS .696/.710/.724. Removing identity supervision gives RT .615; removing memory-support gives RS .720. Removing keep or local-preservation gives integrity .717/.713.

Q3. Why can target deletion be worse than no memory?

A3. Appearance-only deletion lowers similarity more than complete target deletion.

We test revised interventions on 12 sequences per dataset, up from four.

Object similarityMOSEv2VOST
No state.612.839
Delete target appearance.416.451
Delete all target information.597.818
Delete equal-capacity other information.744.762
LaST.813.794

Appearance-only deletion retains geometry and scene cues, consistent with residual conflict. Complete deletion approaches no state: below other-object deletion on MOSEv2 but above it on VOST. On VOST, no state has higher similarity, while LaST has lower image error (.0168 vs .0311).

Q4. How robust are the gains?

A4. Gains remain after excluding the largest contributors.

On a separate set, 5B LaST-minus-Short-KV integrity is .377 overall, .365 excluding the top contributor, and .351 excluding the top five. LaST-minus-no-history is .027/.025/.022. Scene-group exclusion gives LaST-minus-S=0 RS ranges of .021–.039 (1.3B) and .058–.091 (5B).

LaST − Short-KV: paired 95% CIs from 20,000 source resamples.

ScaleIntegrityRevisit spatial
1.3B0.043 [-0.033, 0.122]0.046 [0.009, 0.076]
5B0.377 [0.234, 0.518]0.124 [0.044, 0.208]

Integrity uses eight tasks; revisits use three/six at 1.3B/5B. At 5B, LaST uses eight tasks in Q1; its paired spatial mean on six is .732099.

Q5. How do viewpoint and capacity affect results?

A5. More slots improve reappearance with more objects.

At 5B, RS is .702/.681/.697/.669 for long occlusion/crowding/fast motion/large viewpoint change. Geometry provides 2D cues; relocation uses current observations and camera conditioning. For 7–8 objects, using eight rather than four slots raises RS from .614 to .692 and lowers the identity-switch rate from .29 to .16. Updates admit new objects and can replace old records.

Q6. What is the end-to-end cost?

A6. LaST reduces total time by 9.1%, including memory construction and updates.

At 1.3B on H100, 832×480, eight steps, and the same offload path, cold-start totals are 166.3/182.9 seconds for LaST/Full-KV. GPU peaks are 14.9/16.1 GiB; CPU peaks are 22.8/21.7 GiB. Continuation including decoding/transfer takes 111.6/106.8 seconds; savings come from history preparation. Outputs peak at .11 GiB per run, excluding weights and environments.

AuthorsOpenReview ↗

Additional WorldMem results with the public Minecraft checkpoint

Thank you for highlighting WorldMem as a relevant memory-based approach. Following our earlier responses, we provide the additional evaluation below, together with the checkpoint and evaluation settings.

Q. What does the released WorldMem checkpoint achieve on our natural-scene inputs?

A. We evaluated its public Minecraft checkpoint without fine-tuning on 25 fixed WRBench T0 camera-revisit cases. The authors clarify that their RealEstate10K experiments use a separate model (author clarification). As of November 16, 2026, we could not locate an official public release of that model. We therefore used the released Minecraft checkpoint.

Each case uses one initial image and a 60-degree turn-and-return camera trajectory: 81 frames at 640 × 360, 16 fps, and seed 42. We retain WorldMem's history retrieval and memory attention, with 20 sampling steps, an eight-frame generation window, and eight memory conditions. No future reference images are supplied; task text is used for evaluation rather than as an input to WorldMem.

Automatic scores on all 25 videos

MetricScore ↑Evaluated videos
Visual integrity (D2)0.055825/25
Visible spatial fidelity (D3)0.610225/25
Visible state fidelity (D4)0.460425/25

These scores use the current WRBench evaluator. Its D2 uses global/local DINOv2 features, whereas the LaST main table uses crop-based integrity; the two integrity columns should not be directly compared.

Return evidence across all 25 outputs

AI-assisted frame inspection covered all 81 frames of every video, with ambiguous late frames additionally examined at their original resolution.

Observed return evidenceVideosFraction of all cases
Clearly identifiable return to the original region00%
Late scene fragments; region return remains ambiguous520%
No identifiable late return evidence2080%

These are qualitative inspection counts, not blinded human ratings or official D5/D6 scores. The generated videos showed substantial loss of subjects and scene structure. The automatic evaluator found no eligible videos for conditional D5/D6 scoring; the table above retains all 25 cases and describes the affected outputs explicitly.

This experiment characterizes zero-shot transfer of the released Minecraft checkpoint to natural scenes. We report it separately from the matched-backbone comparisons in our earlier responses, since the training domain, backbone, and input interface differ.

Thank you to the authors for the detailed rebuttal and the new matched controls, ablations, and WorldMem results. I appreciate the transparency in reporting these numbers. The memory-specific gain is concentrated in revisit spatial fidelity, which is evaluated on a small number of tasks without confidence intervals against the matched controls. On VOST, the no-state variant also yields higher object similarity than LaST. Given the modest and weakly supported contribution of the memory itself, I therefore maintain my rating.

AuthorsOpenReview ↗

Follow-up on revisit fidelity and object appearance

Dear Reviewer HtE4,

Thank you for taking another look at the experiments. We especially appreciated your observation that a generated frame can match the reference more closely overall even when the target object is less well preserved. That distinction helped us connect the VOST result to what the record-deletion experiment actually measures.

What does structured memory add at the same budget? LaST achieved higher average revisit spatial fidelity than the learned Flat baseline at both scales, with gains of +0.016 at 1.3B and +0.034 at 5B. The reported interval for the 5B difference is [−0.002, 0.070]. We also examined reference reuse in four additional scenes with a frozen generator. Across the three scenes with revisit scores for both conditions, reusing the initial image improved average revisit spatial fidelity by 0.091. This gives us a complementary view of how visual history helps when the camera returns.

What does VOST tell us about the stored records? Across 12 sequences, LaST reduced full-frame error from 0.0311 to 0.0168, while the no-state condition achieved higher target-object similarity (0.839 vs. 0.794). As you pointed out, those scores capture different aspects of preservation. The deletion results add a useful detail. Removing only the target’s appearance record reduced object similarity to 0.451, while removing all target-related records brought it back to 0.818. This recovery suggests that the effect of an appearance record depends in part on which other cues remain available during generation.

We will add the matched-budget interval and the VOST interpretation to the appendix and highlight the additions in blue in the revised PDF so they are easy to find.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.