acceptodds
Under review as a conference paper at ICLR 2027

Memorizon: Training World Model Beyond Their Context Window

Abstract

Streaming world models should render a place consistently across repeated visits. Directly supervising such revisits requires training samples that capture both visits, often spanning minutes. Yet dense attention over the full span incurs quadratic cost, making long-span supervision expensive. Memorizon breaks this coupling: long spans are needed for supervision, but not for attention, since the two visits can share a forward pass without including every intervening frame. A training sample covers a span of any length but is scored only on its last chunks. Instead of tokenizing the history before them, each scored chunk retrieves its own top- latents by camera co-visibility, and the union of these requests forms a shared bank. The bank is bounded by , so the sequence stays bounded however long the span; at the shortest span the recipe is exactly conventional training. Adding the bank raises the cost of a step once; beyond that, a longer span costs little, and going from 100 to 400 s adds 12% to the step time. Against a sliding-window baseline, retrieval raises revisit consistency on every split, and a span long enough to reach the first visit of each return adds a further 24% to 30%, at some cost in image quality; beyond that span, more length no longer helps. Filling the bank from another episode lowers revisit correlation by 83%, so the model uses what it retrieves. Project page: https://memorizon.github.io/

Primary area:
generative models
Keywords:
interactive world model, long-horizen memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_KgbXOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Memorizon, a training framework for long-horizon world models that decouples the training span from the attention context. It uses per-chunk retrieval and a shared memory bank to incorporate distant history without significantly increasing computational cost. Experiments show improved revisit consistency over sliding-window baselines and competitive results against existing world models.

Critical strengths

Simple and effective design. The proposed framework enables long-span supervision with bounded attention cost. The per-chunk retrieval and shared memory bank provide a practical solution to long-term consistency.

Solid experimental validation. The paper includes comparisons with recent world models and extensive ablations. The results support the effectiveness of retrieval and longer training spans, particularly for revisit consistency.

Critical weaknesses

Limited evaluation on real-world videos. Although the paper evaluates generation from real photographs, most training and quantitative evaluations rely on rendered scenes. It remains unclear how well the method generalizes to real-world videos.

Trade-off between consistency and visual quality. Longer training spans improve revisit consistency but can degrade generation quality. This trade-off may limit the practical benefits of the method.

Questions for authors

Q1: There appears to be a minor writing or notation issue in the abstract: "The bank is bounded by kK." Would "k × K" be clearer? Please also check the preceding sentence for readability.

Q2: Th evaluation includes real-world photographs as initial frames, but lacks experiments using real-world videos. Could the authors clarify whether the method can be trained and evaluated on real-world video data, and discuss the potential challenges?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer KgbX

Thank you for the review. Tables, videos and details: https://memorizon-rebuttal.pages.dev/

Q1 (abstract notation). Fixed. The abstract now reads "bounded by k x K", and the preceding sentence is split for readability.

Q2 (real-world video). Our view, with details and examples at https://memorizon-rebuttal.pages.dev/e6/:

  1. Real video has no accurate camera labels. Cameras for real video are estimated, and an estimation error can reach both camera control and our retrieval, which uses camera poses only: a wrong pose retrieves the wrong frames. As an illustration, we ran the released SolarWM checkpoint (trained on real video with camera annotations from its data pipeline) on prescribed trajectories: the distance its generated camera travels differs from the prescribed one by a factor of 2.0 to 10.8 (median 3.1 over 12 trajectories on 3 scenes), while rotation is closer. We did not isolate the cause, which may also involve how we convert our trajectories to its input format. One example clip is on the page.
  2. Real video has few revisits. Our goal is revisit consistency, and the bottleneck is long-revisit data: real clips rarely return to a place after minutes. Ordinary clip training also sees only a short tail of an episode, which is why Memorizon samples long spans that contain both visits.
  3. A possible recipe is to mix both. Real video without camera labels is used with short windows and no retrieval; UE data with exact cameras and long returns is used with long spans and retrieval. We have not run this mixed training and leave it as future work.

Consistency versus quality. Table 13 attributes the quality loss to a train/test mismatch of the bank (clean at training, self-generated at inference), not to a weaker generator. We tested whether a milder bank noise (sigma <= 0.2, half of the samples clean) narrows this gap at 200 s: against a matched run of the same recipe, Revisit is 0.421 vs 0.388 and imaging quality 0.717 vs 0.749, which is within the run-to-run variation we see between two trainings, so the gap is not clearly narrowed (https://memorizon-rebuttal.pages.dev/e4/).

These clarifications address my main questions. I continue to favor weak acceptance.

Reviewer_m2mqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MEMORIZON, a training framework designed to improve long-term revisit consistency in streaming video world models. The authors argue that conventional short-window training provides insufficient supervision for revisits separated by long time intervals, even when the model can access historical information during inference. To address this issue, MEMORIZON samples training spans from long videos while computing the training loss only on the last several target chunks. Each target chunk independently retrieves its Top-K historical latent frames based on camera co-visibility, and the union of these retrieved frames forms a shared memory bank. Experiments on rendered scenes and video generation initialized from real-world photographs demonstrate improved long-term revisit consistency compared to baseline methods.

Critical strengths
  1. A bounded-cost training framework for long video spans. By combining per-chunk retrieval with a shared memory bank, the proposed method enables training with historical information spanning longer time intervals while keeping the Transformer input sequence length bounded. The analysis of the shared memory bank and the relationship between training span and computational cost provides useful insights.

  2. Systematic experimental analysis of long-term memory capabilities. The paper provides several informative analytical experiments. In particular, the ablations on retrieval and training span, the distinction between starting-pose and mid-path revisits, and the interventions that replace the memory bank contents to verify whether the model actually utilizes historical information all help demonstrate the effectiveness of the proposed method.

Critical weaknesses
  1. The methodological novelty relative to closely related works is not sufficiently clear. Several key techniques employed by MEMORIZON, including sampling training targets from long videos, retrieving relevant historical frames based on camera geometry, and conditioning video generation on a limited number of historical memory frames, have already been explored to varying degrees in prior work, particularly Context-as-Memory (Yu et al., 2025) and WorldMem (Xiao et al., 2025). Although MEMORIZON further introduces per-chunk retrieval with a shared Top-K union bank and explicitly samples variable training spans, the paper does not sufficiently clarify the substantive differences between these designs and existing memory-conditioned training methods, or what additional capabilities they provide.

  2. The experimental comparisons and computational cost analysis are not sufficiently comprehensive. Although Appendix C.5 reports training step times and GPU memory usage, the additional overhead introduced by the proposed techniques relative to the baselines, particularly inference latency and GPU memory usage, has not been sufficiently analyzed. Furthermore, Context-as-Memory and WorldMem are highly relevant to the proposed method, yet neither is directly evaluated under the proposed evaluation protocol. Although Appendix C.6 explains the difficulties in running their released models, a more detailed comparison at the mechanism level, or reproducing their relevant training and retrieval strategies under a unified setting, would strengthen the experimental validation.

Questions for authors
  1. Novelty relative to Context-as-Memory and WorldMem. Could the authors further clarify the specific differences between MEMORIZON and these two closely related works in terms of long-video training sample construction, historical retrieval, and memory-conditioned supervision? What capabilities does the proposed training framework provide that cannot be achieved by existing long-history retrieval-based training methods?

  2. Computational cost analysis. Could the authors provide a more detailed analysis of the additional computational and memory overhead introduced by the proposed techniques relative to the baselines, including inference latency and GPU memory usage? In particular, the proposed per-chunk retrieval appears to substantially increase the memory bank size. Does this lead to a significant increase in computational and memory costs?

  3. Direct comparison with related methods. Context-as-Memory and WorldMem are highly relevant to the proposed method, yet neither is directly evaluated under the proposed evaluation protocol. Although Appendix C.6 explains the difficulties in running their released models, could the authors provide a more detailed mechanism-level comparison or reproduce their relevant training and retrieval strategies under a unified setting to strengthen the experimental validation?

  4. Construction of the evaluation dataset. The evaluation data and protocol used in this paper appear to differ from those adopted in related works. Although the appendix describes the data and evaluation metrics, could the authors provide further details on the construction of the evaluation dataset, particularly the selection criteria and design of the web photographs and camera trajectories? Additionally, why is it necessary to construct a new evaluation dataset or protocol rather than adopting existing evaluation settings?

  5. Evaluation over longer time horizons. Although the paper emphasizes long-term revisits separated by several minutes, the main experimental comparisons only report results for the first 60 seconds of generation. Could the authors provide further comparisons with the methods in Table 1 at longer revisit intervals?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer m2mq

Thank you for the review. Details and tables: https://memorizon-rebuttal.pages.dev/

Novelty relative to Context-as-Memory and WorldMem. We agree that sampling from long videos, geometry-based retrieval and conditioning on a few memory frames are not new in themselves. What Memorizon adds is how they are combined in training:

  • Several returns per sample. Each sample draws a span of m ~ U[9, m_max] chunks (10 s up to 100 / 200 / 400 s) and scores its last 10 chunks. Each scored chunk retrieves its own top-6 by frustum overlap minus pose distance and attends only to those six, so one forward pass supervises ten returns, and no scored chunk reads a clean copy of a latent it is scored on.
  • A shared union bank. The union of the ten requests (at most 60 slots) is one shared bank, so the sequence is bounded by 1 + bank + recent + targets whatever the span. The intervening history is never tokenized.

As far as we can tell from the two papers, Context-as-Memory takes a random segment of a long video as the target and retrieves its context frames (20 in their setup) from the rest of that video by field-of-view overlap, once for the whole segment; the context is concatenated with the target and the diffusion loss is computed on the noisy target latents. WorldMem retrieves from a memory bank by field-of-view overlap minus timestamp difference plus a similarity filter, reads it through memory cross-attention in a diffusion-forcing causal model, and is evaluated on Minecraft and RealEstate10K. We did not find in either paper a retrieval per scored chunk inside one sample, a union bank, or a span drawn per sample.

Generation speed. Memorizon generates chunk by chunk in one causal sequence, and the released model is distilled to 4 steps without classifier-free guidance; it takes about 1.3 s per second of video on one H200 at 1280x704, in bf16 without any quantization, excluding VAE decoding. Context-as-Memory uses a full-sequence (bidirectional) text-to-video model on 77-frame segments, compresses each retrieved context frame separately with the VAE, and slows as the context grows (0.97 FPS at 20 context frames, Table 2 of their paper). The 20-step models with CFG are costed below.

On one backbone, replacing per-chunk retrieval with one retrieval for the whole scored block lowers web-photo Revisit from 0.434 to 0.302, and one shared RoPE index lets a model trained on 10 s read frames 59 s old. We have narrowed the decoupling claims (abstract, Sec. 1, Sec. 5) to the union-bank mechanism.

Computational cost (https://memorizon-rebuttal.pages.dev/e5/).

  • Training: step time 8.02 s -> 9.00 s from 100 s to 400 s spans, peak memory flat (Table 10).
  • Bank size: at most 60 slots in training (n_scored x K). At inference each chunk reads only its own top-6, so per-chunk retrieval does not enlarge the sequence.
  • Retrieval scoring (pose only, one CPU thread): 0.65 ms at 10 s of history, 11 ms at 100 s, 32 ms at 400 s, 70 ms at 800 s per chunk.
  • Inference along a 400 s rollout (one H200, median per chunk of 1 s of video). 20-step models with CFG: sliding window 2.71 s at every history length, Memorizon (200 s model, k = 6) 3.18 s at 10 s of history and 3.19 s at 300-400 s (+17%). Released 4-step model, no CFG, 1280x704, VAE decoding excluded: 0.97 s without a bank, 1.30 s at 10 s of history and 1.31 s at 400 s with k = 6 (+35%), constant in history length. Peak memory: 22.6 GB for both 20-step arms; 22.5 GB (k = 6) vs 18.6 GB (no bank) for the 4-step model. One video each.

Reimplementation and comparison with CaM/WorldMem. We are reimplementing their training and retrieval recipes on our backbone, to compare them with ours under one setting; and we will reply here with the results.

Evaluation data. The 20 web photographs were downloaded from Pinterest (14 anime, 3 realistic, 2 3D cartoon, 1 CG render; street-level outdoor city scenes) and resized and cropped to 864x480. They test generalization: all of our training data is rendered in Unreal Engine, so we want to know whether the revisit ability still holds out of the training distribution. The camera paths are not designed per photograph: each photograph is given a path picked at random from the camera paths of our UE data. We define the protocol on camera paths because a return is then known by construction (the camera comes back within 0.5 units and 15 degrees of an earlier pose, at least 8 s later, having moved away in between), so Revisit can be scored for any model and any photograph.

Longer horizons. Figure 6 shows Memorizon's return intervals beyond 60 s. Running the Table 1 baselines to 400 s needs each model's own long-rollout pipeline, which we have not done; we state this as a limit.

Reviewer_m2mqOpenReview ↗

Official Comment by Reviewer m2mq

I thank the authors for their detailed response and additional clarifications. The explanations regarding computational cost and the evaluation protocol are helpful and address several of my concerns. However, I still have some reservations about the methodological novelty relative to closely related works, particularly Context-as-Memory and WorldMem. Specifically, it remains unclear how the proposed "several returns per sample" provides substantially different supervision from existing long-history memory-conditioned training, and whether the shared union bank offers benefits beyond efficiently organizing per-chunk retrieval results. A clearer discussion of these distinctions would help better establish the unique contribution of the proposed framework.

Overall, I appreciate the authors' efforts and will maintain my original rating.

Reviewer_MjTCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses long-horizon revisit consistency in streaming video world models by proposing Memorizon, a training recipe that decouples the supervision span from the attention sequence length.

Critical strengths
  1. Extending the maximum training span from 100s to 400s adds only 12% to step timewhile keeping peak GPU memory flat at 28.45 GB.
  2. Sharing RoPE index 1 lets a 10s-trained model read 59s-old frames whereas block-shared retrieval drops web-photo Revisit from 0.434 to 0.302.
  3. Gain metrics and fold-back checks isolate memory from camera tracking, bank swaps prove active memory reliance, and Table 13 traces VBench quality drops to train-test mismatch.
Critical weaknesses
  1. Web-photo Revisit peaks at 200s with lower quality, then falls to 0.396 at 400s. Could adding noise or dropout to clean bank latents during training mitigate this mismatch without full Self Forcing?
  2. Co-visibility relies on a fixed depth range and scale, ignoring occlusion and forward motion. How does the model behave in multi-room scenes when frustums overlap through walls?
  3. Recomputing 256-point frustum overlap over all past latents scales as O(n) per chunk without spatial indexing. What fraction of step latency goes to pose scoring at the end of a 400s rollout?
Questions for authors

See the weakness.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer MjTC

Thank you for the review. Tables and details: https://memorizon-rebuttal.pages.dev/

Noise or dropout on clean bank latents. First, the premise is less firm than it looked. A second, independent training of the 200 s and 400 s models flips the order (Revisit 0.388 for 200 s, 0.444 for 400 s; the second runs start from a different initial checkpoint), and the same setting moves by 0.05-0.08 between two trainings, so the drop at 400 s may be run-to-run variation (https://memorizon-rebuttal.pages.dev/e4/).

On the mismatch itself, we trained with bank noise at 200 s (sigma <= 0.2, half of the samples clean) and compared it with a matched 200 s run of the same recipe: Revisit 0.421 vs 0.388 and Gain 0.333 vs 0.312, but imaging quality 0.717 vs 0.749 and aesthetic quality 0.591 vs 0.596. The Revisit change is smaller than the run-to-run variation. Bank noise did not alleviate the drop in image quality. We did not test dropout of bank entries. A stronger noise tried earlier (sigma up to 0.5 on 85% of samples) raised seen Revisit slightly (0.405 -> 0.42-0.44) but cut quality (0.560 -> 0.49), and sweeping sigma at inference changed nothing.

Co-visibility ignores occlusion; multi-room scenes. You are right: frustum overlap measures containment, not visibility, so two rooms separated by a wall can look overlapping. We note this in Sec. 3.3 and in Appendix A.4 (Failure Modes). We also measured it (https://memorizon-rebuttal.pages.dev/e7/). The UE capture has no usable depth, so we used monocular depth (Depth Anything 3) with the known poses, on 399 chunks sampled as in training (spans up to 200 s). Among each chunk's top-6 retrieved frames, 24-49% of the content that falls inside the frustum is occluded (depending on the depth tolerance, tau = 0.25 to 0.1). For the top-6 frames retrieved by frustum overlap alone (no distance term) it is 31-56%. For 79-94% of the chunks that have a frustum hit, at least one retrieved frame has visible overlap >= 0.3. This is indirect evidence: the depth is monocular, not ground truth, and tau is our choice. Note that retrieval is not frustum overlap alone: frames are ranked by S = O - 0.2 D, where O is the frustum overlap and D = ||t_q - t_c|| + 4 * angle(f_q, f_c) is a pose distance (camera centres t, optical axes f). The distance term discourages far-away frames. On ground-truth latents, overlap alone reaches 47.4% of the achievable gain and the mixed score 57.2%. Neither term models occlusion: even with our retrieval, 24-49% of the in-frustum content is still occluded, and we have not tested an occlusion-aware retrieval. We did not run a dedicated multi-room test.

O(n) scoring cost (https://memorizon-rebuttal.pages.dev/e5/). Correct, and we clarify that only attention cost is constant. Measured on one CPU thread: 32 ms per chunk at 400 s of history and 70 ms at 800 s. At 400 s that is about 1% of the 3.2 s needed to generate a chunk. A spatial index would make it sub-linear; we have not needed one at these lengths.

Thank you for the detailed rebuttal and additional experiments. The computational overhead concern has been adequately addressed. The new analyses also clarify the training variability and occlusion limitations, although these issues remain partially unresolved. Overall, I appreciate the authors' efforts and will maintain my rating of 3 (Weak Accept).

Reviewer_btN3OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how to train camera-based video world models to maintain consistency in scene appearance across multiple revisits beyond the training attention window. Memorizon samples longer temporal spans but applies the training loss only to its final segments. Each scored segment utilizes camera overlap regions and relative pose to retrieve historical latent variables, and the union of these requests forms a bounded shared library with segment-specific access permissions. Experiments based on the Wan2.2 model examined the retrieval mechanism, library organization, and training duration, and compared the results with published world models. The retrieval mechanism improved consistency across revisit sessions, and extending the maximum training duration to 200 seconds yielded further improvements, including during a 400-second rolling simulation. These improvements come with trade-offs in image quality, which are explicitly discussed in this paper.

Critical strengths

We have successfully achieved a precise and useful separation between the supervision time span and the transformer context. The shared repository retains the Top-K history entries requested for each segment without having to process the entire intermediate video segment. Proposition 1 establishes selection equivalence and explicitly provides an upper bound on the sequence size. Using the configuration reported in this paper, the maximum sequence contains 105 latent slots regardless of the sampling span. Table 10 provides empirical support: increasing the maximum span from 100 seconds to 400 seconds increases the step time from 8.02 seconds to 9.00 seconds, while peak GPU memory usage remains constant. This paper also distinguishes between bounded Transformer context and retrieval scans performed on an ever-growing history.

These experiments aim to investigate whether memory is utilized efficiently and what benefits longer-span training can bring. Tables 2 and 7 compare retrieval operations with simply expanding the attention window, respectively. Table 12 modifies the repository content while keeping checkpoints, trajectories, and seeds unchanged: When returning midway through a path, replacing the retrieved frame with a random frame from the same history reduces the “revisit” metric from 0.440 to 0.207. Figure 6 evaluates return intervals beyond the standard 60-second reporting window and confirms the additional benefits of 200-second training. The real-world interventions in Table 13 also help explain this loss.

Critical weaknesses

The experimental results did not fully confirm the issue regarding the optimal training span. The results indicate that, in the tested setup, the 200-second configuration performed best; however, they did not prove that coverage saturation was the cause of the performance decline observed at 400 seconds. The reported first-visit coverage still increased from 84% to 93%. Increasing the upper bound of the uniform span sampler also alters the training history and the distribution of distances from anchors to targets. Saturation based solely on dataset size does not prove that the dataset content or training samples remain equivalent. Furthermore, the differences reported in the study are based on different initialization seeds (rollout seeds), rather than independently trained models. These factors limit the persuasiveness of the proposed explanations regarding span selection but do not negate the observed improvements.

External comparisons failed to fully distinguish between the effects of memory quality and actual camera motion. The revisit operation is sensitive to spatial misalignment, but the return pairs were selected based on a predefined trajectory, whereas the compared systems had different control interfaces and camera behaviors. Appendix C.7 provides a useful diagnostic method, but it measured yaw-related image offsets for only one rollout seed and did not include the closest memory comparator, CaR. Therefore, it cannot directly prove the equivalence of return poses across different systems. If control errors are sufficiently consistent, the return-to-origin argument could alleviate this concern, but this hypothesis has not been verified. Therefore, the within-backbone ablations consequently support the memory claims more strongly than the cross-system ranking.

Questions for authors

For the existing 100-, 200-, and 400-second configurations, can you report how often scored chunks actually retrieve a relevant first visit, rather than only whether that visit lies within the maximum span? Please distinguish the observed 200-second optimum from the explanation proposed for the 400-second decline, and clarify whether independent training runs support this ordering.

Using the existing rollouts, do the external-comparison conclusions remain consistent when returns are matched or stratified by realized motion error, or when fold-back and non-fold-back returns are evaluated separately? Including CaR in the camera-following diagnostic would help interpret the closest comparison.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer btN3

Thank you for the careful review. Both of your points are fair, and we ran new analyses for each. Tables and details are on an anonymous page: https://memorizon-rebuttal.pages.dev/

Q1. How often does a scored chunk actually retrieve a relevant first visit? (https://memorizon-rebuttal.pages.dev/e1/) We replayed the training sampler (380 segments, 4,000 spans per setting, ten scored chunks each) with the training retrieval score. A latent counts as relevant if it sees at least half of the chunk's view frustum (0.3 and 0.7 give the same picture).

max spanrelevant first visit existstop-6 contains onegiven one existsmedian age
100 s82.7%76.4%92.5%16.8 s
200 s91.0%82.6%90.7%24.5 s
400 s95.5%86.3%90.3%37.0 s

Longer spans help because more first visits are in range and the retrieved ones are older. Retrieval itself does not get better: given that a visit exists, top-6 finds it about 90% of the time at every span.

The 200 s optimum versus the 400 s explanation. We separate them in the revision. "200 s is best" is an observation of our first runs: a second, independent training of the 200 s and 400 s models flips the order (Revisit 0.388 for 200 s, 0.444 for 400 s; the same setting also moves by 0.05-0.08 between two trainings, https://memorizon-rebuttal.pages.dev/e4/), so independent runs do not support this ordering. "Saturation" is only a hypothesis, and we agree it is not established: raising the cap from 200 s to 400 s also changes the span distribution (spans <= 30 s fall from 10.2% to 5.0%; mean span roughly doubles, 106 s to 206 s). Coverage and span mix are confounded in our runs and we cannot separate them.

Bank content. The paper already tests this for the 100 s model (Table 12, Appendix C.8): with only the bank's frames replaced at inference, mid-path Revisit falls from 0.440 (retrieved top-K) to 0.207 (random frames from the same history), 0.105 (empty bank) and 0.073 (frames from another episode); Gain falls from 0.319 to 0.109, 0.019 and 0.016. We repeated it for the 200 s and 400 s models (https://memorizon-rebuttal.pages.dev/e2/, with example videos for each arm). With no bank, or a bank from another video, Gain falls to about zero (200 s: 0.359 to 0.041 / -0.002; 400 s: 0.285 to 0.013 / 0.005), and random frames from the same video recover only about a third. So all three models read their retrieved content. This does not explain why 400 s is below 200 s.

Q2. Do the external comparisons hold when realized motion is matched? (https://memorizon-rebuttal.pages.dev/e3/) We measured camera following on all five rollout seeds and added CaR. CaR follows the path as well as we do (correlation with the prescribed yaw 0.959 vs 0.955-0.958 for Memorizon); LingBot, Matrix-Game, Infinite-World, DreamX and HY-World are clearly worse (0.70-0.89). CaR's implied horizontal field of view is 60 deg, so its camera is narrower, not worse at following.

We then stratified returns by realized yaw error, with a per-video calibration that removes the field-of-view difference. Returns with error <= 10 deg, Revisit (mean over five seeds):

systemRevisitGain
Memorizon 100 s0.4410.305
Memorizon 200 s0.4830.361
Memorizon 400 s0.4350.310
CaR0.4250.289
HY-World 1.5 / Matrix-Game 3.0 / Infinite-World0.329 / 0.302 / 0.2750.146 / 0.104 / 0.164
LingBot-World-v2 / DreamX-World 1.00.163 / 0.1390.006 / 0.029

Among well-aligned returns, CaR is within seed noise of our 100 s and 400 s models and about 0.06 Revisit below the 200 s model. The other systems stay far lower, but most of their returns fall in the > 25 deg bin, which is the confound you describe. Fold-back and non-fold-back returns separately give the same picture (<= 10 deg: CaR 0.423 / 0.433; Memorizon 100 s 0.442 / 0.441; 200 s 0.481 / 0.489; 400 s 0.442 / 0.409). The yaw estimate comes from phase correlation of the generated video; it sees rotation, not translation, and is not a pose estimate.

Changes to the paper.

  1. The camera-following diagnostic moves from Appendix C.7 into Sec. 4.2, with five seeds and CaR.
  2. The cross-system ranking is described as indicative; the within-backbone ablations are the primary evidence for the memory claims.
  3. We no longer claim an advantage over CaR in camera following: the two follow the prescribed path equally well.
  4. Sec. 4.3 states that 200 s was best in our first runs, reports that a second training does not reproduce it, and presents saturation only as a hypothesis.

Thank you again; your comments made these claims more careful.

Thank you for running these analyses and for reporting the reversal in the second independent training run. The sampler replay answers my first question directly. Longer spans make relevant first visits available more often, and the top-six retrieval rate stays close to 90% conditional on such a visit being available. I agree with limiting the 200-second optimum to the initial runs and presenting coverage saturation as a hypothesis. The bank-replacement experiments across all three spans support the broader conclusion that the models use the retrieved content.

Including CaR and stratifying returns by realized yaw error across five rollout seeds substantially addresses my second concern. CaR is close to the 100- and 400-second models on well-aligned returns. Given the remaining uncertainty about translation, I agree with treating the cross-system ranking as indicative and placing the most weight on the within-backbone ablations.

These clarifications address my main questions. I continue to favor weak acceptance.

AuthorsOpenReview ↗

Thanks for discussion

Thank you for the constructive discussion. We wil make the changes described above in the revision.

Comments on the paper

1 comment

Comment: Hi Reviewers,

The authors have posted their rebuttal along with new results. Please review their responses, engage with the authors to discuss any remaining concerns.

Best, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.