acceptodds
Under review as a conference paper at ICLR 2027

ARMem: Autoregressive Latent Memory for Streaming Video Generation

Abstract

Long-horizon autoregressive (AR) video generation requires compact memory that preserves persistent information while adapting to evolving content. Existing methods typically preserve history through post-hoc selection or compression of previously generated visual representations and reuse them as static context. While effective for preserving past information, these static memories cannot adapt to evolving video dynamics and may lead to quality degradation during long rollouts. In this work, we introduce **ARMem**, an autoregressive memory paradigm that extends AR modeling from video generation to memory evolution. Specifically, memory is represented as compact latent tokens that are jointly denoised with video tokens in a shared diffusion transformer. At each step, the previous memory state guides video synthesis, while newly generated content is incorporated into the next memory state to support subsequent generation. To mitigate optimization conflicts between video generation and memory evolution, we introduce Dual-LoRA Self-Attention, which decouples their parameter adaptations. We further introduce a Historical Readout Task that explicitly supervises the evolving memory to retain future-useful information, such as subject identity. Extensive experiments on LTX and Wan demonstrate improved short-video quality and long-horizon performance under bounded memory constraints.

Primary area:
generative models
Keywords:
autoregressive video model, streaming generation, video generation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_Wra1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ARMem, a framework for streaming video generation that jointly denoises video tokens and compact recurrent memory tokens within a shared diffusion transformer. It combines separate read/write LoRA adapters with historical identity reconstruction supervision to improve generation quality and long-term consistency under a bounded memory budget.

Critical strengths
  1. The paper addresses the tension between long-term consistency and continued content evolution under a limited context budget through a clearly structured recurrent memory framework. Joint video–memory updates, separate read/write parameter adaptations, and identity reconstruction supervision form a coherent technical approach with potential practical value.

  2. The evaluation covers both Wan and LTX backbones, short- and long-video generation, and image-conditioned generation, complemented by a user study and component ablations. This breadth suggests applicability across multiple settings. The ablations also provide evidence supporting the effectiveness of Dual-LoRA and readout supervision.

Critical weaknesses
  1. The incremental novelty over prior work and the necessity of the proposed designs are not sufficiently established. The core idea of jointly denoising video and a recurrent latent state already appears in AdaState, while the motivation and formulation of Historical Readout closely resemble the subject reconstruction objective in Memento. The potential contributions therefore lie primarily in the combination of compact memory representations, asymmetric attention, cross-chunk memory gradients, and read/write LoRA adapters. However, the paper does not compare joint denoising against a separate learnable memory updater under controlled conditions, nor does it independently ablate asymmetric attention or cross-chunk gradient propagation. Although Table 4 supports the benefit of Dual-LoRA, it does not distinguish whether this benefit arises from mitigating read/write optimization conflicts or simply from additional parameter capacity. Consequently, the current evidence does not sufficiently substantiate the positioning of the approach as a new memory-modeling paradigm.

  2. The experimental evidence for the performance advantage and its attribution remains limited. On the Wan long-video benchmarks, the improvements in average score over the strongest baseline are modest: 76.41 versus 75.75 at 30 seconds and 76.39 versus 76.01 at 60 seconds. The method does not consistently outperform the baselines in subject consistency, background consistency, or visual quality, and no uncertainty estimates are reported for these differences. The LTX comparison is also confounded by backbone differences: OmniForcing uses LTX-2, whereas the proposed method uses LTX-2.3, making it difficult to attribute the improvements solely to the proposed approach. Furthermore, AdaState, a particularly close baseline based on jointly denoised recurrent states, is absent from the experimental comparison. These limitations prevent a clear assessment of whether the proposed memory mechanism offers a meaningful advantage over the closest alternatives.

Questions for authors
  1. Under the same backbone, training data, training budget, and a comparable memory budget, which specific design choices account for any performance gains of ARMem over AdaState?

  2. Could the authors provide comparisons with matched backbones and long-video fine-tuning budgets, together with confidence intervals for the differences in the main metrics? Additionally, in Table 4, the full model achieves a 30-second Quality score of 82.14, slightly below the 82.24 obtained without memory. How do the authors explain this result, and could more targeted evaluations of long-term recall or identity retention help establish the contribution of the memory mechanism?

1 comment
Area_Chair_d5UnOpenReview ↗

Reminder: Please Follow Up on Authors’ Responses

Dear Reviewer Wra1,

A friendly reminder to check the authors’ responses and the other reviewers’ comments. It would be helpful to hear your thoughts on the differing assessments and whether they affect your evaluation.

Thank you for your time and effort!

AC

Reviewer_aDBiOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

ARMem introduces recurrent latent memory for streaming video generation. Its contributions include asymmetric attention, separate read/write LoRA adapters, sequential replay for memory learning, and historical identity readout supervision. Experiments on Wan and LTX cover five-second text/image-to-video generation and 30/60s rollouts, with qualitative demonstrations reaching 240s.

Critical strengths
  1. An impressive memory framework. ARMem combines joint video–memory generation, separate read/write adaptation, and explicit retention supervision, offering a substantive advance beyond heuristic history management.

  2. Broad empirical support. Experiments across two backbones show competitive short-video quality and strong long-horizon performance. Ablations and user preferences further support the method’s effectiveness.

Critical weaknesses

Quantitative evaluation primarily covers 30/60s videos; the 240s rollout is qualitative. However, aggregate video metrics do not directly establish whether memory preserves information through prolonged occlusion, scene changes, or delayed reappearance. Targeted retention tests would better substantiate the central claim.

Questions for authors

Beyond aggregate video scores, what evidence demonstrates that the recurrent memory preserves historical information unavailable from the fixed sink and recent window? Does the current evaluation measure how this retention changes with rollout length?

Reviewer_aqSFOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

ARMem addresses streaming autoregressive video generation with bounded historical context. It jointly denoises video tokens and a compact recurrent memory state within a shared diffusion transformer. Asymmetric attention controls information flow, while separate read/write LoRA adapters provide task-specific adaptation. Training combines distribution matching distillation, sequential replay that propagates gradients through memory updates, and historical readout supervision for identity reconstruction. The method is evaluated on Wan and LTX for short- and long-horizon generation, with additional image-to-video experiments on LTX. The paper reports competitive generation quality and improved aggregate long-video scores, supported by component ablations and a user study.

Critical strengths

The method addresses how memory is trained, not just how it is stored. Sequential replay allows later generation losses to supervise earlier memory updates, while historical readout introduces an explicit objective for preserving identity information. Together with the asymmetric attention structure and read/write adaptation, these choices form a coherent approach to learning a compact, evolving memory within the generator.

The empirical evaluation covers two backbone families and multiple generation horizons. On Wan, ARMem achieves the highest reported aggregate scores at both 30 and 60 seconds, with competitive consistency and strong dynamic degree. Component ablations and a blinded user study provide complementary evidence beyond qualitative examples. This breadth supports the method’s practical value, even though improvements are not uniform across all metrics.

Critical weaknesses

The comparison with OmniForcing is not a like-for-like comparison of memory methods. Appendix C states that ARMem uses LTX-2.3, whereas OmniForcing uses an LTX-2 checkpoint, so the LTX gains highlighted in Section 4.2 may partly reflect the backbone difference. This comparison remains informative about the complete systems, but does not isolate ARMem’s contribution. Table 4 provides relevant internal controls; clarifying whether initialization and training schedules are matched would strengthen the attribution.

The evidence for long-term identity retention is less direct than the evidence for generation quality. Although Table 3 reports subject consistency, the component ablations in Table 4 only report aggregate Quality and Overall at 30 seconds; Section 4.3 defines the latter as video–text alignment. These results support the usefulness of the proposed components, but do not directly establish how much recurrent memory and readout supervision improve identity retention. A subject-consistency breakdown of the existing ablations would better support this claim.

Questions for authors

1.Could you provide a comparison with OmniForcing that holds the backbone version fixed? Please also clarify whether the full and w/o Memory variants in Table 4 share the same initialization and training schedule. 2.Could you report 30-second subject consistency for the full model, w/o Memory, and w/o Readout Sup. in Table 4? This would more directly assess their contributions to long-term identity retention.

1 comment
Area_Chair_d5UnOpenReview ↗

Reminder: Please Follow Up on Authors’ Responses

Dear Reviewer aqSF,

A friendly reminder to check the authors’ responses and follow up with any comments or remaining concerns. Your feedback would be greatly appreciated.

Thank you for your time and effort!

AC

Reviewer_FojMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ARMem, a recurrent latent memory for autoregressive video diffusion. A shared diffusion transformer jointly denoises each video chunk and a new memory state. Video tokens read the previous memory, while the new memory also attends to the current video; an asymmetric mask prevents video tokens from reading the live memory. Separate read and write LoRA adapters support these roles, and an auxiliary identity reconstruction task encourages information to remain recoverable from memory. Training adapts Self Gradient Forcing to replay memory updates sequentially with gradients across chunks. Experiments on Wan and LTX cover five-second generation, quantitative evaluation at 30 and 60 seconds, component ablations, and a pairwise user study. A 240-second rollout is shown qualitatively.

Critical strengths
  1. The method gives memory reading, memory writing, and historical reconstruction distinct roles within one generator. The asymmetric attention pattern makes the intended information flow clear, and the authors correctly recognize that future generation losses must reach earlier memory updates when current video cannot read the live memory. Restricting the readout branch's historical conditioning to the previous memory is also a sensible design. Together, these choices form a coherent approach to learning a compact recurrent state, with explicit supervision for information that should survive successive updates.

  2. The evaluation provides useful evidence that the complete system improves video generation. The Wan results include several recent memory mechanisms, including Echo-Infinity, rather than only a sliding-window baseline. ARMem obtains the highest reported average at both long-video horizons, and its Wan average remains almost unchanged from 30 to 60 seconds. Appendix C specifies shared prompts and replica seeds, with five samples for each of 128 long-video prompts. The blinded user study also favors ARMem over three baselines, using randomized presentation and left-right placement. The component ablations and results on a second model family add useful breadth, even though some comparisons need stronger controls.

Critical weaknesses
  1. The advance over joint recurrent-state denoising needs a more direct comparison. AdaState [1], which the paper cites, already denoises a hidden state jointly with video in the same generator and carries that state forward as the next chunk's reference. Thus, learning the memory transition inside the diffusion transformer is not, by itself, a new distinction from prior work. ARMem differs in meaningful ways: its 16-token memory accompanies a fixed sink, and it introduces asymmetric attention, separate adapters, and cross-chunk memory gradients. These differences could be valuable, but AdaState is absent from the comparisons. The readout objective also follows the memory-based subject reconstruction idea in Memento [2], as the authors acknowledge. The contribution therefore depends on showing what the particular combination adds, rather than contrasting learned memory evolution broadly with static or external history management. The existing comparisons with Echo-Infinity are useful, but do not resolve this close architectural comparison.

    The LTX comparison has an additional limitation. Appendix C states that OmniForcing uses a released LTX-2 checkpoint, whereas ARMem uses LTX-2.3. Consequently, the 2.93-point five-second total-score gain does not isolate the effect of the proposed method. Table 4's no-memory variant is a useful control within the authors' training recipe, but it does not compare ARMem with another learned recurrent state. A matched comparison with AdaState, and a clear account of the no-memory variant's training and gradient paths, would substantially strengthen the case for the method.

  2. The experiments demonstrate generation improvements more directly than they demonstrate the claimed long-term memory behavior. ARMem retains an initial sink and recent context. Section 3.4 supervises reconstruction of the subject in the conditioning frame using crops from the same identity pool. This is a reasonable training objective, but it does not establish that the recurrent state preserves information acquired later, after that information has left the recent window. Since memory updates can repeatedly access the fixed sink, subject reconstruction alone cannot distinguish persistent storage from repeatedly recovering information from that reference. Subject-consistency scores are relevant, but do not isolate what the memory stores beyond these explicit references. The reported evaluation includes no controlled delayed-recall test or static-memory comparison addressing this distinction.

    The ablations offer some support, with trade-offs in the reported point estimates. Table 4 evaluates LTX without Streaming Long Tuning. Its 30-second video-text alignment score, labeled Overall, is 24.79 with memory versus 24.14 without, while Quality is 82.14 versus 82.24. With readout supervision, the reported quality scores are higher, while the five-second semantic score is 79.03 versus 79.15 without it. These measures do not directly test identity retention. In Table 3, the Wan 60-second average exceeds Echo-Infinity's, 76.39 versus 75.83, while imaging quality is lower, 68.37 versus 70.85, and video-text alignment is lower, 24.58 versus 24.84. The authors describe Wan consistency as competitive, which is fair; my concern is that aggregate generation scores do not isolate what the recurrent memory retains. Paired uncertainty would also help establish which differences are reproducible. Finally, the LoRA ablation does not include a shared adapter with comparable parameter count, so it does not isolate the benefit of read/write separation from that of additional adaptation capacity, or directly test the gradient-conflict explanation.

I lean toward weak rejection. The complete system is promising, and the Wan comparisons and user preferences provide positive evidence. My main reservation is that the paper has not yet shown a sufficiently clear advance over joint recurrent-state denoising, or directly established the memory behavior used to explain its gains. A matched recurrent-state comparison and a focused retention experiment could materially change my assessment.

References

[1] Y. Dalva and P. Yanardag. AdaState: Self-Evolving Anchors for Streaming Video Generation. arXiv:2605.30349, 2026.

[2] X. Wei et al. Memento: Reconstruct to Remember for Consistent Long Video Generation. arXiv:2606.14667, 2026.

Questions for authors
  1. How does ARMem compare with AdaState under a common backbone, training budget, and explicit context budget? Which changes account for any gain over joint state-video denoising? Please also define Table 4's no-memory variant: does it retain the two-pass context-gradient training, and how are its readout loss, adapters, and auxiliary training data handled when memory is removed? This would help separate the effect of the memory architecture from that of the training recipe. For LTX, how much of the advantage over OmniForcing remains when its training recipe uses LTX-2.3?

  2. Can the memory retain a subject attribute or event introduced after the initial frame, once the relevant frames have left the recent window? A small controlled test, including freezing or resetting memory while keeping the sink and recent context fixed, would address the central memory claim more directly. For the readout objective, how does reconstruction performance change with rollout length, and how much does it depend on the memory input rather than the noisy reconstruction target?

  3. Section 3.3 explains the no-gradient rollout followed by sequential replay with undetached memory states. Could you provide pseudocode making the remaining implementation details explicit? In particular, how is the predicted memory used to advance its noisy latent between denoising steps, and which recorded video and memory tensors remain detached during replay? Appendix B gives learning rates and context sizes, but not the LoRA ranks or the precise routing of projections and token groups through the read and write adapters. The memory and time cost of sequential replay would also clarify its training requirements.

  4. Could you report paired uncertainty for the main long-video and ablation comparisons? For the long-video benchmark, the five samples should remain grouped under each prompt. Please also specify the normalization and weighting used for 30-second Quality in Tables 4 and 5. For the LoRA claim, does the benefit remain against a shared adapter with comparable parameter count? These checks would help determine whether the component gains support the proposed explanations.

1 comment
Area_Chair_d5UnOpenReview ↗

Friendly Reminder: Please Follow Up on Authors’ Responses

Dear Reviewer,

A friendly reminder to check the authors’ responses and follow up with any comments or remaining concerns. Your feedback would be greatly appreciated.

Thank you for your time and effort!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.