acceptodds
Withdrawn from ICLR 2027

Temporal Residual for Consistent Autoregressive Long Video Diffusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Autoregressive video diffusion enables efficient long-video generation, but maintaining visual consistency while allowing scenes to evolve remains challenging. Existing approaches improve robustness to generated histories or retain selected visual anchors, yet bounded context windows lose access to earlier appearance cues, while fixed visual anchors cannot summarize an evolving history and restrict video motion. To resolve the three-way tension among long-range consistency, meaningful dynamics, and the efficiency of bounded-context autoregressive video generation, an efficient global memory that evolves with video progress is needed alongside local context and fixed visual anchors. We introduce Temporal Residual (TR), a lightweight evolving plug-in global state that extends pretrained generators through brief post-training. TR combines three key designs: Evolving Global State, Dual-Path Memory Injection, and Chunk-Synchronous Evolution Schedule. Together, they make evolving visual history available throughout generation while allowing the scene to continue changing. Applied to Self Forcing and LongLive with brief post-training, TR improves long-range consistency and visual quality while preserving meaningful dynamics. Human preferences and ablations further support the value of this global state.

Primary area:
generative models
Keywords:
Autoregressive Video Diffusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9Ln6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

From my understanding, the paper tries to add a small recurrent memory, “Temporal Residual” (TR), to bounded-context autoregressive video DiTs. Each of the 30 Wan2.1-1.3B blocks gets one d-dimensional state token, G^ℓ, initialized to zero. he model reads it in two ways: 1)AdaLN path: an MLP on [timestep embedding; pooled state] adds a correction to the six host modulation vectors. 2)Attention path: the state is projected with the host Q/K/V (RoPE at the chunk’s first frame, scaled by a learned γ^ℓ). Visual queries attend to it alongside the host KV context. The state’s own query attends to itself and the current chunk. TR adds 118.4M parameters and is post-trained jointly with the host for 2,000 iterations (400 generator updates) under the host’s DMD objective. Evaluation is on 60s videos from 256 MovieGenBench prompts × 5 samples using six VBench-Long dimensions.

Critical strengths
  1. A cheap, host-preserving interface that transfers across two cache policies. TR leaves the host’s KV window and cache untouched. It adds negligible attention cost (~0.01% more query–key pairs) and a state that is about 0.003% the size of the KV cache. The same module attaches to a sink-free host (Self Forcing) and a sink-based one (LongLive). The design choices are sensible and well argued: chunk-synchronous writes, so that noisy denoising passes don’t pollute the state, and a read-only clean recache. On Self Forcing the improvement is visible and fairly large: +1.8 subject consistency and +3.8 imaging quality. The user study also favors it strongly (~86% tie-adjusted on consistency and overall quality). If those gains hold up against a matched control, this would be a practical add-on for streaming generators. That is the main thing in the paper’s favor.

  2. TR+LongLive: dynamic degree rises from 35.18 to 52.85, with consistency and quality roughly unchanged (+0.02 to +0.63) while TR+Self Forcing: subject consistency rises from 96.05 to 97.85 and imaging quality from 66.59 to 70.39, while dynamic degree falls from 57.77 to 49.91.

Critical weaknesses
  1. I think the gains can’t be attributed to TR, because there is no matched post-training control. Both host baselines are the released checkpoints. TR variants get an extra 2,000 iterations of joint host+TR training under DMD. For example For Self Forcing, that training includes 93–240-frame extended self-rollouts on half of the iterations (Table 3). The released Self Forcing was trained on short rollouts only, and long-rollout tuning by itself is exactly what LongLive and Self-Forcing++ showed reduces long-horizon drift. MOreover, The closest methods the paper cites, VideoSSM and ARL2 (and also LongLive-RAG and AdaState), are discussed but never compared. Additionally, MALT Diffusion (Feb. 2025) is not cited at all. It uses a fixed-size per-layer memory updated after each segment from the clean segment, with stop-gradient on the prior memory and read through added attention. That is very close to TR’s state, write schedule and stop-gradient. Recurrent Memory Transformer-style memory tokens, TTT layers for minute-long video (CVPR 2025), and block-SSM memory with local attention (Po et al., ICCV 2025) are also absent.

2)I don't think “Long-range consistency” is actually measured, and the dynamic-degree evidence is fragile and read in both directions. VBench-Long splits each 60s video into 30 short clips and scores subject and background consistency within each clip. Those numbers can’t detect whether the child at t=60s matches the child at t=0s, which is the paper’s central claim (Fig. 1).

  1. The ablations make me doubt the headline delta is stable: Four identical state tokens, which the authors acknowledge carry the same information and only change attention mass, drop dynamic degree from 52.85 to 4.96. Restricting TR to the last 10 or first 10 blocks gives 26.78 / 25.83, while the last 20 give 56.00. All of these are single training runs with no seeds or variance.
Questions for authors
  1. Please report each host post-trained with the identical recipe without TR: same iterations, data, extended rollouts and prompt switches, LoRA/full fine-tuning as in Table 3. Without this, I can’t attribute the Table 1 deltas to the memory. A TR-zeroed version of the same run (state forced to 0) would probably also work.

  2. The AI-use statement says LLMs were used “to process the data used for the experiments”. What exactly does that mean: prompt extension, prompt-pair creation for switches, or filtering?

3 Can you add a metric that measures long-range identity directly? For example, DINO/CLIP or face-ID similarity between the first clip and clips at 20/40/60s, or a retrieval test where an object reappears after leaving the cache window. Does TR improve on the host there?

Reviewer_t31bOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Temporal Residual (TR), a persistent state added to pretrained autoregressive video diffusion generators while retaining their bounded local key-value caches. Each transformer block maintains a compact recurrent state intended to summarize generated history. The state is read through adaptive layer normalization and attention, and updated once per committed chunk. Joint post-training integrates TR into Self Forcing and LongLive, two Wan2.1-based generators with different cache policies. Evaluation uses 60-second videos from 256 MovieGenBench prompts, five samples per prompt, automatic metrics, paired human judgments, and LongLive-based configuration ablations.

Critical strengths
  1. The integration interface is concrete and potentially useful (Section 3; Appendix A). Equations (1)-(7) and Algorithm 1 specify how a fixed-size state can persist beyond cache eviction, how it is read, and when chunk writes and clean recaching occur. Implementation on sink-free Self Forcing and sink-based LongLive supports compatibility with two cache policies, although both share the same Wan2.1 backbone. Appendix B appropriately acknowledges limitations of initialization and memory retention.

  2. The evaluation includes complementary automatic dimensions, paired human judgments with ties, and visual comparisons. Table 1 reports a 17.67-percentage-point dynamic-degree increase for LongLive and a 3.80-point imaging-quality increase for Self Forcing; Table 4 reports tie-adjusted overall preferences of 65.7% and 86.8%. These provide preliminary positive signals for the evaluated configurations and examples. Table 2 also usefully illustrates that high measured consistency can coexist with little detected motion. Uncertainty, coverage, and attribution limit the strength of these observations.

Critical weaknesses
  1. The reported gains are not adequately attributed to TR or characterized for uncertainty. Section 3.5 and Appendix A describe joint adaptation, including full-parameter tuning for Self Forcing and LoRA for LongLive. Table 1 does not clearly establish whether host baselines received matched additional training. Without checkpoint provenance and a host-only continuation matching data, teacher, rollout lengths, update budgets, and trainable host parameters, improvements from the complete adaptation recipe cannot confidently be attributed to TR. This is particularly consequential because additional host training can change the same output properties used to evaluate the proposed state.

Statistical precision is also missing. LongLive's subject-consistency, background-consistency, and motion-smoothness changes are only 0.10, 0.03, and 0.02 points. Close point estimates do not establish improvement or non-inferiority. The larger dynamic-degree change measures visual change aggregated over the evaluated clips; as Section 4.2 notes, it can also arise from drift or flicker, and it does not by itself establish motion quality or historical-memory use. Self Forcing improves several quality scores while its dynamic degree decreases by 7.86 points, so the benefit differs by host. Human evaluation provides support on selected examples, but uses only ten different video pairs per host, each rated by 37 participants. Pooled votes do not establish broad prompt coverage. Paired uncertainty accounting for prompt grouping, repeated-training information if available, and per-pair human results are needed to assess robustness. Missing uncertainty does not itself prove that the differences are noise.

  1. The reported results do not isolate dependence on distant historical content. Section 3 already gives an operational read-generate-write interface and defines the three designs before the architectural detail. The names are dense, but that is not the basis of this assessment. The remaining concern is whether the gains depend on historical information beyond the retained local/sink context, rather than on additional adaptation capacity.

Losing direct access to an old frame does not establish that information relevant to future generation is absent from recent frames or cached representations. A fixed opening anchor also does not imply that the host has no long-term reference. The abstract's claim that an evolving global memory "is needed" is therefore stronger than the failure case established. A concrete case identifying the missing historical information and its required use would make the design hypothesis assessable; a universal necessity proof is not required.

Table 2 varies state-token count, layer coverage, and Carry/Rebuild only on LongLive. Appendix B.4 acknowledges symmetry among the four state entries, and Section 5 acknowledges that Carry/Rebuild compares complete training recipes. These experiments show configuration sensitivity, but do not isolate dependence on historical content beyond the retained local/sink context from added adaptation capacity. They also leave the practical increment over the recurrent-memory approaches acknowledged in Section 2 unclear. A persistent state pathway is operationally specified; its distinctive benefit as useful distant-history memory remains insufficiently demonstrated. One well-controlled test of historical-content use would be more decisive than additional configuration sweeps. The absence of a toy example, explicit forget gate, or memory theorem is not by itself a reason to reject the method.

Questions for authors
  1. Training clarification: Did the Table 1 host baselines receive matched additional post-training? Please specify checkpoint provenance and adaptation budgets, and provide matched host-only continuation results if available.

  2. Conceptual clarification: What concrete generation failure requires historical information that the local cache and opening anchor do not adequately supply? Please connect that case to the three named designs and the "compact recurrent history channel," and distinguish the structurally persistent state from the historical-memory function actually supported by the results. This would help identify the specific contribution relative to the closest recurrent-memory methods.

  3. Mechanism evidence: Which existing result most directly shows that TR's benefit depends on historical content beyond the retained local/sink context? If existing evidence cannot separate this from added adaptation capacity, one possible additional test is a capacity-matched comparison removing historical-content dependence, or a controlled history-dependent evaluation. These are alternatives, not a request for exhaustive ablations.

  4. Statistical precision: Please provide available uncertainty estimates accounting for grouping by prompt, clarify generation-seed matching, and report any repeated adaptation runs. Per-video-pair human preference counts and the pair-selection rule would help assess coverage beyond pooled judgments.

  5. Protocol clarification: Section 5 presents Carry/Rebuild as a prompt-switch comparison, and Table 2 labels those rows as the state policy at a prompt switch. Please specify the switching protocol used for Table 2, including switch timing, prompt-pair construction, and whether its evaluation set differs from Section 4.1. Which TR host produced each supplementary video labelled Ours?

Reviewer_2ZRgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Temporal Residual (TR), a small plug-in memory module for autoregressive long-video diffusion models. These models generate video chunk by chunk with a limited KV cache window, so they forget early content once it leaves the window. Fixed anchor frames help with consistency but tend to make the scene static. TR adds one learned state token per DiT block that carries a running summary of the video. The state is read through AdaLN modulation and attention, and is updated once after each chunk is finalized. The authors attach TR to Self Forcing and LongLive with a short post-training stage of 2,000 iterations, adding about 118M parameters. On 60-second MovieGenBench videos, TR improves consistency and visual quality on both models. It also substantially increases motion for LongLive. Ablations suggest that using more state tokens or injecting the state into fewer layers makes videos overly static.

Critical strengths
  1. The design is lightweight at inference. It adds one state token per block and leaves the host’s cache and attention window unchanged, and the same module transfers to two different hosts (Self Forcing without a sink, LongLive with a sink). Note that it still requires post-training together with the host.
  2. The empirical gains are consistent. Consistency and visual quality improve on both hosts, LongLive’s dynamic degree rises substantially (35.18 → 52.85), and the human study agrees.
Critical weaknesses
  1. The experiments don’t support the claims. TR gets 2,000 extra post-training iterations while the hosts get none. No experiment directly tests whether TR remembers content after it leaves the cache, for example a subject reappearing later. Dynamic degree is read as better both when it drops (Self Forcing) and when it rises (LongLive). The consistency gains on LongLive are tiny (+0.02 to +0.10), with no seeds or variance reported. The ablations also show the design is fragile, using 4 tokens, fewer layers, or Rebuild makes the video nearly static, which looks more like a tuning knob than a memory.
  2. The qualitative results don’t show a clear improvement. In Figures 1 and 3, TR+LongLive looks very similar to LongLive, and the claimed gains in consistency and motion are hard to see from the frames. More distinctive examples would make the claims more convincing, such as long-range reappearance or large scene changes, ideally with side-by-side videos where the difference is obvious.
  3. Missing efficiency and reproducibility details. Efficiency is only estimated on paper, with no latency or memory measurements. GPUs, training time, batch size, and the prompt source are not reported, and the code is not released.
  4. Missing related work. Several closely related long-video methods are not cited, including Infinity-RoPE[1], Context Forcing[2], MemRoPE[3], and Stable Video Infinity[4]. Some of these also target long-range consistency with bounded memory, and some are training-free. The paper should discuss how TR differs from them, and compare against those with public code.

[1] Yesiltepe, Hidir, et al. "Infinity-rope: Action-controllable infinite video generation emerges from autoregressive self-rollout." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

[2] Chen, Shuo, et al. "Context forcing: Consistent autoregressive video generation with long context." arXiv preprint arXiv:2602.06028 (2026).

[3] Kim, Youngrae, et al. "Memrope: Training-free infinite video generation via evolving memory tokens." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.

[4] Li, Wuyang, et al. "Stable video infinity: Infinite-length video generation with error recycling." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors

See weekness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.