acceptodds
Under review as a conference paper at ICLR 2027

CausalFold: Letting Later Video Blocks Read and Train Earlier Attention

Abstract

Autoregressive video diffusion models generate a video one latent block at a time, and each new block attends to earlier blocks through a key–value (KV) cache. In every attention layer, the cache returns the Values of earlier tokens, which are projections of their input to that layer. The output that an earlier token computes in the same layer, for example the corridor content that a doorway token selects with its own Query, reaches later blocks only through the next layer, and a later block’s loss reaches that attention only through deeper layers. We introduce CausalFold, which keeps each cached Key and stores the token’s attention output in place of its Value. A token then reads the Values of its own block, whose tokens are denoised jointly, and the outputs of earlier blocks, which are already final. The result is a recursion over blocks that ends after finitely many reads, without truncation or discounting. Each output remains a convex combination of Values, and the Jacobian of a later output with respect to an earlier token’s attention logits is the probability that attention paths visit that token times the differences among its candidates. CausalFold adds no parameters and keeps each backbone’s masks, sampler, and cache schedule. Against Standard-KV controls with identical training on five backbones, it raises transition completion by 2.3 to 4.9 points in all six text- and image-to-video comparisons, including from 37.9 to 42.7 for text-to-video on Wan2.1-1.3B at 5.4% longer generation time. In image-to-video, the trained model depends on the cached outputs, and removing the later-block gradient on earlier attention removes about half of the gain.

Primary area:
generative models
Keywords:
causal video generation, attention states, transformer cache, video diffusion

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_jVbNOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

In block-causal video diffusion with a KV cache, a later block reads an earlier token's Value (a projection of that token's layer input), so what the earlier token selected with its own Query reaches later blocks only one layer later, and later-block losses do not reach the earlier token's attention logits in the same layer. CausalFold keeps the cached Keys and stores, in place of the Value, the token's attention output RR. A token attends with the backbone's own weights over the Values of its own block and the cached outputs of earlier blocks, Rt=DttVt+∑s<tLtsRsR_t = D_{tt}V_t + \sum_{s \lt t} L_{ts}R_s. The recursion terminates in block order; R=(I−L)−1DVR=(I-L)^{-1}DV is a convex combination of Values (Thm. 1), and the Jacobian of a later output w.r.t. an earlier token's logits factorizes as visit probability times candidate differences (Thm. 2). No parameters are added. Each backbone is converted with 200 DMD updates on self-rollouts using a replay that sends gradients through every cache read; a Standard-KV control receives identical training. On Wan2.1-1.3B/14B, MAGI-1 4.5B, HunyuanVideo-1.5 8B and EchoWM 21.8B, TC-Bench transition completion rises by 2.3 to 4.9 points in all six T2V/I2V pairs (e.g. 37.9 to 42.7 T2V), StoryEval and temporal T2V-CompBench categories improve, VBench quality/consistency change by at most 0.3, and dynamic degree drops 0.4 to 0.7. Interventions show the trained model depends on reading outputs (restoring V drops TCR below Standard-KV) and that removing the later-block logit gradient removes about half the gain; MAGI-1, whose writer attends only within its block (zero visit probability), gains least. Long 60-s rollouts, WBench navigation, a 9,027-judgment human study, and 5 to 18% latency / up to 2x cache under GQA are reported.

Critical strengths

1. A simple, parameter-free change with a crisp mechanism, tested as a mechanism rather than as a number. The matched Standard-KV control shares checkpoint, data, sampler, updates and the replay-through-cache gradient path, so the comparison isolates what a cached read returns. The paper states predictions P1 to P4 and tests them with interventions that separate forward reads from the later-block gradient (history reads V; future-logit-off), and uses MAGI-1's self-only writer as a natural experiment for Theorem 2. The breadth (five backbones from 1.3B to 21.8B, T2V/I2V/camera control, 60-second rollouts, a large paired human study) is well beyond what is usual for a method of this size. This is the main reason I recommend acceptance.

2. The theory is modest but exact and does work. The path interpretation of W=(I−L)−1DW=(I-L)^{-1}D, the convexity/contraction result, and the PipΔpP_{ip}\Delta_p factorization are correct and are used to predict where the training signal vanishes and which factor limits it per backbone (Fig. 14).

Critical weaknesses

1. No uncertainty is reported for any headline comparison. Tables 1 to 4 and 7 give means over three training seeds with no standard deviation or interval. TC-Bench has 150 T2V / 120 I2V cases, the judges are VLM-based, the models receive only 200 fine-tuning updates, and the claimed differences are 2.3 to 4.9 points; seed-to-seed variation of that order is plausible. Appendix H describes the statistical units but never gives intervals for the main results. Since the conclusions rest on these differences, per-seed values or paired bootstrap CIs are needed; "all six comparisons positive" is weaker evidence than it sounds if the six are correlated through shared prompts and judges.

2. Trade-offs and training budget are under-examined. Dynamic degree decreases in every pair, VideoPhy-2 physical commonsense drops (3.50 to 3.40), T2V-CompBench object interaction drops, and the human preference for visual realism is only 57%. All comparisons use 200 generator updates after conversion; it is unknown whether the gap persists, grows or closes when both arms are trained longer. For the real-time use case that motivates block-causal generation, +5 to 18% latency and a doubled cache under grouped-query attention are not negligible and deserve more than an appendix table.

Questions for authors

Q1. Please add per-seed TCR (or mean, sd and a paired bootstrap CI over cases) for Tables 1, 2 and 4, and a prompt-level CI for the human-study preferences.

Q2. Please report a training-length sweep for the Wan 1.3B pair (e.g. 200 / 500 / 1000 updates for both arms). Does the TCR gap persist?

Q3. Why do dynamic degree and physical commonsense decrease? Is motion amplitude reduced, or are fewer videos crossing VBench's dynamic threshold while completing transitions?

Q4. Since RtR_t retains information from blocks that have been evicted from the physical window, how does CausalFold behave in interactive settings where the user changes the action (EchoWM/WBench)? Any sign of over-persistence or stale context when the command changes?

Q5. In the "History reads: V" control, the drop below Standard-KV is a distribution shift for a model trained on R; it demonstrates dependence, not that R is intrinsically better. Please phrase it that way (Section 5.3 currently reads as additional evidence for CausalFold).

Q6. Is CausalFold compatible with the rolling-window and sink designs of LongLive / Rolling Forcing? The long-video comparison in Table 7 is unmatched, so the relative merit there cannot be read off.

Suggestion: Report cache memory and latency for the 14B model under GQA if applicable, and discuss whether R could be stored per KV group (averaging query heads) at some quality cost.

4 comments
AuthorsOpenReview ↗

Response to Reviewer jVbN (1/3): uncertainty of the matched comparisons

(1/3) Thank you for a review that engages with the mechanism itself. You described the Standard-KV control as isolating "what a cached read returns", the P1–P4 interventions as testing a mechanism "rather than a number", and MAGI-1's self-only writer as a natural experiment for Theorem 2. That is the reading we hoped for, and your questions pushed us to back each of those points with the statistics and controls they needed. All changes are in blue; new analyses are in Appendix J.

W1 / Q1. Uncertainty for the headline comparisons. You were right that means alone were not enough, and that six positive comparisons sharing prompts and a judge are weaker evidence than they look. Appendix J.1 now reports both levels of variation, with the summary in Section 5.1.

Seeds. Each seed's TCR is a count over 750 T2V or 600 I2V videos. Seed SDs range from 0.6 to 1.2 points across the 12 arms, below the smallest gain (2.3). Every paired seed difference is positive:

PairΔ per seed95% case CIBonferroni low
Wan 1.3B T2V (Δ 4.8)5.1 / 4.5 / 4.8[2.6, 7.2]1.9
MAGI-1 T2V (2.4)2.8 / 2.1 / 2.3[0.8, 4.1]0.2
Wan 1.3B I2V (4.9)5.2 / 5.0 / 4.5[2.5, 7.4]1.6
Wan 14B I2V (3.5)3.0 / 3.5 / 4.0[1.4, 5.7]0.6
MAGI-1 I2V (2.3)2.7 / 2.0 / 2.2[0.5, 4.0]0.1
Hunyuan I2V (3.9)3.5 / 4.5 / 3.7[1.6, 6.1]0.9

Cases, with the correlation you raised. The interval resamples TC-Bench cases with all videos and seeds of a case kept together, and it resamples case indices jointly across comparisons that share a benchmark, so a prompt that favors CausalFold on several backbones counts once per draw. The last column is the lower end of a Bonferroni interval over all six comparisons. All six stay above zero. The two MAGI-1 bounds are the lowest, which is what the zero visit probability between its cache-writing calls predicts.

WBench (Table 10). Consistency, interaction, and physical plausibility exclude zero; the quality interval [-0.3, 0.5] contains it, so navigation quality is unchanged rather than improved.

Human study (Table 11). Resampling the 1,003 prompts with their nine judgments gives intervals that all exclude 50%, e.g. temporal consistency [73.51, 76.30] and visual realism [55.74, 59.10]. A conservative interval that uses only the published counts (variance bounded by m(1−m)) is also given and excludes 50% for every question.

Q5. Wording of the "history reads: V" control. We agree completely: restoring V in a model trained on R is a train–test mismatch. It shows that the trained model uses the outputs it reads, not that outputs are a better quantity to cache. Section 5.3 now says exactly this and points to the matched pairs of Tables 1–2 for the comparison of what to cache; the introduction was reworded the same way.

Parts (2/3) and (3/3) cover training length, the metric trade-offs, cost under GQA, command changes, and the long-video protocols.

AuthorsOpenReview ↗

Response to Reviewer jVbN (2/3): training length and metric trade-offs

(2/3)

Q2. Does the gap persist with longer training? Yes. Standard-KV can in principle relay what an earlier token selected through deeper layers, so this was the right control to ask for. We continued both arms of the Wan 1.3B pair from 200 to 500 updates for paired seeds 1 and 2, and compare them with the 200-update checkpoints of the same seeds (App. J.2, Table 12):

UpdatesStd-KV T2VCausalFold T2VStd-KV I2VCausalFold I2V
20038.142.944.849.8
50039.143.745.950.7

Both arms improve and Standard-KV gains slightly more, but the gap moves only from 4.8 to 4.6 points in T2V and from 5.1 to 4.8 in I2V (gaps are means of the per-seed gaps in Table 12, e.g. 5.2 and 5.0 for I2V at 200 updates, so they can differ by 0.1 from the difference of the rounded means above); every per-seed gap at 500 updates lies between 4.5 and 4.8, and Standard-KV at 500 updates is still below CausalFold at 200. This fits the mechanism: deeper-layer relay can recover part of the information, while the same-layer read and the later-block logit gradient are absent from Standard-KV at any training length. The final version will extend the sweep to 1,000 updates and all three seeds.

Q3. Why dynamic degree decreases. Your second hypothesis is the right one. VBench samples frames at about 8 fps and calls a video dynamic only if at least a quarter of frame pairs exceed the flow threshold, so a transition that completes early and then holds its end state can fail without moving less. Appendix J.3 measures this on VBench-I2V for all four backbones:

  • Amplitude is unchanged: the mean peak frame-pair score moves by at most 0.03 of the threshold, and the share of videos that ever exceed it by at most 0.4 points.
  • Duration changes: videos whose motion is too brief to count rise by 0.3 to 0.8 points, and the share of moving frame pairs falls by 0.7 to 1.8 points.
  • On TC-Bench-I2V, completed transitions pass the test less often than incomplete ones in both models (56.3% vs 68.9% for Standard-KV), and within each stratum the two models differ by under one point. More completions lower the rate by composition.

HunyuanVideo, which has the largest drop, also shows the largest (still small) peak change, so a little of its drop is amplitude; the appendix says so.

Physical commonsense, object interaction, and realism. Splitting VideoPhy-2 by semantic adherence (App. J.4) places the drop from 3.50 to 3.40 in videos that do not follow the prompt (mean PC -0.27), i.e. attempted but incomplete actions. Among adherent videos mean PC changes by -0.03, and more CausalFold videos reach PC ≥ 4 (52.5% vs 49.4%), which is where joint accuracy rises. Object interaction (−0.7) scores contact outcomes such as a collision; that outcome is settled within a few blocks by contact dynamics, not by one entity's state carried across blocks, which is the relation CausalFold's reads add. The 57.43% realism preference is modest, as you say, and its prompt-level interval [55.74, 59.10] lies above 50%, so realism is preserved rather than traded away. Section 5.4 now summarizes these points and Section 6 lists dynamic degree and PC as limitations.

AuthorsOpenReview ↗

Response to Reviewer jVbN (3/3): cost under GQA, command changes, long-video protocols

(3/3)

W2. Latency and cache under GQA deserve the main text. Agreed. Section 5.4 now reports generation cost, training cost, and matched-budget results together, and the Limitations paragraph of Section 6 states the GQA growth directly. Two new results: on MAGI-1, Standard-KV given ten evaluations per block is slower than CausalFold (112.64 s vs 110.09 s) and still 1.8 TCR points lower (App. J.5), and training time rises by 4.7% to 14.9%, below the generation overhead because the score models are unchanged (App. J.8).

Suggestion: 14B under GQA, and R per Key/Value group. Wan2.1-14B uses multi-head attention (as many Key/Value heads as query heads), so its cache has the same size in both arms; its 2.53 GiB higher peak comes from transient buffers of the recursion. MAGI-1 is the GQA case (24 query, 8 Key/Value heads). We implemented your proposal at inference on the trained checkpoint: store one output per Key/Value group, the mean over its three query heads. This returns the cache to Standard-KV size and keeps 56% of the MAGI-1 I2V gain (TCR 49.4 vs 48.1 and 50.4) at 105.37 s instead of 110.09 s. Section 6 now names this as the option for GQA backbones.

Q4. Interactive settings: stale context after a command change? Yes, a small amount, and your question made us measure it (App. J.9). Splitting WBench turns by whether the command changed, CausalFold follows a new command 0.8 frames later on average (33 ms at 24 fps), and 0.9 points more of its changed-command turns still follow the old command in their first second. This is the persistence the path argument predicts. It stays small, and on changed-command turns the interaction score is still higher than Standard-KV's (86.6 vs 85.7), because the scene revealed after the turn stays consistent with what was seen before. Section 6 lists it as a limitation with the number.

Q6. Rolling windows, sinks, and an unmatched Table 7. Yes, they are compatible. CausalFold changes what a cache entry holds, not which entries are kept, so window, sink, and re-cache policies apply unchanged: a sink block caches its within-block aggregation, a re-cache recomputes (K, R) with the new prompt, and Rolling Forcing's jointly denoised frames are evaluated in block order, as for MAGI-1's overlapping chunks (App. J.6). The two act on different quantities. A sink or a longer window decides which Values stay readable; a cached output carries what its block selected from entries that were later evicted (App. B, Fig. 10). You are right that Table 7 cannot isolate this, so App. J.5 does it at matched memory inside one protocol: Standard-KV windows of 24 and 40 frames still have 37/384 and 35/384 failures against 20/384. Matched pairs inside LongLive and Rolling Forcing need each method's own long-rollout training recipe; the final version will add them, and Table 7's caption already marks the external rows as unmatched.

For the AC discussion, in short.

  • The gains exceed seed and case variation, including jointly over the six correlated comparisons (J.1).
  • The gap persists after 500 updates and against Standard-KV given more compute or a longer window (J.2, J.5).
  • The metric trade-offs are explained and bounded: dynamic degree is motion duration, PC is failed attempts (J.3, J.4).
  • Your GQA suggestion works on MAGI-1 even without retraining (J.5).

Thank you again for one of the most constructive reviews we have received. Your recommendation already highlights the mechanism-level evidence; if these additions resolve the concerns on uncertainty and trade-offs, we would be grateful if you would consider raising your score and your soundness rating.

Thank you for the thorough response and the additional analyses. The new results address my main concerns: the per-seed values and case-bootstrap intervals (J.1) show that the gains exceed seed and case variation; the 500-update sweep (J.2) shows the gap persists with longer training; and the decomposition of dynamic degree and physical commonsense (J.3, J.4) gives a convincing explanation of the metric trade-offs. I also appreciate the revised wording of the "history reads: V" control, the measurement of command-change lag (J.9), and the per-KV-group variant for GQA.

I encourage the authors to include the matched LongLive / Rolling Forcing comparisons and the 1,000-update sweep in the final version, as promised. Given these improvements, I maintain my positive assessment and recommend acceptance.

Reviewer_BWsROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CausalFold for block-causal video diffusion. It replaces cached Values from earlier blocks with their completed attention outputs, allowing later blocks to read information selected by earlier tokens within the same layer. The paper analyzes the resulting finite recursion and gradient paths, then evaluates the method on text-to-video, image-to-video, and camera-controlled generation.

Critical strengths
  • This paper is well written and easy to follow.
  • The method is simple and parameter-free, with a clear explanation of both its forward behavior and gradient effects.
  • The evaluation is broad and controlled. Matched Standard-KV comparisons across five model configurations show consistent transition-completion gains, and the paired human study supports improved temporal consistency.
Critical weaknesses
  1. Generation is 5.4–18.0% slower, and MAGI-1’s cache doubles in size. While the paper reports these costs, it does not show whether the gains remain compelling against Standard-KV at the same latency or memory budget.
  2. The benefit depends on the cache-writing protocol. MAGI-1’s committed outputs do not incorporate earlier blocks recursively, and it shows the smallest gains alongside the largest slowdown. This makes it difficult to predict the method’s value for a new backbone.
  3. Dynamic degree declines in every image-to-video comparison, and Wan’s physical-commonsense score also falls. The human study is encouraging, but these metric trade-offs need a clearer explanation.
Questions for authors
  • How does CausalFold compare with Standard-KV at matched end-to-end latency or memory, for example when the latter uses more sampling steps or a longer cache window?
  • Which properties of a backbone’s cache-writing protocol predict whether CausalFold will provide substantial gains?
  • Why does dynamic degree consistently decrease despite improved transition completion and human motion-completeness ratings?
3 comments
AuthorsOpenReview ↗

Response to Reviewer BWsR (1/2): Standard-KV at matched latency and memory

(1/2) Thank you for a clear and fair review. You found the paper "well written and easy to follow", the method simple with "a clear explanation of both its forward behavior and gradient effects", and the evaluation "broad and controlled", with consistent gains across five configurations and a human study that supports better temporal consistency. Your three questions are exactly the ones a practitioner would ask before adopting the method, and answering them made the paper more useful. All changes are in blue; new analyses are in Appendix J.

W1 / Q1. Is CausalFold still better than Standard-KV at the same latency or memory? Yes, in both directions you suggested (App. J.5, summarized in Sec. 5.4). One detail shapes the design: a short Wan video has 21 latent frames and the cache holds 20 earlier ones, so on TC-Bench and VBench Standard-KV already sees every earlier block and a longer window cannot change its output. We therefore matched latency with more sampling steps on short videos, and memory with longer windows on 60-second rollouts, where eviction matters.

More sampling steps (short videos, Table 16). MAGI-1's shortcut model samples natively at several step counts, so it gives a clean latency match without retraining:

ModelNFETCR I2VTime (s)Peak (GiB)
Std-KV848.193.2731.99
Std-KV, same weights1048.6112.6432.02
CausalFold850.4110.0935.66

With ten evaluations per block, Standard-KV is already slower than CausalFold and gains under one point; CausalFold stays 1.8 points ahead. Extra steps refine each block, but they do not change what the block reads from earlier blocks, which is where the transition information lives. The final version will add the same comparison for Wan with a Standard-KV student retrained at five steps.

Longer cache window (60-second rollouts, Table 17). All rows are scored on the 384-video 60-second cohort of App. I.2. Each extra slot holds 0.268 GiB, so a 24-frame window costs about the memory CausalFold adds (1.07 vs 1.27 GiB):

ModelWindowSubjectBackgroundFailuresTime (s)Peak (GiB)
Std-KV2091.693.340/384204.6221.86
Std-KV2492.193.737/384231.0822.93
Std-KV4092.594.035/384319.4727.24
CausalFold2095.196.020/384216.3523.11

Even with twice the window and far more memory and time, Standard-KV remains below CausalFold on both consistency scores and has nearly twice the failures. A longer window keeps more raw Values; a cached output keeps what its block selected from blocks that have since been evicted.

MAGI-1's doubled cache. Storing one output per Key/Value group, applied at inference to the trained checkpoint, returns MAGI-1's cache to the Standard-KV size and keeps 56% of its I2V gain (TCR 49.4, at 105.37 s and 32.71 GiB). Wan and HunyuanVideo use as many Key/Value heads as query heads, so their caches are the same size in both arms.

Part (2/2) answers your questions on backbone choice and on dynamic degree.

AuthorsOpenReview ↗

Response to Reviewer BWsR (2/2): which backbones benefit, and dynamic degree

(2/2)

W2 / Q2. Which properties of the cache-writing protocol predict the gain? You are right that the paper showed MAGI-1's small gain without turning it into guidance for a new backbone. Appendix J.7 now does this. Theorem 2 gives two necessary conditions for the later-block gradient, positive visit probability and unequal candidates, and the first one is decided by the cache-writing call. Both can be computed on the pretrained checkpoint before any training. Table 18 reports the expected number of historical moves at writer rows, E κ_w, together with the forward change TV at initialization:

BackboneWriterE κ_w (init)TV (init)I2V gain
Wan 1.3Brecursive0.820.06424.9
Wan 14Brecursive0.690.11783.5
HunyuanVideorecursive0.570.10463.9
MAGI-1self-only0 (exact)0.02982.3
EchoWMrecursive1.060.0871–

The check separates the self-only writer, which has the lowest TV and the smallest gain and error reduction, from the recursive writers. Among recursive writers, the size of the gain also depends on how much headroom Standard-KV leaves, so TV alone does not rank them. With four backbones on TC-Bench we state this as a screen rather than a fitted predictor: check first that the cache-writing call attends to earlier cached entries (E κ_w > 0); a writer that attends only within its block will gain least. Fewer evaluations per block raise the weight on paths through earlier outputs (Table 5), so few-step students are where the method is most useful. Section 6 now states this rule.

W3 / Q3. Why does dynamic degree fall while completion and motion-completeness ratings rise? Because VBench's dynamic test measures how long motion lasts, and a completed transition stops moving. The test samples frames at about 8 fps and calls a video dynamic only when at least a quarter of frame pairs exceed the flow threshold. Appendix J.3 separates amplitude from duration for all four I2V backbones:

  • the mean peak frame-pair score changes by at most 0.03 of the threshold, so motion is as large;
  • the share of videos whose motion is too brief to count rises by 0.3 to 0.8 points;
  • on TC-Bench-I2V, completed transitions pass the test less often than incomplete ones in both models (56.9% vs 68.3% for CausalFold), and within each stratum the two models differ by under one point.

So CausalFold finishes more transitions and then holds the end state, which the human raters see as more complete motion (71.17%) and the flow test counts as less dynamic. On HunyuanVideo, which has the largest drop, a small amplitude change also appears, and the appendix reports it.

For Wan's physical commonsense (App. J.4), the drop from 3.50 to 3.40 sits in videos that do not follow the prompt, i.e. attempted but incomplete actions. Among prompt-adherent videos, mean PC changes by -0.03, and more CausalFold videos reach PC ≥ 4 (52.5% vs 49.4%), which is why joint accuracy rises from 28.4 to 31.9. Both scores remain in the Limitations of Section 6 with these explanations.

Summary for the discussion. At matched latency and at matched or larger memory, Standard-KV stays clearly behind (J.5); the benefit can be screened on a pretrained model from its cache-writing call (J.7); and the dynamic-degree and PC changes are explained and bounded (J.3, J.4).

Thank you again for questions that made the paper more practical. If these results resolve your concerns about cost and predictability, we would be grateful if you would consider raising your score.

The authors’ response addresses most of my concerns, so I maintain my original positive rating.

Reviewer_EywROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces CausalFold, that replaces values with the attention outputs in block-causal video diffusion models. This enables aggregating the relational history and providing additional gradients to the transformer's earlier blocks. The paper reports consistent improvements in transition completion across several video models.

Critical strengths
  1. The proposed method is simple, and adds no parameters. Also, it has an interesting, plausible motivation in Section 3.2.
  2. It provides solid experimental comparison against the Standard-KV method.
  3. It provides comprehensive evaluation across 5 backbones, demonstrating the generalizability of the proposed method.
Critical weaknesses
  1. I think the training costs should be addressed in the paper. The recursive chain might generate long gradient propagation graph as RNN, so it could increase the training latency or memory. Please report and compare the training costs with the baseline.
  2. It was hard to feel the improvement due to the lack of qualitative, side-by-side comparison with Standard-KV method.
  3. Figure 1 and 2 are pretty confusing. It was almost impossible to understand the proposed method and distinguish it with the existing methods from the Figure 2. Furthermore, I believe Figure 1 cannot demonstrate anything about the proposed method.
Questions for authors

Please see Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer EywR (1/2): training cost and Figures 1-2

(1/2) Thank you for your review and for the encouraging summary. You found the method simple and parameter-free with "an interesting, plausible motivation in Section 3.2", the Standard-KV comparison "solid", and the evaluation across five backbones a demonstration that the method generalizes. Your three points are about how the paper is read and what it costs, and each one led to a visible change. All changes are in blue in the revised PDF.

W1. Training cost: does the recursion build a long RNN-like graph? No, and the paper now says why and reports the measured cost (App. J.8, Table 19; summary in Sec. 5.4).

Why the graph stays short. The recursion R_t = D_tt V_t + Σ_{s<t} L_ts R_s runs over blocks inside one attention call. Its backward pass is one reverse substitution over at most S blocks with the same operations as the forward pass (Algorithm 1, lines 10–12), so its depth equals the number of blocks, not the number of denoising steps or layers. The replay that sends gradients through every cache read across blocks is identical for both arms: Standard-KV is trained with exactly the same replay, so the only training-time difference is this within-layer recursion. Cached entries hold R in place of V, and R is the attention output that standard attention already stores for its backward pass. The accumulated gradient is also bounded by (1 + max E κ) times the entering gradient (Eq. 19), so the recursion cannot blow gradients up.

Measured cost. One run is 200 generator updates with batch 64 on A100 80GB GPUs:

BackboneStd-KV s/updateCausalFold s/updateOverhead
Wan 1.3B318.6333.6+4.7%
Wan 14B618.2665.0+7.6%
MAGI-1402.7462.6+14.9%
HunyuanVideo447.9497.6+11.1%
EchoWM963.51088.1+12.9%

The overhead is below the generation overhead of Table 6 for every backbone, because the real and fake score models are unchanged bidirectional networks and take a fixed share of each update. Peak memory per GPU rises by at most 5.8 GiB (MAGI-1, whose grouped-query cache is larger), and every run fits on the same GPUs with the same sharding as Standard-KV.

W3. Figures 1 and 2 were confusing. We agree, and we replaced Figure 1. The old Figure 1 showed rollouts and said nothing about the method; it moved to Appendix I.4. The new Figure 1 shows one attention layer and the corridor example from Section 1 as two panels side by side: (a) Standard-KV caches (K, V), so the doorway token's selected content reaches the next block only one layer later, and the next block's loss sends no gradient to the doorway's attention in that layer; (b) CausalFold keeps K and caches the attention output R, so the next block reads that content in the same layer, and its loss reaches the doorway's attention with Jacobian P·Δ (Theorem 2). The caption gives the three-token numbers of Appendix C.3 (output 0.20 vs 0.32; gradient 0 vs 0.072), so the difference can be checked by hand. Figure 2 is now introduced as "the same change placed in the full generator": its caption tells the reader to go through panels A to C in execution order, and states that the cache stores (K_t, R_t) where Standard-KV stores (K_t, V_t), and that this is the only difference.

Part (2/2) covers the qualitative comparison.

AuthorsOpenReview ↗

Response to Reviewer EywR (2/2): qualitative evidence and the comparison with Standard-KV

(2/2)

W2. It was hard to feel the improvement without side-by-side comparisons. This is a fair point, and we agree the qualitative material should be easier to find and to connect to Standard-KV. Appendix J.10 now gathers it in one place. The frame sequences already in the paper show the kinds of transitions that the matched comparisons count, each read left to right:

  • Figure 3: an attribute change, an object relation (the frog leaving the lily pad, the dog catching the frisbee), and a background change, completed while the scene persists;
  • Figure 4: a three-event story in which the girl puts down the basket she has just filled with the apples picked two events earlier, the same chain of states as the corridor example;
  • the cake that keeps changing color while it rotates (App. I.1), a left, left, right, right camera sequence that returns toward its starting view (App. F.2), and the long rollouts formerly shown as Figure 1 (App. I.4).

The direct comparison with Standard-KV on identical prompts is made by the paired human study: 3,009 video pairs from 1,003 prompts, each judged by three raters on four questions. Raters prefer CausalFold for temporal consistency (74.93%) and motion completeness (71.17%), and the new prompt-level intervals exclude 50% for every question, including visual realism (App. J.1). Two quantitative views complement it: TC-Bench completion rises in all six matched comparisons, and in 60-second rollouts the five-second segment scores separate as the video goes on (Fig. 19), with failures falling from 40/384 to 20/384.

Summary for the discussion.

  • Training cost is reported for every backbone: +4.7% to +14.9% time per update, with no long unrolled graph (J.8).
  • Figure 1 now explains the method in one layer and shows the difference from Standard-KV directly; Figure 2's caption was rewritten.
  • The qualitative frame sequences are now gathered in App. J.10 and tied to the paired human study, whose preferences exclude 50% at the prompt level.

Thank you again for reading the paper as a user would; the new Figure 1 in particular makes the paper much easier to enter. If the revision resolves your concerns about cost and presentation, we would be grateful if you would consider raising your score.

Thank you for clarifications and revisions. My concern about W1 has been fully addressed, and I no longer consider the training cost to be a major issue. However, I still find Figure 2 confusing and I couldn't find a direct side-by-side qualitative comparison with Standard KV in the revised paper. I will maintain my positive score, but I highly recommend revising Figure 2 and adding the qualitative comparisons.

Warm regards,

Reviewer EywR

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: what changed in the revision

We thank the three reviewers for careful and generous reviews. Each of you read the mechanism closely, and each of you pointed to something concrete that the paper had not yet shown. The revised PDF is uploaded; every change is in blue, and the new analyses are collected in Appendix J.

What the reviews agree on. All three reviews describe CausalFold as simple and parameter-free (jVbN, BWsR, EywR). Reviewer jVbN writes that the matched Standard-KV control "isolates what a cached read returns" and that the theory is "modest but exact and does work"; Reviewer BWsR calls the evaluation "broad and controlled" and the paper "well written and easy to follow"; Reviewer EywR finds the Standard-KV comparison "solid" and the motivation of Section 3.2 "interesting, plausible". All three point to the five backbones as evidence that the change carries over.

Main revisions (blue in the PDF).

  1. Uncertainty for every matched comparison: per-seed differences, seed SDs, case-bootstrap and simultaneous intervals, prompt-level intervals for the human study (App. J.1; summary in Sec. 5.1).
  2. Standard-KV at matched latency and memory: more evaluations per block, longer windows for 60-s rollouts, and a grouped-output variant that returns MAGI-1's cache to Standard-KV size (App. J.5; Sec. 5.4).
  3. Training cost of every pair (App. J.8; Sec. 5.4) and longer training to 500 updates on two paired seeds (App. J.2).
  4. Dynamic degree and physical commonsense, decomposed rather than only reported (App. J.3, J.4; Sec. 5.4 and the Limitations in Sec. 6).
  5. A pre-training diagnostic for which backbones benefit (App. J.7), interactive command changes (App. J.9), and how CausalFold combines with window, sink, and re-cache policies (App. J.6).
  6. A new Figure 1 that contrasts the two caches in one layer, a reworded Figure 2 caption, and a guide to the qualitative frame sequences and paired comparisons already in the paper (App. J.10).
  7. The "history reads: V" control is now described as evidence that the trained model depends on its reads, not that outputs are better to cache (Sec. 1, Sec. 5.3).

The title is unchanged, no claim is extended, and the main text still ends on page 10. Every new experiment checks a statement the submission already made.

New results at a glance.

QuestionResultWhere
Are the TCR gains larger than seed noise?Seed SD 0.6–1.2 pts; all 6 paired intervals exclude 0 after BonferroniJ.1
Std-KV with more compute (MAGI-1, 10 NFE)112.64 s vs 110.09 s, still 1.8 TCR pts lowerJ.5
Std-KV with longer window (60 s)failures 37/384 (24 frames), 35/384 (40 frames) vs 20/384J.5
Longer training (500 updates, 2 paired seeds)gap 4.6 (T2V) / 4.8 (I2V) pts, vs 4.8 / 5.1 at 200J.2
Training cost+4.7% to +14.9% time per updateJ.8
Dynamic degreepeak flow changes ≤ 0.03 of threshold; brief-motion share +0.3 to +0.8 ptsJ.3
Physical commonsenseamong prompt-adherent videos, mean PC changes by -0.03J.4
MAGI-1 cache under GQAgrouped outputs keep 56% of the gain at Std-KV cache sizeJ.5
Changed camera command0.8 frames more lag; interaction still higherJ.9

The per-reviewer replies give the details and the exact settings. Two things came out more clearly than we expected: the gap barely moves when training continues to 500 updates, and the lower dynamic degree is about how long motion lasts after a change completes, not how large it is. We are grateful for the questions that led there, and we are happy to run further checks during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.