acceptodds
Under review as a conference paper at ICLR 2027

FrameFold: Response-Guided KV Cache Consolidation for Autoregressive Long Video Generation

Abstract

In autoregressive video generation, retaining all historical key–value (KV) states increases memory usage and per-step attention costs as videos grow longer. A fixed cache budget bounds these costs but raises the challenge of preserving historical content that supports consistent, high-quality generation. Existing methods control cache size through history selection or consolidation, yet deciding what to retain and how to limit the loss of useful content remains a shared challenge. Our analysis shows that spatial queries depend on different historical content and that higher current attention coverage does not necessarily yield better long-video quality. We introduce FrameFold, a training-free KV cache consolidation method that compares the content retrieved by the same recent historical queries before and after merging to guide both representative construction and merge selection. Exploiting temporal redundancy between adjacent historical entries, FrameFold merges their KV states into shared representatives while retaining complete spatial grids. It fits value-merging coefficients in closed form by matching conditional attention responses, then selects the merge with the lowest remaining response distortion. Under matched KV cache budgets, experiments on Self Forcing and Causal Forcing at 30 and 60 seconds and on LongLive at 120 seconds demonstrate improved consistency and visual quality. Continuous eight-minute Self Forcing rollouts further show less quality degradation than the baselines.

Primary area:
generative models
Keywords:
Autoregressive Video Generation, Long Video Generation, KV Cache Compression, Training-Free Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1aWjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FrameFold is a training-free KV cache consolidation method for AR video generation. It merges adjacent historical entries by evaluating how well the merged representations preserve the original attention responses, then greedily selects the merges that introduce the least change. Experiments show improved video consistency and visual quality under fixed cache budgets, with further improvements in continuous eight-minute generation.

Critical strengths

1. The paper is well organized, with a clear presentation of the motivation, method, and experiments. The figures also help explain the proposed approach and make the paper easy to follow.

2. The long-horizon experiments are valuable. Testing continuous generation for up to eight minutes shows that the benefits extend beyond short clips, with FrameFold maintaining good consistency and visual quality as the video gets longer.

Critical weaknesses

1. The paper shows that FrameFold can achieve better video quality with a smaller KV cache, but reducing the number of cache entries does not necessarily translate into the same reduction in overall GPU memory usage. The actual memory savings are therefore unclear.

2. FrameFold reduces the cache size by introducing additional steps to evaluate candidate merges and update the cache. These operations also incur computational and memory costs, which are not fully quantified in the paper.

Questions for authors

1. Could the authors report the actual GPU memory usage and computational costs compared with the baselines under different cache budgets? This would help clarify how much the reduced cache size translates into practical memory and computation savings.

2. Could the authors provide a breakdown of the additional latency and memory overhead introduced by candidate evaluation, value fitting, and iterative cache updates? This would help better understand the overall cost of FrameFold.

3 comments

(W1 & Q1) GPU memory and computational cost.

Thank you for highlighting the distinction between KV-cache capacity and actual runtime memory consumption.

For a fixed current-chunk size, a larger historical KV cache increases both GPU memory consumption and the computational cost of self-attention, reducing generation throughput. Maintaining a smaller cache can therefore improve generation efficiency and facilitate deployment on memory-constrained consumer GPUs. FrameFold aims to preserve useful historical information within this smaller budget through cache consolidation.

To quantify these practical benefits, we report generation-stage peak allocated GPU memory and throughput after cache saturation across cache budgets:

Cache budgetPeak allocated GPU memory (GB) ↓Single-H20 throughput (FPS) ↑
1214.8895.545
1517.3164.755
1819.7334.166
2122.1553.700

Reducing the cache budget from 21 to 12 entries decreases KV storage by 42.9%, reduces measured peak GPU memory by 32.8%, and increases throughput by 49.9%. These measurements demonstrate that a smaller cache translates into practical memory and throughput benefits, while distinguishing cache-storage savings from the reduction in total runtime memory.

Peak allocated memory is recorded using torch.cuda.max_memory_allocated(), including generator weights, caches, activations, and temporary allocations during denoising and cache consolidation. Throughput is reported as output-equivalent FPS after cache saturation. Text-encoder and VAE weights remain on the CPU; text encoding, VAE decoding, and video saving are excluded.

(Q2 & W2) Additional latency and memory overhead.

Thank you for requesting this breakdown. At a cache budget of 12, FrameFold adds 305.7 ms per chunk relative to FIFO, a 16.5% latency increase. We compare complete chunk generation after cache saturation. Each chunk contains three latent frames, with four denoising forwards and one clean refresh at t=0t=0. Timings exclude text encoding, VAE decoding, and video saving.

MethodFull-chunk latency (ms)
FIFO1858.3
FrameFold2164.0

The additional 305.7 ms represents 14.1% of FrameFold’s total chunk latency. At the operation-category level, the net overhead can be expressed as

$

ΔT=TFF−TFIFO=Teval+Tfit+(TupdateFF−TupdateFIFO)\begin{aligned} \Delta T &= T_{\mathrm{FF}}-T_{\mathrm{FIFO}}\\\\ &= T_{\mathrm{eval}}+T_{\mathrm{fit}} +\left(T_{\mathrm{update}}^{\mathrm{FF}} -T_{\mathrm{update}}^{\mathrm{FIFO}}\right) \end{aligned}

$

where evaluation and fitting are introduced by FrameFold, while the cache-update contribution is the difference between the two methods’ maintenance costs.

We separately profile three non-overlapping categories within FrameFold:

ComponentOperations includedOutputTime (ms/chunk)
Candidate preparation and evaluationPrepare historical queries and candidates; apply scoring RoPE and pooling; compute reference responses and candidate attention; reconstruct fitted responses and evaluate residuals.Distortion score for each candidate pair223.0
Closed-form value fittingConstruct the terms for the closed-form fit; solve for and constrain the optimal mixing coefficient �2��2�.Mixing coefficient for each candidate pair36.2
Iterative merging and cache updatesSelect the lowest-error pair; merge full-resolution K/V; update position and span metadata; reconstruct, copy, and write back the cache.Updated cache79.9

For memory, a separate single-GPU measurement at budget 12 gives peak allocated memory of 14.889 GB for FrameFold versus 10.149 GB for FIFO, an increase of approximately 4.74 GB (46.7%).

Our implementation evaluates candidate pairs in parallel. At each greedy merge iteration, �3��3� eligible historical entries form �4��4� adjacent pairs, whose independent attention computations are batched for parallel evaluation. Compared with evaluating these pairs serially, this reduces candidate preparation and evaluation time from 548.2 to 223.0 ms/chunk, yielding a 2.46× speedup (59.3% latency reduction) for this component.

We will include these measurements and the explicit profiling boundaries in the revised paper.

Thank you for the detailed responses. The additional experiments and analyses have addressed most of my concerns.

Although the methodological novelty is somewhat limited, I find the proposed approach practically useful for long-video generation, and the experimental results on video quality are convincing.

I have two minor suggestions:

  1. I encourage the authors to include more efficiency-related evaluations and analyses in the main paper to provide a more comprehensive understanding of the practical benefits and costs of FrameFold.
  2. Please check the figure references and overall formatting. In particular, Figure 2 is cited before Figure 1 in the Introduction, and Figure 3 could be explicitly referenced when introducing the overall framework in Section 4.

If these suggestions are addressed, I would be willing to increase my rating to 3.

Thank you for the follow-up and for your careful review and constructive suggestions.

We have revised the manuscript to address both suggestions:

  1. Efficiency evaluations and analyses. We added Section 5.5 (pp. 9–10) to the main paper. It reports peak allocated GPU memory and throughput across cache budgets, the latency and memory overhead relative to FIFO at the same budget, and the main latency components within FrameFold. Appendix A.9 provides the detailed profiling protocol, operation-level breakdown, and comparison of serial and batched candidate evaluation. These additions clarify both the practical benefits of reducing the cache budget and the additional costs of consolidation.

  2. Figure references and formatting. The Introduction now cites Figure 1 before Figure 2. We also explicitly reference Figure 3 in Section 4.1 when introducing the two coupled decisions underlying our framework. We checked the remaining figure and table references and reviewed the overall formatting.

Changes are highlighted in blue to help locate the revisions.

Reviewer_zd3DOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FreamFold studies how to make history context most usable under constrain of KV memory cache settings. It preserves the sink and nearest few frames’ latent, merges features between them, and maintains overall spatial grids at the same time. When merging, Keys are merged among temporal dimension, while a scalar value-mixing coefficient is fitted to preserve conditional attention responses under recent historical queries. The method then selects the candidate merge with the smallest remaining response distortion. Experiments cover three video generators and continuous rollouts lasting up to eight minutes.

Critical strengths
  1. A compact method with meaningful component validation. FrameFold uses an inexpensive closed-form fit to determine how to merge values, then evaluates the resulting distortion to decide which entries to merge. Table 2 separately examines merge selection and value construction, while Table 3 compares response objectives. These experiments provide useful support for the proposed combination beyond simply showing an improved final system.
  2. The evaluation demonstrates practical value across backbones and long rollouts. Results extend beyond short sequences to three generators and continuous eight-minute generation. At 480 seconds, FrameFold improves aesthetic and imaging quality over FadeMem by 1.40 and 1.61 points, respectively, while also achieving a higher dynamic degree.
Critical weaknesses
  1. The robustness of several reported improvements is unclear. Many consistency improvements in Table 1 are small, and the isolated benefit of value fitting in Table 2 is also modest. No paired confidence intervals or per-prompt win rates are reported. The stronger eight-minute results are encouraging, but use only 24 prompts whose selection procedure is unspecified. At the same cache budget, FrameFold reduces throughput by approximately 10–13% relative to the evaluated baselines, making the reliability of its quality gains consequential.

  2. The restriction to adjacent candidate pairs is insufficiently validated. Section 3.2 motivates adjacency through temporal redundancy, but this does not establish that adjacent entries are the best merge candidates, especially after repeated consolidation. Table 2 and Appendix A.5 compare K/V/O-based selection only within the same adjacent-pair candidate set. They therefore do not test broader temporal matching or a simpler policy that merges an incoming eligible entry into an existing representative. Moreover, whenever the cache exceeds its budget, the method selects the lowest-distortion eligible pair even if all candidates have large distortion, there is no absolute distortion threshold for declining a merge. This raises a concern about information loss during large motion, scene revisits, or sustained compression

Questions for authors
  1. How does the candidate policy compare with merging a newly eligible historical entry into either its nearest temporal neighbor or the existing representative giving the lowest fitted response distortion? To isolate the policy, please retain the same sink/recent protection, cache budget, scoring criterion, and value fitting, and evaluate an entry only after it leaves the protected recent region.
  2. Is there evidence on events where even the best adjacent merge has substantial distortion, and does this distortion predict error on held-out next-chunk queries or subsequent quality degradation?
  3. How were the 48-prompt ablation set and the 24-prompt long-horizon set selected, and how do they relate to the 128-prompt main evaluation set? Are the value-fitting gains in Table 2 consistent across prompts or driven by a few examples?
  4. Appendix B.2 states that the FadeMem reimplementation follows its original temporal allocation and merge rules. Could you confirm that this also includes its span-weighted representative temporal positions and the corresponding RoPE re-encoding?
5 comments

(Q1) Merge strategy

Thank you for suggesting these alternatives. We evaluated incoming-nearest and incoming-best with the same span-weighted key merging, conditional-response value fitting, query-probe construction, and positional-encoding conventions as FrameFold. The cache budget, sink/recent protection, prompts, seeds, and generation settings were identical. An entry becomes eligible only after leaving the protected recent region. Only the pair-selection policy and its candidate set vary.

We evaluated all three policies on Self Forcing with 60-second videos, 128 prompts, and a budget of 12 frame-equivalent KV entries, including the current chunk. FrameFold achieves the highest subject consistency, background consistency, motion smoothness, aesthetic quality, and imaging quality, while incoming-best achieves the highest dynamic degree.

Selection policySubject consistency ↑Background consistency ↑Motion smoothness ↑Dynamic degree ↑Aesthetic quality ↑Imaging quality ↑
Incoming-nearest96.7896.1197.9760.9757.7768.42
Incoming-best96.5396.0197.7962.5057.8268.58
FrameFold96.9496.1798.2362.1158.2368.80

The policies differ as follows:

Selection policyResponse-guided pair selectionGuarantees contiguous source intervalsCan merge two existing representatives
Incoming-nearestNoYesNo
Incoming-bestYesNot guaranteedNo
FrameFoldYesYesYes

Incoming-nearest merges the newly eligible entry into its immediately preceding representative. All policies use the same response-based value fitting; "Response-guided pair selection" refers only to pair selection.

Incoming-nearest restricts where compression occurs. Consider existing representatives A,B,CA,B,C, ordered chronologically, and an entry xx that has just left the recent region. Incoming-nearest fixes the pair to (C,x)(C,x). Value fitting can reduce the distortion of this prescribed merge, but cannot determine whether another part of history would be less costly to compress.

Suppose (A,B)(A,B) can be merged with low response distortion, whereas CC and xx contain distinct information that is difficult to preserve in a single representative. FrameFold can merge (A,B)(A,B), leaving CC and xx separate. Both operations release the same cache capacity, but FrameFold can allocate compression to history that is more readily consolidated under the current query probes. Leaving the recent region therefore makes xx eligible for merging without requiring its immediate absorption. Moreover, at the same cache state, the nearest pair is included in FrameFold’s candidate set, so FrameFold’s selected merge has no greater fitted response distortion at that decision.

Incoming-best uses response-guided selection but permits noncontiguous historical assignments. In the same example, it compares (A,x)(A,x), (B,x)(B,x), and (C,x)(C,x). Matching xx to an earlier representative may be beneficial when content recurs. However, this can assign separated historical periods to the same representative while retaining the intervening period elsewhere. For example, in an “appears–disappears–reappears” sequence, the two appearance periods may be consolidated together, while the intervening absence remains in another entry.

FrameFold’s adjacency constraint preserves the temporal organization of compressed history. Starting from individual frames, merging two adjacent intervals produces another contiguous interval; this property holds after repeated consolidation. Each representative corresponds to a contiguous source interval, and the representatives remain chronologically ordered. Response-guided selection adjusts temporal granularity by choosing which neighboring intervals to consolidate. This does not guarantee preservation of every event within an interval, but it prevents directly grouping separated periods while skipping intervening retained history.

Incoming-best is also position-aware because its response score includes RoPE. The distinction is the explicit contiguous-interval constraint, together with the requirement that every incoming-based merge involve xx. Both incoming policies exclude an existing–existing merge even when it would incur less fitted distortion and allow xx to remain separate.

FrameFold combines response-guided pair selection with an ordered interval representation and the flexibility to reorganize existing history. It can consolidate low-distortion neighboring intervals while retaining newly eligible content separately when that is less costly. The controlled VBench comparison supports the effectiveness of this combined selection policy.

(Q2) Merge distortion and held-out query errors

We acknowledge that even the lowest-distortion eligible pair may have a large residual cost: selecting a relative minimum does not guarantee a small absolute distortion. Our claim is that the relative ordering provides useful guidance for merge selection under a fixed cache budget.

To test this, we compare normal FrameFold against a maximum-cost variant, keeping the candidate restriction, cache configuration, and K/V construction unchanged. Both variants fit each candidate’s value coefficient; only the selection direction changes. We evaluate both variants on Self Forcing, generating 60-second videos on the same 128 prompts as the main evaluation with a cache budget of 12 frame-equivalent KV entries, including the current chunk.

MethodSubject Consistency ↑Background Consistency ↑Motion Smoothness ↑Dynamic Degree ↑Aesthetic Quality ↑Imaging Quality ↑
Minimum cost (FrameFold)96.9496.1798.2362.1158.2368.80
Maximum cost96.5296.0197.9662.3557.5868.14

Minimum-cost selection achieves higher mean consistency, smoothness, and visual-quality scores, while maximum-cost selection has higher dynamic degree. These trends support the usefulness of the relative ordering. In this comparison, reversing the selection direction does not improve mean consistency or visual quality, unlike the reversal observed in our attention-coverage experiment.

Regarding whether distortion predicts error on held-out next-chunk queries, we measure two errors for each selected merge:

  • �0��0�: the residual distortion used for selection, measured with historical queries from chunk �1��1�.
  • �2��2�: the response distortion of the same merge, measured with held-out queries from chunk �3��3�’s clean-context pass.

For both measurements, we compare the conditional response of the original pre-merge pair with that of its merged representative. The pair, representative, and fitted coefficient remain unchanged; only the evaluation queries change. Both errors use the same pooling and mean-L2 metric. The held-out queries are collected along the actual compressed-cache rollout and never participate in selection or fitting.

We collect these paired measurements on all 128 prompts in the main evaluation set, using 60-second Self Forcing rollouts with a cache budget of 12. We record the final merge of each cache update at four layers, yielding 9,728 paired measurements per layer. Within each layer, we compute the Spearman correlation between �4��4� and �5��5� across these measurements.

Layer0102029
Spearman �6��6�0.9970.9940.9790.996

The correlations show that merge events with smaller historical distortion generally also incur smaller errors on subsequent held-out queries.

(Q3 & W1) Prompt selection and per-prompt win rates

Our main evaluation uses the 128 MovieGenBench prompts provided in the MemRoPE repository. The 48-prompt ablation set and the 24-prompt long-horizon set were randomly sampled from this same pool. We will clarify the prompt provenance and subset selection in the revised manuscript.

To examine how frequently the improvements occur across individual prompts, we consider two paired comparisons: FrameFold against FadeMem in the main evaluation, and O diff + Optim against O diff + Span in the value-fitting ablation. For each metric, a win requires a strictly higher score on the same prompt. Comparisons use scores before rounding; ties are not counted as wins and remain in the denominator.

Main evaluation: FrameFold versus FadeMem. We compare the two methods separately in each backbone–duration setting from Table 1, using the same 128 prompts and matched generation settings. The win rate is the number of prompts on which FrameFold outperforms FadeMem divided by 128.

SettingSubject consistencyBackground consistencyMotion smoothnessDynamic degreeAesthetic qualityImaging quality
Self Forcing, 30s80.47%79.69%81.25%50.00%92.19%93.75%
Self Forcing, 60s79.69%80.47%80.47%50.78%91.41%92.97%
Causal Forcing, 30s81.25%80.47%79.69%49.22%90.63%92.19%
Causal Forcing, 60s80.47%78.91%81.25%50.00%92.97%94.53%

Value-fitting ablation: Optim versus Span. To isolate the contribution of value fitting, we compare O diff + Optim against O diff + Span from Table 2 on the same 48 prompts, using 30-second Self Forcing videos with a cache budget of 12, including the current chunk. This comparison keeps the merge-selection criterion fixed and varies the value-merging rule. The win rate is the number of prompts on which Optim outperforms Span divided by 48.

MetricPrompts won by OptimWin rate
Subject consistency36/4875.00%
Background consistency35/4872.92%
Motion smoothness37/4877.08%
Dynamic degree30/4862.50%
Aesthetic quality25/4852.08%
Imaging quality34/4870.83%

These paired comparisons complement the mean scores in Tables 1 and 2 by reporting the frequency of improvement across prompts. They separately assess the full method and the contribution of value fitting, while allowing differences across metrics to remain explicit.

(Q4) Reimplementation of FadeMem

Thank you for highlighting this ambiguity. Our FadeMem baseline was implemented from arXiv v1 when official code was not yet available. We followed its temporal allocation rule (β = 0.3) and span-weighted K/V merging within MemRoPE, while retaining MemRoPE’s online cache-position indexing. Consequently, merged keys are re-encoded at their assigned cache positions, rather than directly at the span-weighted representative temporal centers.

Although v1 describes representative-center re-encoding, it also presents its memory policy as complementary to online RoPE indexing. It does not specify the bounded temporal remapping procedure found in the currently released implementation. Our comparison retains the same MemRoPE positional convention for FrameFold and the FadeMem baseline, keeping positional handling consistent when evaluating their consolidation policies. The current official mapping differs from ours, and we do not claim that the two mappings are equivalent.

We acknowledge that Appendix B.2 did not make this positional adaptation sufficiently explicit. We will revise it to clearly distinguish the reproduced temporal allocation and merging rules from the attention-time RoPE indexing inherited from MemRoPE.

Reviewer_LbHQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

FrameFold introduces a training-free KV cache consolidation method for autoregressive long video generation. It merges adjacent historical entries while preserving complete spatial grids, fits value-merging coefficients in closed form to match conditional attention responses, and selects merges with minimal residual distortion. Evaluations demonstrate improved consistency and visual quality under matched cache budgets.

Critical strengths
  1. The method checks how merging changes the information read from history, providing a concrete criterion for reducing information loss.

  2. The weights for combining cached values have a closed-form solution, avoiding repeated optimization steps during generation.

Critical weaknesses
  1. Comparisons mainly cover FIFO, Deep Forcing, and FadeMem. Including additional relevant methods would better establish the effectiveness.

  2. Decisions rely on recent, pooled queries, but aggregate video metrics do not directly test whether information needed by substantially different future queries survives compression.

Questions for authors

What evidence shows that information remains recoverable when future queries differ substantially from the recent queries used for compression, for example after scene changes or delayed object reappearance?

3 comments

(W1) Additional Baseline Comparisons

We thank the reviewer for suggesting broader comparisons. We have expanded our evaluation to include MemRoPE, which maintains evolving memory tokens through exponential moving averages, and LongLive-RAG, which retrieves relevant historical latents as additional generation context.

We evaluate these methods on Self Forcing for 60-second generation, using the same 128 prompts as the main evaluation, with matched random seeds, resolution, and denoising schedule across methods. All methods use a matched cache budget of 12 frame-equivalent KV entries, including the current chunk, following the budget convention in Sec. 5.1.

MethodSubject Consistency ↑Background Consistency ↑Motion Smoothness ↑Dynamic Degree ↑Aesthetic Quality ↑Imaging Quality ↑
Self Forcing95.8095.6097.4955.3456.9566.12
+MemRoPE96.8196.0598.1261.8557.0168.04
+LongLive-RAG96.7495.9798.0960.0857.6668.23
+FrameFold96.9496.1798.2362.1158.2368.80

(W2 & Q1) Robustness to future query changes

We agree that aggregate video metrics do not directly establish recoverability after substantial changes in query demands, such as scene changes or delayed object reappearance. We will clarify this limitation. Our goal is to preserve historical content useful to future queries, which involves two complementary questions: how to measure and reduce compression loss given query demands, and how to estimate those demands before future queries are available.

Formally, at a fixed compression step, let �1��1� denote the original history, �2��2� its conditional attention readout, and �3��3� the representations satisfying cache budget �4��4�. An idealized readout-preservation objective is $

min⁡H~∈CB(H)Eq∼Pfuture[∥R(q,H)−R(q,H~)∥22].\min_{\widetilde H\in\mathcal C_B(H)} \mathbb E_{q\sim P_{\mathrm{future}}} \left[\|R(q,H)-R(q,\widetilde H)\|_2^2\right].

$ FrameFold primarily studies the compression criterion and the construction of compressed representations. For the complementary problem of estimating future query demands, we use pooled recent historical queries as practical probes.

Even perfect knowledge of future queries does not make attention-mass ranking equivalent to response preservation. For a fixed query, consider retaining token set �5��5� without changing the retained keys or positions. Let �6��6� be its original attention mass, and �7��7� the conditional readouts of retained and discarded tokens. Then

= (1-m_S)\lVert R_{\bar S}-R_S \rVert_2.$$ Thus, readout distortion depends on both the discarded attention mass and the difference between retained and discarded content. Maximizing retained attention mass does not generally minimize this distortion. Moreover, our analysis in Sec. 3.1/Fig. 2(a) demonstrates that shared attention ranking leaves query-level coverage gaps across spatial queries. These issues can arise even when query demands are known. FrameFold directly evaluates conditional-response distortion to guide value fitting and merge selection, using local, pooled candidate comparisons. Retaining complete spatial grids additionally maintains a representative at every spatial location within each merged entry. This provides structural coverage, although merging remains lossy and does not guarantee recovery after substantial query changes. Our matched-budget generation results support the practical value of this compression approach in the evaluated setting. Robustness to substantially different future queries remains an important limitation. **More accurate future-query proxies are complementary to our contribution and can be incorporated into the same response-preservation mechanism.**

Thank you for your response. However, I still feel that some of the paper’s limitations, such as insufficient experimental comparisons. I will maintain my original score.

Thank you for the follow-up and for your careful review and constructive suggestions. We would like to further clarify the contribution of our analysis and the scope of our experimental evidence.

A central contribution of our work is the analysis of attention-based history retention in autoregressive video generation. Section 3 shows that shared Top-K rankings can overlook history important to individual spatial queries, and higher current attention coverage does not necessarily yield better subsequent generation quality. The latter is supported by controlled Top-K/Bottom-K comparisons on two backbones. These findings motivate evaluating what content remains retrievable after compression, leading to FrameFold’s conditional-response-preserving consolidation. Future-query shifts remain a limitation; our work addresses the complementary question of how to represent and compress history given query probes.

Our comparisons cover several representative approaches: attention-guided token selection (Deep Forcing), temporal history merging (FadeMem), EMA-based memory updates (MemRoPE), and retrieval-based historical context (LongLive-RAG). The latter two comparisons were added during rebuttal using Self Forcing at 60 seconds, with 128 prompts and matched cache budgets and generation settings. FrameFold achieves higher mean scores across all six evaluated dimensions in these additional comparisons. Our ablations further examine the contributions of merge selection and representative construction.

The practical benefits are particularly evident in aesthetic and imaging quality, including during sustained generation. On continuous 480-second Self Forcing rollouts, FrameFold exceeds FadeMem by 1.40 points in aesthetic quality and 1.61 points in imaging quality, with smaller quality declines as the evaluated duration increases. These benefits involve a measurable throughput trade-off: under the same 12-entry budget on an H20, generation throughput is 5.56 FPS versus 6.18 for Deep Forcing and 6.21 for FadeMem, approximately a 10% reduction, excluding VAE decoding and video saving.

We hope these clarifications help convey both the analytical contribution and the practical value supported by our experiments.

Comments on the paper

1 comment

Summary of Rebuttal

We sincerely thank the reviewers and the Area Chair for their careful review and constructive suggestions. We summarize the additional evidence provided during rebuttal and the clarifications incorporated into the revised manuscript.

  1. Broader comparisons and prompt-level evidence(LbHQ, zd3D). We added comparisons with MemRoPE and LongLive-RAG using Self Forcing for 60-second generation, with 128 prompts, matched generation settings, and a 12-entry cache budget. Together with Deep Forcing and FadeMem, these comparisons cover selection, merging, EMA-based memory updates, and retrieval. FrameFold achieves higher mean scores across all six metrics than both added baselines. Paired comparisons against FadeMem further show win rates exceeding 90% for each of aesthetic and imaging quality in every main backbone–duration setting, indicating that the visual-quality gains extend across most evaluated prompts. The existing continuous 480-second experiments complement these results by showing higher quality and smaller declines as the evaluated duration increases.

  2. Validation of merge selection and query probes(LbHQ, zd3D). Controlled comparisons with incoming-only merging policies and a maximum-cost variant support FrameFold’s selection strategy. In particular, minimum-cost selection yields better consistency and visual quality than maximum-cost selection while keeping candidate restrictions and representative construction unchanged. For the same selected merge events, historical-query distortion strongly correlates with distortion measured using held-out next-chunk queries: Spearman coefficients range from 0.979 to 0.997 across four measured layers. This supports the relevance of recent-query probes to subsequent compression error in continuous generation; robustness to substantial future-query shifts remains a limitation.

  3. Quantified efficiency trade-offs(1aWj). We added Section 5.5 and expanded Appendix A.9 to report actual runtime memory, throughput, and consolidation overhead. On an NVIDIA H20 at the same cache budget, FrameFold has approximately 10% lower generation-stage throughput than Deep Forcing and FadeMem, and incurs additional peak memory relative to FIFO. Within FrameFold, reducing the budget from 21 to 12 entries lowers measured peak GPU memory by 32.8% and increases throughput by 49.9%. These measurements clarify the costs of consolidation and the benefits of smaller caches, informing configuration choices for memory-constrained deployment.

  4. Transparent experimental protocols(zd3D). We clarified prompt provenance and subset sampling in Section 5.1, and explicitly documented the reproduced components and adapted positional-encoding convention of our FadeMem baseline in Appendix B.2. We also corrected figure references and reviewed the formatting. Manuscript changes are highlighted in blue.

Our work contributes both an analysis of attention-based history retention and a compression method motivated by that analysis. Section 3 shows that shared attention rankings can overlook history needed by individual spatial queries, and that higher current attention coverage does not necessarily yield better subsequent generation quality. These findings inform cache-policy design for autoregressive video generation and motivate directly evaluating the retrievable content preserved after compression. FrameFold implements this perspective through conditional-response-preserving consolidation, achieving better visual quality under fixed cache budgets and smaller quality declines during long-horizon generation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.