acceptodds
Under review as a conference paper at ICLR 2027

Efficient Residual-Centric Inference for Spacetime Autoregressive Video Generation

Abstract

Spacetime autoregressive video models combine multiscale spatial generation within each clip with temporal autoregression across clips, avoiding repeated full-resolution denoising while enabling previously generated visual states to be reused as context. This design, however, introduces two complementary inference bottlenecks: attention computation becomes concentrated at late, high-resolution scales, while reusable history creates a persistent KV cache. A natural solution is to sparsify attention and compress historical KV, but the two are typically optimized with different proxy objectives. We observe that once omitted attention is replaced by a cheap pooled correction, the relevant quantity is no longer raw attention importance or standalone tensor reconstruction error, but the output residual left unexplained by the approximate attention operator. Based on this observation, we introduce residual-centric inference, which provides a shared post-correction residual view for compute allocation and cache representation. We instantiate this principle with Output-Sensitive Routing (OSR), which allocates token-level attention using a proxy for post-correction residual risk, and Residual-Interaction Quantization (RIQ), which preserves pooled statistics and fits low-bit history representations to residual K–V interactions. On InfinityStar, OSR achieves up to 1.86× end-to-end acceleration at native 720p/5s, while an orthogonal FFN FP8 optimization increases the complete-system speedup to 2.01×. At 480p/10s, OSR+RIQ achieves up to 1.50× speedup and 5.74× retained-history KV compression. Across both settings and tasks, these configurations largely preserve generation quality, with at most 0.31% relative degradation in VBench and VBench-I2V scores.

Primary area:
generative models
Keywords:
efficient inference, video generation, visual autoregressive models, sparse attention, KV cache quantization, low-bit quantization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_D7XJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The main idea of this paper is to have the video model spend more compute where a simple average is not good enough. The proposed method also makes the history cache smaller by keeping an accurate summary and compressing the finer details. These changes help make inference cheaper with as little loss in quality as possible.

Critical strengths
  1. The paper addresses a fundamental problem with clear motivation and a well-grounded formulation. Rather than following conventional technical approaches, it offers a fresh perspective by focusing on new types of errors introduced by acceleration.
  2. The experimental results and qualitative visualizations provide sufficient evidence to support the effectiveness of the proposed method.
Critical weaknesses
  1. The paper would benefit from clearer and more accessible writing, particularly for readers who are not specialists in this area. Streamlining the mathematical exposition, providing more intuitive explanations, and explicitly highlighting the differences from existing methods would make the paper easier to follow.
  2. The proposed method is somewhat complex, and its generalizability warrants further investigation. Additional evaluations across different models and settings would help assess whether its benefits extend beyond the configurations considered in the paper.
Questions for authors

None.

4 comments

Thank you for the suggestions on clarity and generality. We have expanded the intuitive explanations in the main text and added longer autoregressive continuation experiments in Appendix C.10. We additionally report a transfer test on LongCat-Video in this response.

W1 Clearer and more accessible writing

We have added explanations at the points where the mathematical quantities and design choices are introduced, while preserving the original formulation.

  • Introduction: We now explain pooled correction directly: an omitted block is represented by its mean key and value, retaining an inexpensive estimate of its contribution despite skipping token-level computation.

  • Method overview and Section 3.1: We add a short roadmap linking visibility, residual decomposition, routing, and cache representation. We also distinguish SSA’s native visibility mask from the further reduction in computation performed by OSR.

  • Section 3.2: We explain the block mean and token-count factor, then interpret the two residual terms separately: missing value detail and excess normalization mass. A simple example shows that equal logits within a block make pooling exact even when the block receives substantial attention. This illustrates why attention mass alone is insufficient.

  • Section 3.3: We clarify that the content and excess-mass terms estimate what pooling misses, while the mass guard provides conservative protection against underestimation by the compact proxy.

  • Section 3.4: We explain why cached blocks must support both pooled and token-level reads and clarify the complementary roles of reconstruction and interaction-aware key fitting. Key reconstruction error indirectly bounds perturbations of both the K–V cross-moment and key covariance, whereas interaction fitting weights key errors according to their interaction with value residuals to more directly target cross-moment distortion. The reconstruction regularizer retains indirect control of covariance perturbations.

  • Related Work: We add SparSTAR and clarify the distinction between its scale-specific block selection and our combination of post-correction routing with history representation.

These changes are incorporated into the revised ten-page main paper and marked in blue. They provide intuition alongside the equations without expanding the technical formulation.

W2 Generalizability across models and settings

We have added two complementary evaluations.

First, we test longer autoregressive continuation using the 480p InfinityStar-Interact checkpoint. Each method generates five successive five-second continuations from a common reference and fixed continuation prompt, feeding its own latent history into the next round. This evaluates 25 newly generated seconds.

InfinityStar-Interact: complete 25-second continuation latency. Lower time and higher speedup and retained-history compression are better.

\text{Method} & \text{Time (s)} & \text{Speedup} & \text{History comp.} \\\\ \hline \text{Dense} & 141.92 & 1.00\times & 1.00\times \\\\ \text{OSR} & 97.90 & 1.45\times & 1.00\times \\\\ \text{Dense + RIQ} & 143.63 & 0.99\times & 5.37\times \\\\ \text{OSR + RIQ} & 99.49 & 1.43\times & 5.37\times \end{array}$$ The complete-continuation VBench dimensions are: InfinityStar-Interact: VBench dimensions for complete 25-second continuations (higher is better). $$\begin{array}{lrrrr} \text{Dimension} & \text{Dense} & \text{OSR} & \text{Dense + RIQ} & \text{OSR + RIQ} \\\\ \hline \text{Subject consistency} & 93.227 & 92.793 & 94.422 & 93.770 \\\\ \text{Background consistency} & 96.460 & 96.542 & 97.308 & 97.043 \\\\ \text{Motion smoothness} & 99.196 & 99.196 & 99.157 & 99.194 \\\\ \text{Aesthetic quality} & 67.181 & 67.353 & 69.463 & 70.069 \\\\ \text{Imaging quality} & 71.210 & 70.012 & 71.580 & 69.484 \\\\ \text{Text consistency} & 27.838 & 27.224 & 28.171 & 27.737 \end{array}$$ OSR + RIQ achieves 1.43× speedup and 5.37× retained-history compression, with similar subject/background consistency and motion smoothness to Dense. Aesthetic quality increases while imaging quality decreases, making the quality trade-off explicit. This evaluation is included in Appendix C.10, Tables 13–14. Second, we evaluate transfer to LongCat-Video, an independent Diffusion Transformer backbone with autoregressive video continuation. We use 20 paired inputs at 832 × 480 and 15 fps. Both methods share a Dense-generated 93-frame initial clip. Each of ten continuations uses the most recent 73 frames as conditioning and generates 20 more frames, for 200 new frames and 293 total frames (approximately 13.33 and 19.53 seconds, respectively). The inference configuration uses CFG 4.0 and nominally 50 steps, with high-frequency enhancement resulting in 56 denoising calls per continuation. LongCat-Video: latency per continuation. Lower time and higher speedup and KV-cache compression are better. $$\begin{array}{lrrr} \text{Method} & \text{Time (s)} & \text{Speedup} & \text{KV comp.} \\\\ \hline \text{Dense} & 166.95 & 1.00\times & 1.00\times \\\\ \text{OSR + RIQ} & 118.78 & 1.41\times & 5.54\times \end{array}$$ LongCat-Video VBench dimensions (higher is better). $$\begin{array}{lrr} \text{Dimension} & \text{Dense} & \text{OSR + RIQ} \\\\ \hline \text{Subject consistency} & 96.09 & 95.26 \\\\ \text{Background consistency} & 96.11 & 94.95 \\\\ \text{Motion smoothness} & 99.29 & 99.16 \\\\ \text{Aesthetic quality} & 65.31 & 63.38 \\\\ \text{Imaging quality} & 72.10 & 72.51 \\\\ \text{Text consistency} & 29.37 & 29.35 \end{array}$$ This independent-backbone comparison yields 1.41× speedup and 5.54× KV-cache compression on the tested continuations. Motion smoothness and text consistency change by only -0.13 and -0.02 points; imaging quality increases by 0.41 points. Subject consistency, background consistency, and aesthetic quality decrease by 0.83, 1.16, and 1.93 points. We present this as evidence of transfer feasibility under a distinct generation architecture, while keeping the observed quality trade-offs explicit. The revised PDF contains the longer InfinityStar-Interact continuation experiment.

I thank the authors for their rebuttal. They have addressed my concerns, and I have decided to maintain my original score.

Thank you for following up and for taking the time to review our responses. We are glad that our rebuttal has addressed your concerns, and we appreciate your constructive feedback throughout the discussion.

Reviewer_CgM4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses expensive late-scale attention and persistent historical KV storage in spacetime autoregressive video generation. It analyzes the output residual of sparse attention with pooled correction and proposes two training-free mechanisms: Output-Sensitive Routing (OSR) for allocating token-level computation, and Residual-Interaction Quantization (RIQ) for compressing historical KV. On InfinityStar, OSR achieves up to 1.86× end-to-end acceleration at 720p/5s. At 480p/10s, OSR+RIQ achieves up to 1.50× acceleration and 5.74× retained-history compression, with aggregate generation-quality scores close to the dense baseline.

Critical strengths

(1) A coherent, operator-aware formulation. The paper identifies post-correction output residuals as a common basis for attention routing and historical KV compression. The decomposition explains why attention mass alone may be insufficient for selecting blocks and why elementwise reconstruction error may not preserve the information needed by corrected attention. This provides a clear technical rationale connecting the designs of OSR and RIQ.

(2) Meaningful practical efficiency gains without retraining. The approach addresses both late-scale attention computation and persistent history storage. OSR achieves up to 1.86× end-to-end acceleration, while OSR+RIQ achieves 5.74× retained-history compression, with aggregate generation-quality scores close to the dense baseline. The latency measurements include routing, pooled correction, and cache encoding/decoding, making the demonstrated gains practically relevant.

Critical weaknesses

(1) The end-to-end contribution of interaction-aware fitting is not isolated. Figure 5(c) and Table 7 provide relevant local comparisons against K-MSE fitting. However, the representation ablation uses fixed BF16-cache OSR trajectories without feeding the approximated outputs back into generation (Appendix B.6). Interaction fitting reduces the relative error against BF16 OSR from approximately 7.26% to 7.03% in Figure 5(c), but the corresponding end-to-end comparison with other representation and execution choices held fixed is not reported. The system-level results establish the effectiveness of the complete RIQ configuration, while leaving unclear how much this distinctive fitting objective contributes to final video quality under autoregressive feedback.

(1) The OSR ablations do not fully disentangle the excess-mass proxy and the mass guard at matched computational density. The content-only versus full-OSR comparison in Figure 5(a) introduces both terms together, supporting their combined benefit but not separating their individual contributions. Table 6 does include a mass-guard sweep, including γ=0γ=0, but retained density changes with γγ. Consequently, this comparison mixes changes in selection with changes in the amount of token-level computation. A matched-density comparison would better establish the individual contributions of these two terms.

Questions for authors

(1) Could the authors provide an end-to-end comparison between K-MSE fitting and interaction-aware fitting, keeping the INT2 codes, value restoration, original summaries, OSR policy, and execution kernels the same? Reporting paired differences in final video-quality and fidelity metrics would clarify whether the local improvement in Figure 5(c) translates into a benefit after autoregressive feedback.

(2) Could the authors compare content-only, content-plus-excess-mass, content-plus-guard, and full OSR at matched retained-block densities, using the same inputs, pooled correction, and source-aware selection procedure? This would complement the γ sweep in Table 6 and separate the individual contributions without changing the amount of retained token-level computation.

4 comments

Thank you for identifying the two controls needed to better isolate the method’s components. We have added a complete-video key-fitting comparison and expanded the OSR ablation to separate the excess-mass term from the mass guard at matched retained-block densities.

W1 / Q1 Interaction fitting under complete autoregressive generation

We agree that the original Figure 5(c) and local representation ablation answer an operator-level question: how a representation changes attention output on fixed inputs. They do not, by themselves, establish the effect after autoregressive feedback.

The revised manuscript therefore includes a complete-generation comparison at 480p/10s T2V, in which both variants use identical INT2 codes, V residual representations and restoration, original block summaries, OSR policy, execution kernels, and paired seeds. The only intervention is the key-scale fitting objective; each method generates the complete video with its own autoregressive feedback.

Complete-generation key-fitting ablation. Higher VBench, PSNR (dB), and SSIM and lower LPIPS are better. Delta is Interaction minus K-MSE.

\text{Metric} & \text{K-MSE} & \text{Interaction} & \text{Delta} & \text{95\\% CI of Delta} \\\\ \hline \text{VBench} & 80.57 & 81.68 & +1.11 & [-0.9081, +1.4471] \\\\ \text{PSNR} & 25.14 & 27.25 & +2.11 & [+1.9479, +2.3996] \\\\ \text{SSIM} & 0.7203 & 0.7895 & +0.0692 & [+0.063075, +0.079803] \\\\ \text{LPIPS} & 0.1957 & 0.1105 & -0.0852 & [-0.091517, -0.079833] \end{array}$$ Paired Δ is Interaction minus K-MSE. The 95% intervals exclude zero for PSNR, SSIM, and LPIPS, indicating better Dense-relative fidelity under complete autoregressive generation. Figure 5(c) evaluates a different question: on fixed BF16-OSR trajectories, interaction fitting reduces the per-call EΔ from approximately 7.26% to 7.03%, without feedback. Final-video fidelity is measured after subsequent layers, refinement scales, and autoregressive feedback. Such propagation could magnify some differences, but we do not quantify amplification; Table 8 provides the direct end-to-end evidence. The VBench interval includes zero, so we do not claim a statistically significant VBench improvement. Section 3.4 now clarifies why K-MSE is a principled but incomplete control. Minimizing key reconstruction error indirectly bounds perturbations of both leading moments—the K–V cross-moment for content and key covariance for excess mass—but cannot distinguish key-error directions with different effects on the cross-moment. Interaction fitting weights these directions through the value-residual Gram matrix and directly penalizes cross-moment distortion; reconstruction regularization retains indirect covariance control. Original block summaries, restored V residuals, the OSR policy, and execution kernels remain unchanged. Thus, the ablation tests the distinctive fitting objective, not the benefits of the shared RIQ components. The added complete-video comparison appears in Appendix C.5, Table 8.

W2 / Q2 Separating excess mass and the mass guard at matched density

We agree that increasing a parameter while allowing density to increase mixes two effects: changing the selection criterion and performing more token-level computation.

We have replaced Figure 5(a) with the requested four variants:

  • Content only: A.

  • Content + excess mass: A+B.

  • Content + guard: A+G.

  • Full OSR: A+B+G.

All four use the same inputs, pooled correction, source-aware selection procedure, protected tokens, and output normalization. We scan each variant's native τ using retained-block density alone, freeze the configuration, and then evaluate output error. The target densities are 6%, 7.5%, 10%, 12.5%, and 15%. The diagnostic uses six inputs and 144 attention calls, with sampling and aggregation specified in Appendix B.6.

The mean relative attention-output L2 errors in the updated figure are:

Mean relative attention-output L2 error (%, lower is better). Density is retained-block density (%). A = content; B = excess mass; G = mass guard.

\text{Density} & \text{A} & \text{A+B} & \text{A+G} & \text{A+B+G} \\\\ \hline 6\\% & 27.82\\% & 19.52\\% & 25.93\\% & 18.85\\% \\\\ 7.5\\% & 21.53\\% & 15.72\\% & 20.49\\% & 15.54\\% \\\\ 10\\% & 14.55\\% & 11.74\\% & 14.37\\% & 11.60\\% \\\\ 12.5\\% & 10.47\\% & 8.79\\% & 10.45\\% & 8.71\\% \\\\ 15\\% & 7.87\\% & 6.85\\% & 7.73\\% & 6.86\\% \end{array}$$ The excess-mass term accounts for the larger improvement. At 6% density, adding it to content-only scoring reduces error from 27.82% to 19.52%, a 29.8% relative reduction. Adding the guard further reduces error to 18.85%, another 3.4% relative reduction. The guard’s benefit is most visible at low density and becomes small at higher density; at 15%, full OSR and content-plus-excess-mass are essentially tied. Accordingly, we describe the guard as additional protection rather than claiming a uniform improvement at every operating point. Top-p and the fixed-block SVG-EAR-style controls remain in the figure. The revised PDF includes the expanded comparison in Figure 5(a), the interpretation in Section 4.3, and the protocol in Appendix B.6.

Thank you to the authors for the detailed responses and additional experiments. The complete-generation comparison of interaction-aware fitting and the matched-density OSR ablations largely address my concerns about the individual contributions of the proposed components. I will therefore maintain my Weak Accept rating.

Thank you for your careful assessment and constructive suggestions. We are glad that the complete-generation comparison and matched-density ablations have helped address your concerns and clarify the contributions of the individual components. Your feedback has helped us strengthen the experimental evidence and its presentation.

Reviewer_FRYoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes training-free inference optimizations for spacetime autoregressive video generation, targeting late-scale attention computation and persistent historical KV storage. It develops a post-correction residual formulation and introduces Output-Sensitive Routing (OSR) for selecting exact attention blocks, together with Residual-Interaction Quantization (RIQ) for compressing historical KV while preserving block statistics and K–V interactions. Experiments on InfinityStar cover T2V and I2V at 720p/5s and 480p/10s. The reported results include up to 1.86× acceleration with OSR, approximately 2× with additional FFN FP8 optimization, and 5.74× retained-history compression with RIQ.

Critical strengths
  1. A well-motivated operator-aware approach. The paper connects computation allocation and cache representation to the error remaining after pooled attention correction. Selecting blocks according to approximation risk, and fitting quantized keys to preserve K–V interactions, are meaningful alternatives to attention-mass selection and standalone tensor reconstruction.
  2. Substantial implementation and evaluation effort. The paper reports meaningful end-to-end acceleration and cache compression, with relevant baselines, five-seed evaluation, paired bootstrap intervals, component ablations, and separate accounting for retained-history storage and peak inference memory. The system controls help distinguish the proposed methods from common implementation optimizations.
Critical weaknesses
  1. Unequal kernel optimization complicates the latency comparison. RIQ uses fused decoding and layout preparation, whereas KIVI and QVG reconstruct history through different paths. The algorithmic contribution to the speedup is therefore unclear.

  2. Comparisons with the closest alternatives are incomplete. The paper lacks full-system comparisons with SVG-EAR and SparSTAR, as well as storage-matched comparisons with higher-fidelity QVG configurations.

  3. The practical benefit of interaction fitting is insufficiently isolated. Its contribution is evaluated through local attention errors, without a complete-generation ablation against reconstruction-only key fitting.

  4. Generality and long-rollout stability remain unverified. Experiments cover only InfinityStar on H200 and videos up to ten seconds.

Questions for authors
  1. How much of RIQ’s speed advantage remains under comparable decoding and layout-preparation implementations?

  2. How does the method compare with full SVG-EAR, SparSTAR, and QVG configurations at matched resource budgets?

  3. With all other components fixed, does interaction fitting improve complete-video quality over K-MSE fitting?

  4. Is there evidence of stable quality over more autoregressive clips or transfer to another backbone?

4 comments

Thank you for the careful assessment. We have added a controlled decoding/layout comparison, full-system SVG-EAR and SparSTAR baselines, storage-matched QVG, and a complete-generation K-MSE versus interaction-fitting ablation. We also provide longer autoregressive rollout evidence and a LongCat-Video backbone-transfer test to address generality. We address all four concerns below.

W1 / Q1 Unequal kernel optimization and comparable decoding and layout preparation

We agree that the original latency comparison did not sufficiently separate RIQ’s representation from the benefit of fused decoding and layout preparation. RIQ’s quantization objective targets the information retained under compression; fusion is an additional execution optimization.

We therefore added a controlled comparison of OSR with BF16 history, unfused KIVI, unfused QVG, unfused RIQ, and fused RIQ. All arms share the same model, System optimizations, BF16 FFN, and OSR policy. Each unfused decoder materializes contiguous BF16 KV in the same batch–head–token–channel layout, including unquantized tails, before invoking the same layout preparation and OSR executor. Fused RIQ combines decoding/restoration with layout preparation. This standardizes the downstream interface and execution while retaining each format’s decoder.

For operator timing, we use identical captured queries and source KV with a frozen BF16 routing mask at layer 18, on the first visit to scales 40 and 41. Timing includes decoding, layout preparation, attention, and merging, and excludes cache encoding and routing. We report median CUDA-event latency after ten warm-ups and 15 rounds of five repeats.

Operator latency (ms, lower is better). All methods use OSR. QVG-S2 is storage-matched; RIQ-U/F denote unfused/fused RIQ.

\text{Scale} & \text{BF16} & \text{KIVI} & \text{QVG-S2} & \text{RIQ-U} & \text{RIQ-F} \\\\ \hline 40 & 15.430 & 20.443 & 16.592 & 16.349 & 13.460 \\\\ 41 & 22.340 & 27.523 & 23.817 & 23.233 & 19.207 \end{array}$$ All values are milliseconds. Unfused RIQ and storage-matched QVG have similar operator latency. Fusion reduces RIQ latency by another 2.89/4.03 ms at scales 40/41. We also measured complete-generation latency on ten paired 480p/10s T2V inputs, with a full-generation warm-up, synchronized timing, and rotated method order. These measurements include routing and KV encoding/decoding. Complete-generation T2V latency (s, lower is better). All methods use OSR; QVG-S2 is storage-matched. $$\begin{array}{lr} \text{History / decoder} & \text{Time (s)} \\\\ \hline \text{BF16 history} & 49.882 \\\\ \text{KIVI} & 56.895 \\\\ \text{QVG-S1} & 52.548 \\\\ \text{QVG-S2} & 52.662 \\\\ \text{RIQ, unfused} & 49.737 \\\\ \text{RIQ, fused} & 47.367 \end{array}$$ On T2V, unfused RIQ reduces latency by 12.6% relative to KIVI and 5.6% relative to storage-matched QVG. Fusion saves a further 2.37 s, or 4.8% of unfused RIQ latency. These results separate fusion’s incremental benefit from the comparison between cache representations and their decoding paths. They do not imply identical optimization of format-specific decoders or isolate the interaction-fitting objective itself. The revised PDF includes the protocol and results in Appendix C.9, Tables 11–12.

W2 / Q2 Full SVG-EAR SparSTAR and storage-matched QVG

We have added full adapted SVG-EAR and the SparSTAR sparse-attention implementation to the main 720p/5s comparison. SVG-EAR includes semantic clustering, token permutation, error-aware selection, and centroid correction, with all associated overhead included in end-to-end latency. This is distinct from the fixed-block SVG-EAR-style routing control in Figure 5(a).

SparSTAR uses retained densities of 0.625, 0.1875, 0.1875, and 0.09375 at its four final refinement scales. Both baselines share the common System optimizations used by the accelerated methods.

720p/5s sparse baselines. Time is in seconds; density is global (%); PSNR is in dB. VB denotes VBench for T2V and VBench-I2V for I2V. Lower time and higher speedup, PSNR, and VB are better.

\text{Task} & \text{Method} & \text{Time} & \text{Speedup} & \text{Density} & \text{PSNR} & \text{VB} \\\\ \hline \text{T2V} & \text{SVG-EAR} & 93.40 & 0.60\times & 27.72 & 31.33 & 84.63 \\\\ \text{T2V} & \text{SparSTAR} & 36.31 & 1.56\times & 25.06 & 29.92 & 83.96 \\\\ \text{T2V} & \text{OSR} & 30.35 & 1.86\times & 19.45 & 32.31 & 84.82 \\\\ \text{I2V} & \text{SVG-EAR} & 105.15 & 0.54\times & 27.29 & 28.78 & 88.08 \\\\ \text{I2V} & \text{SparSTAR} & 35.48 & 1.59\times & 24.79 & 27.42 & 88.15 \\\\ \text{I2V} & \text{OSR} & 31.11 & 1.82\times & 19.08 & 30.30 & 88.41 \end{array}$$ These results compare deployed operating points, with density explicitly reported, rather than claiming identical compute budgets. OSR achieves a favorable latency–fidelity trade-off in both tasks. The results are added to Table 1, the configurations to Appendix B.3, and the bootstrap comparisons to Appendix C.11, Table 15. SparSTAR is also included in Related Work. For QVG, we additionally match retained-history storage. The original configuration is S1/C256/K64/V64; the added configuration is S2/C128/K32/V32, where S denotes progressive residual stages, C the codebook size, and K/V the group sizes. The latter achieves 5.738× compression, compared with 5.74× for RIQ, under the same cache scope and storage accounting. Storage-matched comparison. All methods use OSR. Comp. denotes retained-history compression; time is in seconds; PSNR is in dB. VB denotes VBench for T2V and VBench-I2V for I2V. Higher Comp., PSNR, SSIM, and VB and lower time and LPIPS are better. $$\begin{array}{llrrrrrr} \text{Task} & \text{History} & \text{Comp.} & \text{Time} & \text{PSNR} & \text{SSIM} & \text{LPIPS} & \text{VB} \\\\ \hline \text{T2V} & \text{QVG-S2} & 5.738\times & 52.38 & 25.83 & 0.7423 & 0.1588 & 81.05 \\\\ \text{T2V} & \text{RIQ} & 5.74\times & 46.57 & 27.25 & 0.7895 & 0.1105 & 81.68 \\\\ \text{I2V} & \text{QVG-S2} & 5.738\times & 50.86 & 26.28 & 0.7667 & 0.1595 & 84.07 \\\\ \text{I2V} & \text{RIQ} & 5.74\times & 47.93 & 28.13 & 0.8150 & 0.1056 & 87.11 \end{array}$$ At approximately equal retained storage, RIQ has higher fidelity and aggregate benchmark scores in both tasks. The latency values above include the deployed RIQ fusion; the controlled execution comparison discussed in our response to W1 separates that additional benefit. Both QVG configurations and RIQ are reported in Appendix C.7, Table 10. The main table retains the original QVG configuration.

W3 / Q3 Complete-generation benefit of interaction fitting

We agree that fixed-trajectory attention replay cannot establish whether an improvement persists after autoregressive feedback. We therefore added a complete-generation comparison on 480p/10s T2V. Both methods use the same packed INT2 codes, restored V representation, original block summaries, OSR policy, execution kernels, and paired generation seeds. Only the key-scale fitting objective changes, and each method generates the complete video with its own autoregressive feedback trajectory.

Complete-generation key-fitting ablation. Higher VBench, PSNR (dB), and SSIM and lower LPIPS are better. Delta is Interaction minus K-MSE.

\text{Metric} & \text{K-MSE} & \text{Interaction} & \text{Delta} & \text{95\\% CI of Delta} \\\\ \hline \text{VBench} & 80.57 & 81.68 & +1.11 & [-0.9081, +1.4471] \\\\ \text{PSNR} & 25.14 & 27.25 & +2.11 & [+1.9479, +2.3996] \\\\ \text{SSIM} & 0.7203 & 0.7895 & +0.0692 & [+0.063075, +0.079803] \\\\ \text{LPIPS} & 0.1957 & 0.1105 & -0.0852 & [-0.091517, -0.079833] \end{array}$$ Here Δ denotes Interaction minus K-MSE. The paired intervals exclude zero for PSNR, SSIM, and LPIPS, supporting a fidelity improvement under autoregressive feedback. These end-to-end results complement the smaller local attention-error reductions in Figure 5(c). The VBench interval includes zero, so we do not claim a statistically significant VBench improvement. **Why local and complete-video metrics differ.** Figure 5(c) measures per-call attention-output error relative to BF16 OSR on fixed BF16-OSR trajectories, without feeding approximated outputs back. Interaction fitting changes EΔ from approximately 7.26% (K-MSE) to 7.03%. Table 8 instead evaluates complete videos under each method's own autoregressive feedback. Historical KV is repeatedly accessed across layers and scales 39–41, so attention perturbations can affect subsequent layers and scale-conditioned tokens. **Why interaction fitting adds value beyond K-MSE.** K-MSE minimizes key reconstruction error, indirectly bounding perturbations of both the K–V cross-moment (leading content residual) and key covariance (leading excess-mass residual). These are norm bounds, not an operator-aligned objective: key errors of equal magnitude can produce different cross-moment distortions. The interaction term in Equation 12 therefore weights key errors through the value-residual Gram matrix, directly targeting directions that matter to the content cross-moment. Its reconstruction regularizer retains indirect covariance control. Both variants share original pooled summaries, V restoration, and OSR routing, so Table 8 isolates this more targeted key-scale fitting objective. The complete-video comparison is included in Appendix C.5, Table 8.

W4 / Q4 Longer autoregressive rollouts and backbone transfer

We added two complementary tests. First, we evaluated the 480p InfinityStar-Interact checkpoint on ten inputs, each undergoing five successive five-second continuations (25 newly generated seconds, 400 frames at 16 fps). Each method feeds its own generated latent history into the next continuation. The context is rebuilt within a bounded window, so this tests repeated autoregressive feedback rather than unbounded KV growth.

InfinityStar-Interact: complete 25-second continuation latency. Lower time and higher speedup and retained-history compression are better.

\text{Method} & \text{Time (s)} & \text{Speedup} & \text{History comp.} \\\\ \hline \text{Dense} & 141.92 & 1.00\times & 1.00\times \\\\ \text{OSR} & 97.90 & 1.45\times & 1.00\times \\\\ \text{Dense + RIQ} & 143.63 & 0.99\times & 5.37\times \\\\ \text{OSR + RIQ} & 99.49 & 1.43\times & 5.37\times \end{array}$$ For the complete 25-second continuations, Dense versus OSR + RIQ scores are 93.227/93.770 (subject consistency), 96.460/97.043 (background consistency), 99.196/99.194 (motion smoothness), 67.181/70.069 (aesthetic quality), 71.210/69.484 (imaging quality), and 27.838/27.737 (text consistency). These results show no major aggregate collapse over the tested repeated continuations, although they do not establish a per-clip drift bound. Full details and the four-arm comparison are in Appendix C.10, Tables 13–14. Second, we transferred OSR + RIQ to LongCat-Video, a separate Diffusion Transformer backbone supporting autoregressive video continuation. We tested 20 paired inputs at 832 × 480 and 15 fps. All methods began from the same 93-frame video generated by Dense. Each of ten continuations conditioned on the latest 73 frames and generated 20 new frames, giving 200 new frames (approximately 13.33 s) and 293 frames in total (approximately 19.53 s). We used CFG 4.0, nominally 50 sampling steps with high-frequency enhancement, resulting in 56 denoising calls per continuation. LongCat-Video: latency per continuation. Lower time and higher speedup and KV-cache compression are better. $$\begin{array}{lrrr} \text{Method} & \text{Time (s)} & \text{Speedup} & \text{KV comp.} \\\\ \hline \text{Dense} & 166.95 & 1.00\times & 1.00\times \\\\ \text{OSR + RIQ} & 118.78 & 1.41\times & 5.54\times \end{array}$$ LongCat-Video VBench dimensions (higher is better). $$\begin{array}{lrr} \text{Dimension} & \text{Dense} & \text{OSR + RIQ} \\\\ \hline \text{Subject consistency} & 96.09 & 95.26 \\\\ \text{Background consistency} & 96.11 & 94.95 \\\\ \text{Motion smoothness} & 99.29 & 99.16 \\\\ \text{Aesthetic quality} & 65.31 & 63.38 \\\\ \text{Imaging quality} & 72.10 & 72.51 \\\\ \text{Text consistency} & 29.37 & 29.35 \end{array}$$ LongCat demonstrates acceleration and retained-KV compression on a different architecture and a ten-step continuation regime. The quality changes are mixed: motion/text consistency are nearly unchanged, imaging quality is higher by 0.41 points, while subject/background consistency and aesthetic quality decrease by 0.83/1.16/1.93 points. These results demonstrate the feasibility of transferring OSR+RIQ to a distinct video generation backbone and suggest its potential for broader applicability. The InfinityStar-Interact experiment is in the revised PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.