acceptodds
Under review as a conference paper at ICLR 2027

CORAL-KV: Compact Cross-Layer Prediction and Residual Coding for Autoregressive Video KV Caches

16 views

Abstract

Autoregressive video diffusion models maintain a growing attention key–value (KV) cache, making memory a key bottleneck for long video generation. Recently, clustering-based quantization has emerged as an effective method to compress video KV caches by grouping similar key and value vectors and quantizing their residuals. However, when later frames attend to the compressed history, small reconstruction errors can compound as the video grows. Beyond the within-layer similarity exploited by clustering, we find strong dependencies across cache layers in video models. This motivates Coral-KV: Calibrated Offline, Residual-coded, and Adapted Locally—a KV-cache compression framework that combines reusable cross-layer predictors with low-bit residual quantization. We use reconstructed cache states to predict keys and values in subsequent layers, then quantize the resulting prediction residuals. We construct compact block-diagonal-plus-low-rank (block-DPLR) predictors: block-diagonal components model channel relationships within each attention head, while a global low-rank term models dependencies between heads. Coral-KV exposes a flexible fidelity–memory trade-off: it can reuse its predictors across videos or refit lightweight block-diagonal predictors to the current video for higher fidelity. Across extensive evaluations on both HY-WorldPlay and MAGI-1, our reusable configuration improves PSNR over QVG-4 by 0.32dB and 4.95dB, respectively, while using 48% and 37% less cache memory. Against QVG-Pro, our video-adapted configuration improves MAGI-1 PSNR by 3.77 dB while using 33.7% less cache memory. Coral-KV also maintains higher reconstruction fidelity over extended generation and reduces temporal discontinuities while retaining comparable generation latency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video diffusion, kv cache compression, quantization, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_oQvFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CORAL-KV, a KV-cache compression method for autoregressive video generation. It proposes block-DPLR that utilizes the redundancy between layers to improve the quantization performance. It introduces low-bit residual quantization and cheap predictors to capture both intra-head and inter-head correlations. Experiments on HY-WorldPlay and MAGI-1 demonstrate improved memory–fidelity trade-offs over existing methods like QVG and QVG-Pro.

Critical strengths
  1. The block-DPLR predictor efficiently exploits cross-layer redundancy without introducing too much overhead.

  2. The method achieves better memory-quality trade-offs than existing baselines, particularly on MAGI-1.

Critical weaknesses
  1. Experiments cover only two video models, leaving the applicability to other models unclear. Meanwhile, performance gains on HY-WorldPlay are relatively small compared to the huge gains on MAGI-1.

  2. The video-adapted variant introduces additional latency (especially on Hy World Play), which is much larger than QVG or QVG Pro. However, on MAGI-1, the latency overhead is on par with QVG and QVG-Pro. Doing further ablation studies on why this happens and how to control the overhead will be critical.

Questions for authors

Please refer to the weakness section. I would recommend that the authors add experiments on at least two more model families to validate the effectiveness and generalizability of their method.

5 comments

Summary of Response to Reviewer oQvF

We thank the reviewer for the constructive feedback and, in particular, for providing concrete suggestions for addressing your concerns. In response, we have carried out the requested additional experiments and analyses, including evaluation on two additional model families and further ablations of the HY-WorldPlay gains and video-adaptation overhead. These additions directly address each of the concerns raised in the review. We organize the new evidence as follows:

  • [Answer 1] Video-adapted CORAL-KV outperforms QVG-Pro in both fidelity and cache storage on Self-Forcing and LongCat-Video.
  • [Answer 2] The HY-WorldPlay quality gains hold across input sources, and longer contexts.
  • [Answer 3] We explain the higher video adaptation cost on HY-WorldPlay and show how fitting predictors on fewer tokens reduces adaptation overhead while retaining higher quality than QVG baselines at the same token budget.

We hope the new evidence addresses the reviewer’s concerns and would appreciate a reassessment in light of these results.

[Answer 1] Video-adapted CORAL-KV outperforms QVG-Pro in both fidelity and cache storage on Self-Forcing and LongCat-Video

We add evaluations of CORAL-KV on Self-Forcing DMD 1.3B and LongCat-Video 13.6B. We fit one predictor per layer for each model using 32 calibration inputs, and evaluate both models on the same 32 held-out MovieGen prompts. We compare both reusable CORAL-KV and video-adapted CORAL-KV to QVG baselines, using the same model settings and BF16 references. Both CORAL-KV configurations use the block-DPLR design with rank r=128r=128.

For Self-Forcing, we retain 144 and 288 latent frames as context, generating 573 and 1,149 decoded frames, respectively. We compress the KV cache history in blocks of 24 latent frames. For LongCat, we use a conditioning prefix of 49 and 97 decoded frames, and generate 80 new decoded frames at each setting. We compress the conditioning prefix’s KV cache once. Consistent with our accounting for HY-WorldPlay and MAGI-1, we report memory as including predictor weights, adaptation state, quantized residuals, quantizer metadata and any uncompressed KV retained during generation.

Self-Forcing DMD 1.3B

Method144 latent frames: PSNR ↑144 latent frames: GB ↓ / compression ↑288 latent frames: PSNR ↑288 latent frames: GB ↓ / compression ↑
BF16Reference41.41 / 1.00×Reference82.81 / 1.00×
QVG-122.7411.85 / 3.49×21.9817.80 / 4.65×
QVG-425.6412.97 / 3.19×25.2320.25 / 4.09×
QVG-Pro26.9613.78 / 3.01×26.8622.02 / 3.76×
CORAL-KV reusable26.6011.71 / 3.54×27.0717.38 / 4.77×
CORAL-KV video-adapted27.1812.15 / 3.41×27.8218.33 / 4.52×

At 144 retained latent frames, reusable CORAL-KV achieves 26.60 dB with 11.71 GB of stored cache, using the least memory among all methods while outperforming QVG-1 and QVG-4 in PSNR. Video-adapted CORAL-KV achieves the highest mean PSNR of 27.18 dB, with 11.80% less storage than QVG-Pro.

At 288 retained latent frames, reusable CORAL-KV achieves 27.07 dB, outperforming all three QVG configurations in both mean PSNR and stored-cache memory, with 21.10% less storage than QVG-Pro. Video-adapted CORAL-KV further improves fidelity to 27.82 dB, gaining 0.95 dB over QVG-Pro with 16.80% less storage. Together, these results show favorable quality–storage scaling for CORAL-KV on Self-Forcing.

LongCat-Video 13.6B

Method49 conditioning frames: PSNR ↑49 conditioning frames: GB ↓ / compression ↑97 conditioning frames: PSNR ↑97 conditioning frames: GB ↓ / compression ↑
BF16Reference15.95 / 1.00×Reference30.67 / 1.00×
QVG-119.812.51 / 6.36×21.234.63 / 6.62×
QVG-422.343.30 / 4.84×23.795.60 / 5.48×
QVG-Pro22.954.05 / 3.94×25.687.04 / 4.36×
CORAL-KV reusable25.152.55 / 6.25×25.504.55 / 6.75×
CORAL-KV video-adapted25.612.83 / 5.64×26.624.93 / 6.22×

At 49 conditioning frames, reusable CORAL-KV achieves 25.15 dB, outperforming all three QVG configurations in mean PSNR while using 36.90% less storage than QVG-Pro. Video-adapted CORAL-KV achieves the highest mean PSNR of 25.61 dB, with 30.10% less storage than QVG-Pro.

At 97 conditioning frames, reusable CORAL-KV becomes the most memory-efficient method, achieving 25.50 dB with 4.55 GB of stored cache. It improves over QVG-1 by 4.27 dB with 1.90% less storage. Video-adapted CORAL-KV achieves 26.62 dB, improving over QVG-Pro by 0.94 dB with 29.90% less storage.

[Answer 2] The HY-WorldPlay quality gains hold across input sources and longer contexts

Building on the HY-WorldPlay quality gains demonstrated in the paper at 48 retained latent frames, we show that these gains also hold at 72 and 256 retained latent frames, alongside substantial reductions in cache storage. At 72 retained latent frames, video-adapted CORAL-KV improves video PSNR over QVG-Pro by 0.17 dB while using 32.70% less cache storage. At 256 retained latent frames, this advantage increases to 0.31 dB with 35.20% less storage. Reusable CORAL-KV likewise improves over QVG-1 by 0.53 dB with 13.70% less storage at 72 frames, and 1.19 dB with 18.20% less storage at 256 frames. These results demonstrate a quality–storage Pareto improvement: reusable CORAL-KV outperforms QVG-1, and video-adapted CORAL-KV outperforms both QVG-4 and QVG-Pro, in fidelity and cache storage at both context lengths. We report results on 32 prompts with matched BF16 references below.

Method72 latent frames: Video PSNR (dB) ↑72 latent frames: Stored cache (GB) ↓256 latent frames: Video PSNR (dB) ↑256 latent frames: Stored cache (GB) ↓
QVG-115.777.9916.4928.42
QVG-416.3013.3617.9147.52
QVG-Pro16.5714.4618.3751.41
CORAL-KV reusable16.306.9017.6923.26
CORAL-KV video-adapted16.749.7418.6833.34

Across these two context lengths, both CORAL-KV variants provide higher mean fidelity with less cache storage than their respective QVG baselines, with larger advantages at 256 retained latent frames.

Quality gains on a different input source. We also evaluate 32 inputs from VBench-I2V, reusing the same predictors trained on MovieGen. On HY-WorldPlay, video-adapted CORAL-KV achieves 16.69 dB PSNR versus 16.20 dB for QVG-Pro, a 0.49 dB gain, while using 4.79% less cache storage (9.71 GB versus 10.20 GB). It also improves SSIM from 0.42 to 0.45 and reduces LPIPS from 0.46 to 0.42. These results show that the quality gains also hold on a different input source.

[Answer 3] We explain the higher adaptation cost on HY-WorldPlay and show that fewer fitting tokens reduce adaptation overhead

Why adaptation is more costly on HY-WorldPlay. HY-Worldplay fits predictors across 24 heads with 256 channels per head, compared with eight heads with 128 channels per head on MAGI. To isolate the effect of this geometry, we tested the block-diagonal fitting implementation on synthetic KV inputs with 7,040 fitting tokens. Each measurement uses three warm-ups and nine repetitions; the reported cost is the sum of the median key and value update times for one layer, normalized to MAGI's geometry.

GeometryAttention headsChannels per headTime for a key-and-value adaptation update, relative to MAGI geometry
MAGI-181281.00×
HY-WorldPlay242562.81×

At the same token count and rank, an adaptation update takes 2.81× as long with HY's geometry. This is consistent with the larger generation overhead reported in Table 1: 29.10% on HY versus 10.00% on MAGI. This shows that HY-Worldplay's adaptation cost comes from having to fit more head-aligned blocks, as well as larger dimension per block.

Reducing adaptation cost with fewer fitting tokens. One way to reduce the cost of updating the block-diagonal component is to use fewer KV tokens for each fitting update. We compare using all available KV tokens in the cache block with using 2,048 token positions spaced evenly across the block. At each layer, we use the KV vectors at these positions to refit the block-diagonal component, then apply the updated predictor to the entire cache block. The shared low-rank factors and residual quantization format remain unchanged.

We measured the associated fidelity trade-off on eight fixed inputs per model, and extended the comparison to QVG-1, QVG-4 and QVG-Pro using the same inputs and BF16 references. For QVG, we fit the centroids using the same 2,048 token positions, while still compressing all KV tokens in the block.

MethodHY: all fitting tokens PSNR ↑HY: 2,048 fitting tokens PSNR ↑MAGI: all fitting tokens PSNR ↑MAGI: 2,048 fitting tokens PSNR ↑
QVG-114.0514.0325.2925.16
QVG-414.2414.0827.1325.90
QVG-Pro14.95‾\underline{14.95}14.49‾\underline{14.49}30.14‾\underline{30.14}28.12‾\underline{28.12}
CORAL-KV video-adapted15.2714.9434.8134.80

We find that using only 2,048 tokens can reduce the online adaptation overhead by 64.90% on HY and 37.40% on MAGI. Additionally, the results above show that at the same token budget, CORAL-KV outperforms all three QVG baselines in mean PSNR on both models. We use evenly spaced tokens without optimizing which tokens to select.

On MAGI, mean PSNR changes by only 0.02 dB, compared with a 2.02 dB decrease for QVG-Pro. The 2,048-token CORAL-KV configuration retains a 4.65 dB gain over QVG-Pro fitted on all tokens (95% CI: [3.17, 5.99]), with better mean SSIM and LPIPS. On HY, using fewer fitting tokens reduces mean PSNR by 0.34 dB, compared with 0.46 dB for QVG-Pro, while lowering CORAL-KV's latency overhead. This simple experiment shows that fitting on fewer tokens can retain most of the fidelity while substantially reducing adaptation time; more sophisticated token selection may further improve this trade-off and is left as future work.

We will incorporate the additional model-family evaluations and HY-WorldPlay quality–storage findings into the revised manuscript, with the detailed adaptation-cost and fitting-token ablations in the appendix.

Reviewer_1jPXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets at Key-Value (KV) cache during long video generation. The authors propose Coral-KV, a novel compression framework that leverages cross-layer dependencies. By utilizing a block-plus-low-rank (block-DPLR) predictor combined with low-bit residual coding, CORAL-KV effectively reduces memory overhead while preserving or even improving reconstruction quality and temporal consistency. Extensive evaluations on models like HY-WorldPlay and MAGI-1 demonstrate substantial memory savings alongside significant PSNR gains.

Critical strengths

A refreshing perspective: Shifting the focus from intra-layer redundancy to cross-layer dependencies is well-motivated and provides a fresh angle for video KV cache compression.

Thoughtful architectural design: The proposed block-DPLR predictor strikes a sensible balance by capturing both intra-head channel relationships and inter-head global interactions without introducing prohibitive complexity.

Compelling empirical results: The consistent performance gains (both in memory reduction and PSNR improvement) across different backbone models (HY-WorldPlay and MAGI-1) are impressive and thoroughly validated.

Critical weaknesses

Runtime and Computational Overhead: While the paper reports significant memory savings and claims "comparable generation latency," I would appreciate a more granular breakdown of the computational overhead. Specifically, how do the extra cross-layer prediction steps and the online video-adapted fine-tuning impact the actual throughput (e.g., tokens/sec or frames/sec) and end-to-end wall-clock time during inference?

Ablation on Block-DPLR Components: The block-DPLR design combines block-diagonal components and a global low-rank term. It would strengthen the paper to see a clearer ablation showing how each component individually contributes to the overall fidelity, and whether the global low-rank term becomes a bottleneck for longer sequence lengths.

Hyperparameter Sensitivity: The trade-off between compression ratio and generation fidelity relies on several choices (such as the rank in the low-rank term and block sizes). A brief discussion or sensitivity analysis regarding how these hyperparameters behave across varying video lengths would be very helpful.

Questions for authors

How does the global low-rank term scale as the video sequence length grows significantly? Does it eventually become a memory or computational bottleneck?

Are the hyperparameters (such as block sizes and low-rank dimensions) robust across different types of video content (e.g., high-motion vs. static scenes), or do they require per-dataset tuning?

5 comments

Summary of Response to Reviewer 1jPX

We thank the reviewer for the detailed suggestions on runtime overhead, the contribution of each predictor component, and hyperparameter sensitivity. We provide additional profiling and content-group analyses, together with the component and rank ablations, in the following responses:

  • [Answer 1] We provide a granular breakdown of the latency for both CORAL-KV methods. Cache encoding and online video-adapted finetuning account for most of the time spent managing the compressed cache, while predictor execution during cache reconstruction contributes a small fraction of total generation time.
  • [Answer 2] Block-DPLR improves fidelity over either component alone, with fixed predictor storage and computation that scales linearly with sequence length.
  • [Answer 3] Additional ablations show that the choice of hyperparameters is robust across different types of content. Increasing the rank size generally improves video quality, but provides diminishing returns beyond rank r=128r=128.

We hope these additional results clarify the latency overhead and hyperparameter choice and address the reviewer's questions.

[Answer 1] Granular breakdown of latency for CORAL-KV

Latency breakdown. We measure the cumulative time spent in each component using GPU events and express it as a percentage of the total generation time for the same method and model in the same profiled run. We repeat each generation three times and report the median percentage for each component. We profile the following events: Cache encoding (predicting the KV, then quantizing and packing the residuals); Online fitting (learning centroids through K-means, or fitting block-diagonal predictors); Forward pass (applying predictor weights during cache reconstruction); Cache decoding (unpacking and dequantizing stored residuals); and the remaining generation latency.

ComponentHY reusableHY adaptedMAGI reusableMAGI adapted
Cache encoding12.99%8.03%7.04%7.12%
Online fitting0.00%8.37%0.00%4.63%
Predictor fwd pass for reconstruction2.46%2.08%0.67%0.69%
Cache decoding6.70%1.54%0.82%0.79%
Remaining generation77.85%79.99%91.47%86.76%
End-to-end latency overhead vs. BF16 (paper, Table 1)+8.70%+29.10%+6.80%+10.00%

For a fixed number of output frames, the normalized frame throughput corresponding to a reported end-to-end latency overhead of h%h\% is 1/(1+h/100)1/(1+h/100) times the BF16 throughput.

This table shows that cache encoding and online fitting are the largest measured cache-management costs. For video-adapted CORAL-KV, the median combined share of these components across repetitions is 16.40% of generation time on HY-WorldPlay and 11.76% on MAGI-1, while predictor execution during reconstruction accounts for 2.08% and 0.69%, respectively. For reusable CORAL-KV, online fitting is eliminated, and reconstruction-predictor execution accounts for 2.46% on HY-WorldPlay and 0.67% on MAGI-1.

One reason the online fitting cost remains low is that we adapt the block-diagonal component while keeping the low-rank component fixed. Each head-aligned block can be fitted independently, allowing the solves to be batched and run in parallel on the GPU. We will update the paper with a more detailed discussion of this design and implementation.

[Answer 2] Block-DPLR improves fidelity over either component alone, with fixed predictor storage and computation that scales linearly with sequence length

We provide an ablation of the components separately in Figure 4 of the manuscript. Our predictor ablations show that combining these two structures improves fidelity over either component alone, even when holding the number of parameters fixed. We summarize the results using the same residual quantizer:

PredictorHY-WorldPlay PSNR (dB) ↑MAGI-1 PSNR (dB) ↑
Quantizer only, without cross-layer prediction14.1425.19
Diagonal-only predictor14.5325.37
Block-diagonal-only predictor16.4727.11
Low-rank-only predictor, matched predictor storage17.0329.22
CORAL-KV block-DPLR, rank 12817.5531.99

This shows that block-DPLR has the best reconstruction over each component alone, improving PSNR over block-diagonal-only prediction by 1.08 dB on HY and 4.89 dB on MAGI. At an equivalent number of parameters, it also improves over low-rank-only prediction by 0.52 dB and 2.78 dB, respectively. These comparisons demonstrate the benefit of combining within-head and cross-head prediction.

The low-rank component scales efficiently to long sequence lengths. CORAL-KV applies its predictors independently to each token: predicting one token’s KV does not directly combine inputs from different token positions. Consequently, the predictor weights and computation per token are independent of sequence length.

For NN reconstructed tokens, channel width CC, and rank rr, the low-rank term requires O(NCr)O(NCr) computation and an O(Nr)O(Nr) intermediate, while its predictor weights occupy only O(Cr)O(Cr) storage. We measured the time spent on computing the low-rank prediction with actual KV tensors at 48, 128, and 256 generated latent frames. Its timing scales approximately linearly with respect to the number of generated frames. These measurements support efficient scaling over the tested context lengths without introducing quadratic dependence on sequence length.

To clarify one point: by “global,” we mean that the low-rank component mixes information across attention heads for the same token. Each reconstructed key or value vector has C=HKVdC=H_{KV}d channels, arranged into HKVH_{KV} attention heads with dd channels per head. For key prediction, the block-diagonal component uses the preceding layer’s corresponding key head. For value prediction, it uses the corresponding current-layer key head and preceding-layer value head. The low-rank component can combine information across all heads of these respective inputs.

[Answer 3] Hyperparameter choices are robust across video content, with diminishing returns beyond rank r=128r=128

Rank sensitivity across video lengths

We conduct an ablation study using ranks r=16, 32, 64, 128, and 256 across varying video lengths. We measure quality over the first 205, 301, and 525 frames on HY-WorldPlay, and the first 192, 288, and 384 frames on MAGI-1, using eight inputs per model.

HY-WorldPlay: mean frame PSNR (dB)

Rank205 decoded frames301 decoded frames525 decoded frames
1616.0415.6716.00
3216.2415.6615.92
6416.6016.0016.15
12816.9116.2916.45
25617.0216.3716.45

MAGI-1: mean frame PSNR (dB)

Rank192 decoded frames288 decoded frames384 decoded frames
1628.6123.5520.29
3230.0625.0921.73
6430.9025.5522.08
12831.5926.3922.83
25631.9626.6323.13

Rank 128 improves mean PSNR over ranks 16, 32 and 64 at every tested length on both models. Increasing the rank to 256 yields at most 0.11 dB additional mean PSNR on HY-WorldPlay and 0.37 dB on MAGI-1. These results support rank 128 as a common setting across the tested video lengths, while allowing for modest additional gains at higher rank.

Robustness across video content and input sources

We tested ranks 16, 32, 64, 128 and 256 on the same 32 prompts per model used in Figure 4, grouping them into three buckets corresponding to varying levels of temporal change. We define temporal change as the average absolute pixel difference between consecutive grayscale frames, sampled at 4 Hz and resized to 64×6464\times64.

ModelTemporal-change groupRank 16 PSNR ↑Rank 32 PSNR ↑Rank 64 PSNR ↑Rank 128 PSNR ↑Rank 256 PSNR ↑
HY-WorldPlayLower19.6519.3719.4720.3119.85‾\underline{19.85}
HY-WorldPlayMiddle16.4116.4216.3416.82‾\underline{16.82}16.94
HY-WorldPlayHigher15.0014.9515.1515.33‾\underline{15.33}15.66
MAGI-1Lower31.0732.3633.4534.4134.15‾\underline{34.15}
MAGI-1Middle29.4930.6531.1032.4632.25‾\underline{32.25}
MAGI-1Higher25.6827.0627.7928.82‾\underline{28.82}29.35

Similar to the above, Rank 128 improves mean PSNR over ranks 16, 32 and 64 in every content group on both models. Rank 256 further improves the higher-change group, while rank 128 performs better in the lower-change group on both models. Thus, we use rank r=128r=128 as a common setting across different types of content.

We also evaluated on prompts from a different dataset, VBench-I2V, and found that the gains are robust to distribution shift:

MethodHY-WorldPlay PSNR ↑HY-WorldPlay SSIM ↑HY-WorldPlay LPIPS ↓MAGI-1 PSNR ↑MAGI-1 SSIM ↑MAGI-1 LPIPS ↓
BF16 (reference)——————
QVG-115.250.350.5626.550.810.22
QVG-415.750.390.5029.110.860.17
QVG-Pro16.20‾\underline{16.20}0.420.4631.910.900.13
CORAL-KV reusable16.160.43‾\underline{0.43}0.44‾\underline{0.44}34.55‾\underline{34.55}0.93‾\underline{0.93}0.11‾\underline{0.11}
CORAL-KV video-adapted16.690.450.4235.870.940.09

No dataset-specific hyperparameter selection is used. These results show that the chosen predictor structure and rank transfer across different types of video content without per-dataset retuning.

Note on block-size selection. Choosing the block size does not require hyperparameter tuning, because it is dependent on the models' architecture. For example, HY-WorldPlay has a KV width of size C=6144C=6144, with 24 heads with 256 channels per head. On the other hand, MAGI has a KV width C=1024C=1024, and eight heads with 128 channels per head. Thus, for key prediction on HY-WorldPlay, our block-diagonal component is block-sparse with 24 unique 256×256256\times256 blocks along the diagonal (instead of requiring a dense 6144×61446144\times6144 matrix), and for MAGI-1, our block-diagonal component has 8 unique 128×128128\times128 blocks along the diagonal (instead of a 1024×10241024\times1024 dense matrix). Thus, the predictor blocks follow the architecture and do not require selecting a new block size for each content category or video length.

We will add the latency breakdown and rank-sensitivity results across video lengths and content groups to the appendix, and clarify the low-rank component’s computational and memory scaling in the manuscript.

Reviewer_2Ab9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CORAL-KV compresses autoregressive video KV caches through cross-layer prediction and low-bit residual coding. Compact block-diagonal-plus-low-rank predictors capture dependencies within and across attention heads, while calibration on reconstructed states matches the decoding process. Predictors can be reused across videos or partially adapted to individual cache blocks. Experiments on HY-WorldPlay and MAGI-1 show improved fidelity–memory trade-offs, supported by predictor and quantizer ablations, temporal diagnostics, and latency measurements.

Critical strengths
  • Clear practical contribution. The compact predictor design addresses the memory and computation costs of dense cross-layer prediction, making an established compression principle more practical for video caches.

  • Strong component validation. Parameter-matched predictor comparisons and controlled quantizer ablations provide convincing evidence for the selected design.

  • Thorough evaluation. Disjoint calibration, selection, and evaluation splits, paired confidence intervals, extended rollouts, and temporal analyses strengthen the results. Memory accounting includes predictors and metadata, while timing includes video-specific fitting.

Critical weaknesses
  • HY-WorldPlay operating-point mismatch. The methods are evaluated with 48 latent frames of retained context, whereas the main-table memory totals represent nine completed blocks covering 72 latent frames. The appendix explains this distinction, but the presentation combines different configurations. I think memory for the actual quality-evaluation configuration should accompany the reported fidelity.

  • Limited distribution-shift evidence. Calibration and evaluation use disjoint prompts from the same MovieGen source under fixed generation configurations. This supports reuse within the tested setting, but leaves robustness to different content distributions or conditioning regimes uncertain. The circumstances requiring recalibration or online adaptation remain insufficiently characterized.

  • Incomplete deployment-memory validation. Reported memory covers persistent cache storage and excludes reconstructed tensors and conversion workspaces. The sixteen-video totals reflect shared-bank accounting rather than measured sixteen-stream execution. Storage savings are therefore better established than reductions in peak GPU memory or increases in serving capacity.

Questions for authors
  • What are the stored-cache totals for the actual 48-latent-frame HY-WorldPlay evaluation configuration, including predictors and any native-precision suffix?
  • Among held-out prompts where reusable CORAL-KV underperforms QVG-Pro, does video-specific adaptation recover the loss? Existing paired outputs could clarify when adaptation is worthwhile.
4 comments

Summary of Response to Reviewer 2Ab9

We thank the reviewer for the careful review and suggestions on fidelity–memory comparisons, transfer beyond the calibration source, and deployment memory. We address these concerns with additional memory accounting, evaluations, and measured peak GPU memory. Our responses are organized as follows:

  • [Answer 1] We provide additional cache-storage measurements at the quality-evaluation context and demonstrate a stronger quality–storage trade-off at longer contexts.
  • [Answer 2] CORAL-KV is robust to distribution shift, and outperforms QVG baselines on VBench-I2V data without any additional training.
  • [Answer 3] CORAL-KV reduces peak GPU memory relative to QVG-4 and QVG-Pro across several context lengths.
  • [Answer 4] Video-specific adaptation further improves CORAL-KV’s fidelity, including on prompts where reusable CORAL-KV trails QVG-Pro.

We hope these results clarify the scope of our memory and transfer claims and address the reviewer's questions.

[Answer 1] Additional storage measurements and a stronger quality–memory trade-off at longer contexts

To complement the analysis of memory storage, we report cache memory at the 48-latent-frame context used for quality evaluation. This additional comparison pairs fidelity and storage under the same configuration, including predictors or centroids and the BF16 suffix.

MethodPSNR (dB) ↑Stored cache: GB ↓ / compression ↑
BF16Reference31.14 / 1.00×
QVG-115.708.88 / 3.51×
QVG-416.669.60 / 3.25×
QVG-Pro17.1610.20 / 3.05×
AQUA-KV17.1215.14 / 2.06×
CORAL-KV reusable16.989.24 / 3.37×
CORAL-KV video-adapted17.389.71 / 3.21×

At this context length, reusable CORAL-KV still improves PSNR over QVG-4 while using 3.67% less storage, and video-adapted CORAL-KV improves over QVG-Pro while using 4.79% less storage. This configuration retains one compressed block of 40 latent frames and the latest eight latent frames in BF16.

Building on our method's quality gains at 48 retained latent frames, we extend the evaluation to long-context generation, testing N=72N=72 and 256256 retained latent frames. In this experiment, we compress every eight latent frames and retain the last NN latent frames as context. All retained frames are compressed, including the most recent frames that would otherwise remain in BF16. These settings generate 301 and 1,037 decoded frames per video, respectively. We report video PSNR and cache storage for each context length below.

Method72 frames: Video PSNR ↑72 frames: GB ↓ / compression ↑256 frames: Video PSNR ↑256 frames: GB ↓ / compression ↑
BF16Reference46.71 / 1.00×Reference166.09 / 1.00×
QVG-115.777.99 / 5.84×16.4928.42 / 5.84×
QVG-416.3013.36 / 3.50×17.9147.52 / 3.50×
QVG-Pro16.5714.46 / 3.23×18.3751.41 / 3.23×
CORAL-KV reusable16.306.90 / 6.77×17.6923.26 / 7.14×
CORAL-KV video-adapted16.749.74 / 4.80×18.6833.34 / 4.98×

In this setting, reusable CORAL-KV uses less stored-cache memory than all three QVG variants, because QVG stores separate centroids for each compressed block, while reusable CORAL-KV shares the same predictors across all blocks. From 72 to 256 retained latent frames, reusable CORAL-KV’s video PSNR gain over QVG-1 increases from 0.53 to 1.19 dB, while its storage savings increase from 13.70% to 18.20%. Video-adapted CORAL-KV’s gain over QVG-Pro increases from 0.17 to 0.31 dB, with storage savings increasing from 32.70% to 35.20%. The adapted configuration achieves the highest mean video PSNR at both context lengths. These results suggest that CORAL-KV benefits from longer video context, achieving higher fidelity and greater relative cache-storage savings as the retained context increases.

[Answer 2] CORAL-KV is robust to distribution shift, and outperforms QVG baselines on VBench-I2V

Using the same predictors trained on MovieGen prompts, we evaluated our method on VBench-I2V VBench-I2V (Huang et al., VBench++) inputs spanning 11 content categories, including people, animals, architecture, scenery, and transportation. HY-WorldPlay uses the conditioning images and captions, while MAGI-1 uses the captions. Importantly, we reuse the same predictors that were learned on the MovieGen dataset, and only our video-adapted variant requires occasionally refitting the block-diagonal predictors to the new inputs. We report the evaluations on 32 inputs below:

MethodHY-WorldPlay PSNR ↑HY-WorldPlay SSIM ↑HY-WorldPlay LPIPS ↓MAGI-1 PSNR ↑MAGI-1 SSIM ↑MAGI-1 LPIPS ↓
BF16 (reference)——————
QVG-115.250.350.5626.550.810.22
QVG-415.750.390.5029.110.860.17
QVG-Pro16.20‾\underline{16.20}0.420.4631.910.900.13
CORAL-KV reusable16.160.43‾\underline{0.43}0.44‾\underline{0.44}34.55‾\underline{34.55}0.93‾\underline{0.93}0.11‾\underline{0.11}
CORAL-KV video-adapted16.690.450.4235.870.940.09

These results show that offline calibration learns transferable cross-layer relationships in the models’ KV activations, and can generalize to new input distributions without any retraining. Reusable CORAL-KV outperforms QVG-1 and QVG-4 on all three metrics for both models without any online adaptation, and also outperforms QVG-Pro across all three metrics on MAGI-1. Video adaptation further improves all three quality metrics on both models.

[Answer 3] CORAL-KV reduces peak GPU memory relative to QVG-4 and QVG-Pro across several context lengths

To address the distinction between persistent storage and deployment memory, we measured peak GPU allocation using torch.cuda.max_memory_allocated(). This includes model weights, activations, reconstructed KV tensors, and temporary buffers for compression, prediction and online adaptation. The table shows the maximum across 32 prompts for each method, across 48, 72, and 256 retained latent frames.

Method48-frame context: peak GPU memory (GiB) ↓72-frame context: peak GPU memory (GiB) ↓256-frame context: peak GPU memory (GiB) ↓
BF1650.5767.39203.07
QVG-129.6635.3378.83
QVG-432.4439.7796.06
QVG-Pro33.0140.6899.58
CORAL-KV reusable30.79‾\underline{30.79}36.47‾\underline{36.47}79.90‾\underline{79.90}
CORAL-KV video-adapted32.3438.9489.58

We observe that peak memory generally increases with the number of frames in context as expected. Across the board, reusable CORAL-KV uses only 1-4% more peak GPU memory than QVG-1. CORAL-KV still uses less peak GPU memory than QVG-4 and QVG-Pro at all three context lengths, and our reusable CORAL-KV method reduces peak memory at 256-frame context by 19.80% relative to QVG-Pro. One explanation for the slightly higher peak memory relative to QVG-1 is that CORAL-KV retains the preceding layer’s reconstructed KV while predicting the current layer, alongside prediction and residual buffers.

Remark about concurrent video generation. In Table 1 of the manuscript, we report total memory of sixteen videos by counting video’s KV data, metadata and video-specific state separately, while reusable predictor weights are counted once. These results demonstrate CORAL-KV’s better storage scaling across videos, because the same predictors can be reused across all sixteen videos without duplicating their weights. These storage savings, together with the peak-memory measurements above, suggest a feasible path toward memory-efficient concurrent serving.

[Answer 4] Video-specific adaptation further improves CORAL-KV’s fidelity

We analyzed the existing outputs for all 355 videos generated by HY-WorldPlay. On the 198 prompts where reusable CORAL-KV trails QVG-Pro, the video-adapted configuration improves PSNR by 0.66 dB on average (95% CI: [0.53, 0.79]), closing 70.40% of the mean deficit to QVG-Pro. It reaches or exceeds QVG-Pro's PSNR on 82/198 prompts. This analysis supports using reusable predictors without having to calibrate each new dataset. Video-specific adaptation is useful when prioritizing additional fidelity.

We will include the 48-frame cache-storage measurements alongside the corresponding quality results in the revised manuscript, and add the long-context, transfer, peak-memory, and adaptation analyses to the appendix.

Dear Reviewer,

We thank you for taking the time to read our paper and offer valuable feedback. We have addressed each of your three main concerns:

  • We report the cache memory total at the 48-latent frame configuration. Video-adapted CORAL-KV still beats QVG-Pro on PSNR while requiring less storage (Answer 1). We also extend our analysis to 72 and 256 latent frames, where we find and even stronger quality-memory advantage because of the reuse of predictors.
  • Distribution shift: On VBench-I2V prompts, reusable CORAL-KV outperforms QVG-1 and QVG-Pro, and video-adapted CORAL-KV has the best quality metrics compared to all the baselines (Answer 2).
  • We measured peak GPU allocation, and find that CORAL-KV uses less peak memory than QVG-4 and QVG-Pro at all tested context lengths.

We additionally conduct analyses to show that video adaptation recovers most of the quality loss on prompts where reusable CORAL-KV underperforms.

As the discussion period closes soon, we kindly ask to review our response and let us know if you have further questions!

Thank you, the authors

Thank you for the reply. I have no further questions and will maintain the positive score.

Comments on the paper

1 comment

We thank the reviewers for the constructive discussion and their positive reassessments. Reviewers have praised the clear practical contribution of compact cross-layer predictors, and strong component validation with thorough evaluation. They have also recognized the unique perspective of exploiting cross-layer dependencies, the structured design of the predictor components, and consistent memory and fidelity improvements across the models. Finally, they have highlighted that CORAL-KV efficiently exploits cross-layer redundancy with limited overhead and achieve better memory-quality tradeoffs.

We have incorporated their feedback to strengthen both the evaluation and presentation of CORAL-KV. We have included additional experiments to strengthen this contribution in three ways:

  1. Generalization across four model families. We have extended our evaluation to Self-Forcing 1.3B and LongCat-Video 13.6B. At the longer contexts, video-adapted CORAL-KV improves PSNR over QVG-Pro by approximately 0.95 and 0.94dB, respectively, while using 16.8% and 29.9% less stored cache memory. We ran additional evaluations to show that the same predictor structure has consistent gains across all four model families.
  2. Stronger fidelity-memory tradeoffs at longer contexts. We show that CORAL-KV enjoys a better quality-memory tradeoff for long-context generation. While keeping 256 latent frames retained as context on HY-Worldplay, video-adapted CORAL-KV achieves the highest mean PSNR with 35.2% less stored-cache memory than QVG-Pro. Reusable CORAL-KV improves video PSNR over QVG-1 by 1.19 dB with 18.2% less memory than QVG-1, and uses 19.8% less peak GPU memory than QVG-Pro, while maintaining comparable quality.
  3. CORAL-KV is robust to distribution shift and offers flexible adaptation costs. We find that the same KV-cache predictors calibrated on MovieGen prompts also generalize to VBench-I2V prompts without recalibration. We also show that video-adapted CORAL-KV can smoothly tradeoff its online adaptation cost by fitting fewer tokens. At the same token budget, CORAL-KV achieves higher mean PSNR than all three QVG variants on HY-Worldplay and MAGI-1.

We have incorporated these results and clarifications into the revised manuscript, with changes highlighted in blue to facilitate review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.