acceptodds
Under review as a conference paper at ICLR 2027

TempoKV: Predictive KV Cache Quantization For Autoregressive Video Generation

13 views

Abstract

Autoregressive (AR) video diffusion models enable streaming generation of long videos but suffer from significant memory overhead due to the rapidly growing KV cache. Existing KV cache quantization methods are primarily designed for LLMs and transfer poorly to video models, where activations exhibit different statistics and strong spatiotemporal redundancy. We observe that the video KV cache is largely predictable over time, with a clear asymmetry. Keys are highly predictable from their temporal history, while values depend primarily on the current key and need a short history of preceding values. Based on these observations, we propose TempoKV, a training-free predictive KV cache quantization framework for AR video generation. TempoKV predicts keys and values according to their distinct dependencies and stores only the residuals at low precision. Both predictors are lightweight, calibrated once offline by linear regression on a small set of BF16 samples, and remain fixed across prompts and generation runs. Experiments on Self-Forcing, Causal Forcing, and LongCat-Video-13B show that TempoKV consistently achieves higher generation quality than existing KV cache quantization methods at comparable or higher compression ratios, reaching up to KV cache compression with the best overall quality–efficiency tradeoff.

Primary area:
generative models
Keywords:
KV cache quantization, autoregressive video generation, video diffusion models, long video generation, memory-efficient inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cCapOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TempoKV, a predictive KV-cache quantization method for autoregressive video generation. The main observation is that K and V have different predictive dependencies: preceding keys predict the current key well, while value prediction benefits much more from the key at the same latent frame and spatial position. TempoKV therefore reconstructs K first, then uses it together with a short V history to predict V. It stores low-bit prediction residuals, supports mixed K/V precision, and fits the predictors offline without retraining the video model. Experiments on Self-Forcing, Causal Forcing, and LongCat-Video-13B report improved fidelity to BF16 outputs and a better overall quality–compression trade-off than the tested quantization baselines.

Critical strengths
  1. The difference in K/V predictability is a useful observation. I found the results in Figure 3 convincing. Key history is already quite predictive, whereas adding the aligned current key improves value prediction much more than the reverse direction improves key prediction. This gives a clear reason to reconstruct K before V, rather than treating both caches in the same way. The layer-wise and temporal analyses, along with the LongCat results in Appendix A.1, suggest that the observation is not limited to one layer or one backbone.

  2. The reported gains hold across the three tested models. The fidelity improvements are substantial in several settings. For example, on Self-Forcing (Table 2), k4v2 improves PSNR from 18.92 to 23.17 and SSIM from 0.73 to 0.85 relative to QVG-Pro, with reported compression ratios of 4.73× and 4.65×, respectively. Similar trends are reported on Causal Forcing and LongCat. The mixed-bit and history-length ablations also help explain why giving K more precision and keeping V history short are sensible choices.

Critical weaknesses

The paper makes a good case for the K-to-V prediction direction and the different history lengths. What is less clear to me is why the temporal terms use only per-channel coefficients, while the K-to-V term uses a full dense matrix. The current ablations show that temporal and cross prediction are useful, but they do not compare these parameterizations with alternatives. A dense temporal map could capture cross-channel temporal dependencies, while a low-rank K-to-V map might retain much of the predictive benefit at lower cost. I do not think an exhaustive architecture search is needed, but a small comparison of reconstruction error, parameter storage, and decoding overhead would make the trade-off easier to judge. This is mainly a gap in the design justification, rather than evidence that the current predictors are ineffective.

Questions for authors
  1. Predictor design. Were per-channel temporal prediction and the dense K-to-V projection compared with alternatives, such as dense temporal or low-rank cross projections? If so, could you share their reconstruction error, parameter storage, and decoding overhead under comparable storage budgets? I would like to understand whether the current choices are driven mainly by prediction accuracy, runtime cost, or both.

  2. Calibration loss. Section 3.3 and Appendix B.1 use reconstructed inputs, making the fitting reconstruction-aware. My concern is the objective: Equation (5) minimizes regularized prediction error, whereas residual quantization distortion is ∑t,s∥Rt,sX−DbX(QbX(Rt,sX))∥22\sum_{t,s}\left\lVert R^X_{t,s}-D_{b_X}\left(Q_{b_X}(R^X_{t,s})\right)\right\rVert_2^2, where Rt,sX=Xt,s−fX(Ct,sX;θX)R^X_{t,s}=X_{t,s}-f_X(C^X_{t,s};\theta_X), bXb_X is the residual bit-width, and the sum is over non-anchor frames. These objectives are not equivalent: lower residual L2 energy does not necessarily imply a smaller per-group range or lower low-bit quantization error. Why use Equation (5) as a surrogate rather than directly optimizing residual quantization distortion? A brief justification, or a comparison at matched bit-widths, would help clarify the calibration strategy.

  3. Difficult motion cases. The 1–32 prompt sweep and action/scene cross-calibration tests in Appendix A.5 are helpful. I am still unsure how the fixed predictors behave on fast object motion, large camera movements, occlusion, or abrupt scene changes, where content at the same spatial position can change quickly. Could you provide a breakdown by motion intensity, or discuss representative difficult cases from the existing evaluation, with cache reconstruction error and final video quality? My concern is whether the average scores hide a substantial drop on these cases, not the small calibration set by itself. A broader motion-focused benchmark would be useful future work, but I am not asking for it as a condition for acceptance.

7 comments
AuthorsOpenReview ↗

Response to Q1 & Weakness

To Reviewer cCap

We sincerely thank the reviewer for the constructive comments and recognition of the significance of our asymmetric K/V predictability analysis. We appreciate the concerns regarding predictor design and calibration goals. We apologize for not presenting the relevant analysis and design considerations in the initial draft. In fact, we explored various predictor configurations during the method development phase, but due to space limitations, we were unable to include these results. We are pleased to have the opportunity to explain these choices and supplement the corresponding experimental evidence here.

Q1&Weakness: Predictor design

To validate the design of our predictor architecture, we conducted two additional ablation experiments on Self-Forcing : (1) channel-wise temporal prediction versus dense temporal prediction; and (2) dense K→V prediction versus low-rank K→V prediction, evaluating reconstruction errors, predictor storage, and decoding latency.

(1) Why channel-wise temporal prediction?

We replace our channel-wise temporal predictor with a full per-head dense temporal predictor. Increasing the predictor capacity does not improve reconstruction fidelity. The channel-wise predictor achieves a K NMSE of 0.000902, compared with 0.002333 for the dense predictor, while requiring only 1.05 MiB instead of 90.35 MiB of predictor storage (85.7× smaller). It also reduces KV decoding time from 3511.7 ms to 3116.2 ms. Quantizing the dense predictor weights to INT4 reduces its storage to 11.95 MiB but further increases decoding time to 4342.8 ms, without improving reconstruction accuracy.

PredictorPredictor storage/ MiBK NMSE↓decode overhead ms↓
Ours1.05470.0009020173116.2
dense_fp3290.35160.002333333511.7
dense_fp1645.35160.002333353605
dense_int823.20310.00233363770
dense_int411.95310.002351874342.8

These results indicate that, under our evaluated setting, the additional cross-channel capacity of dense temporal prediction is unnecessary for accurate K reconstruction and introduces substantial storage and computational overhead. This supports our choice of a lightweight channel-wise temporal predictor.

(2) Why dense K→V prediction?

We further investigate low-rank approximations of the dense K→V projection, including truncated SVD and low-rank refitting. Although low-rank factorization reduces predictor storage, it consistently increases V reconstruction error. For example, rank-16 refitting reduces storage from 22.5 MiB to 5.63 MiB but increases V NMSE from 0.002737 to 0.003750 (+37.0%). Even at rank 32, which halves the storage, V NMSE remains 17.4% higher than the dense baseline. Increasing the rank to 64 requires the same storage as the original dense projection, while still producing slightly higher reconstruction error.

PredictorPredictor storage/ MiBV NMSE↓decode overhead/ms↓
Ours22.50.002736693114.8
svd_r82.81250.01596563150.5
svd_r165.6250.01160023210.1
svd_r3211.250.006970473156.1
svd_r4816.8750.005445483112.1
svd_r6422.50.004041543102.1
low_rank_refit_r82.81250.004311333074.4
low_rank_refit_r165.6250.003749783090.2
low_rank_refit_r3211.250.003214253128.2
low_rank_refit_r4816.8750.002961123144.9
low_rank_refit_r6422.50.002832753243.1

In summary, these experimental findings support our chosen asymmetric predictor design. Since our dense K→V projection operates independently within each attention head rather than across the full hidden dimension, its storage cost remains manageable (22.5 MiB). We will include these architectural ablation studies in the revised manuscript, along with measurements of storage and latency.

AuthorsOpenReview ↗

Response to Q2

Q2: On the calibration objective

We agree with the reviewer that minimizing prediction L2 error and minimizing post-quantization distortion are not strictly equivalent. Nevertheless, we adopt the L2 objective in Eq. (5) as a principled surrogate for two reasons: it controls an upper bound on the quantization distortion, and it keeps calibration a closed-form linear problem. We elaborate on both points below.

First, prediction MSE and quantization distortion are linked by an error bound. Consider one quantization group of GG residuals r∈RGr\in\mathbb{R}^G, and let

M=\max(0,\max_i r_i),\quad \Delta=\frac{M-m}{2^b-1}.$$ Asymmetric $b$-bit quantization places $2^b$ uniformly spaced levels on $[m,M]$ with step $\Delta$. Since every $r_i\in[m,M]$, nearest rounding gives $$|r_i-[Q_b(r)]_i| \leq \frac{\Delta}{2}, \qquad i=1,\dots,G.$$ Because $m\le 0\le M$, we have $(M-m)^2\leq 2(M^2+m^2)$. Moreover, $M^2$ and $m^2$ are each either $0$ or the square of a distinct entry of $r$, so $M^2+m^2\leq\|r\|_2^2$. Summing the per-element bound over the group, $$\|r-Q_b(r)\|_2^2 \leq \frac{G\Delta^2}{4} = \frac{G\,(M-m)^2}{4(2^b-1)^2} \leq \frac{G}{2(2^b-1)^2}\|r\|_2^2 .$$ Summing over all groups, the total quantization distortion is bounded by $\frac{G}{2(2^b-1)^2}$ times the total residual energy, which is exactly the data term of Eq. (5). Thus, for fixed $G$ and $b$, **minimizing the residual L2 loss in Eq. (5) also minimizes an upper bound on quantization distortion.** **Second, with reconstructed inputs fixed, our linear predictors make Eq. (5) a regularized least-squares problem solvable through a linear system.** Directly optimizing quantization distortion couples the quantization codes and group-wise scales to predictor parameters, yielding a non-smooth objective to which the same linear-system solution no longer applies and making the optimization more difficult. We compare Eq. (5) (prediction L2) with direct quantization-aware optimization (QAO) at INT2 and INT4 below. | Precision | Calibration objective | Quantization Error ↓ (K) | Quantization Error ↓ (V) | Calibration Time ↓ | | --------- | --------------------- | ------------------------ | ------------------------ | ------------------ | | INT2 | Eq. (5) | 0.089359248 | 0.092160777 | **5.41 s** | | | QAO | **0.089342692** | **0.091883979** | 93.82 s | | INT4 | Eq. (5) | 0.003352054 | 0.003428926 | **5.07 s** | | | QAO | **0.003351606** | **0.003419001** | 93.83 s | QAO reduces quantization error by only **0.01–0.30%** while requiring **17–19× the calibration time**, supporting Eq. (5) as an efficient and effective calibration objective in these settings.
AuthorsOpenReview ↗

Response to Q3

Q3: Difficult motions case

Evaluation protocol.

We thank the reviewer for suggesting a more detailed evaluation under challenging motion conditions. We quantify motion intensity using the average optical flow magnitude computed from BF16 reference videos:

\sum_{t=1}^{T-1}\sum_{x,y}|F_t(x,y)|_2,$$ where $F_t(x,y)$ denotes the optical flow vector between consecutive frames at spatial position $(x,y)$. We rank 128 MovieGen videos by their motion intensity and divide them into three approximately equal-sized groups: **low, medium, and high motion**. We also manually inspect the high-motion subset to ensure coverage of challenging scenarios, including fast object motion, large camera movements, occlusions, and abrupt scene changes. ### **Motion-stratified evaluation.** We compare TempoKV (k5v3) against QVG INT4 across different motion levels, measuring reconstruction fidelity relative to BF16 reference videos. **Table 1. Reconstruction fidelity across different motion intensities.** | Motion Level | Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | | ------------ | ------------ | --------------- | --------- | ---------- | | High | QVG INT4 | 18.56 | 0.760 | 0.1396 | | | TempoKV k5v3 | **20.68** | **0.796** | **0.1151** | | Medium | QVG INT4 | 23.29 | 0.855 | 0.0861 | | | TempoKV k5v3 | **25.24** | **0.889** | **0.0563** | | Low | QVG INT4 | 29.97 | **0.944** | 0.0179 | | | TempoKV k5v3 | **30.01** | 0.943 | **0.0168** | As motion intensity increases, reconstruction becomes more challenging for both methods. However, TempoKV maintains a clear advantage in the more difficult settings. On the high-motion subset, it improves PSNR by **2.12 dB** and SSIM by **0.036**, while reducing LPIPS by **0.0245** compared with QVG. Similar gains are observed for medium-motion videos, with a **1.95 dB** PSNR improvement and a **0.0298** reduction in LPIPS. In contrast, both methods achieve comparable fidelity on low-motion videos. **Table 2. Reconstruction error of TempoKV across different motion intensities.** | Motion Level | K NMSE ↓ | V NMSE ↓ | |---|---|---| | Low | 2.36e-4 ± 0.46e-4 | 1.40e-2 ± 0.15e-2 | | Medium | 2.88e-4 ± 0.57e-4 | 1.55e-2 ± 0.13e-2 | | High | 3.62e-4 ± 0.84e-4 | 1.76e-2 ± 0.18e-2 | **Cache reconstruction error.** We measure the K/V cache reconstruction error of TempoKV on each motion level. Error increases gradually with motion intensity (K NMSE 2.36e-4 → 3.62e-4, V NMSE 1.40e-2 → 1.76e-2 from low to high), with small per-prompt variance and no collapse on high-motion cases. K error is consistently lower than V error because we use the **K5V3** configuration, which allocates more bits to keys (5-bit) than to values (3-bit). ### **Representative challenging cases.** To further examine performance at the individual-video level, we report four representative prompts covering different motion intensities and scene dynamics. **Table 3. Per-prompt reconstruction fidelity across representative video scenarios.** | Prompt | Optical Flow | QVG INT4 | TempoKV k5v3 | | ------------------------------------------------------------ | ------------ | ------------------- | ----------------------- | | Historical footage of California during the gold rush. | 1.39 | 17.89 / .776 / .147 | **24.26 / .898 / .056** | | Step-printing scene of a person running, cinematic film shot in 35mm. | 3.31 | 20.54 / .745 / .133 | **23.00 / .833 / .091** | | Reflections in the window of a train traveling through the Tokyo suburbs. | 3.68 | 23.31 / .864 / .072 | **25.11 / .873 / .056** | | FPV flying through colorful coral-lined streets of an underwater suburban neighborhood. | 5.77 | 13.49 / .567 / .437 | **16.79 / .630 / .349** | TempoKV consistently outperforms QVG across all three reconstruction metrics in these examples. In the gold-rush scene, it improves PSNR by **6.37 dB** and reduces LPIPS from 0.147 to 0.056. For the FPV scene, which exhibits the highest optical flow magnitude among these examples, TempoKV improves PSNR by **3.30 dB** and reduces LPIPS from 0.437 to 0.349. Together, the motion-stratified results and representative cases demonstrate that TempoKV remains effective under challenging video dynamics. Its improvements are particularly pronounced in medium- and high-motion subsets, supporting the robustness of TempoKV.
Reviewer_QfdfOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

TempoKV compresses the KV cache of autoregressive video diffusion models through asymmetric predictive quantization. Keys are predicted from reconstructed temporal key history, while values are predicted from the aligned reconstructed key and temporal value history. Only quantized residuals are stored, with higher precision for the initial anchor. The predictors are fitted by offline regression on eight calibration prompts without retraining the generator. Experiments on Self-Forcing, Causal Forcing, and LongCat-Video-13B examine compression, fidelity to BF16 outputs, perceptual quality, and long-horizon generation.

Critical strengths
  1. The method follows a well-supported structural observation. The paper measures temporal and cross-KV predictability, then uses the stronger key-to-value dependency in the reconstruction path. Predictor-direction, temporal-history, precision-allocation, and anchor-precision ablations support the design. Calibration on reconstructed inputs also addresses the mismatch between offline fitting and recursive inference.

  2. The compression-quality tradeoff is compelling and reasonably documented. On Self-Forcing, k5v3 achieves 3.65x compression versus QVG's 3.49x, while improving PSNR from 18.86 to 24.17 and LPIPS from 0.15 to 0.06 (Table 2). The reported INT4 peak memory falls from 68.27 GB to 35.59 GB, with 11.9% generation-time overhead over BF16. Results across three evaluated models, 720p experiments, and longer rollouts broaden the evidence. Storage accounting includes predictor parameters and quantization metadata, and the appendix tests calibration quantity and distribution.

Critical weaknesses
  1. Figure 6 already evaluates generation up to approximately 90 seconds, and Figures 3(c)/8(c) support stable feature predictability. A finer breakdown within each rollout would still be useful: the aggregate PSNR/SSIM results do not show whether compression error grows over time or varies substantially across prompts.

  2. The motion metrics and calibration analysis in Appendix A.5 provide related evidence, but do not directly quantify compression error versus motion magnitude. This analysis would be useful for a predictor that relies on temporal dependencies at fixed spatial positions.

Questions for authors
  1. Could you report PSNR/LPIPS against BF16 in successive temporal windows, including per-prompt variability, using the existing matched 90-second rollouts? Keep prompts, seeds, and attention windows fixed to separate compression error from the base model's own long-horizon degradation.

  2. Could you group the existing videos by motion magnitude measured on their BF16 references and report compression fidelity and quality differences at matched durations? Separating object and camera motion where practical would help interpret the results.

5 comments
AuthorsOpenReview ↗

Response to Q1 & W1

To Reviewer Qfdf:

We sincerely thank you for your time, valuable feedback, and constructive comments. We deeply appreciate your careful reading of our submission, and we apologize for any parts of the manuscript that were insufficiently clear. Below we provide detailed clarifications and additional results.

Q1: Temporal reconstruction fidelity and per-prompt variability

Thanks for raising this. We extend our original 1401-frame (90-second) TempoKV(k5v3) evaluation by measuring PSNR, SSIM, and LPIPS against the BF16 reference within successive temporal windows. We keep the prompts, random seeds, and attention windows fixed across methods to isolate compression-induced reconstruction errors from the underlying model's long-horizon degradation. We additionally report the 10th–90th percentile (P10–P90) ranges across prompts to characterize variations in reconstruction fidelity.

Table 1. Temporal reconstruction fidelity and per-prompt variability over 1401-frame rollouts. P10–P90 denotes the variation across prompts within each temporal window.

WindowPSNR ↑PSNR [P10, P90]SSIM ↑LPIPS ↓LPIPS [P10, P90]
0–15 s25.44≈[23.4, 32.7]0.9070.042≈[0.012, 0.077]
15–30 s22.61≈[20.1, 29.3]0.8310.088≈[0.029, 0.158]
30–45 s23.77≈[21.0, 30.1]0.8530.074≈[0.025, 0.133]
45–60 s23.02≈[20.3, 29.5]0.8460.077≈[0.027, 0.133]
60–75 s22.09≈[19.2, 28.8]0.8460.084≈[0.030, 0.142]
75–90 s19.80≈[16.6, 26.4]0.8550.089≈[0.032, 0.153]

The results show that reconstruction error increases over extended rollouts and varies across prompts, as expected in long-horizon autoregressive generation. However, the degradation remains relatively moderate in perceptual metrics: even in the final 75–90 s window, TempoKV maintains an SSIM of 0.855 and an LPIPS of 0.089. Importantly, these findings complement our original long-horizon comparisons, where TempoKV consistently outperforms QVG in Imaging Quality at 700, 1050, and 1400 frames. Together, these results demonstrate that, despite temporal error accumulation and prompt-dependent variability, TempoKV maintains its fidelity advantage over competing quantization methods throughout extended autoregressive rollouts.

AuthorsOpenReview ↗

Response to Q2 & W2

Q2: Reconstruction fidelity under different motion intensities

We appreciate the reviewer's suggestion to investigate whether motion magnitude affects the effectiveness of temporal predictive quantization. We quantify motion intensity using the average optical flow magnitude computed from BF16 reference videos:

\sum_{t=1}^{T-1}\sum_{x,y}|F_t(x,y)|_2,$$ where $F_t(x,y)$ denotes the optical flow vector between consecutive frames at spatial position $(x,y)$. We rank **128 MovieGen videos** by their motion intensity and divide them into three approximately three groups: **low, medium, and high motion**. We also manually inspect the high-motion subset to ensure coverage of challenging scenarios, including fast object motion, large camera movements, occlusions, and abrupt scene changes. **Table 2. Reconstruction fidelity of QVG INT4 and TempoKV k5v3 across different motion intensities.** | Motion Level | Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | | ------------ | ------------ | --------- | --------- | ---------- | | High | QVG INT4 | 18.56 | 0.760 | 0.1396 | | | TempoKV k5v3 | **20.68** | **0.796** | **0.1151** | | Medium | QVG INT4 | 23.29 | 0.855 | 0.0861 | | | TempoKV k5v3 | **25.24** | **0.889** | **0.0563** | | Low | QVG INT4 | 29.97 | **0.944** | 0.0179 | | | TempoKV k5v3 | **30.01** | 0.943 | **0.0168** | As motion intensity increases, reconstruction fidelity decreases for both methods, confirming that motion-intensive videos are more challenging to compress. Importantly, TempoKV exhibits larger improvements in these difficult settings. On the high-motion subset, TempoKV improves PSNR by **2.12 dB** and SSIM by **0.036**, while reducing LPIPS from 0.1396 to **0.1151**. Similar gains are observed for medium-motion videos, with a **1.95 dB** PSNR improvement and an LPIPS reduction from 0.0861 to **0.0563**. In contrast, both methods achieve comparable fidelity on low-motion videos. **Representative challenging cases.** We further examine individual videos with different motion patterns, including object motion and camera movement, as illustrated below. **Table 3. Per-prompt reconstruction fidelity across representative video scenarios.** | Prompt | Optical Flow | QVG INT4 | TempoKV k5v3 | | ------------------------------------------------------------ | ------------ | ------------------- | ----------------------- | | Historical footage of California during the gold rush. | 1.39 | 17.89 / .776 / .147 | **24.26 / .898 / .056** | | Step-printing scene of a person running, cinematic film shot in 35mm. | 3.31 | 20.54 / .745 / .133 | **23.00 / .833 / .091** | | Reflections in the window of a train traveling through the Tokyo suburbs. | 3.68 | 23.31 / .864 / .072 | **25.11 / .873 / .056** | | FPV flying through colorful coral-lined streets of an underwater suburban neighborhood. | 5.77 | 13.49 / .567 / .437 | **16.79 / .630 / .349** | TempoKV consistently outperforms QVG across all three fidelity metrics in these examples. For the running-person scene, TempoKV improves PSNR by **2.46 dB** and SSIM by **0.088**. In the FPV scene, which features substantial camera movement and the highest optical flow magnitude among these examples, TempoKV achieves a **3.30 dB** PSNR improvement and reduces LPIPS from 0.437 to **0.349**. **Camera vs. object motion.** To further distinguish the effects of camera and object motion, we decompose the optical flow of BF16 reference videos into two components. Specifically, we estimate global camera motion by fitting an affine transformation to RAFT optical flow using RANSAC (RANdom SAmple Consensus), and use the residual flow magnitude as a proxy for object motion. We divide videos into four groups based on the median camera and object motion magnitudes. The two components are correlated (Spearman $\rho=0.73$). | Camera Motion | Object Motion | PSNR ↑ | SSIM ↑ | LPIPS ↓ | | ------------- | ------------- | --------- | ---------- | ---------- | | Low | Low | **31.50** | **0.9563** | **0.0140** | | Low | High | 25.71 | 0.9209 | 0.0350 | | High | Low | 24.99 | 0.8645 | 0.0630 | | High | High | 20.62 | 0.7828 | 0.1244 | These findings suggest that TempoKV remains applicable across different motion patterns, although reconstruction becomes more difficult as motion complexity increases.

Thanks for the additional experiments. The duration and motion analyses largely address my questions, and I will keep my score at 3 (weak accept). I share B5T8's concern about novelty. I do see some novelty in the video-specific adaptation, but I find the overall advance incremental and the new insights somewhat limited. This is my main reason for not giving a higher score.

Reviewer_B5T8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies KV cache compression in autoregressive video generation. The proposed TempoKV first predicts and reconstructs K, then predicts the current V from the reconstructed K and the historical V, storing only low-bit prediction residuals. The predictor is fitted offline on a small number of samples and fixed at inference time, and a Triton kernel performs online cache reconstruction.

Critical strengths

1.The authors provide a useful analysis of video KV caches, covering temporal prediction, inter-KV interaction analysis, and bit-width allocation.

2.The experiments are thorough. The authors report video quality metrics, peak memory, and end-to-end latency, and the evaluation is broad. These results support that TempoKV improves the compression–fidelity trade-off under the evaluated settings.

Critical weaknesses

1.The paper lacks a comparison with the most relevant predictive quantization work. AQUA-KV [1] already adopts prediction of K and uses the reconstructed K together with the existing V to predict the current V. TempoKV shifts the prediction dependency from between network layers to the video temporal dimension, and adopts a corresponding predictor and runtime implementation. However, the authors do not discuss this directly related work. The analysis of video temporal dependencies is valuable, but it is not yet clear whether the gains come from a new video compression mechanism or from transferring an existing predictive quantization framework to the temporal dimension.

[1] Alina Shutova, Vladimir Malinovskii, Vage Egiazarian, Denis Kuznedelev, Denis Mazur, Surkov Nikita, Ivan Ermakov, and Dan Alistarh. Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models. In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025.

  1. The proposed predictor itself still requires fitting. How general is it? Can the method as a whole still be called training-free?

  2. The main experiments can support that TempoKV improves the compression–fidelity trade-off. However, the results indicate that the evaluation of the method in long-video settings is insufficient; the authors should clarify this or add more analysis of the experimental results.

Questions for authors

1.What are the main differences between TempoKV and AQUA-KV? What independent advantages does the video temporal dimension bring to TempoKV?

2.What is the specific gain of the temporal predictor over directly using the reconstructed cache of the previous frame?

3.Under the same memory budget, how much more history can TempoKV retain than BF16 and QVG, and does this additional history improve long-horizon generation quality?

9 comments
AuthorsOpenReview ↗

Response to Q1 & W1

To Reviewer B5T8:

Thank you very much for your support and for the thoughtful comments. We truly appreciate the time you spent reviewing our submission and the constructive insights you shared. We apologize for any ambiguities in the manuscript, and we outline additional clarifications and results below.

Q1&W1: Differences from AQUA-KV

Thanks the reviewer for raising this important question. While both AQUA-KV and TempoKV employ predictive residual quantization, they differ fundamentally in the predictive dependencies they utilize and how these dependencies guide compression design. We will discuss the relationship between the two in detail in the Related Work section of the revised manuscript.

PredictorQuantizerCompression Ratio↑PSNR ↑SSIM ↑LPIPS ↓Latency (s) ↓
TempoKVScalar3.65×24.170.8870.061375
AQUA-KV adaptedScalar3.63×20.320.7490.135503
TempoKVHIGGS3.92×23.780.8780.057409
AQUA-KV adaptedHIGGS2.98×20.410.7310.136542

1. Video-Specific K/V Asymmetry

Our analysis in the paper (Fig.3) further demonstrates that K and V exhibit different temporal predictive dependencies. The cosine similarity between historical K values and the current K value is 0.877, while the cosine similarity between historical V values and the current V value is only 0.637. Introducing an aligned current K value improves the predictability of the V value to 0.881. These observations suggest that the K cache depends primarily on its temporal history, while the V cache primarily benefits from the reconstructed current K value. This asymmetry provides the basis for our predictive framework, rather than directly adopting the interlayer formula of AQUA-KV.

2. Predictive Dimensions

AQUA-KV primarily utilizes inter-layer dependencies in LLM to predict KV features from the previous Transformer layer. However, in our experiments, even at low compression ratios, directly applying this inter-layer prediction scheme to AR video leads to a significant reduction in generation fidelity. TempoKV, on the other hand, leverages temporal dependencies between latent frames at corresponding spatial locations to capture the evolution of KV features during autoregressive generation. This demonstrates that the gains of our method come from a new video compression mechanism, rather than transferring an existing predictive quantization framework to the temporal dimension.

3. Asymmetric Quantization Design.

We translate these observations into several specific design choices. K uses a longer temporal history, while V depends on the reconstructed aligned K and a shorter V history. Furthermore, the reconstruction error of K propagates to subsequent V predictions, thus K has a higher precision in our hybrid bit configuration. Combined with high-precision anchor frames , these choices constitute a predictive quantization framework for video. Ablation experiments in Tables 4-5 and Appendix Tables 6-7 validate their effectiveness.

4. Practical Efficiency.

Applying AQUA-KV's full-width dense predictor directly to video models incurs significant parameter storage and computational overhead. The projection parameters are quadratic with the total width of the KV channels, and predictions must be performed across attention layers and spatial positions. In contrast, TempoKV combines lightweight, channel-wise temporal prediction and head-wise dense K-to-V projection, reducing the K-to-V parameter complexity from O(LH2d2)O(LH^2d^2) to O(LHd2)O(LHd^2) while preserving full channel interaction within each head. On the Self-Forcing, our predictor requires only 22.5 MiB of FP32 storage. We also implemented efficient online reconstruction using the Triton kernel.

With scalar quantization, at nearly identical compression ratios (3.65× vs. 3.63×), TempoKV improves PSNR by 3.85 dB and SSIM by 0.138, reduces LPIPS from 0.135 to 0.061, and accelerates generation by 1.34×. With HIGGS, TempoKV achieves an even more substantial compression advantage, increasing the compression ratio from 2.98× to 3.92×(31.5% higher) while improving PSNR by 3.37 dB and SSIM by 0.147, and reducing LPIPS from 0.136 to 0.057. Meanwhile, generation latency decreases from 542 s to 409 s, providing a 1.33× speedup.

In summary, TempoKV goes beyond simply applying predictive quantization to video. It can identify asymmetric temporal dependencies in video KV features and translate them into more effective predictive structures, differentiated quantization strategies, and efficient long-term autoregressive generation.

AuthorsOpenReview ↗

Response to W2

W2. Predictor Generalization and Training-free

Generalization.

Thanks the reviewer for raising this important question. In Appendix A.5 (Fig 9), we have explored the predictor's generalization ability, covering both the amount of calibration data and distribution shifts.

First, Fig. 9(a) evaluates the sensitivity to calibration set size. For K prediction, using only a single calibration prompt achieves a reconstruction error comparable to using 32 prompts. For V prediction, increasing the calibration set from 1 to 8 prompts reduces reconstruction error by 17.6%, while further increasing it from 8 to 32 prompts yields only an additional 2.1% reduction. These results demonstrate that our predictor achieves excellent reconstruction accuracy with only a small amount of calibration data.

Second, Fig. 9(b) examines cross-distribution generalization using disjoint Action and Scene calibration sets. Under both evaluation distributions, the change in K/V reconstruction error due to using a mismatched calibration set is at most 0.77% compared to the matched calibration set; the error is even lower in some cross-distribution settings. This indicates that the fitted predictor does not require the calibration data to be highly consistent with the target content distribution. In the manuscript, Fig3(c) and 8(c) also show that the underlying predictive dependencies remain relatively stable over generation time.

Beyond calibration, our additional motion-stratified evaluation on 128 MovieGen videos demonstrates TempoKV's consistent improvements. We quantify motion intensity using the average optical flow magnitude computed from BF16 reference videos:

\sum_{t=1}^{T-1}\sum_{x,y}|F_t(x,y)|_2,$$ where $F_t(x,y)$ denotes the optical flow vector between consecutive frames at spatial position $(x,y)$. We rank videos by their motion intensity and divide them into three approximately three groups: **low, medium, and high motion**. We also manually inspect the high-motion subset to ensure coverage of challenging scenarios, including fast object motion, large camera movements, occlusions, and abrupt scene changes. | Motion Level | Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | | ------------ | ------------ | --------- | --------- | ---------- | | High | QVG INT4 | 18.56 | 0.760 | 0.1396 | | | TempoKV k5v3 | **20.68** | **0.796** | **0.1151** | | Medium | QVG INT4 | 23.29 | 0.855 | 0.0861 | | | TempoKV k5v3 | **25.24** | **0.889** | **0.0563** | | Low | QVG INT4 | 29.97 | **0.944** | 0.0179 | | | TempoKV k5v3 | **30.01** | 0.943 | **0.0168** | In summary, these results support the generalization ability of our predictor across different content distributions with limited calibration data, and also demonstrate the broad applicability of the framework across different models and video dynamics. ### **Training-free property.** We agree that fitting the predictors requires offline calibration. Our intended meaning is that TempoKV **does not** require retraining or fine-tuning the pretrained video model. Specifically, TempoKV fits lightweight linear predictors through offline regularized least squares using a small number of BF16 KV traces. This process requires no backpropagation through the model or updates to its pretrained parameters, and the fitted predictors remain fixed during inference. We will clarify this distinction in the revised manuscript by explicitly describing TempoKV as **no training required for the model, only lightweight offline predictor calibration**.
AuthorsOpenReview ↗

Response to Q2 & Q3

Q2 Compare with Previous-frame Cache

We thank the reviewer for the valuable suggestions. To clarify the gains provided by the learned temporal predictor, we compared TempoKV against an identity predictor applied to both K and V, which uses each cache’s reconstructed values from the previous latent frame at the same spatial position as its prediction. This baseline does not use the K-to-V projection and stores quantized prediction residuals for both caches.

Quantitative results. Compared to the identity predictor, TempoKV improves PSNR from 21.74 dB to 24.17dB, increases SSIM from 0.812to 0.884, and reduces LPIPS from 0.107 to 0.062.

High-motion scenarios. In videos with significant motion, these improvements are even more pronounced: TempoKV achieves a PSNR gain of +3.89 dB, an SSIM increase of +0.116, and a 52% reduction in LPIPS (from 0.2403 to 0.1151).

These results demonstrate that the performance gains from TempoKV stem from more than just a simple reuse of the reconstructed KV cache from the previous frame. The particularly substantial improvements observed in high-motion scenarios indicate that the "learned temporal predictor" is more effective at capturing feature variations across the temporal dimension—changes that cannot be adequately approximated by a simple "identity-based reuse" strategy.

MethodCompression RatioPSNR↑SSIM↑LPIPS↓
Identity INT43.66x21.740.8120.107
TempoKV k5v33.65×24.170.8840.062
MethodPSNR↑SSIM↑LPIPS↓
high_motionIdentity INT416.790.6790.2403
high_motionTempokv(k5v3)20.680.7960.1151

Q3: Long-horizon generation under a fixed memory budget

We thank the reviewer for raising this important question. To investigate whether KV-cache compression enables longer attention histories and benefits long-horizon generation, we conduct additional experiments on Self-Forcing under a fixed 48 GB GPU memory budget, evaluating both the maximum generation length supported by full-history attention and the resulting generation quality.

Maximum generation length. We first measure the maximum video length each method can generate with full-history attention under the same memory constraint. BF16 supports 132 latent frames (525 RGB frames), while QVG INT4 extends this to 426 latent frames (1701 RGB frames). TempoKV (k5v3) further increases the limit to 450 latent frames (1797 RGB frames), achieving 3.41× the full-history generation length of BF16 and 5.6% more retained history than QVG. These results demonstrate that TempoKV's compression advantage translates into greater usable attention history under a fixed memory budget.

MethodPSNR ↑SSIM ↑LPIPS ↓Subject Consistency ↑Background Consistency ↑Motion Smoothness ↑Dynamic Degree ↑Aesthetic Quality ↑Imaging Quality ↑
BF16 (w132)———97.5997.4099.0711.2557.5859.62
QVG INT4 (w426)17.790.75850.167597.7397.4699.0510.0059.8866.28
TempoKV k5v3 (Full)18.050.76620.162797.7797.6299.0710.7560.2166.32

Long-horizon generation quality. We further evaluate all methods on the same 1797-frame generation task, using their respective memory-constrained attention windows: 132 latent frames for BF16, 426 for QVG INT4, and full-history attention for TempoKV. All configurations remain within the 48 GB memory budget. PSNR, SSIM, and LPIPS are measured against BF16 (w132).

TempoKV achieves the highest Subject Consistency, Background Consistency, Aesthetic Quality, and Imaging Quality, while matching BF16 in Motion Smoothness. Compared with QVG, TempoKV also achieves higher reconstruction fidelity, improving PSNR from 17.79 to 18.05 dB and SSIM from 0.7585 to 0.7662, while reducing LPIPS from 0.1675 to 0.1627.

Discussion. These experiments demonstrate that TempoKV, within the same budget, can retain a more complete history of generation while maintaining better generation performance. Experimental results confirm that TempoKV achieves a superior quality-memory trade-off in long sequence generation scenarios.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.