- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TempoKV, a predictive KV-cache quantization method for autoregressive video generation. The main observation is that K and V have different predictive dependencies: preceding keys predict the current key well, while value prediction benefits much more from the key at the same latent frame and spatial position. TempoKV therefore reconstructs K first, then uses it together with a short V history to predict V. It stores low-bit prediction residuals, supports mixed K/V precision, and fits the predictors offline without retraining the video model. Experiments on Self-Forcing, Causal Forcing, and LongCat-Video-13B report improved fidelity to BF16 outputs and a better overall quality–compression trade-off than the tested quantization baselines.
Critical strengths
-
The difference in K/V predictability is a useful observation. I found the results in Figure 3 convincing. Key history is already quite predictive, whereas adding the aligned current key improves value prediction much more than the reverse direction improves key prediction. This gives a clear reason to reconstruct K before V, rather than treating both caches in the same way. The layer-wise and temporal analyses, along with the LongCat results in Appendix A.1, suggest that the observation is not limited to one layer or one backbone.
-
The reported gains hold across the three tested models. The fidelity improvements are substantial in several settings. For example, on Self-Forcing (Table 2), k4v2 improves PSNR from 18.92 to 23.17 and SSIM from 0.73 to 0.85 relative to QVG-Pro, with reported compression ratios of 4.73× and 4.65×, respectively. Similar trends are reported on Causal Forcing and LongCat. The mixed-bit and history-length ablations also help explain why giving K more precision and keeping V history short are sensible choices.
Critical weaknesses
The paper makes a good case for the K-to-V prediction direction and the different history lengths. What is less clear to me is why the temporal terms use only per-channel coefficients, while the K-to-V term uses a full dense matrix. The current ablations show that temporal and cross prediction are useful, but they do not compare these parameterizations with alternatives. A dense temporal map could capture cross-channel temporal dependencies, while a low-rank K-to-V map might retain much of the predictive benefit at lower cost. I do not think an exhaustive architecture search is needed, but a small comparison of reconstruction error, parameter storage, and decoding overhead would make the trade-off easier to judge. This is mainly a gap in the design justification, rather than evidence that the current predictors are ineffective.
Questions for authors
-
Predictor design. Were per-channel temporal prediction and the dense K-to-V projection compared with alternatives, such as dense temporal or low-rank cross projections? If so, could you share their reconstruction error, parameter storage, and decoding overhead under comparable storage budgets? I would like to understand whether the current choices are driven mainly by prediction accuracy, runtime cost, or both.
-
Calibration loss. Section 3.3 and Appendix B.1 use reconstructed inputs, making the fitting reconstruction-aware. My concern is the objective: Equation (5) minimizes regularized prediction error, whereas residual quantization distortion is , where , is the residual bit-width, and the sum is over non-anchor frames. These objectives are not equivalent: lower residual L2 energy does not necessarily imply a smaller per-group range or lower low-bit quantization error. Why use Equation (5) as a surrogate rather than directly optimizing residual quantization distortion? A brief justification, or a comparison at matched bit-widths, would help clarify the calibration strategy.
-
Difficult motion cases. The 1–32 prompt sweep and action/scene cross-calibration tests in Appendix A.5 are helpful. I am still unsure how the fixed predictors behave on fast object motion, large camera movements, occlusion, or abrupt scene changes, where content at the same spatial position can change quickly. Could you provide a breakdown by motion intensity, or discuss representative difficult cases from the existing evaluation, with cache reconstruction error and final video quality? My concern is whether the average scores hide a substantial drop on these cases, not the small calibration set by itself. A broader motion-focused benchmark would be useful future work, but I am not asking for it as a condition for acceptance.
7 comments
Response to Q2
Q2: On the calibration objective
We agree with the reviewer that minimizing prediction L2 error and minimizing post-quantization distortion are not strictly equivalent. Nevertheless, we adopt the L2 objective in Eq. (5) as a principled surrogate for two reasons: it controls an upper bound on the quantization distortion, and it keeps calibration a closed-form linear problem. We elaborate on both points below.
First, prediction MSE and quantization distortion are linked by an error bound. Consider one quantization group of residuals , and let
M=\max(0,\max_i r_i),\quad \Delta=\frac{M-m}{2^b-1}.$$ Asymmetric $b$-bit quantization places $2^b$ uniformly spaced levels on $[m,M]$ with step $\Delta$. Since every $r_i\in[m,M]$, nearest rounding gives $$|r_i-[Q_b(r)]_i| \leq \frac{\Delta}{2}, \qquad i=1,\dots,G.$$ Because $m\le 0\le M$, we have $(M-m)^2\leq 2(M^2+m^2)$. Moreover, $M^2$ and $m^2$ are each either $0$ or the square of a distinct entry of $r$, so $M^2+m^2\leq\|r\|_2^2$. Summing the per-element bound over the group, $$\|r-Q_b(r)\|_2^2 \leq \frac{G\Delta^2}{4} = \frac{G\,(M-m)^2}{4(2^b-1)^2} \leq \frac{G}{2(2^b-1)^2}\|r\|_2^2 .$$ Summing over all groups, the total quantization distortion is bounded by $\frac{G}{2(2^b-1)^2}$ times the total residual energy, which is exactly the data term of Eq. (5). Thus, for fixed $G$ and $b$, **minimizing the residual L2 loss in Eq. (5) also minimizes an upper bound on quantization distortion.** **Second, with reconstructed inputs fixed, our linear predictors make Eq. (5) a regularized least-squares problem solvable through a linear system.** Directly optimizing quantization distortion couples the quantization codes and group-wise scales to predictor parameters, yielding a non-smooth objective to which the same linear-system solution no longer applies and making the optimization more difficult. We compare Eq. (5) (prediction L2) with direct quantization-aware optimization (QAO) at INT2 and INT4 below. | Precision | Calibration objective | Quantization Error ↓ (K) | Quantization Error ↓ (V) | Calibration Time ↓ | | --------- | --------------------- | ------------------------ | ------------------------ | ------------------ | | INT2 | Eq. (5) | 0.089359248 | 0.092160777 | **5.41 s** | | | QAO | **0.089342692** | **0.091883979** | 93.82 s | | INT4 | Eq. (5) | 0.003352054 | 0.003428926 | **5.07 s** | | | QAO | **0.003351606** | **0.003419001** | 93.83 s | QAO reduces quantization error by only **0.01–0.30%** while requiring **17–19× the calibration time**, supporting Eq. (5) as an efficient and effective calibration objective in these settings.Response to Q3
Q3: Difficult motions case
Evaluation protocol.
We thank the reviewer for suggesting a more detailed evaluation under challenging motion conditions. We quantify motion intensity using the average optical flow magnitude computed from BF16 reference videos:
\sum_{t=1}^{T-1}\sum_{x,y}|F_t(x,y)|_2,$$ where $F_t(x,y)$ denotes the optical flow vector between consecutive frames at spatial position $(x,y)$. We rank 128 MovieGen videos by their motion intensity and divide them into three approximately equal-sized groups: **low, medium, and high motion**. We also manually inspect the high-motion subset to ensure coverage of challenging scenarios, including fast object motion, large camera movements, occlusions, and abrupt scene changes. ### **Motion-stratified evaluation.** We compare TempoKV (k5v3) against QVG INT4 across different motion levels, measuring reconstruction fidelity relative to BF16 reference videos. **Table 1. Reconstruction fidelity across different motion intensities.** | Motion Level | Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | | ------------ | ------------ | --------------- | --------- | ---------- | | High | QVG INT4 | 18.56 | 0.760 | 0.1396 | | | TempoKV k5v3 | **20.68** | **0.796** | **0.1151** | | Medium | QVG INT4 | 23.29 | 0.855 | 0.0861 | | | TempoKV k5v3 | **25.24** | **0.889** | **0.0563** | | Low | QVG INT4 | 29.97 | **0.944** | 0.0179 | | | TempoKV k5v3 | **30.01** | 0.943 | **0.0168** | As motion intensity increases, reconstruction becomes more challenging for both methods. However, TempoKV maintains a clear advantage in the more difficult settings. On the high-motion subset, it improves PSNR by **2.12 dB** and SSIM by **0.036**, while reducing LPIPS by **0.0245** compared with QVG. Similar gains are observed for medium-motion videos, with a **1.95 dB** PSNR improvement and a **0.0298** reduction in LPIPS. In contrast, both methods achieve comparable fidelity on low-motion videos. **Table 2. Reconstruction error of TempoKV across different motion intensities.** | Motion Level | K NMSE ↓ | V NMSE ↓ | |---|---|---| | Low | 2.36e-4 ± 0.46e-4 | 1.40e-2 ± 0.15e-2 | | Medium | 2.88e-4 ± 0.57e-4 | 1.55e-2 ± 0.13e-2 | | High | 3.62e-4 ± 0.84e-4 | 1.76e-2 ± 0.18e-2 | **Cache reconstruction error.** We measure the K/V cache reconstruction error of TempoKV on each motion level. Error increases gradually with motion intensity (K NMSE 2.36e-4 → 3.62e-4, V NMSE 1.40e-2 → 1.76e-2 from low to high), with small per-prompt variance and no collapse on high-motion cases. K error is consistently lower than V error because we use the **K5V3** configuration, which allocates more bits to keys (5-bit) than to values (3-bit). ### **Representative challenging cases.** To further examine performance at the individual-video level, we report four representative prompts covering different motion intensities and scene dynamics. **Table 3. Per-prompt reconstruction fidelity across representative video scenarios.** | Prompt | Optical Flow | QVG INT4 | TempoKV k5v3 | | ------------------------------------------------------------ | ------------ | ------------------- | ----------------------- | | Historical footage of California during the gold rush. | 1.39 | 17.89 / .776 / .147 | **24.26 / .898 / .056** | | Step-printing scene of a person running, cinematic film shot in 35mm. | 3.31 | 20.54 / .745 / .133 | **23.00 / .833 / .091** | | Reflections in the window of a train traveling through the Tokyo suburbs. | 3.68 | 23.31 / .864 / .072 | **25.11 / .873 / .056** | | FPV flying through colorful coral-lined streets of an underwater suburban neighborhood. | 5.77 | 13.49 / .567 / .437 | **16.79 / .630 / .349** | TempoKV consistently outperforms QVG across all three reconstruction metrics in these examples. In the gold-rush scene, it improves PSNR by **6.37 dB** and reduces LPIPS from 0.147 to 0.056. For the FPV scene, which exhibits the highest optical flow magnitude among these examples, TempoKV improves PSNR by **3.30 dB** and reduces LPIPS from 0.437 to 0.349. Together, the motion-stratified results and representative cases demonstrate that TempoKV remains effective under challenging video dynamics. Its improvements are particularly pronounced in medium- and high-motion subsets, supporting the robustness of TempoKV.
Response to Q1 & Weakness
To Reviewer cCap
We sincerely thank the reviewer for the constructive comments and recognition of the significance of our asymmetric K/V predictability analysis. We appreciate the concerns regarding predictor design and calibration goals. We apologize for not presenting the relevant analysis and design considerations in the initial draft. In fact, we explored various predictor configurations during the method development phase, but due to space limitations, we were unable to include these results. We are pleased to have the opportunity to explain these choices and supplement the corresponding experimental evidence here.
To validate the design of our predictor architecture, we conducted two additional ablation experiments on Self-Forcing : (1) channel-wise temporal prediction versus dense temporal prediction; and (2) dense K→V prediction versus low-rank K→V prediction, evaluating reconstruction errors, predictor storage, and decoding latency.
(1) Why channel-wise temporal prediction?
We replace our channel-wise temporal predictor with a full per-head dense temporal predictor. Increasing the predictor capacity does not improve reconstruction fidelity. The channel-wise predictor achieves a K NMSE of 0.000902, compared with 0.002333 for the dense predictor, while requiring only 1.05 MiB instead of 90.35 MiB of predictor storage (85.7× smaller). It also reduces KV decoding time from 3511.7 ms to 3116.2 ms. Quantizing the dense predictor weights to INT4 reduces its storage to 11.95 MiB but further increases decoding time to 4342.8 ms, without improving reconstruction accuracy.
These results indicate that, under our evaluated setting, the additional cross-channel capacity of dense temporal prediction is unnecessary for accurate K reconstruction and introduces substantial storage and computational overhead. This supports our choice of a lightweight channel-wise temporal predictor.
(2) Why dense K→V prediction?
We further investigate low-rank approximations of the dense K→V projection, including truncated SVD and low-rank refitting. Although low-rank factorization reduces predictor storage, it consistently increases V reconstruction error. For example, rank-16 refitting reduces storage from 22.5 MiB to 5.63 MiB but increases V NMSE from 0.002737 to 0.003750 (+37.0%). Even at rank 32, which halves the storage, V NMSE remains 17.4% higher than the dense baseline. Increasing the rank to 64 requires the same storage as the original dense projection, while still producing slightly higher reconstruction error.
In summary, these experimental findings support our chosen asymmetric predictor design. Since our dense K→V projection operates independently within each attention head rather than across the full hidden dimension, its storage cost remains manageable (22.5 MiB). We will include these architectural ablation studies in the revised manuscript, along with measurements of storage and latency.