acceptodds
Withdrawn from ICLR 2027

Memory-Efficient Autoregressive Video Generation via Hierarchical Mixed-Precision KV Cache Quantization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Autoregressive video generation models face a critical KV cache memory bottleneck as the demand for long video synthesis grows. Existing LLM quantization methods perform poorly when applied to video generation models due to the high dynamic range of video KV cache activations and their heightened sensitivity to quantization errors. We propose HiMix, a hierarchical mixed-precision KV cache quantization framework for memory-efficient autoregressive video generation. Specifically, HiMix first determines layer-wise bit budgets offline based on per-layer sparsity analysis, then performs runtime channel-aware quantization guided by Q-K saliency: important channels receive higher bit-widths (4-bit) while less important ones are quantized to 2-bit or 1-bit, with spatiotemporal Hadamard smoothing applied prior to quantization to reduce error. Experiments on the Self-Forcing and Causal Forcing models demonstrate that HiMix achieves nearly 7.1x compression ratio for KV cache while preserving video generation quality and long-range consistency comparable to the BF16 baseline, substantially reducing memory consumption.

Primary area:
generative models
Keywords:
Quantization, ARVD, KV Cache

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_i2biOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HiMix is a training-free KV cache quantization method for autoregressive video generation. It combines token-axis Hadamard smoothing, attention-sparsity-based layer budgets, and query-key channel scoring. HiMix-low uses 4/2/1-bit keys and 2-bit values; HiMix-high doubles these widths. Tests on Self-Forcing and Causal Forcing report 7.09x cache compression with quality close to BF16, including a 30-second evaluation.

Critical strengths
  1. Table 1 reports roughly 7x compression with quality within 0.17 and 0.22 points of the BF16 four-metric averages on the two frameworks. The 30-second results also improve substantially over direct KIVI and QuaRot. Retaining historical tokens at this storage cost is a useful result.

  2. The paper provides separate key/value diagnostics, explicit budget allocation, and fused unpacking/dequantization. Appendix E.2 tests channel ranking at the layer-wide, block-mean scope actually used by the method, beyond the illustrative per-head analysis.

Critical weaknesses
  1. The benefit of the new allocation rules is not well isolated. Equation (5) ranks channels by mean absolute query magnitude times key range, as in MixKVQ, Equations (6)-(8). Table 3 does not compare Q-K versus K-only ranking, or sparsity-based versus alternative layer budgets, at fixed bit budgets. The layer-allocation gain is only 0.10 average points. Against QVG, Table 1 gains 0.08 points on each model as compression rises from 6.96x to 7.09x, without uncertainty estimates. Table 2 also trades better image quality for slightly lower subject consistency. These small differences need stronger evidence.

  2. The long-history resource cost remains unclear. Section 3.4 and Appendix D reconstruct packed history before attention and apply a separate inverse Hadamard transform. The 9.3 GB figure at 60 seconds counts packed data and metadata, but gives no peak-memory breakdown for reconstructed tensors, recent BF16 entries, and buffers. Table 6 times only 63 latent frames, and the compression interval F is unspecified. The results therefore do not establish the feasible history length or its latency cost.

Questions for authors
  1. What distinguishes the channel score from MixKVQ? At fixed bit budgets, how do Q-K and K-only ranking compare, and how much does sparsity-based layer allocation add? Please include paired uncertainty estimates for the small gains over QVG and in the layer-allocation ablation.

  2. What F, cache lengths, and baselines were used for the memory and latency results? For the 30- and 60-second settings, what are peak device memory and end-to-end overhead, including recent BF16 entries, dequantization, inverse Hadamard, RoPE, and buffers?

Reviewer_Gy3fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets the KV cache bottleneck in autoregressive video diffusion, where the cache grows with retained history and reaches tens of GB. HiMix is a training-free hierarchical mixed-precision quantiser with three parts. A block-wise spatiotemporal Hadamard smoothing over 128-token blocks, applied separately to keys and values, exploits the observation that video KV outliers vary along the token axis rather than persisting per channel as in LLMs. An offline layer-wise budget allocates channels at 4/2/1 bit (8/4/2 in the high configuration) using attention sparsity S = 1−D as a sensitivity proxy, with layers K-means clustered into sensitive/normal/robust groups. A runtime channel ranking by joint Q-K saliency s_c = mean_t|Q_t,c|·ΔK_c is motivated by showing key-range-only ranking misses 50–66% of jointly salient channels. Values use uniform per-token quantisation. Experiments pair with Self-Forcing and Causal Forcing on 1,003 MovieGen prompts, reporting VBench dimensions, compression ratio, latency and a 30-second long-video evaluation.

Critical strengths

The KV cache statistics analysis is the strongest part and is genuinely useful. Figure 2 makes a point I have not seen stated this explicitly: the Self-Forcing key cache is token-varying rather than channel-persistent, which is exactly why LLM recipes transfer poorly, and the KIVI/QuaRot numbers (77.95 / 77.63 against a 79.49 BF16 baseline) confirm the problem is real. The follow-up measurement that key-range-only ranking loses 50.6% of jointly selected channels even under the exact runtime-matched protocol of Table 7 is well executed and makes the design choice defensible.

The long-video result is where the method has a real story. Table 2 (30-second Self-Forcing) gives HiMix 68.05 Image Quality and 89.24 Subject Consistency against QVG's 67.23 / 89.34 and BF16's 67.81 / 89.99, with Figure 5 attributing QVG's failure to late subject disappearance. If the mechanism is that quantisation lets you keep the full history rather than truncate it, that is qualitatively different from a 0.1-point VBench gain, and the paper would be stronger leading with it.

Critical weaknesses

The headline improvements are within measurement noise, and no variance is reported. At 2 bits HiMix-low scores 79.32 against QVG's 79.24 on a 0–100 scale — 0.08 points over 1,003 prompts, with no seeds, no intervals, no per-dimension variance. At 4 bits HiMix-high is 79.48 against QuaRot 79.46 and QVG 79.48, i.e. indistinguishable. The framing ("near-lossless") therefore rests on differences a reader cannot separate from VBench noise, and the only ablation with a visible margin (Table 3, 79.32 → 78.83 without channel allocation) is under half a point. This matters more than usual because QVG already does semantic-aware smoothing and progressive residual quantisation at the same target width, so the marginal value of a further hierarchical schedule is exactly what needs establishing, and the current evidence does not establish it.

The most important alternative is not compared. The premise, stated in Section 1, is that quantising history beats truncating it — yet every baseline keeps the full cache and no eviction baseline (H2O, SnapKV, or a recency window matched to the same 9.3 GB budget) is run. That is precisely the comparison a practitioner facing a 65 GB requirement would make, and without it 7.1× does not tell the reader whether hierarchical mixed precision is the right use of the budget or a more elaborate way to spend it.

Two supporting points. The offline budget construction is close to forced: Eqs. (12)–(16) show the "≈2-bit" point follows from the payload identity rather than being chosen, and the "−Layer alloc." ablation moves compression only 7.09× → 6.96× for a 0.10-point quality change, so the layer component's quality contribution cannot be separated from its storage contribution. And the sparsity→sensitivity link is validated by Spearman correlation over 10 prompts of one model, yet the same rule assigns budgets for Causal Forcing.

Questions for authors
  1. Please report variance (seeds or at least per-prompt spread) for Table 1. At a 0.08-point gap to QVG, what shows the improvement is systematic rather than sampling noise?
  2. How do you compare against a truncation/eviction baseline (recency window or H2O-style) at the same 9.3 GB budget? The "keep full history" claim is central and currently untested against the obvious alternative.
  3. Are the layer groups recomputed per model? If the Self-Forcing partition is reused for Causal Forcing, how sensitive are the results?
  4. The compression ratio includes scales, zero-points and reordering indices. What fraction of the 9.3 GB is metadata, and how does that change between group sizes 32 and 128?
  5. Channel identities are reassigned at each compression trigger. Is there any measurable artifact at the boundary between a reordered older span and a newly stored BF16 span?
Reviewer_aM92OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the growing KV-cache memory bottleneck in autoregressive video generation through HiMix, a training-free mixed-precision quantization framework. It assigns layer-wise precision budgets offline using attention sparsity and allocates bit widths to key channels at runtime using joint Q-K importance scores. Hadamard transforms mitigate the impact of outliers on quantization. The implementation uses packed low-bit storage and fuses unpacking, dequantization, and channel-order restoration. The experiments suggest that HiMix substantially reduces KV-cache storage while maintaining video quality close to the original models with modest additional latency.

Critical strengths
  • Addresses a practical memory bottleneck in long-video generation without retraining.
  • Combines layer sensitivity and channel importance in a coherent precision-allocation strategy. Evaluates two generation models and includes long-video experiments and component ablations.
Critical weaknesses
  • The paper lacks a matched-budget comparison of key-only and joint Q-K ranking, leaving the benefit of incorporating queries unclear. An offline fixed-channel baseline is missing, making it difficult to assess whether runtime allocation is necessary.
  • The paper does not compare sparsity-based layer allocation with allocation based on directly measured quantization error, leaving the effectiveness of sparsity as a proxy unclear. Improvements over QVG are modest and vary across metrics.
Questions for authors
  • Can HiMix’s precision allocation be combined with QVG’s quantization techniques, and would the combination improve quality at the same actual storage budget?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.