- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
HiMix is a training-free KV cache quantization method for autoregressive video generation. It combines token-axis Hadamard smoothing, attention-sparsity-based layer budgets, and query-key channel scoring. HiMix-low uses 4/2/1-bit keys and 2-bit values; HiMix-high doubles these widths. Tests on Self-Forcing and Causal Forcing report 7.09x cache compression with quality close to BF16, including a 30-second evaluation.
Critical strengths
-
Table 1 reports roughly 7x compression with quality within 0.17 and 0.22 points of the BF16 four-metric averages on the two frameworks. The 30-second results also improve substantially over direct KIVI and QuaRot. Retaining historical tokens at this storage cost is a useful result.
-
The paper provides separate key/value diagnostics, explicit budget allocation, and fused unpacking/dequantization. Appendix E.2 tests channel ranking at the layer-wide, block-mean scope actually used by the method, beyond the illustrative per-head analysis.
Critical weaknesses
-
The benefit of the new allocation rules is not well isolated. Equation (5) ranks channels by mean absolute query magnitude times key range, as in MixKVQ, Equations (6)-(8). Table 3 does not compare Q-K versus K-only ranking, or sparsity-based versus alternative layer budgets, at fixed bit budgets. The layer-allocation gain is only 0.10 average points. Against QVG, Table 1 gains 0.08 points on each model as compression rises from 6.96x to 7.09x, without uncertainty estimates. Table 2 also trades better image quality for slightly lower subject consistency. These small differences need stronger evidence.
-
The long-history resource cost remains unclear. Section 3.4 and Appendix D reconstruct packed history before attention and apply a separate inverse Hadamard transform. The 9.3 GB figure at 60 seconds counts packed data and metadata, but gives no peak-memory breakdown for reconstructed tensors, recent BF16 entries, and buffers. Table 6 times only 63 latent frames, and the compression interval F is unspecified. The results therefore do not establish the feasible history length or its latency cost.
Questions for authors
-
What distinguishes the channel score from MixKVQ? At fixed bit budgets, how do Q-K and K-only ranking compare, and how much does sparsity-based layer allocation add? Please include paired uncertainty estimates for the small gains over QVG and in the layer-allocation ablation.
-
What F, cache lengths, and baselines were used for the memory and latency results? For the 30- and 60-second settings, what are peak device memory and end-to-end overhead, including recent BF16 entries, dequantization, inverse Hadamard, RoPE, and buffers?