acceptodds
Under review as a conference paper at ICLR 2027

QuantWM: Temporally Consistent 2-Bit KV Cache Quantization for Video World Models

18 views

Abstract

Video generation based world models achieve long-range temporal consistency by storing KV cache during generation, but the continuously growing cache makes KV cache memory become a major deployment bottleneck, which motivates low-bit quantization study for efficiency. Existing 2-bit KV cache quantization methods can achieve nearly quantitatively lossless performance on conventional video benchmarks such as VBench, however, when applied to video world models, we find that they still cause severe temporal flickering and visual degradation. Meanwhile, deeper investigates show that Key quantization produces smaller reconstruction errors than Value, but surprisingly leads to much larger output degradation. We trace this discrepancy to attention in video world models: Key perturbations can change the attention logits, i.e., QK, and shift the temporal-spatial tokens selected by Queries. These observations motivate us to explicitly preserve attention logits and temporal-spatial token selection during KV cache quantization to alleviate the visual degradation problem. To address this issue, we present QuantWM, a training-free 2-bit KV cache quantization framework for video world models. QuantWM introduces two complementary techniques to mitigate the attention shifts. Firstly, quantization-sensitivity-aware clustering (QSAC) jointly considers historical Query sensitivity and residual ranges to select INT2-friendly Key centroids, which reduces quantization errors in channels that are more critical to attention. In addition, principal-subspace attention compensation (PSAC) restores the remaining Key errors along the dominant Query subspace using low-rank projections, which provides a direct and efficient correction to stabilize attention logits. Extensive experiments on LingBot-World-v2, HY-World 1.5, Matrix-Game-2, Longcat-Video and Causal-Forcing demonstrate that QuantWM significantly improves visual quality and temporal consistency, while outperforming existing methods across image and video quality metrics with up to \(6.20\times\) KV cache memory compression and limited additional overhead. The project page will be released soon.

Primary area:
generative models
Keywords:
Model quantization, KV cache, world model, video generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_iVgcOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies an interesting phenomenon: existing KV-cache quantization methods introduce noticeable temporal flickering in visual world models, despite causing minimal degradation in conventional video generation models. Motivated by this observation, the authors investigate the effects of quantizing Keys and Values in Transformers and find that Key quantization has a more pronounced impact on model performance than Value quantization. Further analysis attributes this degradation to attention shifts arising from the interaction between quantized Keys and Queries.

To alleviate this issue, the authors propose Quantization-Sensitivity-Aware Clustering (QSAC), which selects Key centroids by jointly considering historical Query sensitivity and residual quantization difficulty, thereby minimizing attention-logit perturbations. They further introduce Principal-Subspace Attention Compensation (PSAC), which identifies dominant Query subspaces through eigendecomposition and applies low-rank compensation to Key quantization errors, reducing attention shifts and improving video quality and temporal consistency.

Experiments on three world models (LingBot-World-v2, HY-World 1.5, and Matrix-Game-2) and two video generation models (LongCat-Video and Causal-Forcing) demonstrate consistent improvements in video fidelity and temporal consistency while substantially reducing KV-cache memory consumption.

Critical strengths
  1. The paper is well organized and presents a clear motivation. The analysis highlights the different impacts of Key and Value quantization on model performance and identifies attention shifts induced by Key quantization as a major source of performance degradation, providing valuable insights into the challenges of low-bit KV cache quantization.

  2. The proposed QSAC and PSAC are straightforward, with clear connections to the identified attention-shift problem. Their designs are easy to understand and can be applied without additional model training, making the framework practical. Most importantly, the framework demonstrates strong practical engineering value, as it requires no additional training and can be readily integrated into existing video generation architectures.

  3. The paper provides extensive experiments across three video world models and two video generation models, covering different resolutions and generation lengths. The comparisons with existing methods, ablation studies, and efficiency analyses provide solid empirical evidence for the effectiveness of the proposed approach.

Critical weaknesses
  1. The proposed QSAC and PSAC share substantial conceptual similarities with existing approaches, particularly Query-aware quantization and low-rank attention error compensation. Prior works, including MixKVQ [1] and KQ-SVD [2], have already explored closely related principles. Therefore, the methodological novelty of the proposed approach remains unclear. The authors should better differentiate QSAC and PSAC from these existing methods and clarify the unique technical contributions beyond established techniques.

  2. Although the paper is positioned as a KV cache quantization method for video world models, it remains unclear how the proposed QSAC and PSAC specifically address challenges unique to world modeling. Unlike conventional video generation, interactive world models typically require action-conditioned state prediction and consistent responses to user actions. However, both proposed components rely on general attention statistics without explicitly considering action-conditioned dynamics. Moreover, the experiments also include conventional video generation models, while the evaluations primarily focus on visual quality rather than action fidelity or environment consistency. The authors should clarify what makes the proposed approach particularly relevant to world models and provide corresponding evaluations to better justify the paper's positioning and title.

  3. Although QuantWM achieves substantial KV cache compression, PSAC introduces additional storage overhead for INT8 compensation coefficients and BF16 principal-subspace bases. While the paper reports memory consumption, it remains unclear whether the proposed method maintains its advantages over alternative quantization strategies under equivalent memory budgets, such as QVG with higher bit widths or different quantization configurations. Additional comparisons under matched memory constraints would provide stronger evidence that the performance gains stem from the proposed techniques rather than the use of additional storage.

[1] MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoning. ACL 2026.

[2] KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelity. AISTATS 2026.

Questions for authors

My major concern is the relation between the method and world models.

1 comment
Reviewer_iVgcOpenReview ↗

Reply to the Authors

We thank the authors for the reply, however, this does not address my concerns on how specifically QSAC and PSAC are designed for world models. It is more like a general design for video generation models.

The additional experiments in W2W2 provide useful evidence that action-conditioned generation changes the Query statistics and that updating these statistics improves the preservation of action-conditioned responses. However, I believe this does not fully resolve my original concern regarding the world-model-specific design of QuantWM. As your title specifically narrow your scope in world models, I do think it is a huge problem.

In particular, QSAC and PSAC still operate on generic Query statistics and attention sensitivity, without explicitly modeling actions, world states, state transitions, or other structural properties unique to interactive world models. The proposed mechanisms therefore appear applicable to general conditional autoregressive video generation models, where different conditioning signals can similarly induce changes in Query distributions.

The comparison between standard QuantWM and its fixed-Query variant demonstrates the benefit of dynamically updating Query statistics under changing actions. Nevertheless, this primarily establishes the effectiveness of condition-adaptive quantization, rather than demonstrating that the underlying quantization mechanisms are specifically designed for challenges unique to world models.

I would appreciate further clarification on what makes action-conditioned attention in interactive world models fundamentally different from conditional attention in general video generation, and which aspects of QuantWM explicitly exploit this distinction. Otherwise, I believe the contribution would be more accurately characterized as a general KV-cache quantization method that is particularly beneficial for interactive visual generation model, rather than a world-model-specific quantization framework.

Reviewer_Zj2SOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper extends QVG’s centroid–residual KV-cache quantization framework with query-aware centroid assignment (QSAC) and low-rank error compensation (PSAC). QSAC selects centroids using historical query sensitivity and residual quantization ranges, while PSAC stores projected Key errors along dominant query directions. Evaluation covers five video generation and world models at both 480p and 720p.

Critical strengths
  1. The motivation and method are well aligned. The paper shows that Key reconstruction error alone does not explain output degradation and connects attention sensitivity to query-aware assignment and error compensation.
  2. The evaluation covers multiple models and includes a practical implementation. Consistent fidelity improvements across five models and two resolutions, component ablations, and actual bit-packing and reconstruction provide useful evidence of effectiveness and feasibility.
Critical weaknesses
  1. The contribution is incremental over QVG, and the comparison does not control for additional storage. The method retains QVG’s centroid–residual structure while modifying assignment and storing correction factors. However, it does not compare against allocating the same total memory to more QVG centroids or additional PRQ stages. Consequently, whether PSAC uses the additional storage more effectively than strengthening QVG itself remains unclear.
  2. The hardware benefits are less established than the cache compression ratio suggests. Appendix D reconstructs BF16 K/V buffers before invoking FlashAttention. Thus, the reported cache compression of up to 6.20× does not establish the reduction in total peak GPU memory. Moreover, the speedup in Table 7 occurs only for LongCat, which uses cache offloading, and latency comparisons against QVG and KIVI under matched conditions are missing.
Questions for authors
  1. Please provide a sweep over the number of K-means centroids K for both QVG and QuantWM. How do quality, actual memory consumption, and latency change with K? Does QuantWM retain its advantage as QVG’s representational capacity increases?
  2. What is the reduction in total peak GPU memory when BF16 reconstruction buffers, the active chunk, query statistics, and PSAC metadata are included? Does this reduction enable a longer context on the same GPU?
  3. How much runtime is spent on query-statistics collection, eigendecomposition, centroid assignment, and cache reconstruction individually? What is the additional overhead relative to QVG and KIVI under identical conditions?
  4. How does latency change when cache offloading is enabled versus disabled for the same model? Please report repeated measurements with controlled GPU load and warm-up, including mean, variance, and chunk-level latency.
Reviewer_fSYHOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies the visual degradation caused by 2-bit KV-cache quantization in autoregressive video world models. It observes that Key quantization often causes larger output degradation than Value quantization despite smaller reconstruction error, and attributes this to perturbations of QK⊤ and temporal-spatial attention. QuantWM combines Query-sensitive centroid assignment (QSAC) with low-rank Key-error compensation in the principal Query subspace (PSAC), achieving up to 6.20× cache compression across three world models and two autoregressive video-generation models.

Critical strengths
  • The paper identifies a useful failure mode that standard VBench scores do not fully reveal. Across most reported settings, Key reconstruction error is smaller than Value error but has a larger effect on the output, showing that tensor reconstruction error alone is not an adequate quantization objective.
  • The mechanism is intuitive and technically grounded: Key error changes QK⊤ and the attention behavior. QSAC and PSAC directly target this mechanism rather than only minimizing K/V reconstruction error.
  • The evaluation covers five models at 480p and 720p, plus one-minute generation. The ablations on QSAC/PSAC, PSAC rank, diagonal statistics, and candidate-centroid count are useful.
  • The system implementation is practical and transparent, including packed INT2 storage, fused reconstruction, and model-dependent latency analysis.
Critical weaknesses

comments from 18/11: most of my concerns have been resolved by the authors. Thanks for the efforts.

------------------------------previous information----------------------

  • The evidence for the central claim of “temporal consistency” is indirect. PSNR, SSIM, and LPIPS are computed frame-wise against the BF16 trajectory. A temporal LPIPS/warping metric, flicker-frequency measure, or small blinded human preference study would directly support the main claim.

  • The evaluation protocol is insufficiently specified for interpreting the frame-level results. The paper does not clearly report the number of MovieGen prompts/videos evaluated per model, generation seeds, or whether BF16 and all quantized methods use exactly the same initial noise and stochastic trajectory. It states that methods are applied to the same cached tensors, but this does not fully clarify the end-to-end comparison. Since PSNR/SSIM against BF16 are meaningful only under paired generation, these details are essential. Multi-seed uncertainty or confidence intervals should also be reported.

  • Top-1 token-selection shift is an incomplete proxy for attention preservation. A top-1 flip between two nearly tied tokens may have little effect, while a substantial redistribution of attention can occur without changing the top-1 token. Therefore, reducing the top-1 shift ratio from 51.55% to 15.31% is suggestive but does not by itself establish temporal consistency. The paper should report a distribution-level metric (e.g., attention-output error, KL/JS divergence, or top-k overlap) and show its correlation with visual degradation or human preference.

  • The “world-model” specificity is not isolated. Three world models are the main targets and two video-generation models test generalization, which is reasonable. However, the paper does not test whether action-conditioned interaction makes quantization errors more damaging than ordinary autoregressive generation. Matrix-Game-2 uses several action sequences, but no action-wise sensitivity analysis is provided. The paper therefore demonstrates broad applicability, but not a world-model-specific effect.

Questions for authors

Thank you for the thorough rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.