- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 2
Read review
Paper summary
This paper investigates the KV cache compression in models that utilize historical scale caching for predicting subsequent scales, as employed by Infinity et al. The core of the method is: using locally attentive output reconstruction errors to replace the simple cache importance evaluation based solely on attention weights; imposing constraints across different generation scales with layer, attention head, and source scale as allocation units; further incorporating the sharing of conditional and unconditional branch KV from later scales into cache cost and error calculations. The authors employ a small number of prompt tokens for offline calibration and construct static cache configurations through Lagrangian decomposition and dynamic programming. The experiments primarily cover 20%, 10%, and 4% of the cache budgets for Infinity-2B and Infinity-8B, with HART results supplemented in the appendix
Critical strengths
1.The reconstruction-error criterion considers both attention weights and value vectors, including interactions between evicted scales. This provides a principled basis for selecting caches according to their effect on attention outputs and supports the goal of preserving the pretrained model’s behavior.
2.Table 1 shows improvements over HeatKV across both Infinity models and all three budgets. For example, at a 10% budget, Infinity-8B’s PSNR improves from 24.40 to 25.95 dB. These results demonstrate practical value for compressing existing models without retraining.
Critical weaknesses
1.Insufficient positioning and comparisons with alternatives. HMAR, MVAR, Markov-VAR, and InfinityStar already reduce historical scale dependencies. Although these model designs differ from compressing a frozen checkpoint, they should inform the paper’s motivation and scope. More directly, comparisons with FastVAR and simple recent-scale retention on the same checkpoint are missing, leaving the additional value of the proposed selective allocation insufficiently established.
2.Incomplete specification of the hard-budget optimization. Appendix B provides dynamic programming for fixed Lagrange multipliers but does not explain multiplier selection, termination, or recovery of a discrete allocation satisfying every scale’s budget. Solving the relaxed subproblem does not itself establish feasibility for Eq. (11). Since budget-constrained allocation is central to the contribution, this omission limits reproducibility and confidence in the optimization claims.
[1] HMAR: Efficient Hierarchical Masked Auto-Regressive Image Generation, CVPR 2025
[2] MVAR: Visual Autoregressive Modeling with Scale and Spatial Markovian Conditioning, ICLR 2026
[3] Markovian Scale Prediction: A New Era of Visual Autoregressive Generation, CVPR 2026
[4] InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generation, NeurIPS 2025
[5] FastVAR: Linear Visual Autoregressive Modeling via Cached Token Pruning, ICCV 2025
Questions for authors
1.How are the Lagrange multipliers selected, and how is a nested allocation satisfying every constraint in Eq. (11) obtained? Please provide the complete optimization procedure and achieved cache usage at each generation scale.
2.How does ResceneKV compare with FastVAR in output fidelity, generation quality, peak GPU memory, and latency?
4 comments
W2 & Q1. Complete specification of the budget-constrained allocation.
We thank the reviewer for this helpful comment. To make the optimization fully reproducible, we have added the complete offline procedure to Appendix B. Algorithm 1 computes the reconstruction error of every candidate eviction set on the calibration prompts, and Algorithm 2 solves Eq. (11) on these errors. We provide Algorithm 2 below.
Algorithm 2. Budget-constrained allocation
Input: reconstruction errors e_k(c, D), budget ratio b,
initial multipliers λ = (λ_τ, …, λ_K) = 0, initial upper bound λ_0,
growth factor ρ, bisection steps T, relative tolerance δ, iterations S
Output: nested eviction sets D* satisfying Eq. (11)
B ← b · 2LHC_{K−1}, D* ← ∅, J* ← ∞, s ← 0
repeat
λ_prev ← λ, s ← s + 1
for k = τ, …, K do # bisection on λ_k
λ_lo ← 0, λ_hi ← max(λ_k, λ_0), λ_k ← λ_hi
while M_k(DP(λ)) > B do # expand the bracket
λ_hi ← ρ · λ_hi, λ_k ← λ_hi
for t = 1, …, T do
λ_k ← (λ_lo + λ_hi) / 2
if M_k(DP(λ)) ≤ B then λ_hi ← λ_k else λ_lo ← λ_k
λ_k ← λ_hi
D ← DP(λ)
J(D) ← Σ_{k=τ}^{K} Σ_{c∈C} e_k(c, D_k(c))
if M_k(D) ≤ B for all k and J(D) < J* then
D* ← D, J* ← J(D)
until |λ_k − λ_prev,k| ≤ δ · |λ_prev,k| for all k, or s = S
return D*
function DP(λ) # Eqs. (36)–(39)
D ← minimizer of Eq. (35) for each c under λ
M_k(D) ← Σ_{c∈C} Σ_{i<k} m_{c,i}^(k) for k = τ, …, K
return D
- Multiplier selection. We assign one multiplier to the budget constraint of each generation scale . When the other multipliers are fixed, the KV cache usage of the DP solution does not increase as grows, since a larger places a larger penalty on the memory at scale . This monotonicity allows us to find a that meets the budget at scale by bisection. Specifically, we first enlarge the upper bound by a factor of until satisfies the budget, and then run bisection steps to approach the smallest such , so that the budget is used as fully as possible. Since updating one multiplier can change the usage at other scales through the nested eviction sets, we repeat these updates until the multipliers converge.
- Discrete and nested allocation. For fixed multipliers, the DP in Eqs. (36)–(39) operates directly on discrete eviction sets and only allows transitions from to . Its output is therefore always a discrete allocation that satisfies the nesting constraint, so no rounding or recovery step is needed.
- Feasibility. As the reviewer notes, solving the relaxed subproblem alone does not guarantee feasibility for Eq. (11). For this reason, our procedure uses the Lagrangian relaxation only to search for candidate allocations and explicitly checks the budget at every generation scale. Among the candidates visited during the search, it returns the one with the lowest reconstruction error that satisfies every per-scale budget. The returned allocation is therefore feasible by construction. In practice, the first outer iteration already yields a feasible allocation in all our settings, and the subsequent iterations further reduce the reconstruction error.
Achieved cache usage. The table below reports the KV cache usage (MB) at each generation scale on Infinity-2B. Bold values indicate the scales where eviction is active. Once eviction starts, the usage stays within the limit at every scale for all budgets and batch sizes. Figures 5 and 6 in Appendix C visualize the eviction sets obtained by Algorithm 2 on Infinity-2B at the 10% and 4% budgets.
| Budget | Batch | Scale 5 | Scale 6 | Scale 7 | Scale 8 | Scale 9 | Scale 10 | Scale 11 | Scale 12 | Scale 13 | Limit |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 20% | 1 | 42.8 | 90.8 | 198.8 | 390.8 | 690.8 | 974.7 | 974.7 | 974.7 | 974.7 | 975.0 |
| 20% | 8 | 342.0 | 726.0 | 1,590.0 | 3,126.0 | 5,526.0 | 7,798.9 | 7,798.9 | 7,798.9 | 7,798.9 | 7,800.0 |
| 10% | 1 | 42.8 | 90.8 | 198.8 | 390.8 | 486.9 | 486.9 | 486.9 | 486.9 | 486.9 | 487.5 |
| 10% | 8 | 342.0 | 726.0 | 1,590.0 | 3,126.0 | 3,898.9 | 3,898.9 | 3,898.9 | 3,898.9 | 3,898.9 | 3,900.0 |
| 4% | 1 | 42.8 | 90.8 | 198.7 | 198.7 | 198.8 | 198.7 | 198.7 | 198.7 | 198.7 | 198.8 |
| 4% | 8 | 342.0 | 726.0 | 1,589.7 | 1,589.9 | 1,590.0 | 1,590.0 | 1,589.9 | 1,589.8 | 1,589.8 | 1,590.0 |
We would sincerely appreciate it if the reviewer could refer to Appendix B, which also presents the full DP formulation and Algorithm 1. We hope that the added algorithms and these results resolve the reviewer's concern about reproducibility.
I thank the authors for their detailed response and additional experiments. The comparisons with FastVAR across different batch sizes provide further evidence of the proposed method’s effectiveness. Algorithms 1 and 2 also clarify the budget-constrained allocation procedure and resolve my concerns about this aspect of the paper. With these clarifications and additional results, I have no further concerns and am raising my score from 2 to 3.
Thank you for your thoughtful feedback and for raising the score. Your suggestions on related efficient next-scale generation methods, including HMAR, MVAR, Markov-VAR, InfinityStar, and FastVAR, helped us clarify the positioning of our work, which is now reflected in the related work. We will also keep the detailed description of the allocation procedure in the final version.
W1-(a). Positioning with respect to HMAR, MVAR, Markov-VAR, InfinityStar, and FastVAR.
We thank the reviewer for this valuable suggestion. We agree that these works should inform the motivation and scope of our paper, and we have added a new paragraph, Efficient Next-Scale Generation, to the related work (Sec. 2, highlighted in blue). It clarifies our position along two axes.
W1-(b) & Q2. Comparison with FastVAR and recent-scale retention.
Since W1-(b) and Q2 both concern the comparison with FastVAR, we address them together with additional comparisons on Infinity-2B.
Fidelity on MS-COCO 2017 and GenEval (Infinity-2B).
Efficiency on Infinity-2B (RTX 3090). Latency at batch size 1.
vs. recent-scale retention. ResceneKV outperforms it by a large margin at every budget, e.g., 3.72 → 0.91 FID and 21.87 → 29.12 dB PSNR at 20%. Notably, ResceneKV at 4% still outperforms recent-scale retention at 20% (FID 2.40 vs. 3.72, PSNR 22.80 vs. 21.87 dB). Thus, the benefit comes not from discarding distant scales but from selecting which source scales to retain for each layer and head based on reconstruction error.
vs. FastVAR. Even at a 4% budget, ResceneKV outperforms FastVAR on all fidelity metrics and on GenEval, while using 8.3× less KV cache memory. This gap grows with batch size. At batch size 4, ResceneKV uses 11.78 GiB versus 20.54 GiB for FastVAR, and at batch size 8, FastVAR runs out of memory while ResceneKV uses 16.18 GiB. FastVAR achieves a larger speedup (2.14× vs. 1.11×) since it is designed for acceleration, whereas ResceneKV is better suited when GPU memory is the bottleneck.
Peak memory across batch sizes. At small batch sizes, the peak memory of ResceneKV is determined by VAE decoding, which runs after the KV cache is released and stays at about 11.1 GiB regardless of the batch size. The memory of the transformer stage grows linearly by about 1.1 GiB per image at the 4% budget and exceeds this level from batch size 4. Therefore, the 4.4 GiB increase from batch size 4 to 8 corresponds to four additional images at about 1.1 GiB each, rather than a sudden jump.