acceptodds
Under review as a conference paper at ICLR 2027

ResceneKV: Reconstruction-Error Selective KV Cache Eviction across Nested Scales for Visual Autoregressive Modeling

22 views

Abstract

Visual autoregressive (VAR) modeling generates images by progressively predicting token maps from coarse to fine, through a next-scale paradigm that enables efficient sampling and strong zero-shot generalization. However, predicting each scale from all preceding scales requires retaining the corresponding keys and values in the KV cache. As generation proceeds to finer scales, the KV cache therefore grows continuously and becomes a major memory bottleneck for high-resolution image synthesis. Existing KV cache eviction methods typically rely on attention statistics to guide eviction decisions, although eviction candidates with similar attention mass can exhibit substantially different reconstruction errors. To address this mismatch, we propose ResceneKV, which directly measures the output reconstruction error induced by KV cache eviction and uses it for joint KV cache budget allocation. We further exploit the redundancy between the conditional and unconditional branches of classifier-free guidance by reusing the conditional KV cache for the unconditional branch at later scales. Extensive experiments across image generation benchmarks demonstrate that ResceneKV consistently improves pixel-level fidelity over existing KV cache eviction methods while maintaining generation quality across different memory budgets.

Primary area:
optimization
Keywords:
KV Cache Evicition, Visual Autoregressive Modeling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_fFFbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper investigates the KV cache compression in models that utilize historical scale caching for predicting subsequent scales, as employed by Infinity et al. The core of the method is: using locally attentive output reconstruction errors to replace the simple cache importance evaluation based solely on attention weights; imposing constraints across different generation scales with layer, attention head, and source scale as allocation units; further incorporating the sharing of conditional and unconditional branch KV from later scales into cache cost and error calculations. The authors employ a small number of prompt tokens for offline calibration and construct static cache configurations through Lagrangian decomposition and dynamic programming. The experiments primarily cover 20%, 10%, and 4% of the cache budgets for Infinity-2B and Infinity-8B, with HART results supplemented in the appendix

Critical strengths

1.The reconstruction-error criterion considers both attention weights and value vectors, including interactions between evicted scales. This provides a principled basis for selecting caches according to their effect on attention outputs and supports the goal of preserving the pretrained model’s behavior.

2.Table 1 shows improvements over HeatKV across both Infinity models and all three budgets. For example, at a 10% budget, Infinity-8B’s PSNR improves from 24.40 to 25.95 dB. These results demonstrate practical value for compressing existing models without retraining.

Critical weaknesses

1.Insufficient positioning and comparisons with alternatives. HMAR, MVAR, Markov-VAR, and InfinityStar already reduce historical scale dependencies. Although these model designs differ from compressing a frozen checkpoint, they should inform the paper’s motivation and scope. More directly, comparisons with FastVAR and simple recent-scale retention on the same checkpoint are missing, leaving the additional value of the proposed selective allocation insufficiently established.

2.Incomplete specification of the hard-budget optimization. Appendix B provides dynamic programming for fixed Lagrange multipliers but does not explain multiplier selection, termination, or recovery of a discrete allocation satisfying every scale’s budget. Solving the relaxed subproblem does not itself establish feasibility for Eq. (11). Since budget-constrained allocation is central to the contribution, this omission limits reproducibility and confidence in the optimization claims.

[1] HMAR: Efficient Hierarchical Masked Auto-Regressive Image Generation, CVPR 2025

[2] MVAR: Visual Autoregressive Modeling with Scale and Spatial Markovian Conditioning, ICLR 2026

[3] Markovian Scale Prediction: A New Era of Visual Autoregressive Generation, CVPR 2026

[4] InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generation, NeurIPS 2025

[5] FastVAR: Linear Visual Autoregressive Modeling via Cached Token Pruning, ICCV 2025

Questions for authors

1.How are the Lagrange multipliers selected, and how is a nested allocation satisfying every constraint in Eq. (11) obtained? Please provide the complete optimization procedure and achieved cache usage at each generation scale.

2.How does ResceneKV compare with FastVAR in output fidelity, generation quality, peak GPU memory, and latency?

4 comments

W1-(a). Positioning with respect to HMAR, MVAR, Markov-VAR, InfinityStar, and FastVAR.

We thank the reviewer for this valuable suggestion. We agree that these works should inform the motivation and scope of our paper, and we have added a new paragraph, Efficient Next-Scale Generation, to the related work (Sec. 2, highlighted in blue). It clarifies our position along two axes.

  • Architectural redesign vs. training-free compression. HMAR, MVAR, and Markov-VAR reduce cross-scale dependency by redesigning the architecture. HMAR and MVAR condition each scale only on the preceding scale, and Markov-VAR supplements this conditioning with a compact history vector of recent scales. InfinityStar extends next-scale prediction to video generation and similarly limits the history across clips by attending only to the last scale of the preceding clip. While effective, these designs define new models that must be trained, which is costly for large-scale text-to-image VAR models. In contrast, ResceneKV compresses the KV cache of pretrained VAR models without retraining.
  • Latency acceleration vs. KV cache reduction. FastVAR is also training-free, but it is designed to speed up inference by forwarding only pivotal tokens at large scales. It reduces the KV cache only as a by-product of this token pruning. ResceneKV instead directly reduces the KV cache by deciding which cached source scales remain accessible to subsequent scales.

W1-(b) & Q2. Comparison with FastVAR and recent-scale retention.

Since W1-(b) and Q2 both concern the comparison with FastVAR, we address them together with additional comparisons on Infinity-2B.

  • Recent-scale retention retains the sink scales and the most recent source scales that fit within the KV cache budget, applied uniformly across all layers and heads.
  • FastVAR uses the official code with its default setting. Since FastVAR cannot be set to a specific KV cache budget, we compare it based on its measured memory and latency. All efficiency results are measured on a single RTX 3090.

Fidelity on MS-COCO 2017 and GenEval (Infinity-2B).

MethodBudgetFID ↓PSNR ↑SSIM ↑LPIPS ↓Two Obj.PositionColor Attri.Overall ↑
Full Cache100%––––0.8760.4150.5400.720
FastVAR–3.7622.210.7450.1510.8540.4080.5380.705
Recent-scale20%3.7221.870.7680.1420.8740.4150.5070.710
Recent-scale10%5.3019.650.7140.2000.8480.4200.5150.709
Recent-scale4%6.6618.550.6830.2410.8430.4030.5050.699
ResceneKV20%0.9129.120.9180.0390.8760.4070.5520.717
ResceneKV10%1.4326.370.8760.0620.8860.4080.5650.722
ResceneKV4%2.4022.800.8030.1130.8790.4070.5470.717

Efficiency on Infinity-2B (RTX 3090). Latency at batch size 1.

MethodBudgetBatchPeak GPU Mem. (GiB)KV Cache Mem. (GB)Latency (s)
Full Cache100%1 / 2 / 4 / 815.75 / OOM / OOM / OOM4.82 / – / – / –2.96
FastVAR–1 / 2 / 4 / 811.61 / 13.96 / 20.54 / OOM1.65 / 3.30 / 6.61 / –1.38 (2.14×)
ResceneKV10%1 / 2 / 4 / 811.09 / 11.30 / 13.62 / 19.850.49 / 0.98 / 1.95 / 3.902.70 (1.10×)
ResceneKV4%1 / 2 / 4 / 811.06 / 11.25 / 11.78 / 16.180.20 / 0.40 / 0.80 / 1.592.67 (1.11×)

vs. recent-scale retention. ResceneKV outperforms it by a large margin at every budget, e.g., 3.72 → 0.91 FID and 21.87 → 29.12 dB PSNR at 20%. Notably, ResceneKV at 4% still outperforms recent-scale retention at 20% (FID 2.40 vs. 3.72, PSNR 22.80 vs. 21.87 dB). Thus, the benefit comes not from discarding distant scales but from selecting which source scales to retain for each layer and head based on reconstruction error.

vs. FastVAR. Even at a 4% budget, ResceneKV outperforms FastVAR on all fidelity metrics and on GenEval, while using 8.3× less KV cache memory. This gap grows with batch size. At batch size 4, ResceneKV uses 11.78 GiB versus 20.54 GiB for FastVAR, and at batch size 8, FastVAR runs out of memory while ResceneKV uses 16.18 GiB. FastVAR achieves a larger speedup (2.14× vs. 1.11×) since it is designed for acceleration, whereas ResceneKV is better suited when GPU memory is the bottleneck.

Peak memory across batch sizes. At small batch sizes, the peak memory of ResceneKV is determined by VAE decoding, which runs after the KV cache is released and stays at about 11.1 GiB regardless of the batch size. The memory of the transformer stage grows linearly by about 1.1 GiB per image at the 4% budget and exceeds this level from batch size 4. Therefore, the 4.4 GiB increase from batch size 4 to 8 corresponds to four additional images at about 1.1 GiB each, rather than a sudden jump.

W2 & Q1. Complete specification of the budget-constrained allocation.

We thank the reviewer for this helpful comment. To make the optimization fully reproducible, we have added the complete offline procedure to Appendix B. Algorithm 1 computes the reconstruction error ek(c,D)e_k(c, \mathcal{D}) of every candidate eviction set on the calibration prompts, and Algorithm 2 solves Eq. (11) on these errors. We provide Algorithm 2 below.

Algorithm 2. Budget-constrained allocation

Input:  reconstruction errors e_k(c, D), budget ratio b,
        initial multipliers λ = (λ_τ, …, λ_K) = 0, initial upper bound λ_0,
        growth factor ρ, bisection steps T, relative tolerance δ, iterations S
Output: nested eviction sets D* satisfying Eq. (11)

B ← b · 2LHC_{K−1},  D* ← ∅,  J* ← ∞,  s ← 0
repeat
  λ_prev ← λ,  s ← s + 1
  for k = τ, …, K do                                        # bisection on λ_k
    λ_lo ← 0,  λ_hi ← max(λ_k, λ_0),  λ_k ← λ_hi
    while M_k(DP(λ)) > B do                                 # expand the bracket
      λ_hi ← ρ · λ_hi,  λ_k ← λ_hi
    for t = 1, …, T do
      λ_k ← (λ_lo + λ_hi) / 2
      if M_k(DP(λ)) ≤ B then λ_hi ← λ_k else λ_lo ← λ_k
    λ_k ← λ_hi
  D ← DP(λ)
  J(D) ← Σ_{k=τ}^{K} Σ_{c∈C} e_k(c, D_k(c))
  if M_k(D) ≤ B for all k and J(D) < J* then
    D* ← D,  J* ← J(D)
until |λ_k − λ_prev,k| ≤ δ · |λ_prev,k| for all k, or s = S
return D*

function DP(λ)                                              # Eqs. (36)–(39)
  D ← minimizer of Eq. (35) for each c under λ
  M_k(D) ← Σ_{c∈C} Σ_{i<k} m_{c,i}^(k)   for k = τ, …, K
  return D
  • Multiplier selection. We assign one multiplier λk\lambda_k to the budget constraint of each generation scale kk. When the other multipliers are fixed, the KV cache usage MkM_k of the DP solution does not increase as λk\lambda_k grows, since a larger λk\lambda_k places a larger penalty on the memory at scale kk. This monotonicity allows us to find a λk\lambda_k that meets the budget at scale kk by bisection. Specifically, we first enlarge the upper bound by a factor of ρ\rho until MkM_k satisfies the budget, and then run TT bisection steps to approach the smallest such λk\lambda_k, so that the budget is used as fully as possible. Since updating one multiplier can change the usage at other scales through the nested eviction sets, we repeat these updates until the multipliers converge.
  • Discrete and nested allocation. For fixed multipliers, the DP in Eqs. (36)–(39) operates directly on discrete eviction sets and only allows transitions from D\mathcal{D} to D′⊇D\mathcal{D}' \supseteq \mathcal{D}. Its output is therefore always a discrete allocation that satisfies the nesting constraint, so no rounding or recovery step is needed.
  • Feasibility. As the reviewer notes, solving the relaxed subproblem alone does not guarantee feasibility for Eq. (11). For this reason, our procedure uses the Lagrangian relaxation only to search for candidate allocations and explicitly checks the budget at every generation scale. Among the candidates visited during the search, it returns the one with the lowest reconstruction error that satisfies every per-scale budget. The returned allocation is therefore feasible by construction. In practice, the first outer iteration already yields a feasible allocation in all our settings, and the subsequent iterations further reduce the reconstruction error.

Achieved cache usage. The table below reports the KV cache usage (MB) at each generation scale on Infinity-2B. Bold values indicate the scales where eviction is active. Once eviction starts, the usage stays within the limit at every scale for all budgets and batch sizes. Figures 5 and 6 in Appendix C visualize the eviction sets obtained by Algorithm 2 on Infinity-2B at the 10% and 4% budgets.

BudgetBatchScale 5Scale 6Scale 7Scale 8Scale 9Scale 10Scale 11Scale 12Scale 13Limit
20%142.890.8198.8390.8690.8974.7974.7974.7974.7975.0
20%8342.0726.01,590.03,126.05,526.07,798.97,798.97,798.97,798.97,800.0
10%142.890.8198.8390.8486.9486.9486.9486.9486.9487.5
10%8342.0726.01,590.03,126.03,898.93,898.93,898.93,898.93,898.93,900.0
4%142.890.8198.7198.7198.8198.7198.7198.7198.7198.8
4%8342.0726.01,589.71,589.91,590.01,590.01,589.91,589.81,589.81,590.0

We would sincerely appreciate it if the reviewer could refer to Appendix B, which also presents the full DP formulation and Algorithm 1. We hope that the added algorithms and these results resolve the reviewer's concern about reproducibility.

I thank the authors for their detailed response and additional experiments. The comparisons with FastVAR across different batch sizes provide further evidence of the proposed method’s effectiveness. Algorithms 1 and 2 also clarify the budget-constrained allocation procedure and resolve my concerns about this aspect of the paper. With these clarifications and additional results, I have no further concerns and am raising my score from 2 to 3.

Thank you for your thoughtful feedback and for raising the score. Your suggestions on related efficient next-scale generation methods, including HMAR, MVAR, Markov-VAR, InfinityStar, and FastVAR, helped us clarify the positioning of our work, which is now reflected in the related work. We will also keep the detailed description of the allocation procedure in the final version.

Reviewer_iCEcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes ResceneKV, a training-free KV-cache compression framework for VAR models. The key idea is to guide cache eviction using a closed-form reconstruction-error measure of the attention output induced by evicting source scales. Experiments on Infinity-2B/8B show consistent fidelity and quality improvements over recent VAR-specific baselines under tight KV budgets.

Critical strengths
  • The idea is interesing. The proposed method directly optimizs an analytically derived attention-output reconstruction error for eviction decisions and extend it to multi-scale VAR model, which is elegent.

  • The author give comprehensive comparisons including VAR-focused KV methods such as ScaleKV, HACK, HeatKV and a popular LLM heuristic KV pruning method SnapKV on two model scales Infinity-2B/8B

  • The paper is well-written and easy to follow

Critical weaknesses
  • Limited ablations on other hyperparameters, e.g., number of protected sink scales and guidance scales

  • Lack performance comparison with other cache-based VAR methods like FastVAR [A]

  • What is the perfromance on other VAR models like HART model?

[A] FastVAR: Linear Visual Autoregressive Modeling via Cached Token Pruning [ICCV25]

Questions for authors

What is the time cost for offline calibration in the proposed method?

2 comments

W1. Ablations on the number of sink scales and the CFG scale.

We conducted both ablations on Infinity-2B and added them to Appendix D (Tables 7 and 9, highlighted in blue).

Number of sink scales. Table R1 varies the number of sink scales that are always kept in the KV cache. One to three sink scales give nearly identical results at both budgets, with two or three performing best, whereas protecting four or five sink scales degrades every metric. The degradation is larger at the 4% budget, where PSNR drops from 22.80 dB to 22.38 dB with five sink scales. Since sink scales are never evicted but still occupy part of the budget, protecting more of them leaves less memory for the remaining source scales. We use three sink scales in all experiments, the same setting as HeatKV, so that the two methods are compared under identical sink protection.

Table R1. Effect of the number of sink scales (Infinity-2B, MS-COCO 2017).

Budget# Sink scalesFID ↓PSNR ↑SSIM ↑LPIPS ↓
10%11.4426.350.8760.062
21.4326.370.8760.062
3 (default)1.4326.370.8760.062
41.4526.270.8740.063
51.5026.060.8700.066
4%12.4122.810.8030.113
22.3922.830.8040.112
3 (default)2.4022.800.8030.113
42.4522.690.8000.116
52.5922.380.7910.123

CFG scale. All experiments in the paper use a CFG scale of 3.0. To examine whether KV cache sharing generalizes to other guidance strengths, we additionally evaluated ResceneKV with and without KV cache sharing at CFG scales of 2.0 and 4.0. As shown in Table R2, KV cache sharing improves FID, PSNR, SSIM, and LPIPS in all six settings, with PSNR gains of 0.09–0.18 dB.

Table R2. ResceneKV with and without KV cache sharing (Infinity-2B, MS-COCO 2017). Fidelity is measured against the uncompressed model at the same CFG scale.

CFGBudgetKV cache sharingFID ↓PSNR ↑SSIM ↑LPIPS ↓
2.010%✗1.3726.600.8830.059
✓1.3326.780.8860.057
4%✗2.3223.110.8130.106
✓2.2923.220.8170.104
3.010%✗1.4626.200.8730.064
✓1.4326.370.8760.062
4%✗2.4322.680.7990.116
✓2.4022.800.8030.113
4.010%✗1.5925.800.8620.070
✓1.5825.890.8640.069
4%✗2.6122.290.7860.125
✓2.5922.410.7890.123

W2. Comparison with FastVAR.

We have added FastVAR to the related work (Sec. 2, highlighted in blue) and also compared it directly with ResceneKV on the same Infinity-2B model. For FastVAR, we used the official code with its default setting. Since FastVAR cannot be set to a specific KV cache budget, we compare it based on its measured memory and latency. All efficiency results are measured on a single NVIDIA RTX 3090.

Table R3. Output fidelity on MS-COCO 2017 and generation quality on GenEval (Infinity-2B).

MethodBudgetFID ↓PSNR ↑SSIM ↑LPIPS ↓Two Obj.PositionColor Attri.Overall ↑
Full Cache100%––––0.8760.4150.5400.720
FastVAR–3.7622.210.7450.1510.8540.4080.5380.705
ResceneKV20%0.9129.120.9180.0390.8760.4070.5520.717
ResceneKV10%1.4326.370.8760.0620.8860.4080.5650.722
ResceneKV4%2.4022.800.8030.1130.8790.4070.5470.717

Table R4. Efficiency on Infinity-2B (NVIDIA RTX 3090). Latency is measured at batch size 1.

MethodBudgetBatchPeak GPU Mem. (GiB)KV Cache Mem. (GB)Latency (s)Speedup
Full Cache100%115.754.822.961.00×
Full Cache100%2 / 4 / 8OOM–––
FastVAR–1 / 2 / 4 / 811.61 / 13.96 / 20.54 / OOM1.65 / 3.30 / 6.61 / –1.382.14×
ResceneKV10%1 / 2 / 4 / 811.09 / 11.30 / 13.62 / 19.850.49 / 0.98 / 1.95 / 3.902.701.10×
ResceneKV4%1 / 2 / 4 / 811.06 / 11.25 / 11.78 / 16.180.20 / 0.40 / 0.80 / 1.592.671.11×

Even at a 4% budget, ResceneKV outperforms FastVAR on all fidelity metrics (FID 2.40 vs. 3.76, PSNR 22.80 vs. 22.21 dB) and on GenEval (0.717 vs. 0.705), while using 8.3× less KV cache memory. The memory advantage also grows with batch size. At batch size 4, ResceneKV uses 11.78 GiB versus 20.54 GiB for FastVAR, and at batch size 8, FastVAR runs out of memory while ResceneKV uses 16.18 GiB. FastVAR achieves a larger speedup (2.14× vs. 1.11×) since it is designed for acceleration, whereas ResceneKV is better suited when GPU memory is the bottleneck.

W3. Performance on other VAR models.

Beyond Infinity, we evaluate ResceneKV on three VAR models with different architectures and conditioning schemes. These are HART, a text-to-image model with a hybrid tokenizer, the original class-conditional VAR at two model sizes and resolutions, and STAR, another text-to-image next-scale prediction model. All methods are compared under the same KV cache budgets. Below we report the full-cache model, the strongest baseline HeatKV, and ResceneKV. The full results with all baselines are provided in Table 3 (Sec. 5.2) for HART and in Tables 11 and 12 (Appendix E, highlighted in blue) for STAR and VAR.

Table R5. HART on MS-COCO 2017 and GenEval.

MethodBudgetFID ↓PSNR ↑SSIM ↑LPIPS ↓GenEval ↑
Full Cache100%––––0.496
HeatKV20%3.8022.960.8040.1170.496
ResceneKV20%3.0224.440.8460.0900.491
HeatKV10%5.3620.140.7040.1870.495
ResceneKV10%5.5320.200.7050.1860.496
HeatKV4%11.6617.480.5850.3030.490
ResceneKV4%7.9918.290.6270.2560.497

Table R6. VAR-d24 (256×256) and VAR-d36 (512×512) on ImageNet class-conditional generation.

MethodBudgetd24 FID ↓d24 IS ↑d36 FID ↓d36 IS ↑
Full Cache100%2.17315.12.64294.4
HeatKV20%2.15310.23.48271.7
ResceneKV20%2.14312.22.58291.2
HeatKV10%2.30299.43.69262.6
ResceneKV10%2.20307.72.63286.0
HeatKV4%6.61221.24.79236.7
ResceneKV4%5.00243.83.15271.9

Table R7. STAR on MS-COCO 2017 and GenEval.

MethodBudgetFID ↓PSNR ↑SSIM ↑LPIPS ↓GenEval ↑
Full Cache100%––––0.503
HeatKV20%3.1121.910.7630.1200.505
ResceneKV20%2.8522.660.7890.1040.500
HeatKV10%4.5519.430.6780.1830.505
ResceneKV10%3.8920.110.7060.1610.505
HeatKV4%6.2517.490.6060.2590.501
ResceneKV4%5.2218.030.6300.2320.502

Across all three models, ResceneKV outperforms HeatKV in fidelity, and the gap generally widens as the budget decreases. On VAR-d36 at the 4% budget, ResceneKV reduces FID from 4.79 to 3.15, close to the full-cache FID of 2.64. On STAR, it improves FID, PSNR, SSIM, and LPIPS at every budget while keeping GenEval comparable to the full-cache model. The only exception is the FID on HART at the 10% budget, where the two methods are essentially tied and ResceneKV remains better on all other metrics. These results show that ResceneKV generalizes beyond Infinity to VAR models with different tokenizers, conditioning schemes, and resolutions.


Q1. Time cost of offline calibration.

We measured the calibration time of every method that requires calibration, under the 10% KV cache budget with the N=10N = 10 calibration prompts used in the paper. These results, together with those for N=1N = 1 and 55, are added to Appendix D (Table 8). SnapKV is not included, since it selects tokens online from the attention of each sample and requires no calibration. The time covers the whole procedure after the model is loaded, from encoding the prompts and generating the calibration samples while collecting statistics to producing the final eviction configuration. Infinity-2B and HART were measured on a single RTX 3090, and Infinity-8B on a single RTX 6000 Ada Generation.

ModelHACKScaleKVHeatKVResceneKV
Infinity-2B76.6 s86.2 s51.8 s83.9 s
Infinity-8B135.1 s128.8 s78.3 s129.2 s
HART49.4 s57.8 s22.7 s53.3 s

ResceneKV takes longer than HeatKV because it directly measures the output error after eviction for each layer and head, whereas HeatKV only accumulates attention statistics. Even so, calibration completes in about two minutes on a single GPU even for Infinity-8B, which is comparable to HACK and ScaleKV. Moreover, calibration is performed only once, and the resulting configuration is reused for all prompts at inference without any additional cost. In return for this one-time cost, ResceneKV preserves the full-cache output more faithfully than HeatKV, e.g., improving PSNR from 24.40 to 25.95 dB on Infinity-8B at the 10% budget (Table 1).

Reviewer_8JWEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ResceneKV, a training-free KV cache compression method for visual autoregressive image generation. The method estimates eviction-induced reconstruction errors and jointly allocates cache budgets across layers, attention heads, and generation scales. It further introduces KV cache sharing between conditional and unconditional CFG branches to reduce redundant storage. Experiments demonstrate improved reconstruction fidelity under constrained memory budgets. The approach is well motivated and experimentally promising, but some details of the optimization procedure and the relationship between local reconstruction errors and final generation quality require clarification.

Critical strengths

1.The reconstruction-error formulation provides a clear motivation for improving attention-based eviction and is supported by mathematical derivations. 2.The experiments cover multiple cache budgets and model architectures, showing consistent improvements in reconstruction fidelity with competitive inference efficiency.

Critical weaknesses

1.The solution to Eq. (11) is not fully specified. Appendix B describes dynamic programming for fixed Lagrange multipliers, but does not explain how these multipliers are updated or how the final solution satisfies all budget constraints. 2.The reconstruction error is measured at individual attention outputs, while the objective sums errors across layers and generation scales. It would be useful to examine how well this surrogate correlates with final image reconstruction error. 3.KV sharing is motivated by high conditional–unconditional cache similarity, but its sensitivity to CFG strength remains unclear. Testing different guidance scales would help establish when this sharing strategy is reliable. 4.In Table 7, ResceneKV slightly underperforms HeatKV in FID on HART at the 10% budget, unlike its consistent improvements on Infinity. A brief discussion of this discrepancy would help clarify its behavior across architectures.

Questions for authors

See Weaknesses

3 comments

W1. Update of the Lagrange multipliers and satisfaction of all budget constraints.

We thank the reviewer for this helpful comment. To describe how Eq. (11) is solved in full detail, we have added the complete offline procedure to Appendix B (highlighted in blue). Algorithm 1 computes the reconstruction error of every candidate eviction set on the calibration prompts, and Algorithm 2 solves Eq. (11) on these errors. We provide Algorithm 2 below.

Algorithm 2. Budget-constrained allocation

Input:  reconstruction errors e_k(c, D), budget ratio b,
        initial multipliers λ = (λ_τ, …, λ_K) = 0, initial upper bound λ_0,
        growth factor ρ, bisection steps T, relative tolerance δ, iterations S
Output: nested eviction sets D* satisfying Eq. (11)

B ← b · 2LHC_{K−1},  D* ← ∅,  J* ← ∞,  s ← 0
repeat
  λ_prev ← λ,  s ← s + 1
  for k = τ, …, K do                                        # bisection on λ_k
    λ_lo ← 0,  λ_hi ← max(λ_k, λ_0),  λ_k ← λ_hi
    while M_k(DP(λ)) > B do                                 # expand the bracket
      λ_hi ← ρ · λ_hi,  λ_k ← λ_hi
    for t = 1, …, T do
      λ_k ← (λ_lo + λ_hi) / 2
      if M_k(DP(λ)) ≤ B then λ_hi ← λ_k else λ_lo ← λ_k
    λ_k ← λ_hi
  D ← DP(λ)
  J(D) ← Σ_{k=τ}^{K} Σ_{c∈C} e_k(c, D_k(c))
  if M_k(D) ≤ B for all k and J(D) < J* then
    D* ← D,  J* ← J(D)
until |λ_k − λ_prev,k| ≤ δ · |λ_prev,k| for all k, or s = S
return D*

function DP(λ)                                              # Eqs. (36)–(39)
  D ← minimizer of Eq. (35) for each c under λ
  M_k(D) ← Σ_{c∈C} Σ_{i<k} m_{c,i}^(k)   for k = τ, …, K
  return D
  • How the multipliers are updated. We assign one multiplier λk\lambda_k to the budget constraint of each generation scale kk. When the other multipliers are fixed, the KV cache usage MkM_k of the DP solution does not increase as λk\lambda_k grows, since a larger λk\lambda_k places a larger penalty on the memory at scale kk. This monotonicity allows us to find a λk\lambda_k that meets the budget at scale kk by bisection. Specifically, we first enlarge the upper bound by a factor of ρ\rho until MkM_k satisfies the budget, and then run TT bisection steps to approach the smallest such λk\lambda_k, so that the budget is used as fully as possible. Since updating one multiplier can change the usage at other scales through the nested eviction sets, we repeat these updates until the multipliers converge.
  • How the final solution satisfies all budget constraints. Our procedure uses the Lagrangian relaxation only to search for candidate allocations and explicitly checks the budget at every generation scale. Among the candidates visited during the search, it returns the one with the lowest reconstruction error that satisfies every per-scale budget. The returned allocation is therefore feasible by construction. Since the DP only allows transitions from D\mathcal{D} to D′⊇D\mathcal{D}' \supseteq \mathcal{D}, the allocation also satisfies the nesting constraint without any rounding or recovery step. In practice, the first outer iteration already yields a feasible allocation in all our settings, and the subsequent iterations further reduce the reconstruction error.

Achieved cache usage. The table below reports the KV cache usage (MB) at each generation scale on Infinity-2B. Bold values indicate the scales where eviction is active. Once eviction starts, the usage stays within the limit at every scale for all budgets and batch sizes. Figures 5 and 6 in Appendix C visualize the resulting eviction sets at the 10% and 4% budgets.

BudgetBatchScale 5Scale 6Scale 7Scale 8Scale 9Scale 10Scale 11Scale 12Scale 13Limit
20%142.890.8198.8390.8690.8974.7974.7974.7974.7975.0
20%8342.0726.01,590.03,126.05,526.07,798.97,798.97,798.97,798.97,800.0
10%142.890.8198.8390.8486.9486.9486.9486.9486.9487.5
10%8342.0726.01,590.03,126.03,898.93,898.93,898.93,898.93,898.93,900.0
4%142.890.8198.7198.7198.8198.7198.7198.7198.7198.8
4%8342.0726.01,589.71,589.91,590.01,590.01,589.91,589.81,589.81,590.0

We would sincerely appreciate it if the reviewer could refer to Appendix B, which also presents the full DP formulation and Algorithm 1. We hope this resolves the reviewer's concern.

W2. Correlation between the surrogate objective and the final image reconstruction error.

We examined this in two ways on Infinity-2B under the 10% KV cache budget, using MS-COCO 2017.

Across allocations. We constructed eight allocations that all satisfy the same budget. The first is the ResceneKV allocation. Four are random perturbations of the ResceneKV allocation. We randomly select 10%, 20%, 30% or 40% of the KV cache that ResceneKV retains and evict it. We then retain the same amount of KV cache, randomly selected from what ResceneKV evicts. Each perturbed allocation therefore uses the same budget as ResceneKV, but a part of what it retains is chosen at random. The remaining three are the HeatKV allocation, an allocation that enforces the budget on every attention head separately, and an allocation that keeps or evicts each source scale as a whole, retaining entire scales from the most recent one until the budget is filled. For each allocation we computed the value of our objective in Eq. 11 and generated the images to measure the reconstruction error against the full-cache images.

AllocationObjective ↓FID ↓PSNR ↑SSIM ↑LPIPS ↓
ResceneKV242.31.4326.370.8760.062
Random perturbation 10%459.11.7725.030.8490.079
HeatKV allocation778.61.6425.410.8570.073
Random perturbation 20%1012.33.1222.740.7880.124
Random perturbation 30%1182.33.0223.520.8070.109
Random perturbation 40%1622.73.2523.360.8020.112
Per-head budget3330.73.9422.300.7750.135
Whole-scale recency4727.44.6920.990.7460.164

The objective is strongly correlated with all four fidelity metrics, with Spearman correlations of 0.95 with FID, -0.90 with PSNR, -0.90 with SSIM, and 0.90 with LPIPS.

Per image. We further evaluated the objective separately on each MS-COCO image, using the reconstruction errors measured on that image alone, and computed the Spearman correlation between the objective and the reconstruction error of that image over the eight allocations. The table below summarizes the distribution of this per-image correlation.

MetricMeanMedianWeakest 5% of images
PSNR-0.90-0.90-0.83
SSIM-0.90-0.90-0.88
LPIPS0.900.900.88

The correlation has the expected sign on every image and remains strong even for the weakest 5% of images. The objective is therefore strongly correlated with the final reconstruction error not only on average but also on individual images.

W3. Sensitivity of KV cache sharing to the CFG scale.

All experiments in the paper use a CFG scale of 3.0. To examine whether KV cache sharing generalizes to other guidance strengths, we additionally repeated both the similarity analysis and the ablation on KV cache sharing at CFG scales of 2.0 and 4.0.

The cross-branch similarity is stable across CFG scales. Table R1 reports the cosine similarity between the cached keys and values of the conditional and unconditional branches in the uncompressed Infinity-2B model, averaged over all layers and heads, for each source scale. The similarity profile is nearly unchanged across the three CFG scales, and the values at each source scale differ by at most 0.004 between CFG 2.0 and 4.0. In all cases, the two branches differ most at the coarse scales and become more similar toward the finer scales. At the finest scales, where KV cache sharing is applied, the similarity remains at least 0.999 for keys and 0.995 for values regardless of the CFG scale.

Table R1. Cosine similarity between the conditional and unconditional KV caches (Infinity-2B).

CFGs1s_1s2s_2s3s_3s4s_4s5s_5s6s_6s7s_7s8s_8s9s_9s10s_{10}s11s_{11}s12s_{12}
2.0K0.99880.99700.98520.99100.99490.99620.99740.99820.99860.99910.99920.9994
V0.99250.98130.90950.94540.96930.97660.98420.98930.99180.99420.99560.9967
3.0K0.99880.99700.98590.99160.99480.99610.99760.99820.99860.99910.99920.9994
V0.99250.98160.91220.94960.96900.97680.98550.98950.99200.99410.99570.9968
4.0K0.99880.99700.98600.99110.99550.99640.99770.99830.99870.99910.99930.9994
V0.99250.98180.91330.94660.97290.97870.98620.99010.99230.99430.99580.9968

KV cache sharing improves fidelity at every CFG scale. Table R2 compares ResceneKV with and without KV cache sharing under the same KV cache budget. KV cache sharing improves FID, PSNR, SSIM, and LPIPS in all six settings, not only at the default CFG scale of 3.0 but also at 2.0 and 4.0, with PSNR gains of 0.09–0.18 dB. This gain arises because KV cache sharing frees memory at the finest scales, which is then reallocated to the rest of the cache under the same total budget.

Table R2. ResceneKV with and without KV cache sharing on MS-COCO 2017 (Infinity-2B). Fidelity is measured against the uncompressed model at the same CFG scale, so the values are not comparable across CFG scales.

CFGBudgetKV cache sharingFID ↓PSNR ↑SSIM ↑LPIPS ↓
2.010%✗1.3726.600.8830.059
✓1.3326.780.8860.057
4%✗2.3223.110.8130.106
✓2.2923.220.8170.104
3.010%✗1.4626.200.8730.064
✓1.4326.370.8760.062
4%✗2.4322.680.7990.116
✓2.4022.800.8030.113
4.010%✗1.5925.800.8620.070
✓1.5825.890.8640.069
4%✗2.6122.290.7860.125
✓2.5922.410.7890.123

These results show that both the high cross-branch similarity and the benefit of KV cache sharing generalize beyond the default CFG scale of 3.0. We have added Table R2 to Appendix D (Table 9, highlighted in blue).

W4. Behavior of ResceneKV across architectures (Infinity vs. HART).

At the 10% budget in Table 3 (Table 7 in the original submission), ResceneKV and HeatKV are essentially tied on HART. ResceneKV is slightly better on PSNR, SSIM, LPIPS, and GenEval Overall, and only its FID is higher (5.53 vs. 5.36). At the 20% and 4% budgets, ResceneKV clearly outperforms HeatKV on all fidelity metrics.

We attribute the narrower margin at 10% to a structural property of HART that affects both methods. Infinity injects text through cross-attention, so its self-attention cache contains only image tokens. In contrast, HART has no cross-attention and places 300 text tokens inside self-attention, where they share a single softmax with the image tokens. In our measurement, these text tokens receive approximately 25% of the attention mass, and this proportion varies across prompts. As a result, the relative importance of each image source scale depends more strongly on the prompt in HART than in Infinity. Since both HeatKV and ResceneKV determine a static allocation from a small calibration set, this prompt dependence limits how precisely either allocation transfers to unseen prompts, which can narrow the gap between the two methods at individual budgets, as observed at 10%.

Reviewer_S8dqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Visual autoregressive (VAR) models are a powerful paradigm for image generation, but their next-scale prediction formulation requires retaining tokens from all previous scales in the KV cache, leading to prohibitive memory overhead at finer scales. The authors propose ResceneKV, a cache management method that jointly addresses KV eviction and classifier-free guidance (CFG) cache sharing. Prior work largely relies on attention statistics to rank eviction candidates, but the authors show that similar attention mass can correspond to substantially different reconstruction errors. Instead, they score eviction decisions using the reconstruction error induced by cache removal, estimated offline on a small calibration set. The paper further observes that the conditional and unconditional CFG branches become increasingly similar at later scales, enabling cache sharing between the two branches. The sharing error is incorporated into the same reconstruction-based objective, resulting in a unified budget-allocation framework for KV cache eviction and CFG cache sharing.

Critical strengths

The paper provides a convincing motivation for moving beyond attention-based eviction heuristics. The empirical observation that similar attention mass can lead to very different reconstruction errors is compelling and naturally motivates the proposed reconstruction-error objective. Eviction and CFG sharing are handled in one budget-allocation problem rather than as separate tricks.

Critical weaknesses
  • The method achieves clear improvements on reconstruction-based metrics, which is expected given that preserving the original model outputs is directly optimized by the proposed objective. However, the picture is much less clear on generation-quality benchmarks such as GenEval, DPG, HPS, and ImageReward, where differences between methods are often small, inconsistent, or potentially within statistical noise. As a result, it is difficult to conclude that ResceneKV is the best overall approach when generation quality, rather than reconstruction fidelity, is the primary objective.

  • The experimental evaluation is primarily focused on Infinity, with results on HART only reported in the appendix and not discussed in the main paper. While these results are encouraging, it would be important to evaluate the method across a broader set of model families, including the original VAR architecture. In addition, the proposed allocation relies on an offline calibration procedure, yet the robustness of the learned allocation to shifts in data distribution, prompt domain, or model family is not thoroughly explored.

I am willing to increase my score if the authors can address the weaknesses and questions in the rebuttal.

Questions for authors
  • Can you report results on the original VAR model as well? Infinity-2B and Infinity-8B are simply two scales of the same model family and rely on the bitwise tokenization specific to Infinity
  • The results on GenEval, DPG, and ImageReward are often very close across methods. Could you report confidence intervals or error bars (across sampling seeds) to clarify whether the observed differences are statistically meaningful?
  • Could you compare methods from an efficiency perspective beyond inference latency? For ResceneKV specifically, how long does calibration take, how expensive is solving the budget-allocation problem, and how do these costs scale with model size, number of scales, and number of calibration prompts?
  • How are the calibration prompts obtained from Qwen3-VL-2B-Instruct? What prompting strategy is used? Does the prompt distribution matter? For example, if calibration is performed on one domain, does performance degrade on another domain? Since evaluation uses COCO captions, have you tested calibration using real images and teacher-forced token maps rather than model-generated samples?
  • The paper lists the fixed allocation strategy as a limitation. Could you elaborate on possible approaches for making the allocation adaptive on a per-sample basis? It would be particularly interesting to visualize cases where adaptive allocation would differ substantially from the fixed allocation, such as text rendering, highly crowded scenes, or minimalist compositions.
  • The memory numbers for Infinity-8B reported in Figure 1 and Table 1 appear inconsistent. Could you clarify the source of this discrepancy and reconcile the reported values?
6 comments

Q1 & W2. Results on other VAR model families.

We thank the reviewer for this suggestion. To verify that ResceneKV is not specific to Infinity or its bitwise tokenization, we evaluated it on two additional model families beyond HART, namely the original VAR [1] and STAR [2]. All methods are compared under the same KV cache budgets, and the results are added to Appendix E (Tables 11 and 12, highlighted in blue). We have also moved the HART results to the main paper (Sec. 5.2, Table 3).

VAR. The original VAR differs from Infinity in both its tokenizer and its conditioning. It uses a multi-scale VQ tokenizer with a shared codebook instead of bitwise tokenization, and it performs class-conditional generation on ImageNet. We use the official VAR-d24 and VAR-d36 checkpoints, which generate images at 256×256 and 512×512, respectively.

Table R1. VAR on ImageNet class-conditional generation. Bold indicates the best result at each budget.

MethodBudgetd24 FID ↓d24 IS ↑d24 Prec. ↑d24 Rec. ↑d36 FID ↓d36 IS ↑d36 Prec. ↑d36 Rec. ↑
Full Cache100%2.17315.10.820.592.64294.40.820.57
SnapKV20%3.48258.90.770.608.20204.80.770.54
HACK20%2.55284.10.790.607.03212.00.780.56
ScaleKV20%2.79277.40.790.597.09215.50.780.55
HeatKV20%2.15310.20.820.593.48271.70.810.57
ResceneKV20%2.14312.20.820.582.58291.20.820.55
SnapKV10%7.21208.00.710.5912.06170.70.740.55
HACK10%5.84223.60.730.6013.45155.40.740.57
ScaleKV10%5.31228.20.740.596.17223.10.780.56
HeatKV10%2.30299.40.810.593.69262.60.800.55
ResceneKV10%2.20307.70.820.582.63286.00.820.55
SnapKV4%66.8531.20.380.5826.9797.20.650.56
HACK4%21.74114.80.610.6031.1780.80.610.56
ScaleKV4%62.0934.50.400.5828.1592.40.630.58
HeatKV4%6.61221.20.730.584.79236.70.790.56
ResceneKV4%5.00243.80.750.593.15271.90.810.55

At the 20% and 10% budgets, ResceneKV performs on par with the full-cache model. At the 4% budget, it achieves the best FID, IS, and precision on both models, while recall remains similar across most methods.

STAR. STAR [2] extends the scale-wise paradigm of VAR to text-to-image generation at up to 1024×1024. Unlike Infinity, it predicts discrete token maps following VAR instead of bitwise tokens, conditions on CLIP text features through a pooled start token and additional cross-attention layers, and replaces absolute position embeddings with a normalized 2D RoPE. We use the official STAR checkpoint with depth 30 (1.7B parameters) at 1024×1024 and evaluate it on MS-COCO 2017 and GenEval with the same metrics as in our main experiments.

Table R2. STAR on MS-COCO 2017 and GenEval. Bold as in Table R1.

MethodBudgetFID ↓PSNR ↑SSIM ↑LPIPS ↓GenEval ↑
Full Cache100%––––0.503
SnapKV20%7.7419.040.6630.2040.491
HACK20%42.0717.790.6260.3510.375
ScaleKV20%8.3519.310.6740.1940.492
HeatKV20%3.1121.910.7630.1200.505
ResceneKV20%2.8522.660.7890.1040.500
SnapKV10%27.8217.420.6100.3050.431
HACK10%48.0517.050.5990.3850.352
ScaleKV10%14.0517.650.6160.2680.479
HeatKV10%4.5519.430.6780.1830.505
ResceneKV10%3.8920.110.7060.1610.505
SnapKV4%49.6916.350.5810.3960.354
HACK4%137.8415.070.5400.5660.119
ScaleKV4%48.4016.490.5840.3820.361
HeatKV4%6.2517.490.6060.2590.501
ResceneKV4%5.2218.030.6300.2320.502

ResceneKV achieves the best FID, PSNR, SSIM, and LPIPS at every budget, and its FID margin over HeatKV grows as the budget decreases (0.26, 0.66, and 1.03 at 20%, 10%, and 4%). GenEval remains close to the full-cache model at all budgets.

Together with HART in Table 3, these results show that ResceneKV generalizes across model families with different tokenizers, conditioning schemes, and resolutions.

[1] K. Tian et al. Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction. NeurIPS, 2024. [2] X. Ma et al. STAR: Scale-wise Text-conditioned AutoRegressive Image Generation. arXiv:2406.10797, 2024.

Q2. Seed variance on the generation-quality benchmarks.

We report the mean ± standard deviation (SD) over five sampling seeds. The paper reports a single run with the default seed, so its values differ slightly.

Table R5. Infinity-2B, five seeds.

MethodBudgetGenEvalDPGImageReward
Full Cache100%0.716±0.00782.35±0.130.922±0.014
HACK10%0.704±0.00682.09±0.270.896±0.014
ScaleKV10%0.710±0.00682.30±0.150.913±0.013
HeatKV10%0.715±0.00482.34±0.130.923±0.014
ResceneKV10%0.718±0.00582.43±0.100.922±0.016
HACK4%0.704±0.00581.59±0.140.865±0.014
ScaleKV4%0.710±0.00682.17±0.040.904±0.013
HeatKV4%0.711±0.00382.34±0.100.920±0.013
ResceneKV4%0.713±0.00582.31±0.130.922±0.014

Table R6. Infinity-8B, five seeds.

MethodBudgetGenEvalDPGImageReward
Full Cache100%0.783±0.00586.07±0.031.051±0.010
ScaleKV10%0.785±0.00186.08±0.121.027±0.009
HeatKV10%0.785±0.00686.06±0.041.028±0.010
ResceneKV10%0.783±0.00786.09±0.101.039±0.008
ScaleKV4%0.776±0.00586.07±0.071.002±0.013
HeatKV4%0.785±0.00686.05±0.121.023±0.011
ResceneKV4%0.782±0.00686.02±0.031.027±0.011

Generation quality is preserved. On Infinity-2B, all three scores of ResceneKV stay within one SD of the full-cache model at both budgets. On Infinity-8B, its GenEval and DPG stay within 0.001 and 0.05 of the full-cache model, and its ImageReward is the highest among the eviction methods. As the reviewer notes, the strongest methods mostly lie within one SD of each other on GenEval and DPG, so we do not claim superiority there. These benchmarks still show clear drops when quality degrades, such as HACK on Infinity-2B (ImageReward 0.865) and ScaleKV on Infinity-8B (1.002) at the 4% budget.

Fidelity to the full-cache output is clearly improved. Similar scores on these benchmarks do not mean that the generated images stay close to the full-cache output. The fidelity metrics in Table 1 measure this difference, and ResceneKV outperforms HeatKV at every budget on both models. At the 10% budget, it reduces FID from 1.64 to 1.43 and gains 0.96 dB PSNR on Infinity-2B, and reduces FID from 1.81 to 1.31 and gains 1.55 dB PSNR on Infinity-8B. ResceneKV thus maintains generation quality while preserving the full-cache output far more faithfully than existing methods.


Q3. Efficiency beyond inference latency: calibration time, cost of the budget allocation, and scaling.

We measured the offline calibration time at the 10% budget and added it to Appendix D (Table 8). Each entry splits it into the collection time for running the model on the calibration prompts and collecting statistics, and the allocation time for solving the budget allocation. SnapKV needs no calibration.

Table R3. Collection/allocation time (s) with N=10N=10 prompts.

ModelHACKScaleKVHeatKVResceneKV
Infinity-2B76.6/0.086.2/0.051.6/0.283.0/0.9
Infinity-8B135.1/0.0128.8/0.078.1/0.2127.9/1.3
HART49.4/0.057.8/0.022.5/0.252.2/1.1
STAR77.5/0.079.1/0.041.1/0.383.0/1.5

Table R4. Collection/allocation time (s) on Infinity-8B with NN prompts.

NNHACKScaleKVHeatKVResceneKV
115.0/0.015.0/0.09.7/0.215.6/0.4
568.2/0.065.0/0.039.5/0.264.1/0.5
10135.1/0.0128.8/0.078.1/0.2127.9/1.3
15202.5/0.0192.9/0.0116.5/0.2191.7/0.4
20269.3/0.0257.7/0.0155.9/0.2255.2/0.5

Calibration time and model size. ResceneKV completes calibration in about two minutes even for Infinity-8B, comparable to HACK and ScaleKV. It takes longer than HeatKV because it directly measures the output error after eviction for each layer and head. For every method, most of the time is spent running the model, so the cost grows with model size. Calibration is performed once and reused for all prompts.

Cost of the budget allocation. In Tables R3 and R4, the allocation takes at most 1.5 s, a small fraction of the total.

Scaling with the number of prompts. As shown in Table R4, the collection time grows linearly with NN for all methods. The allocation time does not grow with NN, since it operates on errors already averaged over the calibration set.

Scaling with the number of scales. More scales make the model run longer for every method. ResceneKV also evaluates all 2n2^{n} subsets of the nn evictable source scales, which grows exponentially in nn, but nn is small in practice. In Infinity (K=13K=13), excluding the three sink scales and the final scale leaves n=9n=9, giving 512 candidates.

Q4. Calibration prompts and robustness to the calibration distribution.

How the calibration prompts are obtained. We use Qwen3-VL-2B-Instruct as a text-only generator with the following fixed instruction.

Write one vivid prompt for a text-to-image model. It must be a single English sentence describing a concrete visual scene: name the subject, its appearance, what it is doing, and the setting. Do not use lists, quotes, numbering, or any preamble. Reply with the sentence only.

We sample 10 prompts with a temperature of 0.9 and nucleus sampling, using fixed per-attempt seeds for reproducibility. A candidate is rejected if it matches any evaluation prompt from MS-COCO 2017, GenEval, DPG-Bench, HPSv2.1, or ImageReward, or another accepted prompt. The instruction is deliberately generic and does not target any benchmark, so the resulting prompts are general scene descriptions, e.g., "A golden retriever with a white chest and floppy ears sits on a sun-warmed stone bench in a quiet park, gazing at a flock of bluebirds fluttering above a meadow of wildflowers."

Effect of the calibration domain. To examine whether the prompt distribution matters, we restricted the scene in the same instruction to a single domain, namely animals, landscapes, or food, calibrated ResceneKV on each set of prompts, and evaluated all resulting allocations on MS-COCO 2017 and GenEval with Infinity-2B at the 10% budget. Since MS-COCO 2017 and GenEval cover a wide range of content beyond each calibration domain, this setting directly tests whether an allocation calibrated on one domain transfers to others. As shown in Table R7, domain-specific calibration yields nearly identical results to the default calibration, with FID changing by at most 0.02 and GenEval Overall by at most 0.004. To compare the allocations directly, we also report the Jaccard index between the eviction sets obtained from each calibration and those from the default calibration. The Jaccard index is 0.957 or higher for all three domains, which suggests that an allocation calibrated on a set of prompts depends little on the content domain of that set.

Calibration with real images and teacher-forced token maps. Following the reviewer's suggestion, we also calibrated ResceneKV on 10 real images from the MS-COCO 2017 training set, which does not overlap with the validation set used for evaluation, by encoding them into token maps and running the model with teacher forcing. Among all calibration settings, this one yields the lowest fidelity, the lowest GenEval Overall score, and the lowest Jaccard index with the default allocation, although the differences remain small (FID 1.50 vs. 1.43, GenEval 0.717 vs. 0.722, Jaccard index 0.941). We attribute this to a distribution mismatch. At inference, the KV cache is built from the model's own generated tokens, whereas teacher forcing feeds token maps derived from real images, which the model would not necessarily produce. The reconstruction errors measured under teacher forcing therefore reflect the inference-time KV cache less faithfully than those measured on model-generated samples. This result supports our choice of calibrating on samples generated by the model itself.

Table R7. Effect of the calibration data on Infinity-2B at the 10% budget.

MethodCalibration dataFID ↓PSNR ↑SSIM ↑LPIPS ↓Two Obj.PositionColor Attri.Overall ↑Jaccard ↑
Full Cache–––––0.8760.4150.5400.720–
ResceneKVDefault1.4326.370.8760.0620.8860.4080.5650.722–
ResceneKVAnimal1.4526.310.8750.0630.8790.4150.5500.7200.983
ResceneKVLandscape1.4526.330.8750.0630.8840.4070.5520.7180.974
ResceneKVFood1.4526.310.8750.0630.8810.4170.5400.7210.957
ResceneKVReal images (teacher-forced)1.5026.150.8720.0650.8690.4050.5450.7170.941

These results indicate that ResceneKV is robust to the choice of calibration data.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (Revision summary)

We sincerely thank all reviewers for their constructive feedback. We have revised the paper accordingly, and all changes are highlighted in blue. Table and appendix numbers below refer to the revised paper.

Summary of revisions

  • Positioning against efficient next-scale generation (fFFb). We added a new related-work paragraph, Efficient Next-Scale Generation (Sec. 2). It positions ResceneKV against architectural redesigns that require retraining (HMAR, MVAR, Markov-VAR, InfinityStar) and against FastVAR, which targets inference latency and reduces the KV cache only as a by-product.
  • Batch size for KV cache memory (S8dq). We added to the caption of Table 1 that KV cache memory is measured at batch size 8. This resolves the apparent mismatch with Figure 1, which reports memory at batch size 4.
  • HART results in the main paper (S8dq). We moved the HART results from the appendix to the main paper (Table 3 and "Evaluation on HART" in Sec. 5.2) and added HART to the experimental setup (Sec. 5.1).
  • Complete solution procedure for Eq. (11) (8JWE, fFFb). We added Algorithm 1 for the reconstruction-error computation and Algorithm 2 for the budget-constrained allocation, together with all hyperparameters (Appendix B). We also visualize the resulting eviction sets across generation scales on Infinity-2B at the 10% and 4% budgets, which shows the removal order and confirms that the budget is satisfied at every scale (Appendix C, Figures 5 and 6).
  • Ablations on sink scales and the CFG scale (8JWE, iCEc). We added an ablation on the number of sink scales (Table 7) and an evaluation of KV cache sharing at CFG scales of 2.0, 3.0, and 4.0 (Table 9) to Appendix D.
  • Calibration time (S8dq, iCEc). We added the offline calibration time of all calibration-based methods on Infinity-2B, Infinity-8B, and HART for different numbers of calibration prompts (Appendix D, Table 8). ResceneKV completes calibration in about two minutes even for Infinity-8B, comparable to HACK and ScaleKV.
  • Generalization to other VAR models (S8dq, iCEc). We added results on STAR [1] (Table 11) and on the original VAR-d24 and VAR-d36 (Table 12) to Appendix E. Together with HART, these results cover models with different tokenizers, conditioning schemes, and resolutions.
  • Discussion on per-sample adaptive allocation (S8dq). We added a discussion of when a per-prompt allocation deviates from the fixed allocation, with an analysis across scene types (Appendix F, Figure 7).

Corrections to existing tables

While preparing the revision, we found two errors in tables of the original submission. We sincerely apologize for these mistakes and have replaced both tables with the correct results (highlighted in blue).

  • Table 6 (Table 5 in the original submission). The sensitivity to the number of calibration prompts was reported from an earlier version of our experiments. We replaced it with the results of the final experimental setting, which is the same setting used for all other results in the paper. With this correction, the result for N=10N=10 matches Table 1 (FID 1.43, PSNR 26.37 dB).
  • Table 10 (Table 6 in the original submission). The HPSv2.1 scores of Infinity-8B at the 20% and 4% budgets had been mistakenly copied from those of Infinity-2B. We replaced them with the correct Infinity-8B results.

These corrections do not affect any claim of the paper. In the corrected Table 6, FID stays between 1.43 and 1.45 and PSNR between 26.30 and 26.37 dB across 1 to 20 calibration prompts, so performance remains stable with respect to the calibration set size, as originally stated. In the corrected Table 10, ResceneKV still maintains HPSv2.1 scores close to the full-cache model and remains competitive with the baselines at every budget. All main results in Table 1 are unaffected.

We hope these revisions address the reviewers' concerns, and we would be glad to clarify any remaining questions.

[1] X. Ma et al. STAR: Scale-wise Text-conditioned AutoRegressive Image Generation. arXiv:2406.10797, 2024.

AuthorsOpenReview ↗

Author Final Remarks

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all reviewers for their time and thoughtful review of our paper. The constructive feedback throughout the discussion has helped us substantially improve the paper.


Summary of the key issues and our responses

  • Generalization beyond Infinity (S8dq, iCEc, 8JWE). We moved the HART results to the main paper and added results on the original VAR models (VAR-d24, d36) and STAR [1]. These models differ from Infinity in their tokenizers, conditioning schemes, and resolutions, and ResceneKV achieves the best fidelity among eviction methods in nearly every setting.
  • Solution procedure for the budget-constrained allocation (8JWE, fFFb). We provided the complete procedure for updating the Lagrange multipliers and showed that the returned allocation satisfies every per-scale budget by construction, with the achieved KV cache usage staying within the limit at every generation scale.
  • Validity of the reconstruction-error objective (8JWE). We showed that our objective is strongly correlated with the final reconstruction error, both across allocations (Spearman correlation of 0.95 with FID) and on individual images.
  • Positioning and comparison with related approaches (fFFb, iCEc). We clarified that ResceneKV compresses the KV cache of pretrained VAR models without retraining, unlike architectural redesigns such as HMAR, MVAR, Markov-VAR, and InfinityStar, and unlike FastVAR, which targets inference latency. At the 4% budget, ResceneKV outperforms FastVAR on all fidelity metrics and on GenEval while using 8.3× less KV cache memory, and it outperforms a recent-scale retention baseline by a large margin.
  • Ablations on sink scales and the CFG scale (8JWE, iCEc). ResceneKV performs nearly identically with one to three sink scales, while protecting more sink scales degrades fidelity because they occupy part of the budget. The cross-branch similarity remains stable at CFG scales of 2.0, 3.0, and 4.0, and KV cache sharing improves every fidelity metric at all of them.
  • Calibration cost and calibration data (S8dq, iCEc). ResceneKV completes calibration in about two minutes even for Infinity-8B, comparable to HACK and ScaleKV, and calibration is performed only once. Allocations calibrated on different prompt domains are nearly identical, and calibrating on real images with teacher forcing confirms our choice of model-generated samples.
  • Reliability of the generation-quality evaluation (S8dq). Over five sampling seeds, ResceneKV preserves the generation quality of the full-cache model while clearly outperforming existing methods in fidelity to the full-cache output.
  • Limitation of the fixed allocation (S8dq). We analyzed when a per-prompt allocation deviates from the fixed allocation and discussed possible approaches for input-adaptive allocation.
  • Clarity of the reported memory (S8dq). We clarified that the KV cache memory in Figure 1 and Table 1 is measured at different batch sizes.

Revisions to the paper (highlighted in blue)

  • A new related-work paragraph, Efficient Next-Scale Generation (Sec. 2)
  • The batch size of the KV cache memory in the caption of Table 1
  • HART results in the main paper (Table 3)
  • Algorithms 1 and 2 for the offline calibration and budget-constrained allocation (Appendix B)
  • Visualization of the eviction sets across generation scales (Appendix C, Figures 5 and 6)
  • Ablations on sink scales and the CFG scale, and the calibration time (Appendix D, Tables 7 to 9)
  • Results on other VAR models (Appendix E, Tables 11 and 12)
  • Discussion on per-sample adaptive allocation (Appendix F, Figure 7)

Outcome of the discussion

Reviewers S8dq (2 → 3) and fFFb (2 → 3) have indicated that their concerns are resolved. Reviewers 8JWE (3 → 3) and iCEc (3 → 3) recognized the strengths of our work in their initial reviews. Although they have not yet responded during the discussion, we have addressed the key issues they raised, including the complete allocation algorithm, the correlation between our objective and the final reconstruction error, the additional ablations, the results on other VAR models, and the calibration time. We therefore hope that their concerns have also been resolved.

We are grateful for the constructive discussion, which has substantially strengthened our work.

Best regards, The Authors


[1] X. Ma et al. STAR: Scale-wise Text-conditioned AutoRegressive Image Generation. arXiv:2406.10797, 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.