acceptodds
Withdrawn from ICLR 2027

PACE-VL: Preserving Attention via Clustered Evictions for Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models that ingest high-resolution images or multi-page documents produce long, predominantly visual key–value (KV) caches, making decode-time memory and latency primary bottlenecks. Under a strict cache budget, the central challenge is preserving task quality when the vast majority of physical slots must be aggressively evicted. We introduce PACE-VL, a training-free KV-cache compression method that jointly designs retention, residual aggregation, and decode-time reading. Specifically, query states from the question or instruction tokens score the history to retain the highest-scoring entries, called evidence tokens, with their original keys and values. The remaining historical entries are clustered into anchors, cache entries that each store a selected original key, an aggregated value sum, and a multiplicity count—the number of tokens represented. At decode time, attention incorporates cluster multiplicities into normalization, so each cluster maintains its aggregate contribution under a shared-anchor-key approximation. Evidence tokens, anchors, and a short, uncompressed recent window occupy a cache of exactly the budgeted size. Across five visual-heavy benchmarks on three open VLMs spanning the Qwen and Llama-3 lineages, a single fixed configuration of PACE-VL without any fine-tuning outperforms all compared compression baselines in all 21 evaluated settings at a 10% cache budget. It matches or exceeds full-cache decoding in 11 settings and otherwise trails it by at most 1.87 points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV-cache compression, vision-language models, training-free inference, attention, token aggregation, long-context VLMs, decode-time efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_qfVROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PACE-VL is a training-free method for compressing the prefill KV cache of vision-language models. It retains exact evidence KV pairs selected using task-field queries, assigns residual tokens to original-key anchors, and stores aggregate values and token counts for count-corrected decoding. The compressed prompt cache has an exact slot budget, while generated tokens are appended separately. Evaluation covers three VLMs and five benchmarks, with component ablations and eager-backend efficiency measurements.

Critical strengths
  1. Consistent quality preservation under the evaluated compression protocol. At the nominal 10% budget, Table 1 reports the best score among the compared compressed implementations in all 21 settings, with a median margin of 3.06 points over the strongest compressed competitor. PACE-VL matches or exceeds the corresponding full-cache score in 11 settings and trails by at most 1.87 points otherwise. These results suggest practical potential.

  2. A coherent construction with relevant controls. The exact-slot construction and count-corrected representation are clearly specified. Table 7 includes equal-slot controls for residual aggregation, multiplicity, and anchor choices, while Table 4 reports generation latency and decoding throughput. These checks help assess both the mechanism and its practical costs.

Critical weaknesses
  1. Insufficient differentiation from prior clustered and merged KV-cache methods. KVMerger already preserves important KV entries while merging the remainder, and CentroidKV uses clustered representations with count-corrected attention. Multipole combines exact attention with count-aware cluster approximations, although it retains the full original cache. PACE-VL’s main incremental choices appear to be task-field selection and original-key anchors, whose significance beyond these precedents is insufficiently established.

  2. Unclear comparability of actual cache budgets. Nominal retention ratios can correspond to different realized cache sizes, but actual prompt-KV storage is not reported alongside the main quality comparisons. It therefore remains unclear whether PACE-VL’s advantage over the strongest baselines holds at comparable storage cost.

  3. Limited evidence for the general benefit of task-field selection. In Table 7, task-field queries improve the macro-average over Tail32 by only 0.29 points, with the gain driven by GovReport; excluding this dataset, the full method scores 0.48 points lower. This comparison covers only one model and context length, leaving the broader benefit of task-field selection insufficiently established.

Questions for authors
  1. Please clarify the main increments over KVMerger and CentroidKV. What does Table 7’s centroid-key control establish about the representative-key choice, and which differences from the prior algorithms does it not isolate? Please also distinguish PACE-VL’s compressed-storage objective from Multipole Attention’s full-cache setting.

  2. For Table 1’s Qwen3-VL-8B/K32 comparison at the nominal 10% budget, could you report the realized prompt-KV footprint at the start of decoding, aggregated across layers and heads and including auxiliary state or retained KV buffers? Please clarify whether PACE-VL and HybridKV have comparable storage costs, and how any differences should qualify the interpretation of the reported quality gap.

  3. For the GovReport setting in Table 7, please clarify which prompt tokens the task-field and Tail32 probes cover, and whether the query-source change also alters the evidence/anchor allocation. How do these details explain the large score difference, and what scope of benefit is supported by the task-dependent results already reported?

Reviewer_wcrKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PACE-VL compresses the prompt KV cache of a vision–language model without additional training. It keeps a recent window and tokens selected using task-field queries, then groups the remaining tokens around retained key anchors. Each group stores aggregated values and a token count, which is used to adjust attention during decoding. The paper proves exactness when all keys in a group match its anchor and bounds each token’s logit perturbation by its distance from the anchor in key space. Evaluation covers three VLMs across five benchmarks. At the tested budgets, PACE-VL generally stays close to full-cache performance and improves on the compression baselines. New tokens are appended during generation.

Critical strengths

1.The main results are the strongest part of the paper. PACE-VL beats the best compressed baseline in all 21 settings in Table 1, by 0.63–5.97 percentage points (median 3.06). It also matches or exceeds full-cache performance in 11 settings, with a maximum drop of 1.87 points in the others. The consistency across models and tasks is the main reason I lean toward acceptance. 2.The attention correction has a clear motivation: an anchor should represent the number of tokens assigned to it. Table 7 gives a useful check at a fixed physical cache size. Removing the count correction lowers the macro-average from 65.25 to 64.21 and the image-counting score from 44.00 to 30.00. This supports the correction, particularly for counting tasks.

Critical weaknesses
  1. I am less convinced by the evidence for the individual design choices. In Table 7, task-field queries beat Tail32 by 0.29 points on average, largely because of the 14.79-point gain on GovReport. Without GovReport, they trail by 0.48 points. For each of the four component controls, image counting accounts for 67–84% of PACE-VL’s gain summed across datasets; excluding that dataset leaves average gains of 0.10–0.39 points. These tests cover one model and one context scale, with one seed for random anchors. Given the related aggregation schemes in Section 2, I would like clearer evidence that the choices specific to PACE-VL help across tasks.
  2. I also had trouble comparing the cache budgets. Table 4 explicitly uses different physical budgets, and Appendix A.3 gives SparseVLM a 192-token setting while other methods follow their own policies. Actual KV bytes would make these comparisons easier to assess. A separate concern is the change in the full-cache references between Tables 1 and 5: Qwen2.5-VL’s InfographicVQA score rises from 73.73 to 78.07. Section 4.2 notes that this is not a controlled budget sweep, but does not fully explain the change. Neither issue establishes that the within-table results are invalid. More detail on the run configurations and baseline adaptations is needed to reproduce the comparisons.
Questions for authors
  1. Why is the task-field advantage over Tail32 so large on GovReport (14.79 points)? Please show which token spans the two query sources cover. Is the gain related to that dataset’s prompt layout?
  2. Could you report total KV bytes for PACE-VL and the strongest compressed baselines in Table 1, both after prompt compression and after 128 generated tokens, together with their quality scores? Please also give the configurations and implementation changes used for CaM and LOOK-M.
  3. What changed between the full-cache runs in Tables 1 and 5 besides adding text_doc? For Qwen2.5-VL K32, adding 100 samples in [0, 1] to 4,414 unchanged scores cannot lower the mean from 0.4596 below about 0.4494, yet Table 5 gives 0.4014. Please clarify the data, run settings, and averaging used for each table.
Reviewer_p9cwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose a training-free KV Cache compression method for VLMs named PACE-VL, which uses query states from the question or instruction tokens to select important cache entries, then preserves them and groups the remaining history around selected original keys. Each group stores an aggregated value and a token count, which is incorporated into attention normalization during decoding. The experiments demonstrate strong quality preservation at about 10% prompt-cache budget.

Critical strengths
  1. The performance under aggressive compression is promising. For example, PACE-VL outperforms the compared baselines in all evaluated settings at the 10% budget. The coverage across multiple models and both short-form and long-context tasks makes this an encouraging empirical result.
  2. PACE-VL is training-free and well specified. The distinction between exact evidence entries and aggregated residual entries provides an understandable design. In addition, the method also considers the prompt-cache budget and represented token counts.
Critical weaknesses
  1. The method includes query-guided selection, key-based aggregation and count-aware attention correction, which are all closely related to existing baselines. The benefit over ToMe and RazorAttention has not been fully established without controlled comparisons.
  2. Eq 3 uses a fixed importance threshold and clamping bounds. It is a natural problem that whether an adaptive is more suitable.
  3. The experimental analysis is limited, where there is only one overview figure and two tables in the main text. For example, the central aggregation approximation lacks direct validation. The discarded tokens carry non-negligible attention mass motivates aggregation, but does not demonstrate that the proposed anchors preserve their contributions.
Questions for authors
  1. How does PACE-VL compare with existing count-aware compensation methods under the same selection rule and cache budget (W1)?
  2. Can you provide a comparison and discussion about the fixed and adaptive allocation at the same budge (W2)?
  3. Please measure attention-mass distortion and attention-output error after aggregation to demonstrate when the approximation works.
Reviewer_PHHGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PACE-VL, a training-free KV-cache compression method for VLMs with three coupled parts. (1) Queries drawn from the located "task field" score the prompt history; top-scoring entries are kept verbatim as evidence tokens. (2) The remaining history is clustered to anchors, each storing an original key from a task-ranked anchor site, a value sum, and a multiplicity count. (3) Decoding adds the multiplicity to the softmax denominator, so each cluster retains its aggregate contribution. Evidence tokens, anchors, and a recent window fill an exactly budgeted cache. On five visual-heavy benchmarks across three VLMs at a 10% budget, PACE-VL is reported as the best compressed method in different settings.

Critical strengths
  1. Well-motivated design grounded in a two-sided measurement. Table 2 shows attention is highly concentrated (top 1% of history carries 76–81% of decode mass) yet the residual still holds 4–8% and inflates surviving weights by 1.05–1.11×. This justifies verbatim retention and residual aggregation together, rather than assuming either alone suffices.

  2. Clean theory with appropriately modest claims. Propositions 1–3 establish an exact slot budget with full population coverage, exactness when cluster keys equal their anchor, and a Cauchy–Schwarz bound showing nearest-anchor assignment minimizes per-token logit perturbation for fixed anchors.

  3. Rigorous experimental hygiene where it counts. All methods share checkpoints, prompt templates, decoding parameters, and evaluation harness, and the four component ablations hold the physical slot count fixed so they change what fills the cache rather than how much.

Critical weaknesses
  1. Evidence selection relies on task-field queries, but the reported results obtain the task field from benchmark metadata (§3.2). In production the task is often not identifiable: multi-turn dialogue, agentic prompts where the instruction arrives as tool output, or long system prompts that resemble instructions.
  2. Several KV-cache reconstruction and compensation methods are discussed in §2 and Table 3 but never evaluated empirically: notably Attention Matching (arxiv.org/abs/2602.16284), which fits per-entry logit biases and values to preserve attention mass, and RazorAttention (arxiv.org/pdf/2407.15891), which already summarizes discarded entries with a count-weighted mean-key/mean-value compensation token. These are the closest prior art to PACE-VL's core contribution: both reconstruct the evicted population rather than deleting it, and RazorAttention in particular already uses multiplicity weighting. Could you add at least one reconstruction-based baseline under the same budget?
Questions for authors
  1. The evidence/anchor split is governed by τ = 3.5×10⁻⁵ and clamp fractions (0.55, 0.75), with no stated derivation or sensitivity analysis. How were these chosen, and on what data, were they tuned on the evaluation benchmarks themselves?

  2. All measurements appear to be batch size 1. How do prefill overhead, decode throughput, and peak memory behave at larger batch?

  3. Table 4 shows PACE-VL's decode throughput (18.80 tok/s at K8) is below HybridKV (22.22), VL-Cache (22.17), and SparseVLM (20.48) at the same context scale, and the reported 1.31–1.49× generation speedup is measured against an eager-attention Vanilla baseline while FlashAttention support is deferred to future work. Why is decode throughput lower than the pruning baselines when the cache size is comparable?

  4. How does the efficiency picture change against a FlashAttention Vanilla baseline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.