acceptodds
Under review as a conference paper at ICLR 2027

FactorKV: A Marginal Coverage Selector for Low-Budget KV Cache Compression in LVLMs

Abstract

Large vision-language models (LVLMs) rely on growing key-value (KV) caches for autoregressive generation, making cache compression important for long visual contexts. Existing methods improve how limited capacity is allocated across attention heads and how individual KV pairs are scored. Yet under tight budgets, top-k selection can retain tokens from answer-relevant regions while still producing answers with missing or incorrect words. Static ranking can favor overlapping content, leaving complementary answer details insufficiently represented. We propose FactorKV, a KV cache selector based on utility-weighted marginal coverage of the full compressible history in key space. Given a host compressor's budget, protected positions, and utility scores, FactorKV first constructs a pool of high-scoring candidates. It then greedily selects the candidate that provides the greatest additional coverage and updates the represented content after each selection. Historical positions already well covered contribute less to subsequent gains, allowing candidates that represent complementary details to gain priority. Each retention decision therefore accounts for both the utility of the historical content and the coverage supplied by the current selected set. FactorKV integrates with existing compressors without requiring retraining or increasing the cache budget. Experiments show gains across multiple cache allocators and LVLMs, particularly under tight budgets. FactorKV repairs some omission and substitution errors made by the original methods, while others remain unresolved. Under low-budget, decoding latency remains comparable to the base methods, and total latency remains lower than FullKV despite additional prefill computation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LMM, KV cache compression

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_LovTOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

FactorKV looks at a simple question: when the KV-cache budget is very small, are we keeping useful information, or just several tokens that represent roughly the same thing? The method works on top of an existing cache compressor. It takes a pool of high-scoring candidates and selects tokens one at a time, giving priority to those that cover information not already represented by the selected set. Coverage is measured using key similarity, usually before RoPE. The original keys and values are then kept for decoding. The reported gains are strongest on text-rich visual tasks at small cache budgets. The tradeoff is extra computation during prefill. Results with stronger allocators and on text-only tasks are more mixed.

Critical strengths
  1. The main idea well motivated. A token can look important on its own but add little once several similar tokens have already been selected. FactorKV addresses this directly by updating each candidate’s value after every selection. The formulation is clear, and the submodularity argument is sound. The authors also correctly explain that the guarantee applies to their coverage objective, not directly to answer accuracy.

  2. The ablations go beyond simply comparing the full method with a weaker baseline. In particular, static coverage, uniform-weight coverage, and MMR help test whether the benefit comes from updating coverage, weighting the historical content, or just encouraging diversity.

  3. The SparseMM comparisons are mixed, and the LongBench averages are lower than those of the corresponding base methods. Re-running the Video-TT comparisons with the same judge is another good choice. These details make the scope of the contribution easier to assess.

Critical weaknesses
  1. Most baseline scores are taken from earlier papers, while FactorKV is evaluated by the authors. That is not necessarily a problem, but the paper needs to establish that the evaluation settings actually match.

  2. The temporary memory cost also deserves more attention. Keeping the same number of KV entries does not remove the cost of constructing the candidate-to-history similarity matrix. The reported peak-memory result is useful, but only establishes this for the tested setting.

  3. The examples are suggestive. What I am missing is a broader check: does better coverage tend to mean smaller attention-output errors, better preservation of answer logits, or more complete retention of the relevant evidence?

Questions for authors
  1. Could you rerun FullKV, MixKV/Top-�0��0�, and FactorKV in the same evaluation pipeline? In particular, do the TextVQA and VizWiz scoring rules match those used for the published baselines?

  2. What happens to end-to-end latency when the answer is short? Please report time to first token and selection overhead on real image/video inputs, along with temporary peak memory.

  3. How do MMR and the similarity-penalty baseline perform when their tradeoff parameters are tuned on a validation set? Which parts of FactorKV provide the main benefit beyond applying weighted facility-location selection?

Reviewer_Gd4AOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies token selection in KV cache compression for vision-language models, motivated by a concrete failure mode: under tight budgets, top-k selection can retain tokens from answer-relevant regions while still producing answers with missing or incorrect words, and static ranking can favor overlapping content, leaving complementary answer details insufficiently represented. FactorKV is a KV cache selector that, given a host compressor's budget, protected positions, and utility scores, greedily selects the candidate that provides the greatest additional coverage and updates the represented content after each selection, so that positions already well covered contribute less to subsequent gains and complementary details gain priority. The objective is monotone submodular, and greedy selection achieves the classical approximation bound on this surrogate.

Experiments across multiple host compressors, models, and task types show consistent gains over both the base allocators and MixKV. The gains are largest at tight budgets and on the document and scene-text benchmarks, and the authors themselves report that the gains do not extend to multi-hop reasoning and code tasks.

Critical strengths

1. The paper clearly diagnoses a real failure mode and designs a selection rule that targets it directly. As the authors point out, top-k selection can retain tokens from answer-relevant regions while still producing answers with missing or incorrect words, as static ranking can favor overlapping content and leave complementary answer details unselected. FactorKV addresses this by judging a token's value by the important content it adds to the current cache, not by its score alone, and the plug-in design keeps the host's budgets, protected positions, and utility scores unchanged, so the comparison is clean and the gains attributable.

2. The evaluation covers a diverse range of tasks and exposes both the applicability and the limits of the method. Beyond the document and scene-text tasks where the method is strongest, the evaluation includes video understanding and text-only tasks, and reports that multi-step integration and code-completion tasks favor the base methods. As the paper itself concludes, the benefit depends on the task's information structure, and stating this boundary openly makes the claimed gains more credible.

Critical weaknesses

1. Whether similarity implies coverage in compression is not established. The method treats key similarity as positive evidence of coverage. But MixKV and related methods use the same similarity signal in the opposite direction, as a redundancy penalty, and this contradiction is not considered.

Questions for authors
  1. The similarity conflict. The paper uses key similarity as positive evidence of coverage, while MixKV-style methods use it as a redundancy penalty. Is there direct evidence for which direction is right?
  2. Combination with other utility scores. The utility scores used in the experiments are largely attention-based. It would be useful to see FactorKV combined with utility scores of a different nature.
  3. GUI tasks. The evaluation does not include GUI or screen-grounded tasks, where the answers also depend on scattered text elements. How does the method perform in this regime?

Comments on the paper

2 comments

Dear Authors,

The author-reviewer discussion period is ending soon, but no rebuttal has been submitted yet. We encourage you to respond to the reviewers' comments and address their key concerns before the discussion period closes.

AC

AuthorsOpenReview ↗

The revision of the paper

Thank you for the suggestions and responses from the reviewers. Our paper has been revised according to the reviewers' requirements. We have incorporated some of the reviewers' previous questions and supplementary experiments into the paper, and marked them in blue, which is different from the previous version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.