- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
FactorKV looks at a simple question: when the KV-cache budget is very small, are we keeping useful information, or just several tokens that represent roughly the same thing? The method works on top of an existing cache compressor. It takes a pool of high-scoring candidates and selects tokens one at a time, giving priority to those that cover information not already represented by the selected set. Coverage is measured using key similarity, usually before RoPE. The original keys and values are then kept for decoding. The reported gains are strongest on text-rich visual tasks at small cache budgets. The tradeoff is extra computation during prefill. Results with stronger allocators and on text-only tasks are more mixed.
Critical strengths
-
The main idea well motivated. A token can look important on its own but add little once several similar tokens have already been selected. FactorKV addresses this directly by updating each candidate’s value after every selection. The formulation is clear, and the submodularity argument is sound. The authors also correctly explain that the guarantee applies to their coverage objective, not directly to answer accuracy.
-
The ablations go beyond simply comparing the full method with a weaker baseline. In particular, static coverage, uniform-weight coverage, and MMR help test whether the benefit comes from updating coverage, weighting the historical content, or just encouraging diversity.
-
The SparseMM comparisons are mixed, and the LongBench averages are lower than those of the corresponding base methods. Re-running the Video-TT comparisons with the same judge is another good choice. These details make the scope of the contribution easier to assess.
Critical weaknesses
-
Most baseline scores are taken from earlier papers, while FactorKV is evaluated by the authors. That is not necessarily a problem, but the paper needs to establish that the evaluation settings actually match.
-
The temporary memory cost also deserves more attention. Keeping the same number of KV entries does not remove the cost of constructing the candidate-to-history similarity matrix. The reported peak-memory result is useful, but only establishes this for the tested setting.
-
The examples are suggestive. What I am missing is a broader check: does better coverage tend to mean smaller attention-output errors, better preservation of answer logits, or more complete retention of the relevant evidence?
Questions for authors
-
Could you rerun FullKV, MixKV/Top-, and FactorKV in the same evaluation pipeline? In particular, do the TextVQA and VizWiz scoring rules match those used for the published baselines?
-
What happens to end-to-end latency when the answer is short? Please report time to first token and selection overhead on real image/video inputs, along with temporary peak memory.
-
How do MMR and the similarity-penalty baseline perform when their tradeoff parameters are tuned on a validation set? Which parts of FactorKV provide the main benefit beyond applying weighted facility-location selection?
Dear Authors,
The author-reviewer discussion period is ending soon, but no rebuttal has been submitted yet. We encourage you to respond to the reviewers' comments and address their key concerns before the discussion period closes.
AC