acceptodds
Withdrawn from ICLR 2027

MetricKV: Decoded-Space Geometry for Visual KV Compression in Streaming Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Streaming video models observe visual evidence before future queries are known, making compact storage of historical visual key–value (KV) states important for long-running inference. We introduce METRICKV, a query-agnostic visual-KV codec derived from the decoded-KV reconstruction objective. For any positive-definite quadratic decoded-space distortion and full-column-rank decoder, decoded error decomposes exactly into a continuous projection term and an additional finite-rate latent error under a decoder-induced pullback metric. This metric determines the same ordering over finite-rate candidates as the decoded-KV objective; absorbing it into the latent coordinates yields a canonical Euclidean coding space in which standard nearest-neighbor quantization is exactly decoded-error aligned. Within this canonical family, minimizing the coder-independent continuous projection term reduces to PCA in the distortion-transformed KV space. We instantiate the framework with Layer-RMS reconstruction geometry and progressive residual vector quantization (RVQ). The experimental results demonstrate that a single progressive RVQ stream achieves dynamic-KV compression while retaining more than 98% of FP16 performance on both OVO-Bench and StreamingBench. Code will be released soon.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Streaming Video Understanding, KV Compression, Decoded-Space Geometry

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6JYWOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MetricKV studies the numerical compression of historical visual KV states in streaming video understanding. The implementation combines Layer-RMS normalization with progressive RVQ. Fixed-readout experiments report 64× dynamic-KV compression with scores of 69.04 versus 69.94 FP16 on OVO-Bench and 79.10 versus 80.74 on the evaluated StreamingBench subset. Additional experiments cover another backbone, transfer of Layer-RMS to KVTC, and integration with streaming memory policies.

Critical strengths
  1. A clear and reusable codec design framework. The exact distortion decomposition explains which latent metric should accompany a decoder. Transferring Layer-RMS to KVTC without replacing its scalar coder provides empirical support that the reconstruction objective is an independently useful design choice.
  2. Strong dynamic compression under controlled evidence selection. The fixed-readout protocol holds the selected frames and token budget constant, helping isolate representation quality. Retaining approximately 98% of reference scores at 64× dynamic compression, with a second-backbone evaluation and a progressive index stream, is potentially valuable for long-lived visual memory.
Critical weaknesses
  1. Practical overhead is not adequately evaluated. Table 4 reports 0.467 s encoding and 0.027 s decoding per frame at 64× compression, but lacks runtime comparisons with the same uncompressed system. End-to-end ingestion throughput and QA latency should be evaluated to determine whether compression benefits outweigh its computational overhead, especially for bounded histories.

  2. The novelty is insufficiently demonstrated. The theoretical derivation largely builds on classical weighted least squares and PCA, with related ideas already explored in KVTC.

Questions for authors
  1. End-to-end system evaluation: Can the authors compare ReKV or SimpleStream with and without MetricKV under identical hardware, inputs, retrieval policies, and cache configurations? Reporting ingestion throughput, QA latency, and total memory usage would help clarify whether compression benefits outweigh the additional codec overhead.
Reviewer_EGEsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MetricKV addresses the compression of retained visual KV states in streaming video understanding, independently of evidence-retention and retrieval policies. For a fixed positive-definite quadratic reconstruction objective and a full-column-rank linear decoder with a matched weighted least-squares encoder, the paper decomposes reconstruction error into continuous projection error and finite-rate latent error. It derives a decoder-induced metric and constructs Euclidean latent coordinates that preserve candidate ordering under the decoded-space objective. PCA in the distortion-transformed KV space minimizes the continuous projection term.

The implementation combines a Layer-RMS-weighted reconstruction objective, joint grouping across layers and heads, and progressive residual vector quantization, allowing multiple storage budgets through different index-stream prefixes. The codec is fitted on unlabeled LongVideoBench KV samples and evaluated on OVO-Bench and StreamingBench under a fixed-readout protocol, with additional integration into ReKV and SimpleStream. On the main backbone, 64× dynamic-KV compression reduces the two benchmark scores by 0.90 and 1.64 points relative to FP16, respectively. Additional experiments examine the reconstruction objective across coders and evaluate a second backbone.

Critical strengths
  1. The theoretical derivation is clear, and the scope of its guarantees is explicitly defined. With a matched weighted least-squares encoder, the paper develops a coherent connection among reconstruction-error decomposition, metric canonicalization, and continuous subspace selection. It also clearly distinguishes reconstruction-error consistency, finite-rate optimality, and downstream question-answering performance.

  2. The controlled experiments are well designed, and performance remains close to the original KV baseline under substantial compression. Fixing the visual evidence and answer-time token budget helps isolate the effect of the compressed representation. At 64× dynamic-KV compression, the OVO-Bench and StreamingBench scores decrease by 0.90 and 1.64 points relative to FP16, respectively. Applying the reconstruction objective across different coders and evaluating a second backbone provide additional support.

Critical weaknesses
  1. The advantages over a strong baseline and the system-level benefits remain unclear. In Table 3, RVQ + Layer-RMS scores below KVTC + Layer-RMS on StreamingBench at all three budgets. Although progressive coding supports multiple budgets and Table 5 shows advantages at some high-compression operating points, the trade-off between quality and cost needs further clarification. Moreover, the paper does not report total system memory savings after accounting for static codec parameters, particularly in settings with a small fixed memory window.

  2. The theoretical novelty is not sufficiently delineated. The core mathematical derivations primarily build on classical weighted least squares, canonicalization of positive-definite quadratic metrics, and PCA. KVTC already uses an orthogonal transform under the raw squared-error objective to preserve quantization error between the transform and reconstruction domains within the retained subspace. The additional theoretical results and algorithmic contributions obtained by extending this approach to general weighted reconstruction objectives and full-column-rank linear decoders therefore need to be defined more clearly.

Questions for authors
  1. The authors explain that RVQ supports multiple budgets using the same index stream. Compared with KVTC + Layer-RMS, what trade-offs does this flexibility offer in task performance, encoding time, and static storage? Please also specify the grouping, retained rank, and calibration configurations used by both methods.

  2. After clarifying the device placement of codec parameters and how they are shared across sessions, how do actual memory usage and end-to-end processing latency change when static parameters and runtime overhead are included? For ReKV, how much retained history is needed to offset the static storage overhead? For SimpleStream with a fixed four-frame window, are there still net memory savings?

  3. KVTC already uses an orthogonal transform to establish quantization-error consistency under the raw squared-error objective. Relative to this result and classical weighted PCA and transform-coding theory, what substantive theoretical results or algorithmic capabilities are introduced by extending the formulation to general positive-definite quadratic distortion objectives and full-column-rank linear decoders? Which actual performance gains can be clearly attributed to this extension?

Reviewer_w42QOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Streaming video MLLMs keep the visual KV cache of past frames so that later questions can reuse it, but this cache grows with video length. The paper studies how to store each kept KV state in fewer bytes, which is complementary to methods that decide which frames to keep. Its key observation is that existing pipelines choose quantized codes by distance in a compressed latent space, which may not match the real error after decoding back to KV. It therefore builds the codec from the KV error itself, which leads to per-layer normalization, PCA, and a progressive residual vector quantizer. With Qwen3-VL-4B, it keeps about 98% of FP16 accuracy on OVO-Bench and StreamingBench at 64× compression.

Critical strengths
  • Clear and principled formulation. The paper cleanly separates the distortion target, the retained subspace, and the quantizer, and builds the codec from unlabeled KV without touching the frozen MLLM. The Layer-RMS objective also improves another codec (KVTC), which shows this separation is useful in practice.
  • High compression with one progressive stream. One RVQ index stream serves several storage budgets, and at 64× compression the accuracy stays close to FP16. The codec works under both retrieval-based and window-based memory policies without changing them.
Critical weaknesses
  • Limited novelty. The final method is per-layer normalization plus PCA plus vector quantization, which is classical transform coding (Goyal, 2001). The metric mismatch it fixes does not arise for orthonormal PCA codecs such as KVTC.
  • The gain mainly comes from Layer-RMS, not from the new codec. KVTC with Layer-RMS beats MetricKV on StreamingBench at every storage budget. So the main remaining benefit is the single progressive stream.
  • High deployment cost. The codec adds 2.1 GiB of static parameters and up to 0.9 s of write time per frame at 1 fps. For short memory windows such as SimpleStream, this cost is larger than the memory it saves.
Questions for authors

Questions that could change my assessment

  • Comparison with KVTC + Layer-RMS. The paper reports KVTC with Layer-RMS only on StreamingBench, where it beats MetricKV at all three budgets. Please report the same comparison on OVO-Bench.
  • Evidence for the metric mismatch. The main claim is that distance in the latent space can rank codes differently from the real KV error. The only evidence is a synthetic test where an existing codec is artificially rescaled. Is there a real KV codec, such as one with a non-orthogonal or learned decoder, where using the derived metric improves accuracy?
  • Choice of the error target. The codec minimizes the KV reconstruction error weighted only by layer scale. But key errors act through the softmax of unknown future queries, so the same KV error may not cause the same accuracy loss. Have you checked how well this error predicts attention-output error or accuracy, for example by compressing only K or only V?

Clarifications

  • Static storage. The 2.1 GiB static size seems to store the encoder and decoder separately in FP32. Both come from the same PCA basis and a diagonal scaling, so could the basis be stored once? Please also state how long a stream must be before the codec saves memory overall.
  • Related work. Please discuss cross-layer KV compression such as xKV (Chang et al., ICML 2026), which also exploits redundancy across layers through a shared low-rank subspace.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.