acceptodds
Under review as a conference paper at ICLR 2027

LMTC: Quantizing KV Caches for How Attention Reads Them in Omni-Modal LLMs

Abstract

Omni-modal Large Language Models (OmniLLMs) unify the understanding of video, audio, and text, yet their continuous streaming inputs generate an overwhelming volume of tokens, making the Key-Value (KV) cache a major memory bottleneck. Existing KV cache quantization methods are suboptimal for OmniLLMs because they either minimize simple reconstruction error or rely on query activations that are unavailable when streaming tokens are cached, ignoring both the attention reading mechanism and cross-modal heterogeneity. To address these challenges, we propose Logit-Metric Transform Coding (LMTC), a training-free and query-free KV cache quantization framework that minimizes the expected attention error using only the query projection weights. Specifically, LMTC quantizes Keys using a Modality-Conditioned Transform Coder guided by a Query-Free Logit Metric derived directly from query projection weights, thereby encoding each modality in its own basis and allocating precision via reverse water-filling to the directions attention reads. Simultaneously, LMTC handles Values through Phase-Offset Rounding, a metadata-free mechanism that offsets the rounding grids of spatio-temporally adjacent visual tokens so that rounding errors cancel out during attention aggregation rather than accumulate. Extensive evaluations across three OmniLLMs on eight benchmarks show that LMTC is virtually lossless at 3 effective bits per scalar, retaining over 99% of BF16 average accuracy. Compared to BF16 FlashDecoding-v2, LMTC achieves up to a decoding speedup, a KV cache memory reduction, and a throughput boost.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
KV cache quantization, omni-modal large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_axf9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Logit-Metric Transform Coding (LMTC), a training-free and query-free KV-cache quantization method that aims to minimize expected attention-logit distortion. Keys are quantized through modality-conditioned transform coding, guided by a logit metric derived from query projection weights. Values are quantized using phase-offset rounding, which shifts the rounding grids of spatiotemporally adjacent video tokens to encourage error cancellation during attention aggregation. Experiments on three omni-modal LLMs across eight benchmarks demonstrate the method’s effectiveness.

Critical strengths

The problem is well motivated. Measuring key quantization error through its effect on attention logits provides an objective more closely aligned with attention behavior than reconstruction error alone. The paper also distinguishes how key and value errors propagate through attention, providing a useful basis for designing separate quantization strategies.

Critical weaknesses
  1. I found the presentation difficult to follow. Figure 1 is meant to motivate the method, but readers need details from Section 3 to understand it. The caption should explain the weight-derived directions, the calculation of “KL share,” and the aggregation of modality attention-mass changes. The notation is also dense, and it is easy to lose track of symbol definitions and coordinate changes. A notation table and brief reminders would help.

  2. Many derivations are provided in the appendix, but the main text needs more explanation of how the equations connect. In particular, the transition from Equation 1 to the per-token coding objective, and from Equation 4 to Equation 6, should be explained more clearly. It is not always clear which steps are exact and which rely on assumptions or approximations.

Questions for authors
  1. How much task accuracy is lost relative to BF16 as the actual input context becomes longer? The paper includes long-context timing results and reader-position tests, but does not report accuracy loss across input context lengths. This would help assess whether the method remains reliable on long inputs.

  2. Section 4.4 excludes prefill and prefix encoding. How expensive is constructing the quantized cache compared with the baselines? Reporting cache construction time or time to first token would help, especially for long inputs followed by short answers, where decoding savings have less time to offset any additional encoding cost.

  3. In Section 3.2 and Appendix D.2, the argument for omitting basis transformations in the fast coordinates needs further explanation. Small within-pair rotary coherence does not rule out cross-pair correlations. Averaging the query metric over future reader positions also does not establish that keys at their actual cache positions are decorrelated. What additional conditions are needed, and how large are the remaining correlations? Table 7 supports the chosen split empirically, but does not directly check these conditions.

9 comments

W1: I found the presentation difficult to follow. Figure 1 is meant to motivate the method, but readers need details from Section 3 to understand it. The caption should explain the weight-derived directions, the calculation of "KL share," and the aggregation of modality attention-mass changes. The notation is also dense, and it is easy to lose track of symbol definitions and coordinate changes. A notation table and brief reminders would help.

A: We thank the reviewer for this helpful suggestion. We have revised the paper accordingly; all changes are marked in blue.

The caption of Figure 1 now reads:

Attention damage concentrates in a few query-weight directions; uniform noise inflates audio-visual attention. (a) Top-kk query-weight directions capture little INT2 Key error (gray) but much of the attention KL (cyan), far more than the Keys' principal components (ticks). (b) Attention mass change per modality under INT2, showing spurious AV gain; dot: median, bar: interquartile range (IQR).

Following the reviewer's suggestion, we added the missing definitions to Observation (i), where the figure is first discussed: the query-weight directions, the calculation of the KL share and the error share, and the aggregation of the attention-mass change. Figure 1 can now be read without Sec. 3. We then checked the rest of the section for the same problem and revised Observation (ii) and the caption of Figure 2 in the same way.

For the notation, we added a notation table (App. D.1, Table 5) and a reminder of the three Key frames at the start of Sec. 3.2, and we removed overloaded symbols.

Please see the updated paper for the details.

W2: Many derivations are provided in the appendix, but the main text needs more explanation of how the equations connect. In particular, the transition from Equation 1 to the per-token coding objective, and from Equation 4 to Equation 6, should be explained more clearly. It is not always clear which steps are exact and which rely on assumptions or approximations.

A: We thank the reviewer for this helpful suggestion. We revised Sec. 3.2 so that the path between the equations is spelled out in the main text, with each step marked as exact or approximate; all changes are in blue.

(1) From Equation 1 to the per-token coding objective. The goal is to keep the attention unchanged. A Key error eiKe^K_i shifts the logit of token ii by δi=q⊤eiK/d\delta_i=q^\top e^K_i/\sqrt d, and the attention changes only through how unevenly these shifts are spread over the tokens: up to higher-order terms, the attention KL is half their variance (its correlation with the exact KL is 0.99 or more, App. D.2). Weighting all tokens equally, which is the safe choice since concentrated attention is less sensitive to such shifts, and averaging over queries gives Eq. 1. This variance is at most the average of the per-token terms eiK⊤ΣqeiKe^{K\top}_i\Sigma_q e^K_i, with equality because LMTC codes each Key around its modality mean, so that the errors have zero mean. Minimizing Eq. 1 therefore amounts to minimizing one term per token, which is what allows each token to be coded on its own. The only ingredient not available when the cache is written is the query statistic Σq\Sigma_q; we obtain it from the weights (Eq. 2) and average it over the unknown position of the future reader (Eq. 3), which gives the per-token objective eˉi⊤Mˉeˉi\bar e_i^\top\bar M\bar e_i. This weight-derived metric comes within 8% of one built from real queries (see our response to Reviewer FLhK, W1).

(2) From Equation 4 to Equation 6. This transition and the fast-coordinate question (Q3) concern the same step, so we answer them together under Q3 below.

Q1: How much task accuracy is lost relative to BF16 as the actual input context becomes longer? The paper includes long-context timing results and reader-position tests, but does not report accuracy loss across input context lengths. This would help assess whether the method remains reliable on long inputs.

A: We thank the reviewer and added this experiment. Accuracy loss relative to BF16 does not grow with input length from 5K to 33K tokens.

We ran 500 LVBench multiple-choice questions on Qwen2.5-Omni-7B with audio and video, in windows of 30, 60, 120 and 190 s (5.2K–32.5K tokens), with the evidence either right before the question or at the start of the window, where it has to be read across the whole cache. Accuracy in %; Δ\Delta is the change against BF16 in points, flip the share of answers that differ from BF16, and KL is over the option letters:

WindowTokensEvidenceBF16LMTC Δ\DeltaLMTC flip / KLKIVI Δ\DeltaKIVI flip / KL
30 s5.2Kend68.4−0.61.0% / 0.004+0.08.0% / 0.055
start68.2−0.61.4% / 0.004+0.29.2% / 0.044
60 s10.3Kend68.2+0.21.4% / 0.005−1.210.4% / 0.052
start68.6−0.22.2% / 0.004−1.69.2% / 0.043
120 s20.6Kend65.6−0.62.0% / 0.006−1.810.4% / 0.064
start66.8+0.03.2% / 0.005−1.811.2% / 0.048
190 s32.5Kend61.8−0.22.6% / 0.005−0.88.0% / 0.044
start63.4+0.43.2% / 0.004−1.69.6% / 0.048

At every length and evidence position, LMTC stays within −0.6 to +0.4 points of BF16, with a KL of 0.004–0.006 and 1.0–3.2% of the answers changed, against 0.04–0.06 and 8.0–11.2% for 2-bit KIVI.

Q2: Section 4.4 excludes prefill and prefix encoding. How expensive is constructing the quantized cache compared with the baselines? Reporting cache construction time or time to first token would help, especially for long inputs followed by short answers, where decoding savings have less time to offset any additional encoding cost.

A: We thank the reviewer for this important question. Please see our joint response on end-to-end efficiency above, which reports TTFT and the latency to the NN-th answer token for N=16N=16, 6464 and 256256. In short, building the LMTC cache costs 7–31 ms (0.6–1.1% of TTFT), the least of all compressed methods, and LMTC has the lowest end-to-end latency at every prompt length, even for 16-token answers after long inputs.

Q3: In Section 3.2 and Appendix D.2, the argument for omitting basis transformations in the fast coordinates needs further explanation. Small within-pair rotary coherence does not rule out cross-pair correlations. Averaging the query metric over future reader positions also does not establish that keys at their actual cache positions are decorrelated. What additional conditions are needed, and how large are the remaining correlations? Table 7 supports the chosen split empirically, but does not directly check these conditions.

A : We thank the reviewer for these insightful comments. Both concern the same step, so we answer them together.

Eq. 4 de-rotates each Key by its own position, so scoring would need one query projection per Key. To avoid this, slow rotary pairs, which turn by at most 0.75 rad within a segment of 255 positions, share one de-rotation per segment, while fast pairs are stored after rotation, without a basis, and scored with no projection. Eq. 6 is the exact score of the Key reconstructed from these codes. The approximation lies only in the codes, mainly in coding the fast pairs without a basis.

A basis does two things, and omitting it is safe if both already hold without it. First, the distortion must remain a sum of per-coordinate terms, as the bit allocation assumes. As the reviewer notes, averaging over reader positions does not make the coupling between pairs small for an actual reader (RMS 0.16), so we rely instead on the errors being uncorrelated: rounding errors are independent, and for zero-bit coordinates this only needs decorrelated Keys. The measured cross terms are below 0.1% of the distortion on average. Second, no bits should be spent on correlated coordinates, which needs the Keys to be decorrelated after rotation. Rotation averages out the correlation between pairs when the Key positions span many periods. On Qwen2.5-Omni-7B (fast pairs 0–26), the cross-pair correlation (RMS) before → after rotation is:

Fast pairsAudioVideo
Temporal (0–15)0.23 → 0.020.23 → 0.03
Temporal × height0.19 → 0.020.18 → 0.03
Height (16–26)0.20 → 0.080.18 → 0.18

Both conditions hold except the second one for the video height pairs, whose positions take only the few rows of a frame. The first still holds there, since zero-bit coordinates lie almost entirely in the temporal pairs.

The remaining correlation costs little. A basis on the post-rotary fast pairs would lower the KL by only 4–5%, and recovering the pre-rotary correlation requires Eq. 4 itself, which lowers the KL by 39%/43% (Near/Far) but needs 234× more query projections. The default thus sits at the knee of the trade-off (Table 7 of the submission, Table 8 of the revision). Coding the video height pairs as slow pairs lowers the KL by a further 6%/12% for 19% more projection FLOPs; we report this variant in the appendix, and replace the two claims of Sec. 3.2 by the conditions and measurements above.

Reviewer_juFHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Logit-Metric Transform Coding (LMTC), a KV cache quantization framework for omni-modal LLMs. For Keys, it constructs a query-free distortion metric from query projection weights and a prior over relative positions, then uses modality-specific transforms and bit allocation to reduce the resulting logit distortion. For Values, it introduces position-dependent rounding offsets intended to reduce correlated quantization errors among neighboring visual tokens. The authors report retaining over 99% of BF16 average accuracy at three effective bits per scalar across three models and eight benchmarks, together with memory and decoding efficiency improvements.

Critical strengths
  1. The paper addresses a well-motivated direction: exploiting the compression potential of omni-modal KV caches while accounting for differences across modalities.

  2. I appreciate the separate treatment of Keys and Values based on their roles in attention, rather than reconstruction error alone. The query-free formulation for reducing Key-induced logit distortion is particularly appealing for streaming inputs.

Critical weaknesses

The query-free metric relies on assumptions tied to the evaluated architectures. The justification for S^W rests partly on the query bias carrying most of the query energy. It is unclear whether the metric holds for models without query bias or with QK-Norm (e.g., Qwen3-Omni). In addition, the per-modality Key statistics are estimated from calibration data, yet their sensitivity to calibration domain is not evaluated.

Questions for authors
  1. Section 4.4 evaluates decoding efficiency but excludes prefill and media/prefix encoding. Could you clarify the additional cost of constructing the quantized cache as audio and video tokens arrive? In particular, does the transform coding introduce extra overhead during streaming? A qualitative discussion would be helpful; quantitative results, if available, would also be appreciated.

  2. In Sec. 3.2, the justification for the weight-derived metric relies partly on the query bias contributing most of the query energy. Can LMTC be extended to architectures without query bias, such as Qwen3-Omni? If so, are there any modifications needed based on the current method?

  3. How sensitive are the modality-specific Key means and covariances to the calibration domain, e.g., when they are estimated on one benchmark and evaluated on another? A qualitative discussion would also be sufficient.

5 comments

W&Q2: The query-free metric relies on assumptions tied to the evaluated architectures. The justification for SWS^W rests partly on the query bias carrying most of the query energy. It is unclear whether the metric holds for models without query bias or with QK-Norm (e.g., Qwen3-Omni). In addition, the per-modality Key statistics are estimated from calibration data, yet their sensitivity to calibration domain is not evaluated.

A: We thank the reviewer for this insightful comment. Without a query bias, the direction shared by all queries comes from the mean of the normalized hidden states, which is exactly the anisotropy that the isotropic prior of Eq. 2 ignores. Adding this mean to the prior restores LMTC on such models, still without data or queries.

The query mean is the most sensitive direction, since a Key error along it shifts the logits of every reader alike. In Qwen2.5-Omni, this mean is the bias (79% of the query energy, cosine 0.93 with bqb_q), which Eq. 2 keeps exactly. Without a bias it does not vanish: in Qwen3-8B it still holds 60–70% of the query energy, but it comes from the mean of the normalized hidden state x~\tilde x, which the isotropic prior sets to zero. We therefore replace the zero mean and identity covariance of the prior by the mean μx\mu_x and the top-32 covariance directions of x~\tilde x, estimated once, offline, from text the model generates from its end-of-text token. No data and no query is used, and Eq. 2 is the special case μx=0\mu_x=0.

When we weaken the bias of Qwen2.5-Omni-7B counterfactually down to zero, the logit error of Eq. 2 relative to a metric built from real queries rises gently from 1.09 to 1.17, while the extension stays at 1.07–1.09. On models without query biases, the Key attention KL relative to per-channel INT2 is (Near / Far, B=3B=3; Qwen3 models on text input):

MetricQwen3-8BQwen3-Omni-30B-A3BQwen2.5-Omni-7B
Reconstruction (KLT)1.39 / 0.820.345 / 0.2790.366 / 0.438
Eq. 2 (weights only)1.10 / 1.020.270 / 0.2650.105 / 0.144
Extension0.231 / 0.2020.180 / 0.1480.108 / 0.147
Real-query second moment (reference)0.205 / 0.2070.169 / 0.1510.097 / 0.139

Without a bias, the weights alone are not enough; the extension comes close to the real-query reference on both models and leaves Qwen2.5-Omni essentially unchanged. At the task level (Qwen3-Omni, 355 Daily-Omni questions, Keys quantized at B=3B=3), both metrics stay within the confidence interval of BF16 (72.4% vs. 72.1%), and the extension lowers the KL of the answer distribution to BF16 from 0.018 to 0.012. We will add these results to the appendix and present Eq. 2 in Sec. 3.2 as the zero-mean case of this extension.

For the calibration domain, please see our response to Q3 below.

Q1: Section 4.4 evaluates decoding efficiency but excludes prefill and media/prefix encoding. Could you clarify the additional cost of constructing the quantized cache as audio and video tokens arrive? In particular, does the transform coding introduce extra overhead during streaming? A qualitative discussion would be helpful; quantitative results, if available, would also be appreciated.

A: We thank the reviewer for this question. Streaming adds only a small, constant cost per chunk, because LMTC fits nothing online and encodes each token once: a token stays in BF16 while it is among the 128 most recent tokens, then joins the pending group of its modality, and each full group of 32 tokens is encoded by a projection and a rounding. Earlier tokens are never re-encoded.

To measure this, we fed a 120 s audio-visual clip to Qwen2.5-Omni-7B (one A100) in chunks of about 1, 2 or 4 s, and compared the LLM forward of each chunk, which every method pays, with the two steps LMTC adds: planning the chunk's token layout (CPU) and encoding its Keys and Values (GPU). Means over chunks, in ms:

ChunkTokensLLM forwardLMTC plan + encodeShare of forward
about 1 s17039.65.514%
about 2 s34053.25.811%
about 4 s68083.66.07%

LMTC thus adds about 6 ms per chunk, whatever its size; for 2 s chunks, this is 0.3% of the time in which the chunk arrives.

Q3: How sensitive are the modality-specific Key means and covariances to the calibration domain, e.g., when they are estimated on one benchmark and evaluated on another? A qualitative discussion would also be sufficient.

A: We thank the reviewer. The calibration statistics only describe how the Keys of each modality are spread, and LMTC uses them to choose the coding basis and split the bits; which directions are protected comes from the weights. A mismatch therefore makes the coding less tight but cannot misdirect it, and its effect is bounded: every coded coefficient is quantized min–max per group of 32 tokens with scale and offset from the actual Keys, so nothing is clipped or biased, and a misestimated variance shifts the bits only to second order (see our response to Reviewer FLhK, W1).

To test this, we calibrate on Daily-Omni (short daily-life clips, as in the paper) or on LVBench (long movies, sports and documentaries) and evaluate on held-out requests of both; only μm\mu_m, CmC_m change (Qwen2.5-Omni-7B, B=3B=3, Key attention KL relative to per-channel INT2, Near / Far):

Test requests \ Statistics fromDaily-Omni (paper)LVBenchThe request's own prefill
Daily-Omni0.105 / 0.1440.106 / 0.1460.084 / 0.114
LVBench0.104 / 0.1500.099 / 0.1450.084 / 0.121

Calibrating on the other benchmark changes the KL by only 1–5%, far less than the gap to KLT (0.37 / 0.44), and with the Daily-Omni statistics LMTC stays within −0.6 to +0.4 points of BF16 on LVBench at 5K–33K tokens. Estimating the statistics from the request's own prefill needs no calibration data and gives the lowest KL, an option when the deployment domain is unknown.

Dear Reviewers,

As the discussion period is nearing its end, we would like to kindly follow up and ask whether you have any additional questions, concerns, or requests for clarification regarding our responses.

We have uploaded a revised PDF, with changes highlighted in blue. And we would be happy to provide further explanations or additional evidence if it would help address any remaining concerns. We sincerely appreciate the time and effort you have invested in reviewing our work and thank you for your thoughtful feedback.

Thank you again for your consideration.

Regards Authors

Reviewer_FLhKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LMTC, a training-free and query-free KV-cache quantization framework for omni-modal LLMs. It combines a weight-derived logit metric, modality-conditioned transform coding for keys, and phase-offset rounding for values. Experiments on Qwen2.5-Omni-3B/7B and OmniVinci-9B show consistent accuracy improvements over the evaluated quantization baselines at three effective bits per scalar. The paper also reports memory savings and decoding speedups.

Critical strengths

S1: Optimizing attention-relevant distortion rather than reconstruction error addresses a meaningful limitation of conventional quantization.

S2: Detailed ablation studies provide useful evidence for the proposed components.

S3: Results on two model families and eight benchmarks support the method’s effectiveness, while streaming evaluations and packed-cache decoding measurements strengthen its practical relevance.

Critical weaknesses

W1: The weight-derived metric lacks sufficient justification. RMSNorm fixes the hidden-state norm but does not imply isotropic directions, so Equation 2 need not approximate the actual query second moment. The authors should establish that this approximation preserves the attention-sensitive directions needed for bit allocation, including when query biases are weak or absent.

W2: The optimized objective only approximates attention distortion. Replacing attention-weighted logit variance with a uniformly weighted surrogate can underestimate errors on highly attended keys. The authors should establish that minimizing this surrogate preserves attention when probability mass is concentrated on a small subset of tokens.

W3: The practical efficiency claims remain incompletely established. Appendix D.6 retains generated tokens in BF16, leaving robustness to sustained quantization during long reasoning unverified. Decoding timings include cache updates but exclude prefill and prefix encoding, leaving the initial full-precision-to-quantized cache conversion cost unreported. Moreover, efficiency is evaluated only for Qwen2.5-Omni-7B on A100 40GB GPUs, despite broader accuracy evaluations. The authors should establish accuracy under extended quantized decoding and end-to-end efficiency after accounting for cache construction, including whether these gains transfer across hardware and models.

Questions for authors

Q1: Are generated KV states quantized after leaving the BF16 residual window in the main evaluations?

Q2: How much does converting the initial KV cache into the quantized representation cost, and do the speedups persist when this cost is included?

6 comments

W1: RMSNorm does not imply isotropic directions, so Eq. 2 need not match the real query second moment. Does it preserve the attention-sensitive directions needed for bit allocation, including when query biases are weak or absent?

A: Thank you for the suggestion. The metric enters the coder in two decisions only: the directions in which Keys are coded, and the bits each coded coefficient receives. We show that SWS^W makes both decisions as real queries would, and that the resulting cache preserves attention better than coders driven by the Key error.

(1) Directions. The most sensitive direction is the query mean, shared by all readers: a Key error along it shifts every reader's logits alike and does not average out. In real queries, this mean holds 79% of the energy and has cosine 0.93 with bqb_q, which Eq. 2 keeps exactly. SWS^W ranks the remaining directions as real queries do (rank correlation 0.86), and its top-16 subspace overlaps 0.75 with theirs, against 0.47 for the principal directions of the Keys (the basis of KVTC) and 0.13 at random (Figure 6c).

(2) Bits. The allocation is robust to errors in the estimated energies: only their ratios matter, and a misestimate costs only second order (an energy off by a factor of 2 moves its coefficient by half a bit). The one first-order risk is giving zero bits to a direction that queries read. With each coder's basis fixed, we compare its bits with those the same allocator assigns from the real-query energies:

MetricSame bitsExtra logit error from the bitsReal logit energy in zero-bit coefficients
Reconstruction, Σq∝I\Sigma_q\propto I (KLT, as in KVTC)60%+70%2.4%
SWS^W (LMTC)87%+2.5%0.5%

(3) Attention. Together, the two decisions move the quantization error into directions that queries do not read. At the same budget, relative to per-channel INT2 (KIVI):

Key coderKey errorLogit errorKL Near / FarTop-1 flips
Per-channel INT2 (KIVI)111 / 15.8%
Reconstruction metric (KLT)0.430.510.366 / 0.4382.7%
SWS^W (LMTC)0.530.120.105 / 0.1441.9%
Real-query second moment (reference)0.610.1050.097 / 0.1391.8%

LMTC has a larger Key error than KLT but a four times smaller logit error, and comes within 8% / 4% of the reference built from real queries, which it never observes.

(4) Weak or absent biases. Without a query bias, the direction shared by all queries comes from the mean of the normalized hidden states, which the isotropic prior of Eq. 2 sets to zero. Adding this mean, estimated from text the model generates by itself, restores LMTC without data or queries: on Qwen3-Omni, it lowers the Key KL from 0.270 / 0.265 to 0.180 / 0.148 of INT2, against 0.169 / 0.151 for a reference built from real queries. Please see our response to Reviewer juFH (Weakness) for details.

W2: The uniformly weighted surrogate can underestimate errors on highly attended Keys. Does minimizing it preserve attention when the mass concentrates on a few tokens?

A: We thank the reviewer for this valuable and practical comment. LMTC weights all Keys equally because the attention is unknown when the cache is written. We show that this does not hide errors on highly attended Keys.

(1) Concentration lowers the sensitivity. With logit errors δi=q⊤ei/d\delta_i=q^\top e_i/\sqrt d, the attention KL is 12Vara(δ)\tfrac12\mathrm{Var}_a(\delta) to second order. We model the rounding errors as zero-mean and independent across Keys, so that δi\delta_i has variance σi2\sigma_i^2. Taking the expectation over the rounding errors, E[Vara(δ)]=∑iai(1−ai)σi2=F⋅h⋅E[Varunif(δ)].\mathbb{E}\big[\mathrm{Var}_a(\delta)\big]=\sum_i a_i(1-a_i) \sigma_i^2=F\cdot h\cdot\mathbb{E}\big[\mathrm{Var}_{\rm unif}(\delta)\big]. F=NNQ∑i∈Qai(1−ai)F=\tfrac{N}{N_Q}\sum_{i\in Q}a_i(1-a_i), with QQ the NQN_Q quantized Keys among all NN, depends only on the attention: it is at most N/NQN/N_Q and falls to zero as the mass concentrates, since a Key that holds the mass barely moves the softmax. hh is the error variance of attended Keys relative to the average Key. Hence, for equal errors, uniform attention is the worst case, and the surrogate can underestimate only if Fh>1F h>1; since F≤N/NQ≈1F\le N/N_Q\approx1, this essentially requires h>1h>1.

(2) Attended Keys receive no larger errors. A Key's error is set by the range of its group of 32 Keys, not by its attention; min–max rounding never clips, so even outlier Keys stay within half a step; the sink token and the 128 most recent tokens stay in BF16. Measured over 294,784 (layer, head, reader) cells, the median hh is 0.995, and 0.81 in the most concentrated 1%.

(3) Attention is preserved under concentration. We bin the cells by how concentrated their attention is, measured by the mass on the top 1% of Keys (median 0.93, so concentration is the typical case).

Concentration binMass on top 1% of KeysSensitivity FF ↓Attended / average error hh ↓KL, INT2 ↓KL, LMTC ↓LMTC / INT2 ↓
0–20%0.490.661.020.09050.01470.163
40–60%0.920.130.990.04570.00550.119
80–90%0.990.021.010.02960.00170.056
top 1%1.000.060.810.96490.03950.041

The gain of LMTC grows with concentration. In the most concentrated 1%, a small logit error can reorder the few attended Keys and the KL of INT2 jumps to 0.96; LMTC keeps it 24× lower and flips the most attended Key in 5–16% of these cells, against 45% for INT2.

(4) The attention-weighted objective brings no gain. A coder minimizing the attention-weighted objective with the true attention, under otherwise identical information, reaches 0.063 / 0.117 (Near / Far, relative to INT2) against 0.068 / 0.110 for uniform weights, and 0.023 against 0.024 in the most concentrated 10% of cells, at 1.7K–33K tokens alike. Both coders use the readers and Keys of the evaluated request, unavailable to LMTC, hence the values below W1. Uniform weighting is therefore not a bottleneck.

W3: Efficiency is not fully established: (i) generated tokens stay in BF16 (App. D.6); (ii) timings exclude prefill and cache conversion; (iii) only Qwen2.5-Omni-7B on A100 is tested. Do the gains hold under quantized decoding, end to end, and across models and hardware?

A: We thank the reviewer for these practical points. We address accuracy under sustained quantization, end-to-end cost including cache construction, and transfer across models and hardware.

(1) Sustained quantized decoding. We repeat D.6 under sustained quantization, with generated tokens quantized as in deployment, and measure at every generated token the KL of the next-token distribution to BF16 and the top-1 agreement.

Generated tokensKL to BF16, KIVIKL to BF16, LMTCTop-1 agreement, KIVITop-1 agreement, LMTC
1–2560.0360.003691.7%97.7%
257–10240.0440.005894.0%97.5%
1025–20480.0270.003997.5%98.9%

The error of LMTC does not accumulate and stays at 0.004–0.006 up to 2K tokens, about 8× below KIVI throughout. The task accuracy is also preserved. On 200 Daily-Omni questions answered by free step-by-step generation, LMTC matches BF16 (42.5% vs. 42.5%) and changes the final answer in 22% of the questions, against 42.5% for KIVI.

(2) End-to-end efficiency including cache construction. Please see our joint response on end-to-end efficiency below. In short, building the LMTC cache costs 7–31 ms (0.6–1.1% of the time to first token), the least of all compressed methods, and is negligible in the total time.

(3) Other models and hardware. We measured decoding on four further backbones with the same kernels (one A100-40GB, eager mode). This is a speed and memory measurement; the cache has the layout and bit allocation of an LMTC cache. Entries are the decode latency per token in ms, BF16 FlashDecoding-v2 → LMTC (speedup).

Model (layers / query heads / KV heads)4K64K128K256KBatch 48 at 4K (tokens/s)KV memory
Qwen2.5-Omni-7B (28 / 28 / 4)19.2 → 18.1 (1.06×)25.3 → 20.4 (1.24×)35.8 → 23.5 (1.52×)55.2 → 31.4 (1.75×)1055 → 1618 (1.53×)5.3× smaller
OmniVinci-9B (28 / 28 / 4)19.4 → 19.6 (0.99×)26.5 → 21.5 (1.23×)36.9 → 24.7 (1.50×)56.3 → 32.6 (1.73×)1009 → 1515 (1.50×)5.3× smaller
MiniCPM-o 2.6 (28 / 28 / 4)19.6 → 19.4 (1.02×)26.3 → 21.3 (1.23×)36.7 → 24.4 (1.50×)56.0 → 32.3 (1.73×)1026 → 1555 (1.52×)5.3× smaller
Qwen3-8B (36 / 32 / 8)33.4 → 32.9 (1.01×)45.2 → 32.0 (1.41×)69.5 → 39.5 (1.76×)OOM → 57.1OOM → 8945.3× smaller
MiniCPM-o 4.5 (36 / 32 / 8)33.1 → 32.6 (1.02×)45.2 → 32.4 (1.39×)69.4 → 39.5 (1.76×)OOM → 57.2OOM → 8945.3× smaller

The gains transfer: models with the attention shape of Qwen2.5-Omni-7B show the same speedups (1.73–1.75× at 256K); models with 8 KV heads, whose BF16 cache is twice as large, gain earlier (1.4× at 64K) and still run where BF16 runs out of memory.

On a second GPU type, an RTX 5000 Ada (32 GB), the kernels pass the same correctness test, and the speedup grows with the context as on the A100 (Qwen2.5-Omni-7B, batch 1, decode latency per token in ms):

Context16K32K64K128K256K
BF16 FlashDecoding-v230.3433.6638.5948.23OOM
LMTC29.6030.0231.5234.7740.79
Speedup1.03×1.12×1.22×1.39×–
KV cache, BF16 → LMTC (GB)0.90 → 0.181.78 → 0.343.53 → 0.677.03 → 1.33– → 2.64

The kernels were developed and tuned on the A100 and were not re-tuned for this GPU, so these speedups are obtained without any hardware-specific optimization. We are still seeking for more GPU types to do the test.

Q1: Are generated KV states quantized after leaving the BF16 residual window in the main evaluations?

A: We thank the reviewer for this question. Following the evaluation protocol of OScaR, every method keeps the 128 most recent tokens in BF16 (Sec. 4.1), and prompt and generated tokens are treated alike: a token is quantized once it leaves this window. Since most benchmarks are multiple-choice with short answers, few generated tokens leave the window in the main evaluation. MMAU-Pro is the exception: 712 of its 5,305 questions are open-ended or instruction-following, and when an answer exceeds 128 tokens, its earlier tokens are quantized. Even there, LMTC matches BF16 on all three models (47.0 vs. 47.1, 50.0 vs. 50.0, 54.8 vs. 54.8 in Table 1), while KIVI loses 0.8–2.2 points.

Q2: How much does converting the initial KV cache into the quantized representation cost, and do the speedups persist when this cost is included?

A: We thank the reviewer for this question. Please see our joint response on end-to-end efficiency below. In short, converting the cache costs 7–31 ms for prompts of 5K–35K tokens (0.6–1.1% of the time to first token), the least of all compressed methods, because each layer is encoded inside the prefill; with this cost included, LMTC has the lowest end-to-end latency at every prompt and answer length we measured.

Comments on the paper

2 comments

Joint response: end-to-end efficiency including cache construction

We thank the reviewers for raising this point. We measured the full pipeline on real Daily-Omni audio-visual inputs with Qwen2.5-Omni-7B on one A100-40GB. Every baseline uses its authors' released implementation; all methods share the same model code and FlashAttention-2 prefill, so only cache construction and decoding differ. No CUDA Graph is used; entries are medians of 3 runs, averaged over 2 clips per length, in ms.

(1) Cache construction costs under 1% of the time to first token. TTFT covers the audio and vision encoders, the LLM prefill, cache construction and the first token. Parentheses give the extra time over BF16, i.e., the cost of building each compressed cache.

Prompt length (tokens)BF16 (no compression)KIVIOTTOScaRLMTC
5.2K642.5651.0 (+8.5)655.0 (+12.5)658.5 (+16.0)649.5 (+7.0)
10.3K1284.51298.5 (+14.0)1301.5 (+17.0)1313.5 (+29.0)1295.5 (+11.0)
18K2434.02461.5 (+27.5)2465.0 (+31.0)2480.0 (+46.0)2456.0 (+22.0)
35K5206.55260.0 (+53.5)5262.5 (+56.0)5286.0 (+79.5)5237.0 (+30.5)
66K1196812027 (+59)12137 (+169)12189 (+221)11949 (−19)*
105KOOM23355OOMOOM23359

*Within the run-to-run variation: near the 40 GB limit, the BF16 prefill itself slows down.

Building the LMTC cache costs 7–31 ms, the least of all compressed methods at every length and 0.6–1.1% of TTFT for 5K–35K tokens. It is cheap for two reasons: each layer is encoded inside the prefill as soon as its Keys and Values exist, and its BF16 copy is freed; and the token-layout plan is computed on the CPU while the encoders run on the GPU. The cost is linear in the prompt length while prefill attention is quadratic, so its share of TTFT shrinks with length. Since a full-length BF16 cache never exists, LMTC also lowers the peak prefill memory (10.1 GiB extra at 66K, against 11.8–13.6 GiB) and still answers at 105K tokens, where BF16, OTT and OScaR run out of memory.

(2) The speedups persist end to end, even for short answers. Latency from the start of the prefill to the NN-th generated token, greedy decoding; every method runs its released decode path, BF16 uses FlashDecoding-v2.

PromptNNBF16KIVIOTTOScaRLMTC
5.2K16976128412801063905
6421033379333424091761
2566607117361150478065180
18K1627633090310028782722
6438915186514842283592
2568398135361334196117086
35K1655665879599956805513
6467477961844970266407
256114741626918210123939979
66K161240112892132111261412314
641391015576168431395513274
2561994626324314571936517117
105K16OOM24606OOMOOM23812
64OOM28322OOMOOM24855
256OOM43198OOMOOM29027

LMTC has the lowest end-to-end latency at every prompt length and every answer length, already with 16 answer tokens: its construction cost is recovered within the first few decoding steps.

We sincerely thank all reviewers for their constructive feedback during the review and discussion stages, and we thank the Area Chair for their time and dedication in ensuring a fair and rigorous evaluation process.

We are encouraged that two of the three reviewers recommended acceptance (weak accept) in their initial reviews and kept their positive ratings after the discussion. The reviewers also consistently recognized the core contributions of our work:

  1. A well-motivated, attention-aligned objective. All three reviewers agreed that measuring Key quantization error by its effect on attention logits, rather than by reconstruction error, addresses a meaningful limitation of conventional KV-cache quantization (FLhK, juFH, axf9).

  2. A principled, role-aware design. Reviewers valued our separate treatment of Keys and Values according to how their errors propagate through attention (juFH, axf9).

  3. Query-free formulation suited to streaming. Reviewer juFH found the query-free reduction of Key-induced logit distortion particularly appealing for streaming omni-modal inputs.

  4. Thorough and practical validation. Reviewers highlighted our detailed ablations and results on several model families and eight benchmarks (FLhK), and the practical relevance of the streaming and packed-cache decoding measurements.

We addressed the main concerns with new analysis and experiments:

  1. Do the speedups hold once cache construction is included? (FLhK, juFH, axf9) Yes. Building the cache costs 7–31 ms (0.6–1.1% of TTFT), and LMTC has the lowest end-to-end latency at every length, even for 16-token answers.

  2. Is accuracy robust under sustained quantization and long inputs? (FLhK, axf9) Yes. With generated tokens quantized, the KL to BF16 stays flat up to 2K tokens (8× below KIVI); from 5K to 33K input tokens, accuracy stays within −0.6 to +0.4 points of BF16.

  3. Is the query-free metric justified, also without query biases? (FLhK, juFH) Yes. It makes nearly the same coding decisions as real queries and comes close to a real-query reference. For models without query bias such as Qwen3-Omni, a data-free extension cuts the Key KL by about one third, bringing it close to the reference.

  4. Why can fast rotary pairs omit the basis? (axf9, remaining point) The objective is unaffected (cross terms below 0.1% of the distortion); the residual correlation only costs a few bits.

Toward the end of the discussion, all three reviewers confirmed that our responses addressed their concerns. Reviewer FLhK stated that we had largely addressed their concerns and kept their positive score. Reviewer juFH stated that our additional experiments on streaming overhead, models without query bias and calibration-domain sensitivity addressed their main concerns, and maintained their positive rating. Reviewer axf9 confirmed that our experiments addressed their experimental concerns. The only remaining point (item 4) concerns exposition, and we have provided a complete response to it, backed by direct measurement. We respectfully ask the Area Chairs to take this into account when weighing the current ratings.

In summary, LMTC is a training-free and query-free KV-cache quantization framework that directly targets attention distortion in omni-modal LLMs. It keeps over 99% of BF16 accuracy at three bits per scalar and delivers real end-to-end gains in memory and latency. We thank all reviewers and the Area Chairs again for their time and effort.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.