- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents Logit-Metric Transform Coding (LMTC), a training-free and query-free KV-cache quantization method that aims to minimize expected attention-logit distortion. Keys are quantized through modality-conditioned transform coding, guided by a logit metric derived from query projection weights. Values are quantized using phase-offset rounding, which shifts the rounding grids of spatiotemporally adjacent video tokens to encourage error cancellation during attention aggregation. Experiments on three omni-modal LLMs across eight benchmarks demonstrate the method’s effectiveness.
Critical strengths
The problem is well motivated. Measuring key quantization error through its effect on attention logits provides an objective more closely aligned with attention behavior than reconstruction error alone. The paper also distinguishes how key and value errors propagate through attention, providing a useful basis for designing separate quantization strategies.
Critical weaknesses
-
I found the presentation difficult to follow. Figure 1 is meant to motivate the method, but readers need details from Section 3 to understand it. The caption should explain the weight-derived directions, the calculation of “KL share,” and the aggregation of modality attention-mass changes. The notation is also dense, and it is easy to lose track of symbol definitions and coordinate changes. A notation table and brief reminders would help.
-
Many derivations are provided in the appendix, but the main text needs more explanation of how the equations connect. In particular, the transition from Equation 1 to the per-token coding objective, and from Equation 4 to Equation 6, should be explained more clearly. It is not always clear which steps are exact and which rely on assumptions or approximations.
Questions for authors
-
How much task accuracy is lost relative to BF16 as the actual input context becomes longer? The paper includes long-context timing results and reader-position tests, but does not report accuracy loss across input context lengths. This would help assess whether the method remains reliable on long inputs.
-
Section 4.4 excludes prefill and prefix encoding. How expensive is constructing the quantized cache compared with the baselines? Reporting cache construction time or time to first token would help, especially for long inputs followed by short answers, where decoding savings have less time to offset any additional encoding cost.
-
In Section 3.2 and Appendix D.2, the argument for omitting basis transformations in the fast coordinates needs further explanation. Small within-pair rotary coherence does not rule out cross-pair correlations. Averaging the query metric over future reader positions also does not establish that keys at their actual cache positions are decorrelated. What additional conditions are needed, and how large are the remaining correlations? Table 7 supports the chosen split empirically, but does not directly check these conditions.
9 comments
Q1: How much task accuracy is lost relative to BF16 as the actual input context becomes longer? The paper includes long-context timing results and reader-position tests, but does not report accuracy loss across input context lengths. This would help assess whether the method remains reliable on long inputs.
A: We thank the reviewer and added this experiment. Accuracy loss relative to BF16 does not grow with input length from 5K to 33K tokens.
We ran 500 LVBench multiple-choice questions on Qwen2.5-Omni-7B with audio and video, in windows of 30, 60, 120 and 190 s (5.2K–32.5K tokens), with the evidence either right before the question or at the start of the window, where it has to be read across the whole cache. Accuracy in %; is the change against BF16 in points, flip the share of answers that differ from BF16, and KL is over the option letters:
| Window | Tokens | Evidence | BF16 | LMTC | LMTC flip / KL | KIVI | KIVI flip / KL |
|---|---|---|---|---|---|---|---|
| 30 s | 5.2K | end | 68.4 | −0.6 | 1.0% / 0.004 | +0.0 | 8.0% / 0.055 |
| start | 68.2 | −0.6 | 1.4% / 0.004 | +0.2 | 9.2% / 0.044 | ||
| 60 s | 10.3K | end | 68.2 | +0.2 | 1.4% / 0.005 | −1.2 | 10.4% / 0.052 |
| start | 68.6 | −0.2 | 2.2% / 0.004 | −1.6 | 9.2% / 0.043 | ||
| 120 s | 20.6K | end | 65.6 | −0.6 | 2.0% / 0.006 | −1.8 | 10.4% / 0.064 |
| start | 66.8 | +0.0 | 3.2% / 0.005 | −1.8 | 11.2% / 0.048 | ||
| 190 s | 32.5K | end | 61.8 | −0.2 | 2.6% / 0.005 | −0.8 | 8.0% / 0.044 |
| start | 63.4 | +0.4 | 3.2% / 0.004 | −1.6 | 9.6% / 0.048 |
At every length and evidence position, LMTC stays within −0.6 to +0.4 points of BF16, with a KL of 0.004–0.006 and 1.0–3.2% of the answers changed, against 0.04–0.06 and 8.0–11.2% for 2-bit KIVI.
Q2: Section 4.4 excludes prefill and prefix encoding. How expensive is constructing the quantized cache compared with the baselines? Reporting cache construction time or time to first token would help, especially for long inputs followed by short answers, where decoding savings have less time to offset any additional encoding cost.
A: We thank the reviewer for this important question. Please see our joint response on end-to-end efficiency above, which reports TTFT and the latency to the -th answer token for , and . In short, building the LMTC cache costs 7–31 ms (0.6–1.1% of TTFT), the least of all compressed methods, and LMTC has the lowest end-to-end latency at every prompt length, even for 16-token answers after long inputs.
Q3: In Section 3.2 and Appendix D.2, the argument for omitting basis transformations in the fast coordinates needs further explanation. Small within-pair rotary coherence does not rule out cross-pair correlations. Averaging the query metric over future reader positions also does not establish that keys at their actual cache positions are decorrelated. What additional conditions are needed, and how large are the remaining correlations? Table 7 supports the chosen split empirically, but does not directly check these conditions.
A : We thank the reviewer for these insightful comments. Both concern the same step, so we answer them together.
Eq. 4 de-rotates each Key by its own position, so scoring would need one query projection per Key. To avoid this, slow rotary pairs, which turn by at most 0.75 rad within a segment of 255 positions, share one de-rotation per segment, while fast pairs are stored after rotation, without a basis, and scored with no projection. Eq. 6 is the exact score of the Key reconstructed from these codes. The approximation lies only in the codes, mainly in coding the fast pairs without a basis.
A basis does two things, and omitting it is safe if both already hold without it. First, the distortion must remain a sum of per-coordinate terms, as the bit allocation assumes. As the reviewer notes, averaging over reader positions does not make the coupling between pairs small for an actual reader (RMS 0.16), so we rely instead on the errors being uncorrelated: rounding errors are independent, and for zero-bit coordinates this only needs decorrelated Keys. The measured cross terms are below 0.1% of the distortion on average. Second, no bits should be spent on correlated coordinates, which needs the Keys to be decorrelated after rotation. Rotation averages out the correlation between pairs when the Key positions span many periods. On Qwen2.5-Omni-7B (fast pairs 0–26), the cross-pair correlation (RMS) before → after rotation is:
| Fast pairs | Audio | Video |
|---|---|---|
| Temporal (0–15) | 0.23 → 0.02 | 0.23 → 0.03 |
| Temporal × height | 0.19 → 0.02 | 0.18 → 0.03 |
| Height (16–26) | 0.20 → 0.08 | 0.18 → 0.18 |
Both conditions hold except the second one for the video height pairs, whose positions take only the few rows of a frame. The first still holds there, since zero-bit coordinates lie almost entirely in the temporal pairs.
The remaining correlation costs little. A basis on the post-rotary fast pairs would lower the KL by only 4–5%, and recovering the pre-rotary correlation requires Eq. 4 itself, which lowers the KL by 39%/43% (Near/Far) but needs 234× more query projections. The default thus sits at the knee of the trade-off (Table 7 of the submission, Table 8 of the revision). Coding the video height pairs as slow pairs lowers the KL by a further 6%/12% for 19% more projection FLOPs; we report this variant in the appendix, and replace the two claims of Sec. 3.2 by the conditions and measurements above.
A: We thank the reviewer for this helpful suggestion. We have revised the paper accordingly; all changes are marked in blue.
The caption of Figure 1 now reads:
Following the reviewer's suggestion, we added the missing definitions to Observation (i), where the figure is first discussed: the query-weight directions, the calculation of the KL share and the error share, and the aggregation of the attention-mass change. Figure 1 can now be read without Sec. 3. We then checked the rest of the section for the same problem and revised Observation (ii) and the caption of Figure 2 in the same way.
For the notation, we added a notation table (App. D.1, Table 5) and a reminder of the three Key frames at the start of Sec. 3.2, and we removed overloaded symbols.
Please see the updated paper for the details.
A: We thank the reviewer for this helpful suggestion. We revised Sec. 3.2 so that the path between the equations is spelled out in the main text, with each step marked as exact or approximate; all changes are in blue.
(1) From Equation 1 to the per-token coding objective. The goal is to keep the attention unchanged. A Key error eiK shifts the logit of token i by δi=q⊤eiK/d, and the attention changes only through how unevenly these shifts are spread over the tokens: up to higher-order terms, the attention KL is half their variance (its correlation with the exact KL is 0.99 or more, App. D.2). Weighting all tokens equally, which is the safe choice since concentrated attention is less sensitive to such shifts, and averaging over queries gives Eq. 1. This variance is at most the average of the per-token terms eiK⊤ΣqeiK, with equality because LMTC codes each Key around its modality mean, so that the errors have zero mean. Minimizing Eq. 1 therefore amounts to minimizing one term per token, which is what allows each token to be coded on its own. The only ingredient not available when the cache is written is the query statistic Σq; we obtain it from the weights (Eq. 2) and average it over the unknown position of the future reader (Eq. 3), which gives the per-token objective eˉi⊤Mˉeˉi. This weight-derived metric comes within 8% of one built from real queries (see our response to Reviewer FLhK, W1).
(2) From Equation 4 to Equation 6. This transition and the fast-coordinate question (Q3) concern the same step, so we answer them together under Q3 below.