- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes LIVE-MLA, a framework for converting pretrained MHA/GQA models into Multi-Head Latent Attention (MLA) while reducing KV-cache memory and preserving model performance. The authors observe that conventional low-rank conversion methods introduce reconstruction residuals that remain uncorrected during inference. The proposed method contains two main components. First, attention-weighted conversion prioritizes the reconstruction of keys and values receiving greater attention mass. Specifically, the low-rank mapping is optimized through an attention importance weighted way.
Second, Residual Feedback (RF) exploits a small exact recent-token window to observe value reconstruction errors. Using paired latent representations and residuals, the method fits a sequence-specific ridge regression, which predicts residual corrections for older compressed tokens without updating model parameters. The correction can also be absorbed into the attention computation to avoid explicitly reconstructing every historical value.
Experiments on Llama-3.1-8B-Instruct and Qwen3-8B-Base demonstrate substantial improvements in training-free perplexity under 50%, 75%, and 87.5% KV-cache reduction. On Qwen3-8B-Base, LIVE-MLA achieves perplexity 13.90 at 87.5% compression, compared with 103.5 for CARE. With additional recovery training, the method achieves competitive or superior results using 4--60 times fewer training tokens than the evaluated baseline adaptations. All persistent residual and exact-window states are included in the reported cache budgets.
Critical strengths
The proposed Residual Feedback mechanism exploits sequence-specific reconstruction errors observed during inference, providing a practical approach to correcting older compressed values without gradient-based updates. Its integration with low-dimensional attention computation is computationally efficient and conceptually well motivated.
LIVE-MLA achieves substantial perplexity improvements under aggressive KV-cache compression across two 8B models. The component ablations support the contributions of attention-weighted conversion, the exact window, and Residual Feedback. The reduced recovery-training requirements further demonstrate practical potential.
Critical weaknesses
The main downstream evaluations focus on 2K--4K contexts, despite the motivation of long-context KV-cache compression. Residual Feedback estimates corrections for older tokens using only a small set of recent observations, and its effectiveness under long-range distribution shifts or abrupt topic changes remains unclear. The theoretical analysis does not guarantee consistent improvements under such conditions.
Questions for authors
Can the authors evaluate Residual Feedback on substantially longer contexts (e.g., 32K--128K tokens) and analyze how correction accuracy changes with the temporal distance between recent residual observations and older compressed tokens?
5 comments
Official Comment by Authors (2/3)
Correction accuracy over distance
We measured correction accuracy on 16 individual 32K documents, aggregating squared errors over nine query positions and ten layers. Raw R² measures individual residual correction. Functional R² measures correction of the residual sum formed using the converted model's attention weights. Both use no correction as the reference, so negative values indicate increased error.
Each cell reports raw R² / functional R² for the original LIVE-MLA configuration.
| Mem. | 16–64 | 64–256 | 256–1K | 1K–4K | 4K–16K | 16K–32K |
|---|---|---|---|---|---|---|
| −50% | 0.56 / 0.75 | 0.30 / 0.37 | 0.26 / 0.44 | 0.24 / 0.56 | 0.22 / 0.63 | 0.15 / 0.55 |
| −75% | 0.51 / 0.79 | 0.20 / 0.25 | 0.17 / 0.21 | 0.15 / 0.37 | 0.12 / 0.45 | 0.06 / 0.33 |
| −87.5% | 0.44 / 0.75 | 0.08 / −0.15 | 0.05 / −0.31 | 0.03 / 0.15 | 0.01 / 0.26 | −0.05 / 0.06 |
Distances are measured from the query to the target token. Fitting observations lie 16–31 tokens before the query, so the first bin overlaps the fitting region. For a target at distance d > 31, its separation from the observations ranges from d − 31 to d − 16.
These results measure residual correction error at 32K, whereas Table 26 measures final prediction NLL at 2K. Improvements in prediction NLL therefore do not require lower residual correction error in every distance bin.
Both measures remain positive through 16K–32K under 50% and 75% memory reduction. Under 87.5% reduction, accuracy varies across bins and layers, with positive functional results beyond 1K depending on the final two evaluated layers.
With BR, both raw and functional correction errors increase relative to no correction at 16K–32K under 75% and 87.5% reduction, so overall perplexity improvements do not imply accurate correction at every distance.
RF's contribution at 32K and 128K
We evaluated Qwen3-8B-Base with and without RF, without recovery training, in both configurations. The RF-disabled settings use W = 15. The 128K comparison uses two WikiText-2 windows with YaRN factor 4, with RF enabled and disabled evaluated on the same windows using the same harness.
| Context | Mem. | LIVE-MLA, RF enabled | LIVE-MLA, RF disabled | LIVE-MLA + BR, RF enabled | LIVE-MLA + BR, RF disabled |
|---|---|---|---|---|---|
| 32K | −50% | 19.29 | 20.93 | 6.27 | 6.28 |
| −75% | 26.47 | 47.80 | 8.09 | 8.65 | |
| −87.5% | 58.76 | 155.10 | 13.46 | 15.88 | |
| 128K | −50% | 23.34 | 27.05 | 6.64 | 6.71 |
| −75% | 32.05 | 73.81 | 8.77 | 9.82 | |
| −87.5% | 54.52 | 208.81 | 14.95 | 20.45 |
Values are perplexity, with the better result within each RF enabled/disabled pair in bold. RF improves both configurations at every evaluated compression level at 32K and 128K. With BR, RF's benefit is small under 50% reduction and increases under stronger compression.
Contribution of distant corrections
We restricted correction to targets within a maximum distance, keeping the converted parameters and fitting observations unchanged. The table reports NLL over three 32K windows under 87.5% memory reduction.
| Method | Correct through 256 | Correct through 2K | Correct through 16K | Correct all distances |
|---|---|---|---|---|
| LIVE-MLA | 4.991 | 4.861 | 4.164 | 4.101 |
| LIVE-MLA + BR | 2.625 | 2.589 | 2.575 | 2.571 |
For both configurations, extending correction to more distant tokens reduces NLL in these three windows. With BR, the additional improvement beyond 16K is small.
This experiment evaluates prediction quality on a different set of samples, so its results need not follow the same pattern as the residual correction errors above.
Scope of the theoretical analysis
Appendix D characterizes the conditions under which RF improves the attention output, rather than guaranteeing improvement under arbitrary distribution shifts. Proposition D.4 relates the gain to bias reduction, observation noise, and the covariance between fitting and target residuals. Our experiments evaluate RF's effect on prediction quality. RF lowers perplexity in all evaluated on/off comparisons at 32K and 128K, and also improves perplexity after domain changes at 8K and 32K. These results support its effectiveness under the evaluated conditions.
Official Comment by Authors (3/3)
Overall quality at longer contexts
We evaluated overall quality at 8K, 32K, and 128K, using YaRN with factor 4 at 128K. Each LIVE-MLA + BR row uses the same converted parameters or recovery checkpoint as the preceding LIVE-MLA row. All recovery checkpoints were trained without BR.
Without recovery training
Bold indicates the best converted-model result within each memory reduction level. Original is shown in italics as a reference.
−50% memory
| Method | PPL 8K ↓ | PPL 32K ↓ | PPL 128K ↓ | NIAH 8K ↑ | NIAH 32K ↑ |
|---|---|---|---|---|---|
| Original | 6.24 | 5.98 | 7.07 | 99.25 | 96.00 |
| TransMLA | 105.63 | 297.27 | 586.24 | 0.12 | 0.00 |
| MHA2MLA | 934.57 | 1258.27 | 1411.46 | 0.00 | 0.00 |
| CARE | 9.39 | 9.25 | 14.96 | 79.06 | 56.09 |
| LIVE-MLA | 8.67 | 19.29 | 23.34 | 23.81 | 1.38 |
| LIVE-MLA + BR | 6.52 | 6.27 | 6.64 | 91.40 | 76.44 |
−75% memory
| Method | PPL 8K ↓ | PPL 32K ↓ | PPL 128K ↓ | NIAH 8K ↑ | NIAH 32K ↑ |
|---|---|---|---|---|---|
| Original | 6.24 | 5.98 | 7.07 | 99.25 | 96.00 |
| TransMLA | 2365.22 | 5370.94 | 8372.40 | 0.00 | 0.00 |
| MHA2MLA | 2148.50 | 2247.90 | 2044.08 | 0.00 | 0.00 |
| CARE | 20.58 | 26.21 | 54.37 | 23.38 | 3.97 |
| LIVE-MLA | 10.87 | 26.47 | 32.05 | 1.25 | 0.00 |
| LIVE-MLA + BR | 8.02 | 8.09 | 8.77 | 20.28 | 8.88 |
−87.5% memory
| Method | PPL 8K ↓ | PPL 32K ↓ | PPL 128K ↓ | NIAH 8K ↑ | NIAH 32K ↑ |
|---|---|---|---|---|---|
| Original | 6.24 | 5.98 | 7.07 | 99.25 | 96.00 |
| TransMLA | 4268.43 | 13466.64 | 12774.41 | 0.00 | 0.00 |
| MHA2MLA | 4252.93 | 4654.58 | 4758.09 | 0.00 | 0.00 |
| CARE | 111.18 | 168.31 | 235.91 | 4.38 | 0.00 |
| LIVE-MLA | 18.03 | 58.76 | 54.52 | 0.00 | 0.00 |
| LIVE-MLA + BR | 13.01 | 13.46 | 14.95 | 0.00 | 0.00 |
After recovery training
The tables report the final baseline checkpoints. Comparisons at matched token budgets are provided in our response to Reviewer 8Y2M. Each LIVE-MLA + BR row uses the same recovery checkpoint as the preceding LIVE-MLA row.
−50% memory
| Method | Tokens | PPL 8K ↓ | PPL 32K ↓ | PPL 128K ↓ | NIAH 8K ↑ | NIAH 32K ↑ |
|---|---|---|---|---|---|---|
| Original | — | 6.24 | 5.98 | 7.07 | 99.25 | 96.00 |
| TransMLA | 3B | 6.63 | 6.91 | 9.84 | 85.50 | 52.13 |
| MHA2MLA | 6B | 7.59 | 188.64 | 1179.17 | 29.66 | 0.00 |
| LIVE-MLA | 100M | 6.42 | 15.94 | 37.00 | 76.78 | 13.25 |
| LIVE-MLA + BR | 100M | 6.36 | 6.16 | 6.44 | 97.60 | 83.62 |
−75% memory
| Method | Tokens | PPL 8K ↓ | PPL 32K ↓ | PPL 128K ↓ | NIAH 8K ↑ | NIAH 32K ↑ |
|---|---|---|---|---|---|---|
| Original | — | 6.24 | 5.98 | 7.07 | 99.25 | 96.00 |
| TransMLA | 6B | 6.63 | 7.32 | 11.02 | 85.47 | 44.63 |
| MHA2MLA | 6B | 7.76 | 234.09 | 2270.61 | 27.25 | 0.00 |
| LIVE-MLA | 300M | 6.55 | 9.01 | 12.49 | 75.69 | 12.88 |
| LIVE-MLA + BR | 300M | 6.53 | 6.39 | 6.71 | 96.78 | 76.00 |
−87.5% memory
| Method | Tokens | PPL 8K ↓ | PPL 32K ↓ | PPL 128K ↓ | NIAH 8K ↑ | NIAH 32K ↑ |
|---|---|---|---|---|---|---|
| Original | — | 6.24 | 5.98 | 7.07 | 99.25 | 96.00 |
| TransMLA | 6B | 6.91 | 8.32 | 12.08 | 64.50 | 27.50 |
| MHA2MLA | 6B | 8.50 | 259.01 | 2182.34 | 14.75 | 0.00 |
| LIVE-MLA | 1.5B | 6.94 | 7.14 | 8.32 | 58.91 | 43.59 |
| LIVE-MLA + BR | 1.5B | 7.07 | 7.21 | 7.95 | 65.36 | 40.00 |
BR improves perplexity substantially without recovery training, while its effect after recovery training varies.
YaRN affects the models differently. At 32K, factor 4 increases Original's perplexity from 5.98 to 6.67 and LIVE-MLA + BR's from 6.27 to 6.57. We therefore do not interpret perplexity below Original at 128K as evidence of superior long-context capability. RF on/off comparisons cover 32K and 128K, while the distance-resolved correction analyses use 32K inputs.
We thank the reviewer again for the insightful questions, which motivated a closer examination of RF across context lengths, correction distances, and domain changes, strengthening our understanding of when and how it helps.
Official Comment by Authors (1/3)
We thank the reviewer for the careful evaluation and constructive feedback. We provide additional experiments on RF at 32K and 128K, correction accuracy over distance, and domain changes.
W1 & Q1. RF at longer contexts and under domain changes
To adapt LIVE-MLA to longer contexts, we evaluated blockwise rotation (BR), which applies RoPE to the content coordinates of compressed keys using the center position of each 256-token block, while retaining token-specific RoPE on the rotary coordinates. This blockwise treatment allows the rotation to be folded into the query, preserving LIVE-MLA's absorbed decoding. BR uses the same converted parameters or recovery checkpoint without additional fitting, training, or cache storage. We report results for both the original configuration and LIVE-MLA + BR.
We measured BR's decoding latency using the same procedure and kernel settings as Table 16, with RF enabled in both configurations.
Abrupt domain changes
We constructed inputs from fiction, government reports, meetings, and code using LongBench contexts. Each input ends with 2,048 tokens from a different document. The preceding text comes either from the same domain or a different domain. Both conditions change documents at the same boundary, with 32 inputs per condition at each length. The RF-disabled settings use W = 15.
The table reports perplexity over offsets [512, 2047) after the boundary using the original LIVE-MLA configuration without recovery training.
RF lowers aggregate perplexity in both conditions at every evaluated compression level. In this interval, fitting observations come from the new document, while correction targets include the preceding text. The different-domain condition therefore evaluates correction of older tokens from a different domain. These measurements use the original positional configuration, whose perplexity also degrades at 32K in the overall quality evaluation.
We also evaluated RF with BR on the 32K domain-change inputs described above.
Values are perplexity after the document boundary, without recovery training. RF lowers aggregate perplexity in both domain conditions with BR.