acceptodds
Under review as a conference paper at ICLR 2027

LIVE-MLA: LATENT INFERENCE WITH VALUE ERROR FEEDBACK FOR MULTI-HEAD ATTENTION

Abstract

Most pretrained models use Multi-Head Attention (MHA) or Grouped-Query Attention (GQA). Recent methods convert these models to Multi-Head Latent Attention (MLA) through low-rank decomposition, reducing KV-cache storage without training comparable MLA models from scratch. The resulting low-dimensional bottleneck leaves reconstruction residuals that fixed low-rank mappings cannot adapt to during inference. We consider both how to correct these residuals during decoding and how to account for their effect when fitting the low-rank mappings. First, when a token leaves a small exact recent region, its latent representation and value residual are available together and can guide corrections for older compressed tokens. Second, KV reconstruction error alone is insufficient to assess compression, since keys and values matter through the attention outputs they produce. These observations lead to LIVE-MLA, which combines Residual Feedback with attention-weighted conversion. Residual Feedback fits sequence-specific value corrections from the observed pairs in closed form. Attention-weighted conversion weights each token's joint key-value reconstruction objective by the attention mass it receives. All persistent states are included in the same fixed cache budget. Experiments show that LIVE-MLA narrows the perplexity gap left by training-free conversion. Shorter recovery training further narrows the gap to the original model while using to fewer recovery-training tokens than the evaluated baseline adaptations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Head Latent Attention, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_n3MLOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LIVE-MLA, a framework for converting pretrained MHA/GQA models into Multi-Head Latent Attention (MLA) while reducing KV-cache memory and preserving model performance. The authors observe that conventional low-rank conversion methods introduce reconstruction residuals that remain uncorrected during inference. The proposed method contains two main components. First, attention-weighted conversion prioritizes the reconstruction of keys and values receiving greater attention mass. Specifically, the low-rank mapping is optimized through an attention importance weighted way.

Second, Residual Feedback (RF) exploits a small exact recent-token window to observe value reconstruction errors. Using paired latent representations and residuals, the method fits a sequence-specific ridge regression, which predicts residual corrections for older compressed tokens without updating model parameters. The correction can also be absorbed into the attention computation to avoid explicitly reconstructing every historical value.

Experiments on Llama-3.1-8B-Instruct and Qwen3-8B-Base demonstrate substantial improvements in training-free perplexity under 50%, 75%, and 87.5% KV-cache reduction. On Qwen3-8B-Base, LIVE-MLA achieves perplexity 13.90 at 87.5% compression, compared with 103.5 for CARE. With additional recovery training, the method achieves competitive or superior results using 4--60 times fewer training tokens than the evaluated baseline adaptations. All persistent residual and exact-window states are included in the reported cache budgets.

Critical strengths

The proposed Residual Feedback mechanism exploits sequence-specific reconstruction errors observed during inference, providing a practical approach to correcting older compressed values without gradient-based updates. Its integration with low-dimensional attention computation is computationally efficient and conceptually well motivated.

LIVE-MLA achieves substantial perplexity improvements under aggressive KV-cache compression across two 8B models. The component ablations support the contributions of attention-weighted conversion, the exact window, and Residual Feedback. The reduced recovery-training requirements further demonstrate practical potential.

Critical weaknesses

The main downstream evaluations focus on 2K--4K contexts, despite the motivation of long-context KV-cache compression. Residual Feedback estimates corrections for older tokens using only a small set of recent observations, and its effectiveness under long-range distribution shifts or abrupt topic changes remains unclear. The theoretical analysis does not guarantee consistent improvements under such conditions.

Questions for authors

Can the authors evaluate Residual Feedback on substantially longer contexts (e.g., 32K--128K tokens) and analyze how correction accuracy changes with the temporal distance between recent residual observations and older compressed tokens?

5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

We thank the reviewer for the careful evaluation and constructive feedback. We provide additional experiments on RF at 32K and 128K, correction accuracy over distance, and domain changes.

W1 & Q1. RF at longer contexts and under domain changes

To adapt LIVE-MLA to longer contexts, we evaluated blockwise rotation (BR), which applies RoPE to the content coordinates of compressed keys using the center position of each 256-token block, while retaining token-specific RoPE on the rotary coordinates. This blockwise treatment allows the rotation to be folded into the query, preserving LIVE-MLA's absorbed decoding. BR uses the same converted parameters or recovery checkpoint without additional fitting, training, or cache storage. We report results for both the original configuration and LIVE-MLA + BR.

We measured BR's decoding latency using the same procedure and kernel settings as Table 16, with RF enabled in both configurations.

Context × batchLIVE-MLA, Table 16 (ms/token)LIVE-MLA + BR (ms/token)
8K × 112.8413.12
8K × 412.7213.01
8K × 1616.6116.95
32K × 113.8713.85
32K × 415.3315.31
32K × 1626.4526.80

Abrupt domain changes

We constructed inputs from fiction, government reports, meetings, and code using LongBench contexts. Each input ends with 2,048 tokens from a different document. The preceding text comes either from the same domain or a different domain. Both conditions change documents at the same boundary, with 32 inputs per condition at each length. The RF-disabled settings use W = 15.

The table reports perplexity over offsets [512, 2047) after the boundary using the original LIVE-MLA configuration without recovery training.

ContextMem.Different domain, RF enabledDifferent domain, RF disabledSame domain, RF enabledSame domain, RF disabled
8K−50%10.4110.848.969.28
−75%14.1919.6612.3916.04
−87.5%31.0669.0424.5245.06
32K−50%37.8143.3124.1827.62
−75%71.37273.0640.78106.03
−87.5%192.98990.54106.84308.33

RF lowers aggregate perplexity in both conditions at every evaluated compression level. In this interval, fitting observations come from the new document, while correction targets include the preceding text. The different-domain condition therefore evaluates correction of older tokens from a different domain. These measurements use the original positional configuration, whose perplexity also degrades at 32K in the overall quality evaluation.

We also evaluated RF with BR on the 32K domain-change inputs described above.

Mem.Different domain, RF enabledDifferent domain, RF disabledSame domain, RF enabledSame domain, RF disabled
−50%5.775.795.485.52
−75%8.149.098.008.95
−87.5%15.7320.3914.7218.35

Values are perplexity after the document boundary, without recovery training. RF lowers aggregate perplexity in both domain conditions with BR.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

Correction accuracy over distance

We measured correction accuracy on 16 individual 32K documents, aggregating squared errors over nine query positions and ten layers. Raw R² measures individual residual correction. Functional R² measures correction of the residual sum formed using the converted model's attention weights. Both use no correction as the reference, so negative values indicate increased error.

Each cell reports raw R² / functional R² for the original LIVE-MLA configuration.

Mem.16–6464–256256–1K1K–4K4K–16K16K–32K
−50%0.56 / 0.750.30 / 0.370.26 / 0.440.24 / 0.560.22 / 0.630.15 / 0.55
−75%0.51 / 0.790.20 / 0.250.17 / 0.210.15 / 0.370.12 / 0.450.06 / 0.33
−87.5%0.44 / 0.750.08 / −0.150.05 / −0.310.03 / 0.150.01 / 0.26−0.05 / 0.06

Distances are measured from the query to the target token. Fitting observations lie 16–31 tokens before the query, so the first bin overlaps the fitting region. For a target at distance d > 31, its separation from the observations ranges from d − 31 to d − 16.

These results measure residual correction error at 32K, whereas Table 26 measures final prediction NLL at 2K. Improvements in prediction NLL therefore do not require lower residual correction error in every distance bin.

Both measures remain positive through 16K–32K under 50% and 75% memory reduction. Under 87.5% reduction, accuracy varies across bins and layers, with positive functional results beyond 1K depending on the final two evaluated layers.

With BR, both raw and functional correction errors increase relative to no correction at 16K–32K under 75% and 87.5% reduction, so overall perplexity improvements do not imply accurate correction at every distance.

RF's contribution at 32K and 128K

We evaluated Qwen3-8B-Base with and without RF, without recovery training, in both configurations. The RF-disabled settings use W = 15. The 128K comparison uses two WikiText-2 windows with YaRN factor 4, with RF enabled and disabled evaluated on the same windows using the same harness.

ContextMem.LIVE-MLA, RF enabledLIVE-MLA, RF disabledLIVE-MLA + BR, RF enabledLIVE-MLA + BR, RF disabled
32K−50%19.2920.936.276.28
−75%26.4747.808.098.65
−87.5%58.76155.1013.4615.88
128K−50%23.3427.056.646.71
−75%32.0573.818.779.82
−87.5%54.52208.8114.9520.45

Values are perplexity, with the better result within each RF enabled/disabled pair in bold. RF improves both configurations at every evaluated compression level at 32K and 128K. With BR, RF's benefit is small under 50% reduction and increases under stronger compression.

Contribution of distant corrections

We restricted correction to targets within a maximum distance, keeping the converted parameters and fitting observations unchanged. The table reports NLL over three 32K windows under 87.5% memory reduction.

MethodCorrect through 256Correct through 2KCorrect through 16KCorrect all distances
LIVE-MLA4.9914.8614.1644.101
LIVE-MLA + BR2.6252.5892.5752.571

For both configurations, extending correction to more distant tokens reduces NLL in these three windows. With BR, the additional improvement beyond 16K is small.

This experiment evaluates prediction quality on a different set of samples, so its results need not follow the same pattern as the residual correction errors above.

Scope of the theoretical analysis

Appendix D characterizes the conditions under which RF improves the attention output, rather than guaranteeing improvement under arbitrary distribution shifts. Proposition D.4 relates the gain to bias reduction, observation noise, and the covariance between fitting and target residuals. Our experiments evaluate RF's effect on prediction quality. RF lowers perplexity in all evaluated on/off comparisons at 32K and 128K, and also improves perplexity after domain changes at 8K and 32K. These results support its effectiveness under the evaluated conditions.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

Overall quality at longer contexts

We evaluated overall quality at 8K, 32K, and 128K, using YaRN with factor 4 at 128K. Each LIVE-MLA + BR row uses the same converted parameters or recovery checkpoint as the preceding LIVE-MLA row. All recovery checkpoints were trained without BR.

Without recovery training

Bold indicates the best converted-model result within each memory reduction level. Original is shown in italics as a reference.

−50% memory

MethodPPL 8K ↓PPL 32K ↓PPL 128K ↓NIAH 8K ↑NIAH 32K ↑
Original6.245.987.0799.2596.00
TransMLA105.63297.27586.240.120.00
MHA2MLA934.571258.271411.460.000.00
CARE9.399.2514.9679.0656.09
LIVE-MLA8.6719.2923.3423.811.38
LIVE-MLA + BR6.526.276.6491.4076.44

−75% memory

MethodPPL 8K ↓PPL 32K ↓PPL 128K ↓NIAH 8K ↑NIAH 32K ↑
Original6.245.987.0799.2596.00
TransMLA2365.225370.948372.400.000.00
MHA2MLA2148.502247.902044.080.000.00
CARE20.5826.2154.3723.383.97
LIVE-MLA10.8726.4732.051.250.00
LIVE-MLA + BR8.028.098.7720.288.88

−87.5% memory

MethodPPL 8K ↓PPL 32K ↓PPL 128K ↓NIAH 8K ↑NIAH 32K ↑
Original6.245.987.0799.2596.00
TransMLA4268.4313466.6412774.410.000.00
MHA2MLA4252.934654.584758.090.000.00
CARE111.18168.31235.914.380.00
LIVE-MLA18.0358.7654.520.000.00
LIVE-MLA + BR13.0113.4614.950.000.00

After recovery training

The tables report the final baseline checkpoints. Comparisons at matched token budgets are provided in our response to Reviewer 8Y2M. Each LIVE-MLA + BR row uses the same recovery checkpoint as the preceding LIVE-MLA row.

−50% memory

MethodTokensPPL 8K ↓PPL 32K ↓PPL 128K ↓NIAH 8K ↑NIAH 32K ↑
Original—6.245.987.0799.2596.00
TransMLA3B6.636.919.8485.5052.13
MHA2MLA6B7.59188.641179.1729.660.00
LIVE-MLA100M6.4215.9437.0076.7813.25
LIVE-MLA + BR100M6.366.166.4497.6083.62

−75% memory

MethodTokensPPL 8K ↓PPL 32K ↓PPL 128K ↓NIAH 8K ↑NIAH 32K ↑
Original—6.245.987.0799.2596.00
TransMLA6B6.637.3211.0285.4744.63
MHA2MLA6B7.76234.092270.6127.250.00
LIVE-MLA300M6.559.0112.4975.6912.88
LIVE-MLA + BR300M6.536.396.7196.7876.00

−87.5% memory

MethodTokensPPL 8K ↓PPL 32K ↓PPL 128K ↓NIAH 8K ↑NIAH 32K ↑
Original—6.245.987.0799.2596.00
TransMLA6B6.918.3212.0864.5027.50
MHA2MLA6B8.50259.012182.3414.750.00
LIVE-MLA1.5B6.947.148.3258.9143.59
LIVE-MLA + BR1.5B7.077.217.9565.3640.00

BR improves perplexity substantially without recovery training, while its effect after recovery training varies.

YaRN affects the models differently. At 32K, factor 4 increases Original's perplexity from 5.98 to 6.67 and LIVE-MLA + BR's from 6.27 to 6.57. We therefore do not interpret perplexity below Original at 128K as evidence of superior long-context capability. RF on/off comparisons cover 32K and 128K, while the distance-resolved correction analyses use 32K inputs.

We thank the reviewer again for the insightful questions, which motivated a closer examination of RF across context lengths, correction distances, and domain changes, strengthening our understanding of when and how it helps.

Reviewer_8Y2MOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

LIVE-MLA converts pretrained MHA/GQA models to MLA using attention-weighted low-rank fitting and inference-time value correction. A small exact window exposes latent–residual pairs as tokens leave the window; a ridge mapping fitted on these observations corrects older compressed values. Experiments on two 8B backbones show substantial training-free perplexity improvements. Recovery experiments on Qwen3-8B-Base also achieve competitive performance using fewer tokens than the evaluated baseline runs.

Critical strengths

The idea of using residuals exposed by the exact window as sequence-specific supervision is appealing. The byte-matched larger-window comparison (Table 9) and corrections restricted to disjoint distance bands (Table 10) provide convincing evidence that RF offers more than simply retaining additional recent tokens, particularly under aggressive compression.

The empirical and analytical support is unusually careful. The component ablations account for all persistent cache state, while the analysis distinguishes tokenwise reconstruction error from attention-output error and explicitly limits its guarantees. The consistent training-free perplexity gains across both backbones make the contribution practically interesting.

Critical weaknesses

The decoding-efficiency protocol needs clarification. Under the paper’s BF16 cache assumption, the Original model at 32K × 16 requires approximately 72 GiB for KV alone, using the standard Qwen3-8B-Base configuration. This exceeds the 48 GB capacity of the RTX 6000 Ada specified in Appendix B.6. It is unclear how this measurement was obtained. Moreover, quality is evaluated only up to 4K, whereas timing uses 8K and 32K, leaving the quality–latency tradeoff at those lengths unverified.

The recovery-efficiency comparison does not establish the minimum training budget needed by the baselines. The reported 4–60× ratios accurately describe the selected token budgets, but compare LIVE-MLA checkpoints with baseline endpoints after 3–6B tokens. Without matched-budget checkpoints or recovery curves, it remains unclear whether the baselines already reach comparable quality much earlier. Token counts also do not establish an equivalent reduction in training time.

Questions for authors

How was the Original 32K × 16 measurement accommodated on the reported GPU? Please clarify cache precision, device-resident memory, and whether Table 7 measures full-model decoding or a layerwise benchmark. Can you also provide quality results at the timing lengths?

Can you provide matched-budget recovery curves for TransMLA and MHA2MLA, particularly at 100M and 300M tokens? Were their training data, trainable parameter sets, and optimization settings matched to LIVE-MLA?

5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

We thank the reviewer for the careful evaluation and constructive feedback. We clarify the measurement conditions and provide additional quality results at the timing lengths and matched recovery budgets.

W1 & Q1. Decoding measurement conditions and quality at longer contexts

The original submission incorrectly listed the measurement device as an RTX 6000 Ada. The measurements in Table 16 were obtained on an NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition. We have corrected the device name in Appendix C.6 of the revised manuscript.

The reviewer's estimate of approximately 72 GiB for the Original model's BF16 KV cache at 32K × 16 is correct. We additionally measured peak GPU memory using the Table 16 harness on the same GPU, running each configuration separately. LIVE-MLA uses the original Table 16 configuration under 87.5% memory reduction, with RF enabled and W = 15.

MethodPeak allocated (GiB)Peak reserved (GiB)
Original90.2990.33
LIVE-MLA31.7432.65

These measurements include model weights, cache states, and temporary allocations. Model weights and resident buffers occupy 15.26 GiB. Both configurations fit within the device's CUDA-reported capacity of 94.97 GiB. All cache states reside in GPU memory, without CPU offloading.

A separate memory breakdown of LIVE-MLA at 32K × 16 measured 7.95 GiB for the latent and rotary cache state. Each stored token uses 161 latent and 64 rotary BF16 values, with no width padding. Including the stored key-normalization factors, calculated to occupy approximately 0.28 GiB, the compressed token state is approximately 8.2 GiB, or about 11.4% of the Original KV cache.

Table 16 reports decoding latency of the entire model in ms/token. Each configuration generates 64 tokens, and the reported value is the median step time after discarding the first eight steps, using CUDA graph replay and synchronization. Prefill is excluded. The Original KV cache and all persistent LIVE-MLA cache states are stored in BF16. FP32 tensors used for ridge computation are temporary workspace.

We evaluated quality at 8K and 32K to match the context lengths used in Table 16. Full quality results before and after recovery training are provided in our comment titled “Official Comment by Authors (3/3)”. Under 87.5% memory reduction, the original configuration used in Table 16 has perplexities of 13.90 at 2K, 18.03 at 8K, and 58.76 at 32K without recovery training, showing degradation as context length increases.

We also evaluated blockwise rotation (BR), which applies RoPE to the content coordinates of compressed keys using the center position of each 256-token block, while retaining token-specific RoPE on the rotary coordinates. In a diagnostic without compression, removing RoPE from the Original model's coordinates corresponding to LIVE-MLA's content coordinates increases its 32K perplexity from 5.98 to 20.31. Under 87.5% memory reduction, BR lowers 32K perplexity from 58.76 to 13.46 without recovery training. After recovery training on 1.5B tokens, 32K perplexity is 7.14 for LIVE-MLA and 7.21 for LIVE-MLA + BR, compared with 5.98 for Original.

Because the content rotation is shared within each block, it can be folded into the query before projection absorption, preserving absorbed decoding and the same cache storage. BR uses the same converted parameters or recovery checkpoint without additional fitting or training. All recovery checkpoints were trained without BR. We measured its decoding latency using the same procedure and kernel settings as Table 16 under 87.5% memory reduction.

Context × batchLIVE-MLA, Table 16 (ms/token)LIVE-MLA + BR (ms/token)
8K × 112.8413.12
8K × 412.7213.01
8K × 1616.6116.95
32K × 113.8713.85
32K × 415.3315.31
32K × 1626.4526.80
AuthorsOpenReview ↗

Official Comment by Authors (2/3)

W2 & Q2. Recovery at matched token budgets and training conditions

We evaluated the baselines at 100M tokens under 50% memory reduction and 300M tokens under 75% memory reduction, matching LIVE-MLA's token budgets. The quality tables in “Official Comment by Authors (3/3)” include these checkpoints alongside the longer baseline runs.

At matched token budgets, the original LIVE-MLA configuration outperforms both baselines on PPL 2K, LongBench 2K, and RULER 2K and 4K. For example, under 50% memory reduction, LIVE-MLA achieves a 2K perplexity of 6.99 at 100M tokens, compared with 7.57 for TransMLA at 100M tokens and 7.22 at 3B tokens.

TransMLA performs better on PPL 32K and NIAH 8K and 32K at these matched budgets, while enabling BR on the same LIVE-MLA checkpoints improves all three of these metrics beyond the matched TransMLA results. Additional baseline training generally improves perplexity, RULER, and NIAH, while LongBench 2K changes less consistently.

All methods use the TransMLA 6B-token mixture for recovery training. We follow the recovery recipes described in the respective baseline papers, so the trainable parameters and optimization settings differ.

SettingLIVE-MLATransMLAMHA2MLA
Trainable parametersAttention-related parametersAll parametersAll parameters
Learning rate2 × 10⁻⁵2 × 10⁻⁵1 × 10⁻⁴
Learning-rate scheduleConstantCosineOfficial decay schedule

The 4–60× ratios compare the training token budgets of the evaluated runs. To complement this comparison, we provide quality results at matched token budgets and direct measurements of training step time and peak GPU memory.

We measured recovery training step times and peak GPU memory on two RTX PRO 6000 Blackwell GPUs using FSDP, sequence length 4,096, 128 sequences per optimizer step, and gradient checkpointing.

Mem.MethodStep time (s)Peak GPU memory (GiB)
−50%TransMLA165.674.4
MHA2MLA136.171.5
LIVE-MLA116.954.9
−75%TransMLA164.172.7
MHA2MLA135.470.7
LIVE-MLA112.452.6
−87.5%TransMLA161.671.8
MHA2MLA133.970.3
LIVE-MLA110.451.4

At the same budget of 100M tokens under 50% memory reduction, the measured step times imply approximately 6.2 hours for LIVE-MLA, 8.8 hours for TransMLA, and 7.2 hours for MHA2MLA on two GPUs. These estimates use 524,288 tokens per step, exclude setup and checkpoint saving overhead, and reflect the respective training scopes and recipes.

LIVE-MLA's measurements include RF. These costs also reflect its attention-only training scope, compared with full parameter fine-tuning for the baselines.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

Additional quality results for W1 & Q1 and W2 & Q2

Quality without recovery training

LIVE-MLA + BR uses the same converted parameters, with BR enabled only at inference. Bold marks the best converted-model result per memory reduction level. Original is italicized as a reference.

−50% memory

MethodPPL 2K ↓PPL 8K ↓PPL 32K ↓LongBench 2K ↑RULER 2K ↑RULER 4K ↑NIAH 8K ↑NIAH 32K ↑
Original7.006.245.9822.6394.3993.6999.2596.00
TransMLA86.62105.63297.271.002.140.700.120.00
MHA2MLA635.80934.571258.271.060.030.020.000.00
CARE10.729.399.2515.9584.5377.8579.0656.09
LIVE-MLA7.598.6719.2920.4884.4063.9523.811.38
LIVE-MLA + BR7.346.526.2721.9188.8988.1391.4076.44

−75% memory

MethodPPL 2K ↓PPL 8K ↓PPL 32K ↓LongBench 2K ↑RULER 2K ↑RULER 4K ↑NIAH 8K ↑NIAH 32K ↑
Original7.006.245.9822.6394.3993.6999.2596.00
TransMLA1213.602365.225370.941.340.060.090.000.00
MHA2MLA1729.902148.502247.900.820.020.030.000.00
CARE20.4420.5826.219.1337.4127.8123.383.97
LIVE-MLA9.3710.8726.4714.1427.7916.141.250.00
LIVE-MLA + BR8.888.028.0915.5638.4333.7320.288.88

−87.5% memory

MethodPPL 2K ↓PPL 8K ↓PPL 32K ↓LongBench 2K ↑RULER 2K ↑RULER 4K ↑NIAH 8K ↑NIAH 32K ↑
Original7.006.245.9822.6394.3993.6999.2596.00
TransMLA2217.404268.4313466.641.860.050.060.000.00
MHA2MLA3759.404252.934654.581.030.000.000.000.00
CARE103.50111.18168.315.516.393.554.380.00
LIVE-MLA13.9018.0358.769.993.592.800.000.00
LIVE-MLA + BR13.4013.0113.4610.723.793.760.000.00

BR substantially improves perplexity without recovery training, although retrieval remains difficult under strong compression.

Quality after recovery training

Each LIVE-MLA + BR row uses the preceding LIVE-MLA checkpoint. Bold marks the best result per memory reduction level across all converted-model runs.

−50% memory

MethodTokensPPL 2K ↓PPL 8K ↓PPL 32K ↓LongBench 2K ↑RULER 2K ↑RULER 4K ↑NIAH 8K ↑NIAH 32K ↑
Original—7.006.245.9822.6394.3993.6999.2596.00
TransMLA100M7.576.987.2119.7789.0684.3880.4735.22
TransMLA3B7.226.636.9119.1293.3289.6485.5052.13
MHA2MLA100M8.5510.503336.5216.8774.2366.8712.500.00
MHA2MLA6B7.807.59188.6417.2286.7085.4329.660.00
LIVE-MLA100M6.996.4215.9421.5593.9088.1876.7813.25
LIVE-MLA + BR100M7.006.366.1622.0793.1692.6397.6083.62

−75% memory

MethodTokensPPL 2K ↓PPL 8K ↓PPL 32K ↓LongBench 2K ↑RULER 2K ↑RULER 4K ↑NIAH 8K ↑NIAH 32K ↑
Original—7.006.245.9822.6394.3993.6999.2596.00
TransMLA300M7.747.037.6815.8086.1277.5876.9425.16
TransMLA6B7.396.637.3215.6792.4189.2085.4744.63
MHA2MLA300M10.159.79331.8713.6965.3861.1715.380.00
MHA2MLA6B9.317.76234.0910.3971.1562.2727.250.00
LIVE-MLA300M7.166.559.0119.0693.2888.0275.6912.88
LIVE-MLA + BR300M7.176.536.3917.3992.7890.4996.7876.00

−87.5% memory

MethodTokensPPL 2K ↓PPL 8K ↓PPL 32K ↓LongBench 2K ↑RULER 2K ↑RULER 4K ↑NIAH 8K ↑NIAH 32K ↑
Original—7.006.245.9822.6394.3993.6999.2596.00
TransMLA6B7.866.918.3210.7585.5176.6164.5027.50
MHA2MLA6B10.118.50259.019.3365.4754.1814.750.00
LIVE-MLA1.5B7.556.947.1415.8982.9176.9058.9143.59
LIVE-MLA + BR1.5B7.527.077.2114.1877.0566.7865.3640.00

We thank the reviewer for helping clarify our measurements and extend evaluations to longer contexts and matched recovery budgets.

Reviewer_zHuqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper refines MHA/GQA-to-MLA conversion using an adaptive residual correction fitted on recent KV items, an attention-weighted compresison, as well as an exact window where MLA is not performed. They evaluate this approach on Llama-3.1-8B-Instruct and Qwen3-8B-Base, and report improvements on various tasks and across 3 compression ratios (x2, x4, x8).

Critical strengths
  • Principled techniques: All techniques introduced in the paper are reasonably justified, and rely on meaningful observations about caveats of existing SVD-based adaptation methods. Attention-based conversion is particularly elegant.
  • Interesting post-compression results: LIVE-MLA seems to decent performance loss for significant compression ratios, compared to other training-free methods.
Critical weaknesses
  • Question-execution gap: This paper is framed as an improvement of MHA/GQA-to-MLA conversion. However, the resulting model after LIVE-MLA conversion is not strictly an MLA model. It relies on exact KV cache within recent values. Given that the baselines worked within the constraint of producing a strict MLA model, it makes the comparison unfair, and it questions the relevance of the specific ridge regression method, which brings minimal improvements compared to both attention-weighting and windowing.
  • Presentation / Novelty: This work reads as the combination of 3 roughly orthogonal methods, each relatively incremental, applied on top of a naive MLA conversion approach. Beyond compounding improvements for post-conversion performance, it is unclear what the deep connection or common ground these techniques share. Moreover, the theory presented in Appendix C is never properly refered to in the core sections, making it unclear what its purpose is in the global flow of the paper.
Questions for authors
  • Is there any overhead in training/prefilling due to the parallel ridge regression?
  • In Figure 1c), the reconstruction error is eerily stable. Do you have any intuition/explanation why?
  • In Table 7, it seems LIVE-MLA is bolded even when it does not have the best latency. What do the bolded scores represent?
  • In Table 7, why do MHA2MLA and TransMLA have different latencies?
8 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank the reviewer for the careful evaluation and constructive feedback. We address each concern below and provide additional experiments and clarifications.

W1. MLA structure, comparison fairness, and the contribution of RF

We agree that LIVE-MLA differs structurally from conventional strict MLA because it retains exact KV for recent tokens. Our work preserves the core MLA computation through joint latent compression and projection absorption, while extending the structure to use residuals observed from recent exact KV to correct older compressed tokens. The purpose of our comparison is therefore to evaluate the quality and decoding latency of this extension under the same total cache budget, rather than to compare only low dimensional mappings within an identical structure. We will clarify this scope and the structural differences from conventional MLA in the revised manuscript.

Our main comparisons match the total cache budget, including the exact window and residual buffer. We reduce the latent width to accommodate these states and report decoding latency that includes the additional computation.

RF provides additional gains after attention weighting and the exact window have already been applied. In Table 24, removing RF while keeping the latent width and exact window unchanged increases perplexity from 13.90 to 16.84 on Qwen3 and from 15.29 to 19.59 on Llama under 87.5% memory reduction. Thus, RF reduces perplexity by approximately 17% and 22%, respectively.

Table 25 further compares RF against reallocating its residual buffer memory to a larger exact window under the same total cache budget. At 87.5% memory reduction, RF reduces perplexity by approximately 4.8% on Qwen3 and 8.4% on Llama compared with this alternative. It performs better in 142 of 146 evaluation windows on Qwen3 and all 141 windows on Llama.

Additional experiments at 32K also show a substantial RF contribution. On Qwen3-8B-Base without recovery training, RF lowers perplexity from 20.93 to 19.29, from 47.80 to 26.47, and from 155.10 to 58.76 under 50%, 75%, and 87.5% memory reduction, respectively.

RF uses the relationship between latent representations and residuals observed in the exact window to correct the contribution of older compressed values to the attention output. Table 26 shows improvements even when correction is applied only to distance bands that do not overlap with the fitting observations. These results support the use of recent observations to correct older tokens outside the observation region. Ridge regression implements this correction using a small number of observations and a small linear system. The core contribution is to use residual observations available during inference to correct the contribution of older compressed tokens and establish the benefit within the same cache budget.

W2. Common principle, connections between components, and the role of Appendix D

The three components are designed together to reduce the effect of compression residuals on attention outputs. Attention-weighted conversion gives more weight to reconstruction errors of tokens that receive more attention, but a fixed low dimensional mapping cannot eliminate all residuals. The exact window makes the actual residuals left by this mapping observable, and RF uses these observations to correct the contribution of older compressed values to the attention output. The exact window and RF therefore form a single process for observing and correcting errors that remain after conversion.

The objective of attention-weighted conversion is also connected to this process. Tokens inside the exact window use their original KV, so the conversion weights depend on the attention each token receives after leaving the window. This determines which reconstruction errors the mapping should prioritize. As tokens leave the window, we observe the residuals left by that same mapping and use them for RF. The exact window therefore determines both which attention weights are used for conversion and where residuals are observed, while RF extends the benefit of those observations to older compressed tokens.

Appendix D analyzes this design in terms of attention output error. With attention weights held fixed, Eq. 33 in Appendix D.7 shows that the attention-weighted value reconstruction error provides an upper bound on the squared output error in the value path. This supports the attention weighting for value reconstruction under fixed attention weights.

For RF, the analysis explains why residuals left by a fixed mapping can retain predictable structure within a sequence and examines the conditions under which a small set of observations can estimate the required attention-weighted residual sum. In particular, Proposition D.2 interprets RF's observation weights through regularized feature moment matching. We will connect these analyses directly to the corresponding design descriptions in the main text to clarify their role.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

Q1. Overhead during prefill and recovery training

RF solves a small ridge system of at most 16 × 16 for each query, with fits for multiple queries computed together. The submitted implementation uses batched Cholesky factorization for prefill and recovery training. The prefill timings below use an optimized Triton path that solves the same per-query ridge systems using Gauss–Jordan elimination. We verified numerical agreement between the inverses produced by the two paths. Decoding uses Gauss–Jordan elimination in both versions. During recovery training, we do not backpropagate through the observation pairs used for fitting or the ridge solve.

All timing measurements below use the original LIVE-MLA configuration. We measured prefill times on Qwen3-8B-Base under 87.5% memory reduction. We kept the same attention kernels, latent aggregation path, and exact window, enabling or disabling only RF fitting and correction.

Context × batchRF enabled (s)RF disabled (s)
8K × 10.6050.591
8K × 42.4252.416
8K × 169.7019.434
32K × 13.5133.449
32K × 413.96713.804
32K × 1655.85755.229

We also measured recovery training step times on two RTX PRO 6000 Blackwell GPUs using FSDP, a sequence length of 4,096, 128 sequences per optimizer step, and gradient checkpointing.

Mem.RF enabled (s/step)RF disabled (s/step)
−50%116.9108.8
−75%112.4106.5
−87.5%110.4104.7

Q2. Stable reconstruction error in Figure 1(c)

Reconstruction error depends on how well each token’s KV is represented in the fixed low dimensional space. Since distance from the query does not directly determine reconstruction accuracy, we do not necessarily expect a large difference between recent and distant tokens. On the evaluated data, relative errors were similar across distance bins.

Unlike Figure 1(c), the attention output error in Figure 1(d) decreases substantially with distance. This difference motivates accounting for importance in attention, rather than reconstruction error alone, during conversion.

Q3. Bold formatting in Table 16

The bold formatting in Table 16 highlights our proposed method, LIVE-MLA. It does not indicate the lowest latency in each column. We will remove this formatting to avoid confusion.

Q4. Latency differences between MHA2MLA and TransMLA

The same target cache budget does not imply identical computational cost. MHA2MLA and TransMLA allocate this budget differently between latent and rotary dimensions, which affects the computation required during absorbed decoding and can contribute to the latency differences in Table 16.

We thank the reviewer again for the constructive discussion, which helped us clarify the architectural contribution of LIVE-MLA and strengthen the evaluation of RF's benefits and computational costs.

Thank you for clarifying these points, and I apologize for the late reply.

  • W1. I think ablating RF in the recovery training setup would be a more effective comparison, but these comparisons indeed show the benefits of RF in the training-free setup.
  • W2. I am not entirely convinced by the authors' response here. I believe a clearer articulation of how the combination of methods makes sense here would greatly improve the value of this paper.
  • Q1. As I understand, the authors verified the throughput gap on a kernel optimized for RF, when RF was disabled. This seems like an unfair comparison, as the vanilla baseline setup could benefit for a specific implementation. They also do not discuss memory usage and maximum batch size, which raises suspicions about the fairness of the comparison when maximizing for the throughput of each method.
  • Q2. In my opinion, this point would benefit from a more thorough analysis, as the relative squared error looks strictly constant across distance bins, not merely similar.
  • Q3. Thank you for clarifying this and I recommend adopting a more conventional formatting (bold=best) to avoid misleading readers.
  • Q4. A more detailed profiling of the differences, and a more thorough discussion would solidify the results here.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Discussion

We sincerely thank the reviewers for their constructive feedback and thoughtful discussion, and the Area Chair for their time and consideration. The reviewers' questions helped us strengthen the experimental evidence and explain more clearly how LIVE-MLA's components work together.

LIVE-MLA uses reconstruction errors observed during inference to correct the values of older compressed tokens under a fixed total cache budget. Attention-weighted conversion defines the shared representation, the exact window provides observations of the errors left by that conversion, and RF uses these observations to adapt the effective value decoder to the current sequence. This correction preserves absorbed decoding without updating model parameters or rewriting stored latent representations. Reviewer 8Y2M recognized the larger-window comparison and corrections at disjoint distances as convincing evidence that RF provides benefits beyond retaining more recent tokens.

The revised manuscript strengthens this contribution in two key respects.

  • Evidence that recent residual observations remain useful at longer contexts and across domain changes. We extended RF evaluations to 32K and 128K and added experiments across document and domain boundaries, together with analyses of correction accuracy and prediction quality across distance (Section 4.5 and Appendices B.1–B.4). At 32K under 87.5% memory reduction in the training-free setting, RF lowers perplexity from 155.10 to 58.76 in the original configuration and from 15.88 to 13.46 with blockwise rotation (BR). BR is presented separately as an additional inference configuration using the same parameters without additional training or persistent cache storage, and RF is evaluated independently within each configuration. Reviewer n3ML stated that these experiments substantially resolved their main concern about generalization on WikiText-2, positively assessed the domain-change experiments, and raised their rating to 4 with strong support for acceptance.

  • Stronger validation of the method's benefits, mechanism, and costs. In response to Reviewers 8Y2M and zHuq, we added comparisons at matched training token budgets, deeper reconstruction-error analysis, and measurements of computational and memory costs. At 50% memory reduction and 100M training tokens, LIVE-MLA achieves perplexity of 6.99 at 2K while updating only attention-related parameters, compared with 7.57 for TransMLA and 8.55 for MHA2MLA, both of which update all model parameters. At 32K, LIVE-MLA + BR achieves perplexity of 6.16 compared with 7.21 for TransMLA, and NIAH scores of 83.62 compared with 35.22, with the original and BR configurations reported separately (Tables 8–9). The reconstruction-error analysis shows a strong association with token identity and confirms comparable averages across distance bins even when those bins contain disjoint source tokens (Appendix B.7). Paired decoding measurements show that RF adds approximately 0.5% to 1.4% latency, alongside explicit measurements of peak memory and batch capacity (Appendix C.8).

We also clarified the computational connection between conversion, the exact window, and RF in Section 3 and linked it to the analysis in Appendix D. Appendices C.6–C.7 document the decoding procedure, prefill costs, and training time and memory, while detailed decoding profiles are provided in the discussion.

All references above use the revised manuscript numbering. We sincerely appreciate the reviewers' time and engagement. Their feedback has helped us explain LIVE-MLA more clearly and strengthen the evidence supporting its contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.