- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MuLA, a sparse KV cache migration framework for multi-LoRA agent systems, where agents share a base model with role-specific LoRA adapters and each downstream agent inherits the upstream agent's KV cache. The method observes that LoRA-induced KV deviations grow with depth, so shallow-layer deviation does not locate deep-layer error hotspots as it does for prefix-induced deviation, and that the upstream agent's attention concentration correlates with these hotspots. At each layer, MuLA selects the top- positions of each agent's segment by upstream observation-window attention and recomputes them layer-wise under the target adapter, storing upstream hidden states only for newly selected tokens. Experiments on LongBench, API-Bank, RULER, and a five-agent GSM8K pipeline show near-full-prefill quality at 10–25% recomputation, with 3.5–7× downstream prefill speedup.
Critical strengths
-
The analysis of prefix-induced versus parameter-induced KV deviation is informative. The paper shows that LoRA switching injects deviation at every layer that grows with depth, and that shallow and deep deviation sets barely overlap (Figures 4–5). This helps explain why detection-layer methods such as CacheBlend, RelayCaching, and DroidSpeak transfer less effectively to multi-LoRA handoff.
-
The evaluation is broad, covering one-shot, two-agent, and five-agent serial settings, a LoRA-divergence sweep (Figure 9), and ablations showing that both per-layer reselection and attention ranking contribute (Table 5).
Critical weaknesses
-
(Quality) The method does not appear to use the layer-lag observation that motivates it, and the evidence seems to support attention as an importance signal rather than a predictor of deviation.
a. The key observation is that shallow-layer attention anticipates deep-layer deviation, while the algorithm selects positions at each layer from the same layer's attention (Eq. 9). It is not fully clear how the lag is used.
b. Figure 8 (right) shows that repairing high-attention tokens reduces logit error more than repairing random tokens. This would also be expected from an importance criterion, since the output reads those positions more regardless of how much their KV deviates.
c. The top-30% overlap in Figure 8 (left) is about 0.65 against a random level of 0.30, without controlling for sink, delimiter, or high-norm tokens, which are excluded in Figure 11.
d. The results therefore support attention-based importance as an effective repair criterion in this setting. This is useful, though somewhat different from the predictor claim of Section 3.2.
-
(Quality / Significance) The repair may not fully remove the deviation accumulated before a token's onset layer, and part of the selection cost is placed outside the measured path.
a. A token first selected at layer is recomputed from the upstream hidden state , which already contains the upstream adapter's effect from earlier layers. Given the paper's finding that the deviation accumulates with depth, only the later part may be corrected.
b. Each repaired query also attends over mostly uncorrected keys, and the residual deviation of repaired tokens is not quantified.
c. Selection is described as near-zero cost because per-layer attention and hidden states for new tokens are collected during upstream decoding and excluded from TTFT. This may require attention probabilities not exposed by fused attention kernels such as FlashAttention, as well as an buffer per turn.
d. The memory analysis in Figure 10 is simulated rather than measured, and its right panel is labeled as including stochastic noise, which makes the overhead difficult to assess.
Questions for authors
-
(ref Weakness 1) Since selection at layer uses , how does the layer lag enter the algorithm? How would MuLA compare with a fixed top- selected from shallow layers only?
-
(ref Weakness 1) Does upstream attention predict deviation magnitude after controlling for importance, for example by the rank correlation among tokens with similar attention, or the overlap after excluding sink and delimiter tokens?
-
(ref Weakness 2) What is the residual KV deviation of repaired tokens relative to full target prefill, as a function of their onset layer?
-
(ref Weakness 2) What are the measured upstream decoding slowdown and peak memory when attention and hidden states are collected? Could the authors clarify why Figure 10 includes stochastic noise?
-
(Presentation, minor) Eq. (11) appears to be missing a complement (). The running header reads "ICLR 2026," and the backbone is named "Qwen3-1.7B-Base" in Figure 3 and "Qwen3-1.7B" in Table 1. A few possessives are missing, such as "MuLAs," "agents forward pass," and "Agent 1s."
-
(Typo, minor) The headline states 'ICLR 2026' not 2027.
4 comments
W1(b–d), Q2: Deviation prediction versus token importance
It's quite intuitive that high attention tokens are often very important for the model output. However, in fact, token importance alone does not establish whether attention is a reliable signal for proactive KV repair. Our analysis provides two key insights beyond this importance-based interpretation.
(1) Attention precedes significant KV deviations. If substantial deviations appeared earlier than attention concentration, attention-guided repair could begin too late to effectively correct accumulated errors. Our layer-wise analysis reveals that attention concentration often emerges in shallower layers, while substantial LoRA-induced deviations at corresponding positions become apparent only in deeper layers. We further explain this pattern through the progressive accumulation of adapter-induced discrepancies across layers (Our explanation of training dynamics is shown in Appendix C). This observed ordering is crucial: it makes attention useful not merely for identifying important tokens, but for identifying where to initiate repair before large deviations accumulate. MuLA provides an appropriate starting point for repair.
(2) Attention predicts deviations across agents. The attention scores used by MuLA are collected under the upstream agent's LoRA, whereas the deviations of interest arise under a different downstream adapter. High attention within one agent does not necessarily imply large cross-LoRA KV deviations. Our empirical analysis establishes this non-obvious cross-agent relationship, enabling MuLA to identify potential downstream deviation hotspots using upstream information, without executing the target adapter beforehand.
Together, these findings show that upstream attention serves not merely as a measure of token importance, but as an early indicator of downstream KV deviations. This distinction is central to MuLA's proactive repair strategy. This is actually not as straightforward as it seems.
Furthermore, in response to the reviewers' question, we have supplemented the experiments: we exclude attention sinks, delimiters, and high-norm tokens (25.6% of positions). The average Top-30% overlap remains high, decreasing only slightly from 0.656 to 0.642, supporting that this correlation is not primarily driven by special tokens.Results are shown as below.
| Attn.(L6-10) | Avg.(L16-L19) | Peak(L18) | random overlap | Exclude rate |
|---|---|---|---|---|
| overlap raw | 0.656 | 0.697 | 0.30 | |
| overlap excl.(exclude sink/delimiter/high-norm) | 0.642 | 0.671 | 0.30 | sink=1.6% delim=19.0% highnorm=5%,total=25.6% |
After excluding Attention Sink, Delimiter, and special positions, we conducted a statistical analysis of the positional overlaps between top-upstream-attn and top-deviation positions.
W2(a–b), Q3: Residual deviation after sparse repair
When a token first enters the repair set at layer L, its computation starts from the upstream hidden state at that layer. Consequently, small inital deviations introduced in preceding layers are not necessarily eliminated. Furthermore, repaired queries attend to a cache containing both corrected and inherited KV entries.
Our objective is therefore not to guarantee exactly zero element-wise KV deviation, but to reduce the large deviations that materially affect downstream generation while avoiding full historical recomputation. The near-full-prefill quality observed on LongBench, API-Bank, and GSM8K demonstrates the effectiveness of this approximation at the task level. The residual KV deviation of repaired tokens relative to full target prefill, as a function of their onset layer is shown in the first table of W1(a), Q1.
W2(c–d), Q4: Upstream overhead and memory measurements
As defined in Appendix B.2, our reported handoff TTFT starts after the preceding agent finishes decoding. It includes the non-overlapped selection, sparse repair, and downstream sparse prefill. Attention-statistics collection and temporary hidden-state buffering performed during upstream execution are therefore not fully reflected in this TTFT measurement.
Comparison of decoding time between "Vanilla Flash-Attn" and "MuLA's Kernel". The numbers represent the average upstream decode time taken (variance, ms), for decoding 64 tokens.The results are shown as below.
| model | Context length | Vanilla(variance) | MuLA's kernel |
|---|---|---|---|
| Llama-3.2-3B | 10,240 | 1178.5 (20.2) | 1190.8 (20.6) |
| Llama-3.2-3B | 20,480 | 1172.9 (20.4) | 1183.2 (28.2) |
| Llama-3.2-3B | 40,960 | 1666.8 (2.8) | 1685.8 (3.9) |
| Llama-3.2-3B | 81,920 | 2759.6 (0.6) | 2796.0 (0.6) |
| Qwen3-1.7B | 10,240 | 1522.7 (46.2) | 1524.6 (21.9) |
| Qwen3-1.7B | 20,480 | 1466.0 (20.1) | 1477.7 (29.6) |
| Qwen3-1.7B | 40,960 | 1630.6 (8.0) | 1646.8 (16.8) |
| Qwen3-1.7B | 81,920 | 2735.0 (6.0) | 2773.3 (5.4) |
We thank the reviewer for highlighting this ambiguity. We would like to clarify that Figure 10 reports actual GPU memory measurements, rather than theoretical memory estimates. The term "simulated" refers only to the use of synthetic (mocked) input sequences of varying lengths to construct controlled workloads. All reported memory values were obtained from actual GPU executions of MuLA.
Furthermore, the "stochastic fluctuations" in the right panel are not artificially injected noise. They arise naturally from variations in the overlap between token sets selected for recomputation at adjacent Transformer layers. Specifically, the temporary hidden-state storage depends on the number of newly selected tokens at each layer, i.e., the onset set . As the overlap between and varies across runs, the required hidden-state storage also varies slightly, producing the observed fluctuations in measured memory overhead.
We agree that the current figure caption may have inadvertently suggested a simulated memory model or artificially introduced noise. We will revise the caption and accompanying discussion to explicitly distinguish synthetic workloads from actual GPU measurements, and clarify the source of the observed memory fluctuations.
Q5–Q6: Presentation and typographical issues
Thank you for identifying these issues. We will correct Eq. (11) to make the set complement/difference explicit, standardize the backbone naming, update the conference header, and fix the listed grammatical and possessive errors.
We appreciate the reviewer's suggestions, which will help us more precisely distinguish our motivation, the approximation inherent in sparse repair, and the measured scope of our system-level efficiency claims.
Thank you for the detailed response and the additional experiments.
The residual deviation and the repair coverage are partially addressed, as the former is reported for four representative tokens and the latter would be easier to interpret with a budget-matched random baseline. The concerns on the upstream decoding overhead, the memory measurement in Figure 10, the influence of special tokens, and the presentation issues are addressed by the new results and clarifications.
However, my main concern on the interpretation of the layer lag remains. In the new comparison with fixed selection, shallow-6 (0.2321) and deep-6 (0.2291) perform similarly and both remain below MuLA (0.2505), which suggests that the gain comes mainly from per-layer reselection rather than from shallow attention anticipating deep-layer deviation. Moreover, since LoRA-induced deviation is small in shallow layers for all tokens (Figure 4), the argument that attention precedes deviation would apply to most selection criteria, and the cross-agent relationship may also be explained by importance as both agents share the base model, while a control for importance, such as the rank correlation among tokens with similar attention, is not yet provided. I would therefore suggest framing Section 3.2 around layer-adaptive attention-based selection, which the results support well.
Overall, the response clarifies the efficiency concerns, and I will keep my positive assessment.
We thank the reviewer for the constructive feedback and recognition of our key insight and comprehensive evaluation. We address the concerns below.
W1(a), Q1: How does layer lag inform MuLA's selection strategy?
We notice that the term "layer-lag" may be somewhat confusing to the reviewer, and we will refine it in the official PDF version. In fact, "Layer lag" refers to the observation that the attention concentration occurs before significant deviations in the KV cache occur on layers. MuLA does not hard-code a fixed cross-layer offset or lag parameter. Instead, the lag observation motivates the use of attention-guided proactive repair rather than observed KV deviation as an earlier selection signal.
Specifically, substantial LoRA-induced KV deviations may emerge clearly observable only in middle or deep layers, while upstream attention can identify the corresponding critical positions from earlier layers. Therefore, waiting until KV deviations become significant may miss the opportunity to mitigate their accumulation. MuLA addresses this by selecting repair positions using upstream attention at earlier layers and propagating the repaired hidden state forward; thus, MuLA's correction starts at each token's onset layer before substantial deviations fully manifest and is carried to consecutive deeper layers in which this token remains selected. In fact, per-layer reselection avoids assuming a fixed uniform lag across tokens, layers, or LoRA adapters, while allowing the repair set to adapt to their heterogeneous evolution. Such a design of MuLA provides a better and more accurate starting point for the repair of KV migration; we will make this clearer in the official PDF version.
We provide two additional pieces of evidence supporting this design:
(1) MuLA's Onset Earlier repair is more effective at reducing accumulated deviations. Our new Table 1 examines the effect of manually varying repair starting layers. For instance, for token 823 at layer 20(L20T823), initiating repair at layers 4, 12, and 20 yields V-cache deviation reductions of 92.3%, 87.6%, and 13.7%, respectively. These representative cases suggest that delayed repair can leave substantially larger residual deviations. However, starting earlier also increases recomputation, motivating MuLA's selective strategy that balances repair quality and computational cost.
The table reports relative V-cache error reduction across different repair onset layers on Llama-3.2-3B-Instruct with a 30% recomputation budget. Each row represents a token, with its dynamically selected MuLA repair interval (onset and end layers). Columns L4–L28 denote manually specified onset layers. The reduction ratio is defined as 1−∣Vgt−Vr=0∣∣Vr=0.3−Vgt∣. Earlier repair generally yields greater error reduction at additional computational cost, highlighting the benefit of early but not necessarily first-layer repair.
(2) Dynamic selection aligns repair with the evolution of KV deviations. Table 2 analyzes significant-deviation spans and MuLA's repair coverage. Across the reported groups, MuLA achieves 84.9%–93.5% interval coverage, indicating that the repair intervals selected by MuLA generally cover the positions where significant deviations occur.
The table shows the distribution of significant KV deviation spans and their coverage by MuLA's dynamic selection. Tokens are grouped by consecutive deviation span. Token Ratio indicates the proportion of tokens in each group, while Coverage Rate measures the fraction whose deviation intervals are fully covered by MuLA's repair intervals. The consistently high coverage demonstrates that MuLA effectively aligns repair intervals with KV deviations across layers.
(3) Comparison with fixed shallow-layer selection. To directly address the reviewer's question, we additionally compare MuLA with fixed Top-K selection using shallow-layer and deep-layer attention, under matched recomputation budgets. We test it on Longbench and present the average score. For shallow/deep-n, we use the first/last n-layers attetion score to compute a mean attention score and choose the top-25% tokens to fix. Results are shown as below.