acceptodds
Under review as a conference paper at ICLR 2027

MuLA: One Prefill, Many Agents via Efficient KV Cache Reuse in Multi-LoRA Agents Systems

16 views

Abstract

The multi‑LoRA agent framework, in which a collection of specialized agents share a single base LLM while each is equipped with its own LoRA adapter, has been widely adopted in complex real‑world applications. In such pipelines, a downstream agent has to prefill the context produced by upstream agents before generating its own response. Because upstream agents typically accumulate long contexts, re‑prefilling the entire context from scratch for every downstream agent incurs prohibitive computational overhead and inference latency. Yet we find that this repeated prefill is largely redundant: the KV cache of upstream agents can be substantially reused, requiring only lightweight sparse corrections. To this end, we propose **MuLA** (Multi‑LoRA‑Agents Sparse KV Cache Migration), a framework that efficiently reuses upstream KV caches by selectively recomputing only a small subset of critical token positions. Our empirical analysis shows that LoRA‑switch‑induced KV‑cache deviations are sparse over tokens and delayed to deep layers. Critically, cheap attention‑concentration signals collected in the upstream agent’s forward pass strongly anticipate these error hotspots. Building on this insight, we exploit attention concentration as a cheap early indicator to proactively identify tokens requiring correction when switching LoRA adapters between agents, thereby completing sparse recomputation before the target deviations fully manifest, while balancing both accuracy and efficiency in the agents workflow. Results show that **MuLA** preserves nearly full‑prefill quality with a small fraction of token recomputation, delivering substantial downstream prefill speedup and promising end‑to‑end speedup.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent, lora, NLP, efficiency, KV cache

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_26a5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MuLA, a sparse KV cache migration framework for multi-LoRA agent systems, where agents share a base model with role-specific LoRA adapters and each downstream agent inherits the upstream agent's KV cache. The method observes that LoRA-induced KV deviations grow with depth, so shallow-layer deviation does not locate deep-layer error hotspots as it does for prefix-induced deviation, and that the upstream agent's attention concentration correlates with these hotspots. At each layer, MuLA selects the top-rr positions of each agent's segment by upstream observation-window attention and recomputes them layer-wise under the target adapter, storing upstream hidden states only for newly selected tokens. Experiments on LongBench, API-Bank, RULER, and a five-agent GSM8K pipeline show near-full-prefill quality at 10–25% recomputation, with 3.5–7× downstream prefill speedup.

Critical strengths
  1. The analysis of prefix-induced versus parameter-induced KV deviation is informative. The paper shows that LoRA switching injects deviation at every layer that grows with depth, and that shallow and deep deviation sets barely overlap (Figures 4–5). This helps explain why detection-layer methods such as CacheBlend, RelayCaching, and DroidSpeak transfer less effectively to multi-LoRA handoff.

  2. The evaluation is broad, covering one-shot, two-agent, and five-agent serial settings, a LoRA-divergence sweep (Figure 9), and ablations showing that both per-layer reselection and attention ranking contribute (Table 5).

Critical weaknesses
  1. (Quality) The method does not appear to use the layer-lag observation that motivates it, and the evidence seems to support attention as an importance signal rather than a predictor of deviation.

    a. The key observation is that shallow-layer attention anticipates deep-layer deviation, while the algorithm selects positions at each layer ll from the same layer's attention a(l)a^{(l)} (Eq. 9). It is not fully clear how the lag is used.

    b. Figure 8 (right) shows that repairing high-attention tokens reduces logit error more than repairing random tokens. This would also be expected from an importance criterion, since the output reads those positions more regardless of how much their KV deviates.

    c. The top-30% overlap in Figure 8 (left) is about 0.65 against a random level of 0.30, without controlling for sink, delimiter, or high-norm tokens, which are excluded in Figure 11.

    d. The results therefore support attention-based importance as an effective repair criterion in this setting. This is useful, though somewhat different from the predictor claim of Section 3.2.

  2. (Quality / Significance) The repair may not fully remove the deviation accumulated before a token's onset layer, and part of the selection cost is placed outside the measured path.

    a. A token first selected at layer ll is recomputed from the upstream hidden state H(l)H^{(l)}, which already contains the upstream adapter's effect from earlier layers. Given the paper's finding that the deviation accumulates with depth, only the later part may be corrected.

    b. Each repaired query also attends over mostly uncorrected keys, and the residual deviation of repaired tokens is not quantified.

    c. Selection is described as near-zero cost because per-layer attention and hidden states for new tokens are collected during upstream decoding and excluded from TTFT. This may require attention probabilities not exposed by fused attention kernels such as FlashAttention, as well as an L×N×dmodelL \times N \times d_{\text{model}} buffer per turn.

    d. The memory analysis in Figure 10 is simulated rather than measured, and its right panel is labeled as including stochastic noise, which makes the overhead difficult to assess.

Questions for authors
  1. (ref Weakness 1) Since selection at layer ll uses a(l)a^{(l)}, how does the layer lag enter the algorithm? How would MuLA compare with a fixed top-kk selected from shallow layers only?

  2. (ref Weakness 1) Does upstream attention predict deviation magnitude after controlling for importance, for example by the rank correlation among tokens with similar attention, or the overlap after excluding sink and delimiter tokens?

  3. (ref Weakness 2) What is the residual KV deviation of repaired tokens relative to full target prefill, as a function of their onset layer?

  4. (ref Weakness 2) What are the measured upstream decoding slowdown and peak memory when attention and hidden states are collected? Could the authors clarify why Figure 10 includes stochastic noise?

  5. (Presentation, minor) Eq. (11) appears to be missing a complement (Fl∩Fl−1‾F_l \cap \overline{F_{l-1}}). The running header reads "ICLR 2026," and the backbone is named "Qwen3-1.7B-Base" in Figure 3 and "Qwen3-1.7B" in Table 1. A few possessives are missing, such as "MuLAs," "agents forward pass," and "Agent 1s."

  6. (Typo, minor) The headline states 'ICLR 2026' not 2027.

4 comments

We thank the reviewer for the constructive feedback and recognition of our key insight and comprehensive evaluation. We address the concerns below.

W1(a), Q1: How does layer lag inform MuLA's selection strategy?

We notice that the term "layer-lag" may be somewhat confusing to the reviewer, and we will refine it in the official PDF version. In fact, "Layer lag" refers to the observation that the attention concentration occurs before significant deviations in the KV cache occur on layers. MuLA does not hard-code a fixed cross-layer offset or lag parameter. Instead, the lag observation motivates the use of attention-guided proactive repair rather than observed KV deviation as an earlier selection signal.

Specifically, substantial LoRA-induced KV deviations may emerge clearly observable only in middle or deep layers, while upstream attention can identify the corresponding critical positions from earlier layers. Therefore, waiting until KV deviations become significant may miss the opportunity to mitigate their accumulation. MuLA addresses this by selecting repair positions using upstream attention at earlier layers and propagating the repaired hidden state forward; thus, MuLA's correction starts at each token's onset layer before substantial deviations fully manifest and is carried to consecutive deeper layers in which this token remains selected. In fact, per-layer reselection avoids assuming a fixed uniform lag across tokens, layers, or LoRA adapters, while allowing the repair set to adapt to their heterogeneous evolution. Such a design of MuLA provides a better and more accurate starting point for the repair of KV migration; we will make this clearer in the official PDF version.

We provide two additional pieces of evidence supporting this design:

(1) MuLA's Onset Earlier repair is more effective at reducing accumulated deviations. Our new Table 1 examines the effect of manually varying repair starting layers. For instance, for token 823 at layer 20(L20T823), initiating repair at layers 4, 12, and 20 yields V-cache deviation reductions of 92.3%, 87.6%, and 13.7%, respectively. These representative cases suggest that delayed repair can leave substantially larger residual deviations. However, starting earlier also increases recomputation, motivating MuLA's selective strategy that balances repair quality and computational cost.

The table reports relative V-cache error reduction across different repair onset layers on Llama-3.2-3B-Instruct with a 30% recomputation budget. Each row represents a token, with its dynamically selected MuLA repair interval (onset and end layers). Columns L4–L28 denote manually specified onset layers. The reduction ratio is defined as 1−∣Vr=0.3−Vgt∣∣Vgt−Vr=0∣1-\frac{|V_{r=0.3}-V_{gt}|}{|V_{gt}-V_{r=0}|}. Earlier repair generally yields greater error reduction at additional computational cost, highlighting the benefit of early but not necessarily first-layer repair.

tokenMuLA Onset/End Layer/residual reduction ratioL4L8L12L16L20L24L28
L20T82312/28/0.8760.9230.8910.8760.5120.137--
L16T2149/20/0.9020.9190.8990.7710.244---
L23T35312/24/0.9090.9330.8820.9090.6990.192--
L28T3948/28/0.8620.8890.8620.6020.4640.1990.1430.169

(2) Dynamic selection aligns repair with the evolution of KV deviations. Table 2 analyzes significant-deviation spans and MuLA's repair coverage. Across the reported groups, MuLA achieves 84.9%–93.5% interval coverage, indicating that the repair intervals selected by MuLA generally cover the positions where significant deviations occur.

The table shows the distribution of significant KV deviation spans and their coverage by MuLA's dynamic selection. Tokens are grouped by consecutive deviation span. Token Ratio indicates the proportion of tokens in each group, while Coverage Rate measures the fraction whose deviation intervals are fully covered by MuLA's repair intervals. The consistently high coverage demonstrates that MuLA effectively aligns repair intervals with KV deviations across layers.

significant KV deviation layer span lengthToken RatioTypical Deviation Span End LayerMuLA's Repair Interval Coverage
1–46.9%~L2692.4%
5–819.6%~L2493.5%
9–1238.8%~L2290.3%
13–1617.9%~L2085.4%
17–2015.4%~L2485.5%
21–281.4%~L2884.9%

(3) Comparison with fixed shallow-layer selection. To directly address the reviewer's question, we additionally compare MuLA with fixed Top-K selection using shallow-layer and deep-layer attention, under matched recomputation budgets. We test it on Longbench and present the average score. For shallow/deep-n, we use the first/last n-layers attetion score to compute a mean attention score and choose the top-25% tokens to fix. Results are shown as below.

MuLA 25%Shollow-6 (layers)deep-6deep-2deep-4shallow-4shallow-2
Longbench avg.0.25050.23210.22910.22270.22710.22970.2189

W1(b–d), Q2: Deviation prediction versus token importance

It's quite intuitive that high attention tokens are often very important for the model output. However, in fact, token importance alone does not establish whether attention is a reliable signal for proactive KV repair. Our analysis provides two key insights beyond this importance-based interpretation.

(1) Attention precedes significant KV deviations. If substantial deviations appeared earlier than attention concentration, attention-guided repair could begin too late to effectively correct accumulated errors. Our layer-wise analysis reveals that attention concentration often emerges in shallower layers, while substantial LoRA-induced deviations at corresponding positions become apparent only in deeper layers. We further explain this pattern through the progressive accumulation of adapter-induced discrepancies across layers (Our explanation of training dynamics is shown in Appendix C). This observed ordering is crucial: it makes attention useful not merely for identifying important tokens, but for identifying where to initiate repair before large deviations accumulate. MuLA provides an appropriate starting point for repair.

(2) Attention predicts deviations across agents. The attention scores used by MuLA are collected under the upstream agent's LoRA, whereas the deviations of interest arise under a different downstream adapter. High attention within one agent does not necessarily imply large cross-LoRA KV deviations. Our empirical analysis establishes this non-obvious cross-agent relationship, enabling MuLA to identify potential downstream deviation hotspots using upstream information, without executing the target adapter beforehand.

Together, these findings show that upstream attention serves not merely as a measure of token importance, but as an early indicator of downstream KV deviations. This distinction is central to MuLA's proactive repair strategy. This is actually not as straightforward as it seems.

Furthermore, in response to the reviewers' question, we have supplemented the experiments: we exclude attention sinks, delimiters, and high-norm tokens (25.6% of positions). The average Top-30% overlap remains high, decreasing only slightly from 0.656 to 0.642, supporting that this correlation is not primarily driven by special tokens.Results are shown as below.

Attn.(L6-10)Avg.(L16-L19)Peak(L18)random overlapExclude rate
overlap raw0.6560.6970.30
overlap excl.(exclude sink/delimiter/high-norm)0.6420.6710.30sink=1.6% delim=19.0% highnorm=5%,total=25.6%

After excluding Attention Sink, Delimiter, and special positions, we conducted a statistical analysis of the positional overlaps between top-upstream-attn and top-deviation positions.

W2(a–b), Q3: Residual deviation after sparse repair

When a token first enters the repair set at layer L, its computation starts from the upstream hidden state at that layer. Consequently, small inital deviations introduced in preceding layers are not necessarily eliminated. Furthermore, repaired queries attend to a cache containing both corrected and inherited KV entries.

Our objective is therefore not to guarantee exactly zero element-wise KV deviation, but to reduce the large deviations that materially affect downstream generation while avoiding full historical recomputation. The near-full-prefill quality observed on LongBench, API-Bank, and GSM8K demonstrates the effectiveness of this approximation at the task level. The residual KV deviation of repaired tokens relative to full target prefill, as a function of their onset layer is shown in the first table of W1(a), Q1.

W2(c–d), Q4: Upstream overhead and memory measurements

As defined in Appendix B.2, our reported handoff TTFT starts after the preceding agent finishes decoding. It includes the non-overlapped selection, sparse repair, and downstream sparse prefill. Attention-statistics collection and temporary hidden-state buffering performed during upstream execution are therefore not fully reflected in this TTFT measurement.

Comparison of decoding time between "Vanilla Flash-Attn" and "MuLA's Kernel". The numbers represent the average upstream decode time taken (variance, ms), for decoding 64 tokens.The results are shown as below.

modelContext lengthVanilla(variance)MuLA's kernel
Llama-3.2-3B10,2401178.5 (20.2)1190.8 (20.6)
Llama-3.2-3B20,4801172.9 (20.4)1183.2 (28.2)
Llama-3.2-3B40,9601666.8 (2.8)1685.8 (3.9)
Llama-3.2-3B81,9202759.6 (0.6)2796.0 (0.6)
Qwen3-1.7B10,2401522.7 (46.2)1524.6 (21.9)
Qwen3-1.7B20,4801466.0 (20.1)1477.7 (29.6)
Qwen3-1.7B40,9601630.6 (8.0)1646.8 (16.8)
Qwen3-1.7B81,9202735.0 (6.0)2773.3 (5.4)

We thank the reviewer for highlighting this ambiguity. We would like to clarify that Figure 10 reports actual GPU memory measurements, rather than theoretical memory estimates. The term "simulated" refers only to the use of synthetic (mocked) input sequences of varying lengths to construct controlled workloads. All reported memory values were obtained from actual GPU executions of MuLA.

Furthermore, the "stochastic fluctuations" in the right panel are not artificially injected noise. They arise naturally from variations in the overlap between token sets selected for recomputation at adjacent Transformer layers. Specifically, the temporary hidden-state storage depends on the number of newly selected tokens at each layer, i.e., the onset set Ol=Fl∖Fl−1O_l=F_l\setminus F_{l-1}. As the overlap between FlF_l and Fl−1F_{l-1} varies across runs, the required hidden-state storage also varies slightly, producing the observed fluctuations in measured memory overhead.

We agree that the current figure caption may have inadvertently suggested a simulated memory model or artificially introduced noise. We will revise the caption and accompanying discussion to explicitly distinguish synthetic workloads from actual GPU measurements, and clarify the source of the observed memory fluctuations.

Q5–Q6: Presentation and typographical issues

Thank you for identifying these issues. We will correct Eq. (11) to make the set complement/difference explicit, standardize the backbone naming, update the conference header, and fix the listed grammatical and possessive errors.

We appreciate the reviewer's suggestions, which will help us more precisely distinguish our motivation, the approximation inherent in sparse repair, and the measured scope of our system-level efficiency claims.

Thank you for the detailed response and the additional experiments.

The residual deviation and the repair coverage are partially addressed, as the former is reported for four representative tokens and the latter would be easier to interpret with a budget-matched random baseline. The concerns on the upstream decoding overhead, the memory measurement in Figure 10, the influence of special tokens, and the presentation issues are addressed by the new results and clarifications.

However, my main concern on the interpretation of the layer lag remains. In the new comparison with fixed selection, shallow-6 (0.2321) and deep-6 (0.2291) perform similarly and both remain below MuLA (0.2505), which suggests that the gain comes mainly from per-layer reselection rather than from shallow attention anticipating deep-layer deviation. Moreover, since LoRA-induced deviation is small in shallow layers for all tokens (Figure 4), the argument that attention precedes deviation would apply to most selection criteria, and the cross-agent relationship may also be explained by importance as both agents share the base model, while a control for importance, such as the rank correlation among tokens with similar attention, is not yet provided. I would therefore suggest framing Section 3.2 around layer-adaptive attention-based selection, which the results support well.

Overall, the response clarifies the efficiency concerns, and I will keep my positive assessment.

Reviewer_xrf7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies KV-cache reuse across different LoRA adapters. In a multi-agent workflow with a shared backbone, a downstream agent often repeats prefilling over history already processed upstream. MuLA uses upstream attention scores to select a small set of tokens at each layer. It recomputes these positions with the downstream adapter and reuses the remaining cache. The implementation stores hidden states needed for repair and includes a sparse prefix-attention operator. Experiments cover LongBench, API-Bank, RULER, and a five-agent GSM8K workflow, measuring answer quality and time to first token at handoffs.

Critical strengths
  1. The paper addresses repeated processing of long contexts in multi-LoRA agent workflows. It repairs a cache already produced under an upstream adapter and fits systems that share a backbone and pass history sequentially. This is a useful problem to solve.

  2. The main experiments show that limited recomputation recovers much of the average performance. At ten and twenty-five percent budgets, LongBench averages on both backbones are close to full prefill. Tests on other tasks and different levels of adapter divergence give the evaluation reasonable breadth.

  3. The ablations support the main design choices. Table 5 compares attention guidance, fixed token sets, and random selection under the same repair pipeline and budget. The results show that token choice and per-layer changes to the selected set matter for repair quality.

  4. The authors measure actual handoff latency and test successive migrations among five role-specific adapters. These experiments cover repeated adapter switching in sequential agent workflows and report both quality and timing results.

Critical weaknesses
  1. The added contribution over existing methods could be explained more clearly. RelayCaching, CacheBlend, and LRAgent are directly related, so the paper should identify what MuLA adds in cross-LoRA repair and per-layer selection. Matched recomputation-budget comparisons are provided, but the baseline adaptations and their applicable conditions remain unclear. Actual latency at comparable quality is also missing. This makes it difficult to judge the advantage over existing methods. Original sources: RelayCaching https://arxiv.org/html/2603.13289v1 ; CacheBlend https://arxiv.org/html/2405.16444v3 ; LRAgent https://arxiv.org/html/2602.01053v1 .

  2. Efficiency of the complete system is not yet fully evaluated. The timing mainly measures waiting at agent handoffs, while some attention collection and hidden-state storage have already taken place upstream. The memory results are also mainly simulated. Handoff acceleration is useful, but complete task latency and measured peak memory are still needed to assess the total deployment benefit.

  3. The theoretical cost analysis needs a narrower conclusion. Appendix B.1 links the repair fraction to speedup approaching 1/r, but causal-attention work also depends on selected-token positions. The repair fraction alone is insufficient. State the required assumptions or revise the conclusion. This does not affect the measured handoff timings, but matters when explaining or generalizing the speedup.

  4. The method description needs a clearer account of the complete execution flow. Selection, recomputation, and state storage are described separately, but it is difficult to follow hidden-state propagation when selection changes or caches pass through successive agents. Complete pseudocode and an example covering input processing through the end of a handoff would help reproduction and clarify the method's applicable conditions.

  5. Average scores close to full prefill do not yet show stable quality preservation across tasks. Some retrieval tasks still lose substantial quality, while some differences from the closest method are small. Main results lack repeated-run variation or confidence intervals. Discuss where quality losses occur and when a larger repair budget may be necessary.

Questions for authors
  1. Does CacheBlend retain gradual filtering? How was RelayCaching's 2:8 allocation chosen, and what happens when it changes? Which LRAgent sharing scheme is used, and does it include the Q/K/V/O extension and satisfy the associated training requirements?

  2. How much do attention collection and hidden-state storage slow upstream generation? Could the authors report complete workflow time, measured peak memory, and the latency of each method at comparable quality?

  3. Please clarify Equation (12) and explain state propagation and updates when tokens re-enter selection or undergo repeated handoffs. With identical context and position handling, shared input representations, and all historical tokens recomputed at every layer, do caches and logits recover full target-model prefilling within numerical tolerance?

  4. Could uncertainty be reported for the main quality differences, together with an explanation of the larger retrieval losses? Please also account explicitly for selected-token positions in Appendix B.1's cost analysis.

4 comments

W1 / Q1: Baseline Comparison and Configuration Selection

MuLA's primary distinction lies in its treatment of parameter-induced KV discrepancies. Unlike prefix-induced deviations, LoRA switching introduces new discrepancies throughout the network, making shallow-layer KV mismatch an unreliable indicator of deep-layer error hotspots. MuLA addresses this setting by using upstream attention statistics for proactive selection and performing layer-adaptive sparse recomputation under the target adapter.

We clarify that our original CacheBlend baseline did not enable gradual filtering. To assess whether this choice affects the comparison, we additionally evaluate CacheBlend with gradual filtering enabled. Sweeping the filtering retention ratio (fr) which means how much token will be retaned, while keeping the total KV recomputation budget fixed at 20%. As the table shown blow, enabling gradual filtering does not improve performance in our cross-LoRA KV repair setting.

CacheBlend fixed(fr=1)fr=0.90.80.70.50.3
F1-Score0.14530.13980.13660.13100.13620.1218

For Relaycaching, after conducting tests on the calibration set, we chose a ratio of 20:80. For other parameters, if the fixed recalculation ratio is set at 25%, the results on Longbench are as follows. The table below shows that their performance is broadly comparable in these tasks, and each has its own strengths and weaknesses.

dataset100:080:2050:5020:800:100
narrativeqa0.14380.16180.16590.16160.1642
qasper0.28760.31220.30990.31410.3217
multifieldqa_en0.47870.45860.46000.45190.4554
hotpotqa0.39640.39140.37460.36330.3808
2wikimqa0.26250.28500.27170.27070.2840
musique0.22920.20430.22580.23570.2243
trec0.01000.02000.03000.02000.0300
triviaqa0.19870.34880.34140.34140.3514
samsum0.21560.36620.32150.24300.2467
passage_count0.01000.00000.00000.00000.0000
passage_retrieval_en0.31000.23000.25000.23000.2300
lcc0.28410.28700.29450.31250.3067
repobench-p0.17960.18830.18180.18430.1894
avg.0.23120.25030.24820.24070.2450

For LRAgent configuration. We follow LRAgent's QKVO-based KV-sharing scheme in our implementation. However, our evaluation considers a more general multi-LoRA setting, where LoRA adapters are not restricted to attention projections (Q/K/V/O) but are also applied to MLP/FFN modules. This differs from the training configuration considered in the original LRAgent evaluation and may affect its performance.

Importantly, we retain this broader setting to evaluate cache reuse under more general adapter configurations, rather than restricting LoRA placement to favor a particular baseline. Unlike approaches that rely on specific adapter-placement assumptions, MuLA's attention-guided sparse repair does not require LoRA updates to be confined to Q/K/V/O projections. We will clarify this distinction and its implications for baseline applicability in the revised paper.

W2 / Q2: Complete workflow efficiency and measured GPU memory

Our reported handoff TTFT (Fig6) measures the latency speedup from the end of upstream decoding to the downstream agent's first token. It captures the handoff-critical operations but does not fully include upstream attention collection and temporary hidden-state buffering. We will clarify this measurement boundary and report the associated upstream overhead separately. Comparison of decoding time overhead between "Vanilla Flash-Attn" and "MuLA's Kernel". The numbers represent the average upstream decode time taken (variance, ms), for decoding 64 tokens. The results indicate that MuLA incurs only minimal overhead at the upstream stage. The results are shown as below.

modelContext lengthvanilla kernel(ms)MuLA's kernel(ms)
Llama-3.2-3B10,2401178.5±20.21190.8±20.6
Llama-3.2-3B20,4801172.9±20.41183.2±28.2
Llama-3.2-3B40,9601666.8±2.81685.8±3.9
Llama-3.2-3B81,9202759.6±0.62796.0±0.6
Qwen3-1.7B10,2401522.7±46.21524.6±21.9
Qwen3-1.7B20,4801466.0±20.11477.7±29.6
Qwen3-1.7B40,9601630.6±8.01646.8±16.8
Qwen3-1.7B81,9202735.0±6.02773.3±5.4

We would also like to clarify a misunderstanding regarding Figure 10: About Peak memory overhead, We would like to clarify that Figure 10 reports actual GPU memory measurements, rather than theoretical memory estimates. The term "simulated" refers only to the use of synthetic (mocked) input sequences of varying lengths to construct controlled workloads. All reported memory values were obtained from actual GPU executions of MuLA.

Furthermore, the "stochastic fluctuations" in the right panel are not artificially injected noise. They arise naturally from variations in the overlap between token sets selected for recomputation at adjacent Transformer layers. Specifically, the temporary hidden-state storage depends on the number of newly selected tokens at each layer, i.e., the onset set Ol=Fl∖Fl−1O_l=F_l\setminus F_{l-1}. As the overlap between FlF_l and Fl−1F_{l-1} varies across runs, the required hidden-state storage also varies slightly, producing the observed fluctuations in measured memory overhead.

We agree that the current figure caption may have inadvertently suggested a simulated memory model or artificially introduced noise. We will revise the caption and accompanying discussion to explicitly distinguish synthetic workloads from actual GPU measurements, and clarify the source of the observed memory fluctuations.

W4 / Q3: Complete execution flow and state propagation

At each handoff, MuLA first inherits the upstream KV cache and the metadata describing previously selected repair positions. It then performs target-adapter sparse recomputation layer by layer.

For a token selected at consecutive layers, the recomputed hidden state is propagated from the preceding layer. For a token that newly enters or re-enters the repair set, recomputation starts from the corresponding stored upstream hidden state. KV entries at selected positions are updated in-place, while unselected entries remain inherited.

After sparse repair, the downstream agent processes its incremental prompt and continues generation. Its resulting cache and newly collected selection information are subsequently used for the next handoff.

Regarding Repeated Handoffs (Inter-handoff): Each upstream agent jj selects top-kk positions Sl,jS_{l,j} exclusively within its own new segment ΔCj\Delta C_j and caches onset hidden states Hil,jH_{i}^{l,j} once for i∈Sl,j−Sl−1,ji \in S_{l,j} - S_{l-1,j}. At handoff to Agent mm, the cumulative repair set is the disjoint union Fl,m=∪j=1m−1Sl,jF_{l,m} = \cup_{j=1}^{m-1} S_{l,j} (Eq. 10). Agent mm runs Eq. (12) over Fl,mF_{l,m} using the cached onset states Hil,1:mH_i^{l, 1:m} and its own LoRA weights ΔWm\Delta W_m to overwrite KV entries at Fl,mF_{l,m} in-place, preventing multi-hop error accumulation without re-selection. We will make this point more clearly and explicitly in the final version (Eq. 10 & 12).

Then we compare the error reduction ratio (defined as 1−∥Vr−Vgt∥∥Vgt−Vr=0∥1-\frac{\|V_r-V_{gt}\|}{\|V_{gt}-V_{r=0}\|}) over reduction repair ratio. We present two types of statistics, the average of reduction ratio of all the tokens and the average of reduction ratio of selected tokens. The results are shown below, the selected tokens can be well repaired even if r is small.

rV(all)V(selected)K(all)K(selected)
0.000.0000.0000.0000.000
0.100.1320.6820.1600.798
0.250.3640.8660.4180.894
0.500.6880.8990.7320.917
1.001.0001.0001.0001.000

W3 / Q4: Position-dependent computational complexity

We agree that the repair fraction alone does not fully determine sparse causal-attention FLOPs. Since a repaired query at position (i) attends to all preceding positions, the precise attention cost is proportional to ∑l=1L∑i∈Fli\sum_{l=1}^{L}\sum_{i\in F_l}i, rather than simply rN2rN^2, We will revise Appendix B.1 to introduce the position-weighted effective recomputation ratio ∑l∑i∈FliLN(N+1)/2\frac{\sum_l\sum_{i\in F_l}i}{LN(N+1)/2}, while 1r\frac{1}{r} follows only under an additional assumption that the selected tokens have approximately the same average position as the full sequence.

We show the repair profiling breakdown of 16k context as below.The acceleration of FFN is nearly linear. However, the acceleration of attn deviates from linearity more significantly when r is relatively small (< 0.10). While FFN and projections scale nearly linearly with 1/r1/r, attention scales sub-linearly at small rr (< 0.10) for two reasons: (1) HBM bandwidth bound: even with rNrN queries, each query block must still stream KV tiles across its causal prefix [0,pi][0, p_i], making KV memory loading the bottleneck at low query density; and (2) Average positional skew: attention-selected tokens (including retained trailing observation-window tokens) skew toward later sequence positions, incurring above-average prefix scan lengths compared to a uniform distribution.

r (16k)attn (ms)proj (ms)ffn (ms)attn / proj / ffn proportion%attn speedupproj speedupffn speedup
0.032.3710.5841.48053 / 13 / 3310.1×19.0×22.9×
0.104.4821.1963.83247 / 13 / 405.3×9.3×8.8×
0.259.2822.9178.89444 / 14 / 422.6×3.8×3.8×
0.5015.9125.88817.68840 / 15 / 451.50×1.88×1.91×
0.7520.9148.61425.97238 / 16 / 471.14×1.29×1.30×
1.0023.87511.07633.87035 / 16 / 491.00×1.00×1.00×

W5 / Q4: Quality stability and retrieval-task degradation

We agree that average benchmark performance does not imply uniformly small degradation across every subtask.

The following table presents MuLA's performance on the passage_retrieval_en dataset under different budget settings. We made extensive efforts to investigate the reasons behind the performance gap on the passage_retrieval_en dataset and conducted additional experiments on other retrieval tasks, including the NIAH retrieval task in RULER(Table 7 in original paper). Although MuLA still exhibits a performance gap compared with the full-repair setting, its performance remains much better than the 0%-repair setting. Meanwhile, other baselines also exhibit relatively poor performance on this dataset, suggesting that the task itself may be particularly special(Table 1 in original paper). Despite our efforts to conduct further experiments, we did not observe similarly large performance gaps on other retrieval tasks. This suggests that the observed issue may be specific to passage_retrieval_en. Due to time constraints, It remains unclear to identify the underlying cause of this phenomenon or develop a corresponding solution.

The performance improvement in terms of the budget for repair after adding it to passage_retrieval_en is as follows:

Repair budget0(Direct)10%25%40%50%60%80%
score0.060.490.520.560.580.590.62

The performance of the Retrieval task on the Ruler dataset in the original paper is as follows:

TaskQwen3-0.6B (Full prefill)Direct ReuseMuLA (10%)MuLA (25%)
NIAH multi-key (1)99.267.299.298.4
NIAH multi-query100.084.8100.099.2
NIAH multi-value99.293.6100.0100.0
NIAH needle-noise62.452.863.263.2
NIAH single (1)100.0100.0100.0100.0
NIAH single (2)100.097.6100.0100.0
TaskLlama-3.2-3B (Full prefill)Direct ReuseMuLA (10%)MuLA (25%)
NIAH multi-key (1)99.294.499.299.2
NIAH multi-query100.085.6100.0100.0
NIAH multi-value100.087.2100.0100.0
NIAH needle-noise72.074.472.872.8
NIAH single (1)100.0100.0100.0100.0
NIAH single (2)100.0100.0100.0100.0

We thank the reviewer again for these helpful suggestions. They will help us strengthen the presentation of baseline applicability, system-level efficiency, theoretical assumptions, and the correctness of sparse state propagation.

Thank you for the detailed responses. Based on the new experiments, I would like to maintain an overall positive assessment. However, I still strengthen that I am not very familiar with the specific topic studied in this paper, and therefore I could not raise my score to 4.

Reviewer_WyhnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MuLA studies KV-cache reuse between agents sharing a backbone but using different LoRA adapters. It uses upstream attention statistics to select tokens for repair, allows repair positions to vary across layers, and caches onset hidden states to support sparse recomputation under the receiving adapter. Experiments on LongBench, API-Bank, and a five-agent GSM8K workflow evaluate task quality and handoff latency.

Critical strengths
  1. The paper addresses a relevant problem: different LoRA adapters introduce cache mismatch even when agents share the same backbone. Supporting efficient reuse in this setting is useful for specialized multi-agent workflows.

  2. The method combines attention-based selection with layer-dependent repair and onset-state caching. The selection ablations provide evidence that these choices improve quality over fixed or random repair sets.

  3. The evaluation includes two model families, multiple repair budgets, and comparisons with several cache-reuse methods, including CacheBlend. Reporting both task quality and handoff latency helps characterize the practical trade-off.

Critical weaknesses
  1. The contribution beyond existing selective cache repair is not sufficiently isolated. MuLA follows the reuse-and-recompute framework used by CacheBlend. Although the paper includes a direct comparison and selection ablations, these do not fully separate the gains from the existing framework from those introduced by MuLA’s selection strategy, layer-dependent repair, and onset-state caching. A controlled, incremental ablation within a common implementation would clarify the contribution of each component.

  2. The custom Triton kernel lacks an independent efficiency evaluation. The paper presents an implemented sparse attention kernel, but the timing breakdown groups attention with projections and feed-forward computation. Kernel-level profiling and microbenchmarks are needed to characterize its efficiency across context lengths and repair ratios, and to distinguish reduced computation from implementation-level improvements.

  3. The workloads provide limited coverage of interaction patterns. While LongBench, API-Bank, and the five-agent chain cover different tasks, the evaluated communication is largely based on controlled handoffs and fixed sequential workflows. More varied role transitions, interaction lengths, and evolving context would strengthen the evidence that the method generalizes beyond these configurations.

Questions for authors
  1. How much improvement comes from each MuLA-specific component when added to a CacheBlend-style repair pipeline under matched recomputation and storage budgets?

  2. What fraction of repair latency is spent in the Triton attention kernel, and how does it compare with an appropriate reference implementation under the same precision and causal-attention semantics?

  3. How would MuLA extend to dynamic multi-agent frameworks, where the next agent is chosen at runtime or multiple incoming histories are merged? Would its inherited repair sets and onset states remain sufficient, or would reselection or additional recomputation be required?

5 comments

We thank the reviewer for their feedback and for acknowledging the practical relevance of cross‑LoRA KV migration, the effectiveness of our selection strategy, and the scope of our evaluation. Our responses to the concerns are given below.

W1 / Q1: Separating MuLA's contributions from Prior Repair Methods

While MuLA shares the general reuse-and-recompute paradigm with CacheBlend, the key distinction lies in how repair positions are identified under parameter-induced KV deviations. CacheBlend addresses prefix-induced discrepancies, for which shallow-layer KV deviation can serve as useful indicators of subsequent errors. In contrast, switching LoRA adapters injects new discrepancies at each Transformer layer, making shallow-layer deviation detection less reliable for identifying mid- and deep-layers error hotspots (Figures 4–5).

MuLA therefore introduces two complementary mechanisms: (i) upstream-attention-guided proactive selection, which does not require target-adapter prefill execution to identify candidate repair positions, and (ii) layer-adaptive sparse recomputation, which allows the repair set to evolve with layer-dependent attention-predicted-error patterns. Additionally, onset-state caching reduces the hidden-state storage required to support sparse recomputation, enabling the repair set to perform repair starting from the start layer where deviation arises.

Importantly, our existing Table 5 in paper already evaluates selection variants under MuLA's sparse-recomputation pipeline and budget. At a 10% repair budget on API-Bank, MuLA achieves 0.6799 Macro-F1, compared with 0.5272 without dynamic selection and 0.1280 with random selection. Additionally, we introduce onset caching, a storage-efficient mechanism that substantially reduces memory overhead by retaining hidden states only at the positions where tokens newly enter the layer-wise repair set, rather than storing the full hidden states across all layers. As demonstrated in Figure 10, this design effectively controls the additional memory footprint of sparse KV recomputation, making MuLA practical for long-context multi-agent workflows.More ablation results are shown as below.

Pipeline ConfigurationSelection SignalLayer-Adaptive Set?Hidden-State Storage FootprintMacro-F1(10%)Macro-F1(25%)
0. Random BaselineUniform RandomNo(Fixed)O(r · N · d)0.12800.1200
1. CacheBlend-style BaselineShallow KV DeviationNoO(r · N · d)0.1620.155
2. + Upstream Attention Guidance (w/o Dynamic)Upstream AttnNo(Fixed from shallow onset)O(r · N · d)0.52720.6478
3. + Layer-Dependent Reselection (w/o Onset Cache)Upstream AttnYes(Per-layer)O(L r· N · d)(Full states)0.67990.6840
4. + Onset-State Caching (Full MuLA)Upstream AttnYes(Per-layer)∑i∣Oi∣⋅d≪L⋅rNd\sum_i \lvert O_i\rvert \cdot d \ll L \cdot rNd0.67990.6840

This controlled progression isolates each contribution clearly: (i) replacing shallow KV-deviation selection (CacheBlend) with upstream attention guidance yields the largest jump because shallow KV error fails to anticipate deep LoRA hotspots; (ii) enabling layer-dependent reselection adds another +0.153 Macro-F1 by tracking cross-layer error evolution; and (iii) onset-state caching is lossless for generation quality while slashing the multi-layer hidden-state memory requirement from O(L⋅N⋅d⋅r)O(L \cdot N \cdot d \cdot r) down to only newly entering onset tokens ∑l∣Ol∣⋅d\sum_l |O_l| \cdot d.

W2 / Q2: Independent evaluation of MuLA's custom Triton sparse-attention kernel

Following this suggestion, we conduct separate evaluations for the computational gains of sparse recomputation and the efficiency of our GPU implementation.

As detailed in Appendix B.3 and Algorithm 1, we implement a FlashAttention-style sparse causal-attention kernel using Triton for MuLA, incorporating tiled KV processing and online softmax to efficiently handle selected queries with variable causal-prefix lengths. Importantly, our reported acceleration is measured against standard full-prefill FlashAttention, rather than an unoptimized dense-attention implementation, ensuring a strong and practical performance baseline.The complete prefill performance test results of customized kernel and FA2 clearly show that MuLA's speedup is primarily driven by reduced computation through sparse repair, rather than superior full-prefill kernel performance.

Context LengthFA2MuLA's kernelFA2 / MULA
81926.065 ms6.214 ms0.98×
1638423.339 ms24.075 ms0.97×
2457655.739 ms56.501 ms0.99×
32768102.357 ms103.527 ms0.98×

To further isolate the efficiency of our customized operator, we additionally conduct kernel-level profiling, separating sparse attention computation from Q/K/V projections and feed-forward operations. The detailed timing breakdown and microbenchmark results are presented below:

r (16k)attn(ms)proj(ms)ffn(ms)attn / proj / ffn proportion%attn speedupproj speedupffn speedup
0.032.3710.5841.48053 / 13 / 3310.1×19.0×22.9×
0.104.4821.1963.83247 / 13 / 405.3×9.3×8.8×
0.259.2822.9178.89444 / 14 / 422.6×3.8×3.8×
0.5015.9125.88817.68840 / 15 / 451.50×1.88×1.91×
0.7520.9148.61425.97238 / 16 / 471.14×1.29×1.30×
1.0023.87511.07633.87035 / 16 / 491.00×1.00×1.00×

W3 / Q3: Extension to dynamic agent routing and merged histories

Beyond fixed two-agent pairs, our evaluation already stresses MuLA across diverse semantic transitions and evolving context scales: (i) Heterogeneous role transitions: In our 5-agent GSM8K cascade (Appendix G.3, Fig. 11), context evolves continuously across four distinct functional transitions, switching between natural language reasoning, formal logic, and executable code. (ii) Wide-ranging interaction lengths: We evaluate evolving histories from 4K up to 80K tokens across 5–10 sequential turns (Fig. 6, Fig. 10) as well as 23 long-context tasks in LongBench and RULER (Tables 1 & 7). (iii) Varying adapter discrepancy: Our LoRA divergence sweep (Appendix G.1, Fig. 9) systematically varies source–target adapter distance, showing that MuLA keeps the quality gap within 0.5 pp even under extreme adapter divergence (where Direct Reuse drops by >60 pp).

About this question, we would like to distinguish runtime agent selection from multi-branch history merging, as they introduce different requirements.

(1) Runtime-selected downstream agents. MuLA does not require the identity of the next agent to be known when collecting upstream attention statistics. Selection is based on the upstream agent's attention and its local context segment. Once the next agent is selected at runtime, the inherited cache can be sparsely recomputed using that agent's LoRA parameters. Thus, runtime routing is compatible with MuLA's handoff mechanism as long as the cache has a well-defined predecessor, the backbone and positional semantics remain compatible, and the corresponding repair metadata and onset states are available. The current experiments validate sequential handoffs rather than only arbitrary routing policies, which we will clarify.

(2) Parallel fan-out. MuLA naturally supports fan-out scenarios. For example, when Agent A's output is passed to both Agent B and Agent C, both downstream agents can reuse Agent A's KV cache and independently perform sparse repair using their respective LoRA adapters. Each branch then maintains its own repaired cache for subsequent generation. No changes to MuLA's selection or repair mechanism are required, since each branch is simply an independent A-to-B or A-to-C handoff.

(3) Multi-branch Fan-in. Fan-in is more challenging because the outputs of different branches must be combined into a single context. For example, suppose Agent B and Agent C independently process Agent A's output, and Agent D needs to read both results. Since B and C were generated independently, their cached representations do not capture attention interactions between the two branches. Therefore, simply concatenating their KV caches would not be equivalent to prefilling the merged context.

A practical solution is to reuse the KV cache of the shared history from Agent A, while fully prefilling the outputs of B and C under the merged context for Agent D. This avoids recomputing the potentially long shared history while ensuring that the merged branch outputs are processed with the necessary cross-branch dependencies. We discuss this extension in Appendix H; optimizing general fan-in cache reuse remains future work.

We will make these applicability conditions more explicit to distinguish supported routing patterns from extensions requiring additional recomputation or cache-alignment mechanisms.

We thank the reviewer again for these helpful suggestions.

Comments on the paper

1 comment
Area_Chair_Ga9XOpenReview ↗

Action Required: Respond to Author Rebuttals Before the Discussion Period Ends

Dear Reviewers,

The authors have posted their rebuttals. As the author–reviewer discussion period is coming to a close soon, please read their responses and participate in the discussion.

All reviewers are expected to respond, at minimum with a brief acknowledgment confirming that the reviewers have read and considered the authors’ rebuttals. If your concerns have been addressed or remain unresolved, please briefly indicate this in your response.

Thank you for your time and effort.

Best,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.