acceptodds
Under review as a conference paper at ICLR 2027

AIRMAS: Inter-Agent Intermediate Activation Reuse for LoRA-Based Multi-Agent System

Abstract

LLM-based multi-agent systems assign a different role to each agent. Fine-tuning a separate model for every role is expensive in both compute and storage. To reduce the burden, recent systems share one frozen base model and give each agent an additional role-specific LoRA adapter. When the agents run in sequence, each agent receives the context produced so far and appends its own output. Because every adapter transforms the same tokens differently, each agent prefills the shared context again and keeps its own KV cache. Prefill cost and memory therefore increase with the context length. To address this issue, recent work splits the value cache into two parts. All agents share the base cache computed with the frozen backbone weights, and each agent keeps a small low-rank part from its own LoRA adapter. Reusing the low-rank part across agents requires every adapter to share a single down-projection. Adapters with independently trained down-projections cannot reuse it. We propose AIRMAS, an inference framework that addresses this limitation. Instead of sharing a single down-projection matrix, AIRMAS maps the stored low-rank cache of an earlier agent to the current agent with an matrix precomputed from the two down-projections. On agentic question-answering benchmarks, AIRMAS improves accuracy by up to 3.33% over BaseLRShared, the low-rank cache sharing scheme of LRAgent. It requires no change to the adapters, and its time-to-first-token stays within 2.4% of that scheme.

Primary area:
optimization
Keywords:
Multi-agent system, LoRA, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WheUOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes AIRMAS, an LoRA-based multi-agent inference framework that reuses both base cache and LR cache for efficiency. Based on LRAgent (Jeon et al., 2026), AIRMAS proposes the transformation matrix for LR cache reuse, thereby overcomes the limitation of BaseLRShared. Furthermore it recovered part of the accuracy drop in BaseLRShared maintaining the inference efficiency.

Critical strengths
  1. AIRMAS presents the limitations of the baseline clearly. It reports an improvement in accuracy compared to the baselines, while nearly maintaining the efficiency of BaseLRShared which requires a shared down-projection.
Critical weaknesses
  1. In Section 1, the authors identify the limitations of BaseLRShared, which are: 1. a single shared down-projection can limit specialization, and 2. adapters with independently trained down-projections also cannot use this scheme. AIRMAS resolves the second limitation, but questions remain regarding the first limitation. AIRMAS transforms the stored LR caches of the earlier agents into the LR caches for the present agent by multiplying them by a matrix T∈Rr×rT \in \mathbb{R}^{r \times r}. This is a process of reconstructing the hidden state XX from XAiXA_i and then computing XAjXA_j. XAj≈X′Aj=XAiAi†Aj=ZiAi†Aj=ZiTi→jXA_j \approx X'A_j = XA_iA_i^{\dagger}A_j = Z_iA_i^{\dagger}A_j = Z_iT_{i \to j}. However, when XX is reconstructed from ZiZ_i, since ZiZ_i already depends only on the projection of XX onto the column space of AiA_i, the row space of the reconstructed X′X' is also contained in the column space of AiA_i, whose dimension is at most rr. The approximated XAjXA_j obtained from this loses the components of the original XX that lie outside the column space of AiA_i but are captured by AjA_j. The approximation therefore depends only on the projection of XX onto the column space of AiA_i. When the LoRA contribution to the value vector is written as an equation, this has, in fact, the same effect as two agents sharing the same down-projection AiA_i. VLoRA=XAjBj≈ZiTi→jBj=XAiTi→jBj=XAiBj′V_{\mathrm{LoRA}} = XA_jB_j \approx Z_iT_{i \to j}B_j = XA_iT_{i \to j}B_j = XA_iB_j', where Bj′=Ti→jBjB_j' = T_{i \to j}B_j. That is, through appropriate training, a new agent jj, (Ai,Bj′)(A_i, B_j'), with the same specialization capacity as agent j in AIRMAS can be constructed on the shared down-projection, and this is also possible in closed form. Therefore, the accuracy improvement over BaseLRShared that the authors show in the experiments requires supplementary mathematical justification.
Questions for authors
  1. As I stated in Critical Weaknesses, could you make a supplementary mathematical justification of why AIRMAS is superior to BaseLRShared in terms of the accuracy and diversity?

  2. When evaluating in the AutoAct framework, the trajectories and answers can vary even with greedy decoding and a fixed seed, because the Tool-Agent invokes external tools (e.g., web search) and the observations can change over time (please correct me if I am wrong). Could you provide the mean and variance of the accuracy over multiple runs?

10 comments
AuthorsOpenReview ↗

Response to Reviewer WheU (Part 1)

We sincerely thank you for the constructive feedback. We appreciate your valuable recognition of our clear presentation of the baseline's limitations. We address all the concerns and questions below.

(W1) & (Q1) Equivalence on Reused Prefixes, and Justification of Accuracy and Diversity

(1) The proposition holds on reused prefixes. For any r×rr\times r matrix TT, ZiTBj=XiAi(TBj)Z_i T B_j = X_i A_i (T B_j). On tokens produced by agent ii, AIRMAS is therefore equivalent to an adapter that shares AiA_i, with Bj′=TBjB'_j = T B_j, and Xi(I−AiAi+)AjX_i(I - A_iA_i^{+})A_j cannot be recovered deterministically. We have added this proposition as Proposition 1 in Appendix F.1 of the revised paper, credited to the reviewer, and we do not claim that AIRMAS is mathematically superior to BaseLRShared [1]. The proposition limits what the transform can recover, but the transform still recovers part of it. On our LLaMA·ScienceQA adapters, T=Ai+AjT = A_i^{+}A_j is small but not zero (∥T∥F/r\lVert T\rVert_F/\sqrt{r} = 0.077, root mean square over layers, modules and agent pairs, against 0.044 for random subspaces and 1 for the identity; largest singular value 0.45). On actual inputs, the transformed LoRA output points along a direction that holds 89% of the exact output's energy (cos⁡2=0.89\cos^2 = 0.89 under teacher forcing, i.e. fed Non-Shared's recorded tokens; Table A), although it is much smaller than the exact output. Appendix F.2 of the revised paper reports these values. Two differences from BaseLRShared remain. On its own tokens, each agent uses its independently trained AjA_j, and Bj′B'_j is obtained in closed form without retraining. The first difference is visible in measurement (Table A). On the tokens the action and reflect agents compute themselves, AIRMAS's LoRA outputs differ from Non-Shared's by only 15–16%, because both use the same adapters there. BaseLRShared's differ by 64–76%, about as much as those of Non-Shared with shared-A adapters, which reuses nothing. The planner is left out because its adapter is the same file in both adapter families. Appendix G of the revised paper now states that on reused prefixes, specialization is restricted to col(Ai)\mathrm{col}(A_i), which answers the first limitation you raise.

Table A compares the LoRA outputs of both methods with Non-Shared's, under teacher forcing on Non-Shared's trajectories. It also includes Non-Shared with shared-A adapters, which differs from Non-Shared only in its adapters and reuses nothing.

Table A. Relative error of the LoRA output against Non-Shared under teacher forcing (cosine similarity in parentheses). All rows except the first are for the action and reflect agents.

LoRA outputNon-Shared (shared-A)AIRMASBaseLRShared
Value, reused from other agents0.65 (0.81)0.85 (0.95)0.66 (0.80)
Value, computed by the same agent earlier0.67 (0.80)0.16 (0.99)0.64 (0.81)
Value, computed in the current call0.69 (0.74)0.16 (0.99)0.66 (0.76)
Query, computed in the current call0.78 (0.64)0.15 (0.99)0.76 (0.66)

Table 23 in Appendix E.5 of the revised paper reports these values.

AuthorsOpenReview ↗

Response to Reviewer WheU (Part 2)

(W1) & (Q1), continued

(2) What can be proven, and what it assumes. Under your equivalence, AIRMAS on a prefix produced by agent ii is the shared-A adapter (Ai,Bj′)(A_i, B'_j) with Bj′=T∗BjB'_j = T^{*}B_j and T∗=Ai+AjT^{*} = A_i^{+}A_j. T∗T^{*} minimizes ∥Aj−AiT∥F\lVert A_j - A_iT\rVert_F, so AiT∗A_iT^{*} is the orthogonal projection of AjA_j onto col(Ai)\mathrm{col}(A_i). Among adapters that share AiA_i and are built from (Aj,Bj)(A_j, B_j) without data, AIRMAS's is therefore the closest to agent jj's own adapter in weight space. Proposition 2 in Appendix F.1 of the revised paper proves a slightly stronger statement. Among all adapters with down-projection AiA_i, this one is the closest to AjBjA_jB_j. This guarantee carries over to the LoRA output when the inputs are isotropic, but ours are far from isotropic. Appendix F.2 of the revised paper gives the exact condition with Eqs. 19 and 20. The value-projection inputs have an effective rank of about 20 out of 4096 dimensions (Table B). On such inputs, it need not be optimal. For example, with Aj=(1,1)⊤A_j = (1, 1)^{\top}, Ai=(1,0)⊤A_i = (1, 0)^{\top} and x=(1,−0.9)x = (1, -0.9), the exact value is 0.1 and the transformed value is 1. On our inputs, it is still clearly better than reusing the cache without a transform (T=IT = I; Table C). It also does not make AIRMAS more accurate than a shared-A adapter trained on data, as in BaseLRShared. Which method is more accurate therefore has to be measured.

Table B. Isotropy of the value-projection inputs, on Non-Shared's trajectories processed as in AIRMAS (mean over layers).

MeasureOur inputsIsotropic inputs
Effective rank (of 4096 dimensions)20.14096
Energy in the top 8 directions38%0.2%
Largest eigenvalue / mean eigenvalue8771

Eq. 18 and Table 28 in Appendix F.2 of the revised paper define the effective rank as the participation ratio and describe how it is measured.

Table C. Same adapters, with and without the transform. The first three rows are under teacher forcing on Non-Shared's trajectories; accuracy is from actual runs.

MeasureNo transform (T=IT = I)AIRMAS
Relative error (cosine similarity) of xAB1.00 (0.11)0.85 (0.95)
Component along Non-Shared's xAB (% of its norm)215
Margin of Non-Shared's final answer3.473.75
Accuracy (%)65.00 ± 0.0066.39 ± 0.00

Table 21 in Appendix E.5 of the revised paper reports these results.

(3) What we measured, with Non-Shared as the reference for both methods. In Table A, on reused tokens, AIRMAS's LoRA output points closer to Non-Shared's than BaseLRShared's does, but it is smaller (magnitude ratio 0.16 vs. 0.52). For isotropic inputs, the cosine similarity would equal the magnitude ratio; on our inputs it is about six times larger. The inputs concentrate on a few directions (Table B), and the trained A matrices put 85–92% of their input-weighted energy in the top 8 of them (37% for random A). So the part that the projection keeps points nearly the same way as the exact output. The transform therefore loses magnitude but keeps direction. Appendices E.6 and F.2 of the revised paper analyze this direction–magnitude split. BaseLRShared's error is almost the same as that of Non-Shared with shared-A adapters. Of its squared error of 0.44, 0.43 is already present without reuse, and reuse adds 0.004. Its deviation therefore comes from its adapters, not from reuse.

AuthorsOpenReview ↗

Response to Reviewer WheU (Part 3)

(W1) & (Q1), continued

(4) Accuracy against BaseLRShared is measured, not guaranteed. On reused prefixes, both methods have a shared A. About 96% of reused rows are produced by the planner, and on these both methods use the planner's A, so they differ only in B. AIRMAS derives B in closed form, and BaseLRShared trains it. Overall accuracy also depends on each agent's own tokens, where AIRMAS uses its independently trained adapters. Structure alone cannot decide this. Measured, the gap splits exactly into the adapter effect (Non-Shared minus Non-Shared with shared-A adapters) and the difference between the two methods' reuse losses (each method's drop from its own Non-Shared). Table D gives this split over five adapter-training seed.

Table D. Split of the accuracy gap AIRMAS − BaseLRShared over five seeds (ScienceQA, pp, mean ± SD).

ModelNon-SharedNon-Shared (shared A)BaseSharedBaseLRSharedAIRMASDifference
LLaMA71.29 ± 0.5467.94 ± 3.3268.55± 3.2162.83 ± 2.2766.78 ± 0.83+3.95
Ministral68.34 ± 0.5667.95 ± 0.5359.33 ± 1.7856.78 ± 1.1758.33 ± 1.55+1.55

Tables 19 and 22 in Appendices E.4 and E.5 of the revised paper report these results.

Both components contribute positively to the measured mean accuracy gap in both models, but their relative contributions differ. The adapter effect accounts for most of the gap on LLaMA, whereas the difference in reuse losses accounts for most of the gap on Ministral. In both cases, AIRMAS incurs a smaller mean accuracy drop from its own Non-Shared reference than BaseLRShared does from its corresponding reference.

On seed 42, BaseLRShared's reuse loss shows up in its planner's decision margin (Table E). Under the same teacher forcing, at the steps where Non-Shared's planner decides to answer, BaseLRShared's planner is less confident in that decision, even though the LoRA output it reuses there is nearly identical to Non-Shared's (cosine 0.99). Non-Shared with shared-A adapters uses the same planner and reuses nothing, so this drop comes from reuse. At the final answer, AIRMAS is also more confident in the answer Non-Shared chose.

Table E. Margin under teacher forcing, defined as the logit of Non-Shared's choice minus the highest other logit.

DecisionNon-SharedNon-Shared (shared a)AIRMASBaseLRShared
Accuracy (%)73.0666.6766.3965.39
Planner's decision margin1.911.912.031.77
Non-Shared's final answer–-3.753.28

Table 24 in Appendix E.5 of the revised paper reports these margins.

The episodes in which BaseLRShared never answers repeat the same search until the 15-step limit. Exact reuse of the LR term therefore does not remove the cost of reusing hidden states computed by other agents.

Reviewer_eZhSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AIRMAS, an inference framework for sequential multi-agent systems based on Large Language Models (LLMs) that enables the reuse of intermediate activations across independently trained LoRA adapters. AIRMAS employs a lightweight pseudo-inverse transformation to approximate the low-rank activations of a target agent using previously cached activations. Furthermore, the authors integrate this transformation into a fused Flash-LoRA-Attention operator, thereby minimizing inference overhead.

Critical strengths
  1. It addresses a critical limitation in existing cross-agent cache sharing mechanisms. The paper identifies a significant flaw in LRAgent: it requires the sharing of LoRA down-projection matrices, which restricts its applicability to independently trained adapters and may hinder agents from achieving role specialization. AIRMAS removes this constraint without requiring modifications to existing adapters or any additional training. This is a practically valuable contribution for multi-agent systems where distinct roles necessitate independent, specialized adapters.

  2. An efficient implementation. The proposed transformation mechanism is highly lightweight; its integration into Flash-LoRA-Attention avoids the overhead of additional kernel launches and the writing of intermediate data to global memory. Experiments demonstrate that this implementation significantly reduces redundant computation and lowers memory consumption.

Critical weaknesses
  1. The contribution of the pseudo-inverse transformation has not been sufficiently isolated or verified.

The core technical contribution of this paper lies in the "pseudo-inverse mapping" between independently trained LoRA adapters. However, in the primary accuracy comparison against BaseLRShared, the experimental setup simultaneously alters the adapter training process: BaseLRShared employs a shared down-projection, whereas AIRMAS utilizes independently trained adapters. As shown in Appendix E, independently trained adapters can achieve significantly higher accuracy on their own, even without cross-agent cache reuse. Consequently, while this comparison demonstrates a system-level advantage, it is impossible to definitively determine what portion of the performance gain is truly attributable to this specific "pseudo-inverse transformation."

  1. The trade-off between accuracy and efficiency in the evaluated benchmarks is not sufficiently compelling.

Although AIRMAS consistently outperforms BaseLRShared in terms of accuracy, it still falls short of the "Non-Shared" and "BaseShared" approaches across all four primary evaluation scenarios. For instance, on the ScienceQA task using the LLaMA-3.1-8B-Instruct model, AIRMAS achieves an accuracy of 66.39%, compared to 73.06% for the Non-Shared approach and 70.56% for the BaseShared approach.

More importantly, Appendix F.1 reveals that on actual question-answering (QA) benchmark traces, the BaseShared approach not only yields higher accuracy but also exhibits a slightly lower average model latency than AIRMAS. The substantial throughput improvements highlighted in the paper are primarily observed in controlled test sequences characterized by context lengths far exceeding those encountered in typical QA tasks. Therefore, it remains unclear whether this loss in accuracy is adequately compensated for by significant gains in end-to-end efficiency within realistic application scenarios.

Questions for authors
  1. To what extent can the improvement in accuracy be specifically attributed to the pseudo-inverse transformation?

  2. Under what practical workloads does AIRMAS offer a superior "accuracy-latency" trade-off compared to BaseShared?

Appendix F.1 indicates that, on the evaluated question-answering (QA) benchmarks, BaseShared achieves not only higher accuracy but also slightly lower average model latency. Could the authors clarify the specific context lengths or workload scenarios in which AIRMAS demonstrates a significant practical advantage over BaseShared?

  1. How robust are the reported improvements in accuracy?

The primary evaluation uses 300 questions from HotpotQA and 360 from ScienceQA, with some results showing only marginal gains over BaseLRShared. Could the authors provide paired confidence intervals, or illustrate the variance in results when adapters are trained independently using different random seeds?

4 comments
AuthorsOpenReview ↗

Response to Reviewer eZhS (Part 1)

We sincerely thank you for the constructive feedback. We appreciate your valuable recognition of the key limitation we address in existing inter-agent cache sharing and the efficiency of our fused implementation. We address all the concerns and questions below.

(W1) & (Q1) Isolating the Pseudo-Inverse Transformation

We agree that the comparison with BaseLRShared [1] changes the adapters and the reuse rule at the same time, so it cannot isolate the transformation. We separate them in two ways.

(1) Same adapters, only the reuse rule changed. We keep the independently trained adapters fixed and change only how the reused LR cache is handled (Table A). Zero transform (T=0T = 0) gives 64.33 ± 0.15%, below BaseLRShared (65.39 ± 0.15%). Identity transformation (T=IT = I) gives 65.00 ± 0.00% without mapping and 66.39 ± 0.00% with the pseudo-inverse. The transformation therefore adds 2.06% over T=0T = 0 and 1.39% over T=IT = I.

Table A. Same adapters, different reuse rules (LLaMA-3.1-8B, ScienceQA). Accuracy represent as the mean ± SD. The cosine similarity of the reused LoRA output (xAB) to Non-Shared's is measured on each method's own trajectories. The last two columns are measured at the final answer, under teacher forcing on Non-Shared's trajectories.

Reuse ruleAccuracy (%)xAB cosine similarity to Non-SharedMargin of Non-Shared's final answer
T=0T = 0 (prefix LR dropped)64.33 ± 0.15–3.29
T=IT = I65.00 ± 0.000.123.47
AIRMAS (pseudo-inverse)66.39 ± 0.000.953.75

The three reuse rules differ more clearly in the reused LoRA output. It is nearly orthogonal to Non-Shared's under T=IT = I but nearly aligned under AIRMAS. For the last two columns, every rule is run with teacher forcing on the same Non-Shared trajectories (fed Non-Shared's recorded tokens). The margin is the difference between the logit of the Non-Shared response and the highest logit among the others, so it does not depend on whether that answer is correct. AIRMAS is therefore the most confident of the three in the answer Non-Shared chose. Its reused LoRA output also carries the largest part of Non-Shared's. Table 21 in Appendix E.5 of the revised paper reports these results, where the reuse rules are compared as transformations.

(2) Adapter and reuse contributions. The gap to BaseLRShared splits exactly into two parts. The first is adapter effect, the loss from Non-Shared. The second is the difference between the two methods' reuse losses, where a method's reuse loss is its drop from its own Non-Shared. Table B gives the four accuracies on seed 42.

Table B. Accuracy (%) by adapter family and KV reuse method for different agent, seed 42 (LLaMA-3.1-8B, ScienceQA; mean of 5 runs).

AdaptersNo reuseWith reuse
IndependentNon-Shared 73.06AIRMAS 66.39
Shared-ANon-Shared (shared-A) 66.67BaseLRShared 65.39

Appendix E.5 of the revised paper extends this split to five adapter-training seeds with Eqs. 13–14 and Table 22.

The same split appears in the value cache on the action and reflect agents against Non-Shared (Table C). Non-Shared with shared-A adapters differs only in its adapters. Its relative error is 0.33, 0.45 across reused rows, the agent's own rows and the current call. AIRMAS differs only through reuse. Its error is 0.43, 0.47. BaseLRShared has both. It deviates most on the rows it computes itself 0.42, 0.52. These follow the order of the accuracy losses in Table B.

Table C. Relative error against Non-Shared under teacher forcing (cosine similarity in parentheses).

QuantityNon-Shared (shared-A)AIRMASBaseLRShared
xAB, reused from other agents0.65 (0.81)0.85 (0.95)0.66 (0.80)
Value cache, reused from other agents0.330.430.42
Value cache, computed by same agents earlier0.450.470.52
Value cache, computed in current call0.420.420.46

Table 23 in Appendix E.5 of the revised paper reports these values.

We have added the transformation ablation and this split to Appendix E.5 of the revised paper.

[1] Jeon et al., "LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents", ICML 2026.

AuthorsOpenReview ↗

Response to Reviewer eZhS (Part 2)

(W2), (Q2) Accuracy–Latency Trade-off against BaseShared

We agree with this assessment for the evaluated benchmark lengths, and our new experiments confirm it. On ScienceQA, BaseShared [1] is more accurate than AIRMAS. Relative to BaseShared, AIRMAS saves only the recomputation (prefill) of other agents' prefixes; its decode cost is equal or slightly higher, and its memory is similar (23.35 vs. 23.46 GiB at 66.4k tokens).

To answer Q2 and Q3 directly, we lengthened the tool outputs on ScienceQA with LLaMA-3.1-8B, by truncating the tool response to different lengths (Table A). We measured accuracy and model latency, with the deployed kernels (Flash-LoRA for BaseShared and BaseLRShared, the fused kernel for AIRMAS). All measurements were run on H100 NVL GPU. Three warmup episodes excluded from calculating timing.

Table A. Long tool outputs on ScienceQA. Times are totals over the 360 questions.

Average tool length per responseAverage context lengthMethodAccuracy (%)Total model time (s)
2311352BaseLRShared65.282142
2301154BaseShared70.561882.8
2271184AIRMAS66.392109.6
5673678BaseLRShared52.223600
5853502BaseShared56.673420
5673432AIRMAS56.113636
168114412BaseLRShared36.675616
164713452BaseShared38.065796
166913175AIRMAS40.565652
295230634BaseLRShared25.338208
298129722BaseShared26.679216
305430234AIRMAS25.568260

At short contexts, AIRMAS is less accurate and slower than BaseShared. With long tool outputs or context, which are common in LLM agent systems, AIRMAS becomes faster. On ScienceQA, AIRMAS achieves less model latency with similar or higher accuracy in long context benchmark run. Most importantly, AIRMAS reaches higher accuracy than BaseLRShared, the state-of-the-art multi-LoRA agent system, at nearly the same latency (Table A above and Table 3 of the paper). Appendix E.8 and Table 27 of the revised paper report these runs, and its last paragraph explains how the number of agent steps of each scheme affects the time on real trajectories.

[1] Jeon et al., "LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents", ICML 2026.

AuthorsOpenReview ↗

Response to Reviewer eZhS (Part 3)

(Q3) Robustness and Seed Variance

To assess the robustness of the reported accuracy improvements, we conducted additional repeated-run evaluations and experiments using five adapter-training seeds on ScienceQA. Tables C and D report these results.

Repeated-run evaluation. Table C reports five runs for LLaMA–ScienceQA and four runs for each remaining setting. AIRMAS achieves higher mean performance than BaseLRShared in all four settings. The mean improvements are 1.00 and 2.67 percentage points for LLaMA on ScienceQA and HotpotQA, respectively, and 2.15 and 0.67 percentage points for Ministral.

Table C. AIRMAS vs. BaseLRShared accuracy over repeated runs (mean ± SD, %).

SettingRepeatsAIRMASBaseLRSharedDifference
LLaMA · ScienceQA5 runs66.39 ± 0.0065.39 ± 0.15+1.00
LLaMA · HotpotQA4 runs37.75 ± 0.535.08 ± 0.32+2.67
Ministral · ScienceQA4 runs58.54 ± 0.2756.39 ± 1.11+2.15
Ministral · HotpotQA4 runs34.17 ± 0.5833.50 ± 0.58+0.67

Table 18 in Appendix E.4 of the revised paper reports these runs and analyzes the sources of run-to-run variation.

Variation across adapter-training seeds. Table D reports mean accuracy and standard deviation across five adapter-training seeds on ScienceQA. For LLaMA, AIRMAS achieves 66.78 ± 0.83%, compared with 62.83 ± 2.27% for BaseLRShared, yielding a mean improvement of 3.95 percentage points. For Ministral, AIRMAS achieves 58.33 ± 1.55%, compared with 56.78 ± 1.17%, yielding a mean improvement of 1.55 percentage points.

Table D. Accuracy on ScienceQA over five adapter-training seeds (mean ± SD, %).

ModelNon-SharedBaseSharedBaseLRSharedAIRMASDifference
LLaMA71.29 ± 0.5468.55± 3.2162.83 ± 2.2766.78 ± 0.83+3.95
Ministral68.34 ± 0.5659.33 ± 1.7856.78 ± 1.1758.33 ± 1.55+1.55

Tables 19 and 20 in Appendix E.4 of the revised paper report these results and list the random seed of every training job.

[1] Jeon et al., "LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents", ICML 2026.

AuthorsOpenReview ↗

Follow-up on our response to Reviewer eZhS

Thank you again for your review. As the discussion period ends soon, we would like to briefly note an update related to your W1 and Q1.

In response to additional feedback from Reviewer WheU, the main text now separates the two effects that the comparison with BaseLRShared mixes. Section 4.2 splits the gap into the accuracy that the shared-A constraint costs and AIRMAS keeps, and the difference between the accuracy that each method loses to reuse. It reports the effect of the pseudo-inverse transformation only with the adapters fixed: 2.06% over T=0T=0 and 1.39% over T=IT=I (Appendix E.5, Tables 21 and 22). The main text also no longer presents AIRMAS as a more accurate cache reuse. All changes relative to the original submission are shown in blue in the updated PDF.

For W2 and Q2, Section 4.3 states that BaseShared has a lower model latency at the benchmark lengths and that AIRMAS becomes faster with long tool outputs (Appendices E.1 and E.8). For Q3, the repeated runs and the five adapter-training seeds are in Appendix E.4 (Tables 18–20). All section, appendix, table and equation numbers are unchanged, so the references in our earlier responses remain valid. If any questions remain, please let us know.

Reviewer_twK6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LRAgent’s BaseLRShared reuses the low rank value cache across LoRA-based agents, but only when all adapters share the same down projection. AIRMAS removes this requirement. It applies a small precomputed transformation, derived from the two agents’ down-projection weights, to convert an earlier agent’s stored LR cache for the current agent, and fuses this step into Flash-LoRA-Attention. On AutoAct with LLaMA-3.1-8B and Ministral-8B (HotpotQA, ScienceQA), AIRMAS is 1.1–3.3 pp more accurate than BaseLRShared at essentially identical latency and memory.

Critical strengths
  • Forcing a shared down projection at training time is restrictive, and a training-free way to reuse caches across independently trained adapters would be useful.
  • The fused kernel keeps TTFT, TBT, and memory within a few percent of BaseLRShared across 1.9k–66.4k tokens.
Critical weaknesses
  • Incremental contribution with unclear positioning. All efficiency benefits come from LRAgent, and the only new component is the transformation matrix. ICaRus already enables exact cache reuse without a shared down projection. The paper argues that AIRMAS instead "keeps existing adapters", but its experiments train new adapters anyway, and no comparison with ICaRus is provided.
  • The accuracy gain does not show that the transform works. AIRMAS loses 6.67 pp relative to Non-Shared with the same adapters (9.72 pp on Ministral/ScienceQA). Its small edge over BaseLRShared reflects stronger starting adapters rather than an accurate transformation. The margins (e.g., 4 of 360 questions) come from a single greedy run, and AIRMAS even exceeds exact Non-Shared in some settings, suggesting the effects are within noise.
Questions for authors
  • How does AIRMAS compare with ICaRus in accuracy and efficiency under the same agent setup? Since adapters are retrained in your experiments, why is approximate reuse preferable to a training-time method that enables exact reuse?
  • What is the relative error of the transformed cache versus the recomputed one on real trajectories, and what are the accuracies with an identity transform or with the prefix LR contribution dropped? Were the role adapters’ down-projections initialized with the same seed?
7 comments
AuthorsOpenReview ↗

Response to Reviewer twK6 (Part 1)

We sincerely thank you for the constructive feedback. We appreciate your valuable recognition of our motivation. We address all the concerns and questions below.

(W1), (W2) & (Q1) Positioning, Comparison with ICaRus, and Where the Accuracy Comes From

We agree that LR cache reuse and Flash-LoRA-Attention come from LRAgent (BaseLRShared) [1]; AIRMAS adds pseudo-inverse transform of each agent and the fused kernel that integrates it into Flash-LoRA-Attention. Section 2 of the revised paper now states this positioning: AIRMAS builds on the LR cache and Flash-LoRA-Attention of LRAgent and adds the transformation and its fused kernel.

Exact-reuse methods obtain exactness by changing the training procedure. ICaRus [2] prefills with the frozen base and decodes with LoRA adapters. LRAgent trains all agents to share the same down projection matrix A , which makes its cache reuse more exact than AIRMAS’s. This comparison was missing, and we have now run ICaRus under the same agent setup. It uses LLaMA-3.1-8B, ScienceQA, the three AutoAct [3] roles, and the same training data and hyperparameters. LoRA placement follows the ICaRus paper (q, o and FFN). Table A reports the mean and standard deviation across 5 runs.

Table A. LLaMA-3.1-8B, ScienceQA accuracy.

MethodTraining procedureAccuracy (%)
Non-Shared (non shared A)standard LoRA (q, v)73.06 ± 0.00
Non-Shared (shared A)A fixed to the plan agent's A, only train B66.67 ± 0.19
BaseLRShared (LRAgent)A fixed to the plan agent's A, only train B65.39 ± 0.15
ICaRusbase prefill, LoRA decode (q, o, FFN)64.84 ± 0.65
AIRMASunchanged66.39 ± 0.00

In our evaluation, adapters trained for exact reuse were weaker. ICaRus is 1.55% below AIRMAS. AIRMAS keeps BaseLRShared's efficiency, with negligible r×rr \times r transform cost. On the controlled traces of the paper's Table 3, ICaRus has lower throughput than AIRMAS (Table B), because it decodes two rows per token and places LoRA on five modules.

Table B. Throughput (tokens/s) on the controlled traces of Table 3 (LLaMA-3.1-8B).

Method1.9k9.1k66.4k
BaseLRShared267.01180.24051.2
AIRMAS264.81158.24027.7
ICaRus203.0911.03797.4

Appendix E.7 (Table 26) reports this comparison.

AuthorsOpenReview ↗

Response to Reviewer twK6 (Part 2)

(W1), (W2) & (Q1), continued

Even when adapters are retrained, we prefer approximate reuse for two reasons.

  1. Accuracy. In the settings we measured, the two training-time exact-reuse procedures (LRAgent and ICaRus) lost accuracy relative to Non-Shared; that is, making KV reuse exact cost more accuracy than approximating it.
  2. Modularity. AIRMAS works with standard q/v adapters as they are. T=Ai+AjT = A_i^{+}A_j depends only on the two adapters' A matrices, so it needs no retraining, calibration or joint training. In AIRMAS, single adapter only requires computing T for the affected pairs. Appendix E.7 of the revised paper gives the details.

On W2, we agree that AIRMAS is more accurate than BaseLRShared mainly because it uses independently trained adapters. AIRMAS pays no procedure cost, whereas BaseLRShared already loses 6.39% before any reuse (Table A). Keeping the adapters unconstrained is the point of a training-free transform, and the transform is what lets AIRMAS keep this advantage. Appendix E.5 of the revised paper separates these two effects with Eqs. 13–14 and Table 22. During reuse, AIRMAS's action and reflect agents still produce LoRA outputs close to Non-Shared's on the tokens they compute themselves (relative error 0.15–0.16, cosine 0.99), whereas BaseLRShared's adapters give outputs far from them (0.64–0.76). Table 23 in Appendix E.5 reports these values. With the same adapters, the zero transform (T=0T = 0) reaches only 64.33%, below BaseLRShared (65.39%). The identity transform (T=IT = I) reaches 65%, and AIRMAS 66.39% (Table E in our answer to Q2). Table 21 in Appendix E.5 reports these results.

You are right that the margins in the submission came from a single greedy run. We have now repeated the runs on the same test set. Over 5 runs, AIRMAS reaches 66.39 ± 0.00% and BaseLRShared 65.39 ± 0.15%, and every divergence between runs started from a tool response. T=0T = 0 and T=IT = I have standard deviations at most 0.15% (Table E). Although the submission reported a single run, the submitted values (AIRMAS 66.39%, BaseLRShared 65.28%) lie within the range of the repeated runs. Appendix E.4 and Table 18 of the revised paper report all repeated runs, including the original run of Table 2.

On the term "keeps existing adapters", you are right that our experiments retrain adapters; we did so to match LRAgent's data and hyperparameters. We have revised the claim to "works with standard multi-LoRA systems, without architectural constraints or retraining." Because Related Work moved to the main text, this sentence now appears in Section 2 of the revised paper.

[1] Jeon et al., "LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents", ICML 2026.

[2] Woo et al., "ICaRus: Identical Cache Reuse for Efficient Multi-Model Inference", ICLR 2026.

[3] Qiao et al., "AutoAct: Automatic Agent Learning from Scratch for QA via Self-Planning", ACL 2024.

AuthorsOpenReview ↗

Response to Reviewer twK6 (Part 3)

(Q2) Cache Error, Identity and Dropped-LR Accuracies, and Initialization

This is the most direct test of the transform, and the submission did not report it. We measured the cache error on the trajectories each method generates itself. For each quantity in Table C, YY is its value under the method and Y∗Y^{*} is its value of Non-shared when recomputes the same context. For AIRMAS, T=IT = I and T=0T = 0, Y∗Y^{*} is therefore the value under Non-Shared. For BaseLRShared [1], it is the value under Non-Shared with shared-A adapters, so that column shows only BaseLRShared's reuse error. Errors are relative Frobenius errors, e=∥Y−Y∗∥F/∥Y∗∥Fe = \lVert Y - Y^{*}\rVert_F / \lVert Y^{*}\rVert_F, aggregated per question over layers, producers and calls, then averaged.

Table C. Relative error against exact recomputation.

QuantityAIRMAST=IT = IT=0T = 0BaseLRShared
LR cache xA0.864 (cos = 0.886)1.173 (cos = 0.050)10.156
xAB0.855 (cos = 0.952)0.997 (cos = 0.116)10.115
Value cache0.3720.3810.3810.332
Attention output (rows computed in the call)0.1530.1590.1600.129
Last-layer attention output0.1180.1230.1240.107
Last-layer hidden state at decisions0.3600.3800.3870.321

Table 25 in Appendix E.6 of the revised paper reports these errors.

With the magnitude ratio s=∥Y∥F/∥Y∗∥Fs = \lVert Y\rVert_F / \lVert Y^{*}\rVert_F and the cosine cos⁡=⟨Y,Y∗⟩F/(∥Y∥F,∥Y∗∥F)\cos = \langle Y, Y^{*}\rangle_F / (\lVert Y\rVert_F\\, \lVert Y^{*}\rVert_F), the error splits as e2=(s−cos⁡)2+(1−cos⁡2)e^2 = (s - \cos)^2 + (1 - \cos^2), and cos⁡2\cos^2 is the fraction of ∥Y∗∥F2\lVert Y^{*}\rVert_F^2 that lies along the direction of YY. Eq. 15 in Appendix E.6 of the revised paper derives this split step by step. This fraction is 0.91 for AIRMAS at xAB, versus 0.01 for T=IT = I and 0 for T=0T = 0. For AIRMAS at xAB, 87% of e2e^2 comes from magnitude. The transformed LR term points in nearly the right direction (cos⁡=0.95\cos = 0.95) but is too small (s≈0.15s \approx 0.15).

BaseLRShared's lower error is measured against its own shared-A adapters. Table D measures both methods against Non-Shared instead, under teacher forcing on Non-Shared's trajectories. It also includes Non-Shared with shared-A adapters, which differs from Non-Shared only in its adapters and reuses nothing.

Table D. Relative error against Non-Shared under teacher forcing (cosine similarity in parentheses).

QuantityNon-Shared with shared-AAIRMASBaseLRShared
xAB, rows reused from other agents0.65 (0.81)0.85 (0.95)0.66 (0.80)
Value cache, reused from other agents0.330.430.42
Value cache, already computed on same agents in past0.450.470.52
Value cache, present agent0.420.420.46

Table 23 in Appendix E.5 of the revised paper reports these values.

BaseLRShared's reused LoRA output is as far from Non-Shared's as that of Non-Shared with shared-A adapters. BaseLRShared differs from Non-Shared in two ways, its adapters and its reuse. Non-Shared with shared-A adapters has the same adapters as BaseLRShared but no reuse, and its xAB error is almost the same (0.65 vs. 0.66). BaseLRShared's xAB error therefore comes from its adapters, and reuse adds almost nothing to it. On the value cache, AIRMAS and BaseLRShared are about equally close to Non-Shared on reused rows (0.43 vs. 0.42). On the other rows, AIRMAS is closer (0.47 vs. 0.52), because there it uses its own adapters, which are the same as Non-Shared's.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Discussion

We thank all reviewers and the Area Chair for their time and constructive feedback. As the discussion period ends, we briefly summarize our main responses and revisions below.

Contribution and positioning. AIRMAS builds on BaseLRShared of LRAgent and removes its shared-A constraint. A training-free r×rr \times r pseudo-inverse transformation maps the stored low-rank cache of an earlier agent to the current agent, and a fused kernel applies it within Flash-LoRA-Attention at nearly the same cost as BaseLRShared. Following the discussion, the main text now presents AIRMAS as a method that keeps each agent's own down-projection under cache reuse, rather than as a more accurate cache reuse. With Non-Shared as the reference, the accuracy gain over BaseLRShared and ICaRus is modest; we view the main contribution as removing the training-time constraint of exact reuse at nearly the same cost, rather than as a large accuracy gain.

  • Interpretation of the accuracy gain (WheU W1/Q1; twK6 W2; eZhS W1/Q1). The comparison with BaseLRShared in Table 2 changes the adapters and the reuse together. Section 4.2 now separates the accuracy that AIRMAS keeps by leaving the adapters unconstrained from the accuracy that it loses to reuse. On ScienceQA with LLaMA-3.1-8B-Instruct and the adapters of Table 2, the whole gain over BaseLRShared comes from keeping the unconstrained adapters. The effect of the transformation itself is measured only with the adapters fixed: it adds 2.06% over T=0T=0 and 1.39% over T=IT=I (Appendix E.5).
  • Robustness (WheU Q2; twK6 W2; eZhS Q3). Over repeated runs, AIRMAS is above BaseLRShared in all four settings by 0.67–2.67%. Over five adapter-training seeds, it is 3.95% and 1.55% above BaseLRShared on ScienceQA with LLaMA-3.1-8B-Instruct and Ministral-8B-Instruct (Appendix E.4). The run-to-run variation comes mostly from the external search tools.
  • What the transformation recovers (WheU W1; twK6 Q2). Propositions 1 and 2 (Appendix F.1) state what the transformation can and cannot recover. On real trajectories, the transformed LoRA output keeps the direction of the exact one (cosine similarity 0.95), and its error lies mostly in the magnitude (Appendix E.6).
  • Comparison with ICaRus (twK6 W1/Q1). Under the same agent setup, ICaRus is 1.55% below AIRMAS, and its throughput is 0.77–0.94× that of AIRMAS (Appendix E.7).
  • Trade-off against BaseShared (eZhS W2/Q2). At the benchmark lengths, BaseShared is more accurate and has a lower model latency. With long tool outputs, AIRMAS takes 2.5% and 10.4% less model time than BaseShared at about 13k and 30k tokens (Appendix E.8).

All changes relative to the original submission are shown in blue. We thank the reviewers and the Area Chair again for their time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.