- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes AIRMAS, an LoRA-based multi-agent inference framework that reuses both base cache and LR cache for efficiency. Based on LRAgent (Jeon et al., 2026), AIRMAS proposes the transformation matrix for LR cache reuse, thereby overcomes the limitation of BaseLRShared. Furthermore it recovered part of the accuracy drop in BaseLRShared maintaining the inference efficiency.
Critical strengths
- AIRMAS presents the limitations of the baseline clearly. It reports an improvement in accuracy compared to the baselines, while nearly maintaining the efficiency of BaseLRShared which requires a shared down-projection.
Critical weaknesses
- In Section 1, the authors identify the limitations of BaseLRShared, which are: 1. a single shared down-projection can limit specialization, and 2. adapters with independently trained down-projections also cannot use this scheme. AIRMAS resolves the second limitation, but questions remain regarding the first limitation. AIRMAS transforms the stored LR caches of the earlier agents into the LR caches for the present agent by multiplying them by a matrix . This is a process of reconstructing the hidden state from and then computing . . However, when is reconstructed from , since already depends only on the projection of onto the column space of , the row space of the reconstructed is also contained in the column space of , whose dimension is at most . The approximated obtained from this loses the components of the original that lie outside the column space of but are captured by . The approximation therefore depends only on the projection of onto the column space of . When the LoRA contribution to the value vector is written as an equation, this has, in fact, the same effect as two agents sharing the same down-projection . , where . That is, through appropriate training, a new agent , , with the same specialization capacity as agent j in AIRMAS can be constructed on the shared down-projection, and this is also possible in closed form. Therefore, the accuracy improvement over BaseLRShared that the authors show in the experiments requires supplementary mathematical justification.
Questions for authors
-
As I stated in Critical Weaknesses, could you make a supplementary mathematical justification of why AIRMAS is superior to BaseLRShared in terms of the accuracy and diversity?
-
When evaluating in the AutoAct framework, the trajectories and answers can vary even with greedy decoding and a fixed seed, because the Tool-Agent invokes external tools (e.g., web search) and the observations can change over time (please correct me if I am wrong). Could you provide the mean and variance of the accuracy over multiple runs?
10 comments
Response to Reviewer WheU (Part 2)
(W1) & (Q1), continued
(2) What can be proven, and what it assumes. Under your equivalence, AIRMAS on a prefix produced by agent is the shared-A adapter with and . minimizes , so is the orthogonal projection of onto . Among adapters that share and are built from without data, AIRMAS's is therefore the closest to agent 's own adapter in weight space. Proposition 2 in Appendix F.1 of the revised paper proves a slightly stronger statement. Among all adapters with down-projection , this one is the closest to . This guarantee carries over to the LoRA output when the inputs are isotropic, but ours are far from isotropic. Appendix F.2 of the revised paper gives the exact condition with Eqs. 19 and 20. The value-projection inputs have an effective rank of about 20 out of 4096 dimensions (Table B). On such inputs, it need not be optimal. For example, with , and , the exact value is 0.1 and the transformed value is 1. On our inputs, it is still clearly better than reusing the cache without a transform (; Table C). It also does not make AIRMAS more accurate than a shared-A adapter trained on data, as in BaseLRShared. Which method is more accurate therefore has to be measured.
Table B. Isotropy of the value-projection inputs, on Non-Shared's trajectories processed as in AIRMAS (mean over layers).
| Measure | Our inputs | Isotropic inputs |
|---|---|---|
| Effective rank (of 4096 dimensions) | 20.1 | 4096 |
| Energy in the top 8 directions | 38% | 0.2% |
| Largest eigenvalue / mean eigenvalue | 877 | 1 |
Eq. 18 and Table 28 in Appendix F.2 of the revised paper define the effective rank as the participation ratio and describe how it is measured.
Table C. Same adapters, with and without the transform. The first three rows are under teacher forcing on Non-Shared's trajectories; accuracy is from actual runs.
| Measure | No transform () | AIRMAS |
|---|---|---|
| Relative error (cosine similarity) of xAB | 1.00 (0.11) | 0.85 (0.95) |
| Component along Non-Shared's xAB (% of its norm) | 2 | 15 |
| Margin of Non-Shared's final answer | 3.47 | 3.75 |
| Accuracy (%) | 65.00 ± 0.00 | 66.39 ± 0.00 |
Table 21 in Appendix E.5 of the revised paper reports these results.
(3) What we measured, with Non-Shared as the reference for both methods. In Table A, on reused tokens, AIRMAS's LoRA output points closer to Non-Shared's than BaseLRShared's does, but it is smaller (magnitude ratio 0.16 vs. 0.52). For isotropic inputs, the cosine similarity would equal the magnitude ratio; on our inputs it is about six times larger. The inputs concentrate on a few directions (Table B), and the trained A matrices put 85–92% of their input-weighted energy in the top 8 of them (37% for random A). So the part that the projection keeps points nearly the same way as the exact output. The transform therefore loses magnitude but keeps direction. Appendices E.6 and F.2 of the revised paper analyze this direction–magnitude split. BaseLRShared's error is almost the same as that of Non-Shared with shared-A adapters. Of its squared error of 0.44, 0.43 is already present without reuse, and reuse adds 0.004. Its deviation therefore comes from its adapters, not from reuse.
Response to Reviewer WheU (Part 3)
(W1) & (Q1), continued
(4) Accuracy against BaseLRShared is measured, not guaranteed. On reused prefixes, both methods have a shared A. About 96% of reused rows are produced by the planner, and on these both methods use the planner's A, so they differ only in B. AIRMAS derives B in closed form, and BaseLRShared trains it. Overall accuracy also depends on each agent's own tokens, where AIRMAS uses its independently trained adapters. Structure alone cannot decide this. Measured, the gap splits exactly into the adapter effect (Non-Shared minus Non-Shared with shared-A adapters) and the difference between the two methods' reuse losses (each method's drop from its own Non-Shared). Table D gives this split over five adapter-training seed.
Table D. Split of the accuracy gap AIRMAS − BaseLRShared over five seeds (ScienceQA, pp, mean ± SD).
| Model | Non-Shared | Non-Shared (shared A) | BaseShared | BaseLRShared | AIRMAS | Difference |
|---|---|---|---|---|---|---|
| LLaMA | 71.29 ± 0.54 | 67.94 ± 3.32 | 68.55± 3.21 | 62.83 ± 2.27 | 66.78 ± 0.83 | +3.95 |
| Ministral | 68.34 ± 0.56 | 67.95 ± 0.53 | 59.33 ± 1.78 | 56.78 ± 1.17 | 58.33 ± 1.55 | +1.55 |
Tables 19 and 22 in Appendices E.4 and E.5 of the revised paper report these results.
Both components contribute positively to the measured mean accuracy gap in both models, but their relative contributions differ. The adapter effect accounts for most of the gap on LLaMA, whereas the difference in reuse losses accounts for most of the gap on Ministral. In both cases, AIRMAS incurs a smaller mean accuracy drop from its own Non-Shared reference than BaseLRShared does from its corresponding reference.
On seed 42, BaseLRShared's reuse loss shows up in its planner's decision margin (Table E). Under the same teacher forcing, at the steps where Non-Shared's planner decides to answer, BaseLRShared's planner is less confident in that decision, even though the LoRA output it reuses there is nearly identical to Non-Shared's (cosine 0.99). Non-Shared with shared-A adapters uses the same planner and reuses nothing, so this drop comes from reuse. At the final answer, AIRMAS is also more confident in the answer Non-Shared chose.
Table E. Margin under teacher forcing, defined as the logit of Non-Shared's choice minus the highest other logit.
| Decision | Non-Shared | Non-Shared (shared a) | AIRMAS | BaseLRShared |
|---|---|---|---|---|
| Accuracy (%) | 73.06 | 66.67 | 66.39 | 65.39 |
| Planner's decision margin | 1.91 | 1.91 | 2.03 | 1.77 |
| Non-Shared's final answer | – | - | 3.75 | 3.28 |
Table 24 in Appendix E.5 of the revised paper reports these margins.
The episodes in which BaseLRShared never answers repeat the same search until the 15-step limit. Exact reuse of the LR term therefore does not remove the cost of reusing hidden states computed by other agents.
Response to Reviewer WheU (Part 1)
We sincerely thank you for the constructive feedback. We appreciate your valuable recognition of our clear presentation of the baseline's limitations. We address all the concerns and questions below.
(W1) & (Q1) Equivalence on Reused Prefixes, and Justification of Accuracy and Diversity
(1) The proposition holds on reused prefixes. For any r×r matrix T, ZiTBj=XiAi(TBj). On tokens produced by agent i, AIRMAS is therefore equivalent to an adapter that shares Ai, with Bj′=TBj, and Xi(I−AiAi+)Aj cannot be recovered deterministically. We have added this proposition as Proposition 1 in Appendix F.1 of the revised paper, credited to the reviewer, and we do not claim that AIRMAS is mathematically superior to BaseLRShared [1]. The proposition limits what the transform can recover, but the transform still recovers part of it. On our LLaMA·ScienceQA adapters, T=Ai+Aj is small but not zero (∥T∥F/r = 0.077, root mean square over layers, modules and agent pairs, against 0.044 for random subspaces and 1 for the identity; largest singular value 0.45). On actual inputs, the transformed LoRA output points along a direction that holds 89% of the exact output's energy (cos2=0.89 under teacher forcing, i.e. fed Non-Shared's recorded tokens; Table A), although it is much smaller than the exact output. Appendix F.2 of the revised paper reports these values. Two differences from BaseLRShared remain. On its own tokens, each agent uses its independently trained Aj, and Bj′ is obtained in closed form without retraining. The first difference is visible in measurement (Table A). On the tokens the action and reflect agents compute themselves, AIRMAS's LoRA outputs differ from Non-Shared's by only 15–16%, because both use the same adapters there. BaseLRShared's differ by 64–76%, about as much as those of Non-Shared with shared-A adapters, which reuses nothing. The planner is left out because its adapter is the same file in both adapter families. Appendix G of the revised paper now states that on reused prefixes, specialization is restricted to col(Ai), which answers the first limitation you raise.
Table A compares the LoRA outputs of both methods with Non-Shared's, under teacher forcing on Non-Shared's trajectories. It also includes Non-Shared with shared-A adapters, which differs from Non-Shared only in its adapters and reuses nothing.
Table A. Relative error of the LoRA output against Non-Shared under teacher forcing (cosine similarity in parentheses). All rows except the first are for the action and reflect agents.
Table 23 in Appendix E.5 of the revised paper reports these values.