acceptodds
Under review as a conference paper at ICLR 2027

LAM-RAG: LoRA Messages for Multi-Document Retrieval-Augmented Generation

Abstract

Multi-agent debate is built to reach consensus. Multi-document RAG often needs the opposite: when evidence is ambiguous or conflicting, different documents support different correct answers, all of which should be kept. Transcript-based debate works against this goal: from the same local answers, transcript exchange lowers answer-set F1 below the no-exchange level as agents copy their peers' answers and supported alternatives disappear. We propose LAM-RAG (LoRA-as-Message RAG), which keeps local answers and final aggregation in text but carries inter-agent influence as transient LoRA messages. A frozen translator maps each answer state to low-rank factors that a reliability-aware controller prunes and scales. Because a LoRA message exposes no copyable peer answers, it couples agents weakly: peer-answer adoption drops sharply and distinct supported answers survive. Yet the messages still carry useful information: on Qwen2.5-1.5B, one round of exchange raises F1 by 1.5–2.5 points over no exchange, and zero, random, or cross-question messages lower it by 1.3–1.9. On RAMDocs, AmbigDocs, and FaithEval with three backbones, LAM-RAG beats the strongest text-exchange or single-pass baseline by 2.86–5.20 average F1. Under added distractors or peers, LoRA messages degrade less than peer text.

Primary area:
foundation or frontier models, including LLMs
Keywords:
retrieval-augmented generation, multi-agent systems, agent communication, LoRA, multi-document reasoning, conflicting evidence

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CYGrOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes LAM-RAG, a multi-agent framework for multi-document, multi-answer QA. Document-specific agents communicate through LoRA-based messages, with reliability-based filtering and scaling, before final answer aggregation. The method aims to reduce conformity during inter-agent communication while preserving complementary correct answers.

Critical strengths

The paper identifies an important failure mode of multi-LLM debate in multi-answer QA: agents may discard correct, locally supported answers after adopting their peers’ responses.

The experimental evaluation is extensive, covering multiple datasets and model backbones, with a broad range of baselines, ablations, and analyses of agent behavior.

Critical weaknesses

The paper does not adequately explain why low-rank parameter updates should specifically address conformity. The intuition connecting this representation to the preservation of independent judgments, while still allowing useful information exchange, remains underdeveloped.

The reduced loss of correct answers may reflect weaker peer influence under a restricted communication channel. Although improvements over no exchange demonstrate the value of sharing complementary information, they do not establish why LoRA is preferable to other forms of partial interaction. The current experiments do not sufficiently distinguish the proposed mechanism from this alternative explanation. Comparisons that control the amount or strength of communication would help justify the design choice.

Table 2 shows modest, setting-dependent gains over the no-exchange setting, which already aggregates independent local answers. This raises the question of whether similar performance could be achieved through simpler procedures, such as retaining the union of independently supported answer candidates or having a central agent verify independently generated document summaries and proposed answers. More direct comparisons are needed to establish the value of mutual exchange over these alternatives.

Questions for authors

What makes LoRA particularly suitable for reducing conformity beyond limiting peer influence? Can the authors provide a clearer explanation and controlled comparisons with alternative forms of partial interaction?

How does LAM-RAG compare with preserving the union of initial answer candidates or centrally verifying document-level summaries and answers, under comparable evidence and computational budgets?

Reviewer_rypQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that transcript-based debate is a bad fit for multi-document RAG when the right output is a set of answers: agents copy each other's answers, and correct alternatives get lost. LAM-RAG keeps one agent per document and keeps answers and final aggregation in text. Between rounds, agents exchange LoRA factors instead of transcripts. A frozen hypernetwork maps each agent's answer and rationale to rank-2 factors, which are averaged in factor space and injected into every agent for the next round. Which messages survive and how much each counts is set by a controller. It scores each message by answer confidence weighted by whether the answer appears verbatim in the agent's own document, then applies MAD pruning and sigmoid scaling. On RAMDocs, AmbigDocs, and FaithEval with three backbones up to 8B, LAM-RAG beats the best text-exchange or single-pass baseline.

Critical strengths
  • The conformity result is the strongest part of the paper. Starting from the same no-exchange point, MADAM-RAG and LLM-Debate lose F1 with every extra round on all three benchmarks and on two backbones. Table 9 ties this drop to agents adopting peer answers rather than to noise. That is worth knowing for anyone using debate on ambiguous or conflicting QA.

  • The controls are also more thorough than usual, paired CIs on the content-free message controls, a translator-input ablation, an honest null result on FactorMean versus UpdateMean, and a check that factorized injection gives identical outputs. The limitations section is upfront that the content gain is small and that fact transfer isn't shown.

Critical weaknesses

The main comparison leaves out the strongest baseline, which is no exchange at all. Table 1 compares against text debate at T=3T=3, where the paper itself shows debate does worst. Against MADAM-RAG at T=1T=1, LAM-RAG's margin is much smaller: roughly +1.8/+1.6/+3.6 on Qwen-1.5B and +2.2/+0.8/+1.9 on Qwen-7B. On 7B FaithEval the exchange actually hurts (−0.38 against LAM-RAG's own T=1T=1), so that gain comes from pruning, not communication. On 1.5B RAMDocs, "LoRA msg. only" (58.14) matches text no-exchange (58.10). Much of the headline gap is the cost of debate rather than a benefit of LoRA messages, and the practical lesson could just be "don't let the agents talk, and filter by lexical support."

"Weak coupling" isn't separated from "the channel carries very little." The messages are rank-2 compressions of a final hidden state, every agent gets the same pooled update including its own message, and the evidence for content is a drop of under 2 F1 when it is removed. Low peer adoption is also what you'd expect from a nearly empty channel. To support the carrier claim, the paper needs a text baseline with similarly weak coupling, for example sharing rationales with the answers redacted, or telling agents to keep their answer unless their document contradicts it.

Questions for authors
  1. Can you report no-exchange baselines, with and without the same substring/MAD filter, for all three backbones, along with LAM-RAG's paired margin over them? This is the main thing that would change my score.
  2. Is the random-message control norm-matched to the real messages? If not, the drop may reflect perturbation size rather than missing content.
  3. When an agent produces a new gold answer in round two (like in Table 15), how often is it in its own document versus only in a peer's? This would show whether anything is actually being transferred.
  4. How does the canonical system do with peers-minus-self pooling, so agents don't receive their own message back?
Reviewer_dc9JOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a failure mode of text-based multi-agent debate in multi-document retrieval-augmented generation, where exchanging explicit answers can cause agents to converge prematurely and discard distinct, evidence-supported alternatives. To address this issue, the authors propose LAM-RAG, which replaces intermediate textual communication with transient LoRA messages generated by a frozen Text2LoRA translator. A reliability-aware controller prunes and scales the messages, while local answer generation and final answer-set aggregation remain textual. Experiments on RAMDocs, AmbigDocs, and FaithEval show improvements over conventional text-exchange baselines, with additional analyses of answer conformity, message content, robustness, and computational cost.

Critical strengths

[#1]. The paper identifies an interesting limitation of multi-agent debate in multi-document QA.

The observation that consensus can be harmful when multiple answers are supported by different documents is well motivated. I also find the decision to investigate the communication channel, rather than introduce another debate or voting strategy, interesting. The separation between textual answer generation and parameter-space communication gives the method a clear conceptual direction.

[#2]. The experimental analysis is relatively thorough and supports the main observation.

The paper evaluates three backbones and three benchmarks, and includes several useful diagnostics beyond final accuracy. In particular, the communication-depth results in Table 2 and the receiver analysis in Appendix D.3 show how textual communication increases peer-answer adoption while reducing the retention of supported answers. The zero, random, and cross-question message controls in Appendix D.2 also provide evidence that the LoRA messages contain a useful question-conditioned signal. I appreciate that the authors report limitations and exceptions to their findings, including the stronger LatentMAS result on Llama-3.1-8B.

Critical weaknesses

[#1]. The main performance improvement appears to come primarily from avoiding harmful text exchange, rather than from substantial information transfer between agents.

This is my main concern. Table 2 shows that conventional text-based debate often performs worse than independent answer generation. LAM-RAG avoids much of this degradation, which is a useful result. However, the improvement from introducing LoRA messages over the zero-message condition is only around 1.77–2.29 F1 points in the reported Qwen2.5-1.5B depth experiments (Section 4.3).

The message-content controls in Appendix D.2 show that the messages are not arbitrary, but they do not directly demonstrate that agents successfully acquire complementary information from other documents. For example, it remains unclear how often a receiver produces a previously missing correct answer specifically because another agent communicated relevant evidence through a LoRA message.

This distinction matters because preserving independent answers and transferring useful information are different capabilities. The current experiments provide strong evidence for the former, but more limited evidence for the latter. The paper itself acknowledges in Section 5 that faithful transfer of individual peer facts has not been demonstrated.

[#2]. The contribution of the proposed message-composition and control mechanisms is not sufficiently established.

LAM-RAG combines several components: a pretrained Text2LoRA translator, FactorMean composition, reliability-based pruning, and soft scaling. However, the experimental evidence does not clearly establish why this particular combination is necessary.

For example, Table 2(b) shows that the benefit of individual control components is not consistent across backbones. On Qwen2.5-7B, adding MAD pruning or scaling individually produces slightly lower F1 than using LoRA messages alone, while the full method provides only a 0.29-point improvement. Appendix F.2 further reports that enabling the controller does not improve task-level robustness in the fixed-round source-matched experiment.

I also have concerns about FactorMean. As the authors explain in Section 3.2, averaging the LoRA factors separately introduces cross-message products between factors from different agents. These interactions are not equivalent to averaging the original parameter updates. Although Appendix D.4 compares FactorMean with UpdateMean, the results do not establish a consistent advantage for the proposed operator, particularly in the multi-answer setting that motivates the paper.

Consequently, it is difficult to determine whether the specific factor-space composition and reliability-control design are essential to the method's success, or whether a simpler parameter-space communication mechanism could achieve similar results.

Questions for authors

Q1. To what extent does the proposed theory capture the informational value of communication, rather than primarily modeling its token consumption?

The central argument of this paper is that reading additional peer responses reduces the generation budget available for independent reasoning, potentially making larger multi-agent groups less effective. While this trade-off is well motivated, the theoretical formulation appears to characterize communication mainly through its token cost. In particular, the per-agent ability function is calibrated using independent single-agent reasoning under different generation budgets, whereas the effects of receiving peer information are not explicitly incorporated into this function.

This raises several important questions:

  • Communication quality versus communication quantity: How would the predicted optimal group size change if peer messages differed substantially in their informational usefulness while consuming the same number of tokens? For instance, a short message identifying a specific reasoning error may be more valuable than a longer message repeating an existing solution. Can the current framework distinguish these cases, or would they be treated equivalently when their token costs are matched?

  • Communication-induced improvements in reasoning: The paper assumes that individual reasoning ability can be estimated from single-agent generation-cap measurements. However, an agent receiving useful feedback may achieve higher accuracy with fewer reasoning tokens than an agent reasoning independently. Could the authors extend the ability function to depend on both the remaining generation budget and the information received from peers? This would clarify whether the predicted preference for independent reasoning remains valid when communication improves the effectiveness of subsequent reasoning.

  • Interpretation of the existing communication analysis: Appendix D.2 examines changes in the accuracy ceiling after reading peer responses and reports small, mixed effects across four model-task settings. However, does this finding indicate that communication has intrinsically limited informational value, or could it reflect the effectiveness of the particular debate protocol? For example, would the conclusion remain unchanged if agents exchanged targeted corrections, complementary evidence, or concise critiques instead of conventional peer responses?

  • Conditions under which communication becomes beneficial: The theory identifies conditions under which the cost of reading additional messages outweighs the benefits of adding agents. Can the authors also characterize conditions under which sufficiently informative communication reverses this conclusion? In particular, is there a measurable threshold of communication-induced accuracy improvement beyond which reading more becomes preferable to allocating the same tokens to independent reasoning?

I believe this distinction is important for interpreting the paper's central contribution. The current results convincingly demonstrate that communication can be expensive under a shared token budget. However, establishing that independent reasoning is generally preferable requires separating the cost of receiving information from the value of the information received. A more explicit treatment of this distinction would substantially strengthen the theoretical contribution and clarify the scope of the conclusions.

Q2. How accurately does the proposed theoretical model describe actual multi-agent interactions, and under which communication protocols does its optimal group-size prediction remain valid?

The empirical evaluation demonstrates that TB-MAD achieves competitive regret when selecting group sizes, particularly under limited calibration budgets. This is a meaningful practical result. However, successfully selecting a near-optimal configuration does not necessarily imply that the theoretical model accurately captures the mechanisms determining collective reasoning performance.

The proposed framework combines single-agent ability estimates with a correlated-voting model to predict group accuracy. In actual multi-agent debate, however, agents do not simply produce independent predictions followed by aggregation. Their intermediate reasoning processes may be influenced by peer responses, and this interaction can change both individual correctness and the correlation structure of their errors.

I would appreciate clarification on the following points:

  • Predicted versus observed group-accuracy curves: Beyond reporting regret for the selected group size, could the authors directly compare the theoretically predicted accuracy with the empirically observed accuracy across different group sizes and context windows? For example, plotting predicted and observed group-accuracy curves would reveal whether the model captures the overall performance landscape or merely identifies a competitive configuration. It would also be useful to report prediction errors and examine whether they systematically increase with the number of agents or the amount of communication.

  • Validity of single-agent calibration after interaction: The individual-agent ability function is fitted using single-agent measurements without debate. However, receiving peer responses may alter an agent's probability of producing a correct answer even when its remaining generation budget is held fixed. Have the authors examined whether the single-agent ability estimates remain calibrated for agents after communication? If substantial discrepancies arise, how sensitive is the predicted optimal group size to these deviations?

  • Changes in error correlation during debate: Although the voting model permits correlated errors, actual communication may dynamically increase or decrease correlations among agents. For example, agents may converge toward a shared incorrect solution, or they may correct complementary mistakes through interaction. Does the proposed correlation model adequately capture these communication-induced changes? Could the authors compare the assumed correlation structure with empirical pairwise error correlations measured before and after debate?

  • Dependence on the shared-window assumption: The formulation assumes that reading additional peer responses directly reduces the generation budget available for reasoning. This is a reasonable setting, and the existing ablations on reading cost and token pricing are useful. Nevertheless, many practical multi-agent systems use selective communication, message compression, or separate resource allocations for reading and generation. How robust is the predicted optimal group size when these assumptions are relaxed? In particular, would the interior optimum and the observed performance collapse persist if agents could exchange short, highly informative messages without proportionally reducing their reasoning budgets?

  • Distinguishing model accuracy from decision robustness: Could the authors investigate cases where the theory substantially mispredicts group accuracy but still selects the correct group size, as well as cases where relatively small prediction errors lead to an incorrect selection? Such an analysis would help distinguish the accuracy of the theoretical characterization from the robustness of the resulting configuration-selection procedure.

These questions are particularly relevant because the paper presents TB-MAD as a theoretical explanation of the relationship between reasoning, communication, and collective accuracy, rather than solely as a practical group-size selection method. The existing experiments provide strong evidence that the proposed predictor is useful under the evaluated settings. However, more direct validation of its intermediate assumptions and predicted accuracy curves would be necessary to establish that the theory faithfully explains the behavior of interacting agents.

Clarifying these issues would also help identify whether the reported preference for independent reasoning reflects a general property of multi-agent reasoning or a consequence of the specific communication and resource-allocation protocols considered in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.