- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes MERGE, a group-aware evidence routing framework for multi-agent LLM systems. MERGE learns how the marginal benefit of sharing evidence changes with the number of agents already holding it, using paired interventions to estimate its effect on final team performance. The learned group effect is combined with agent-specific evidence utility and communication cost through a joint routing procedure. Experiments across multi-hop QA benchmarks, multiple LLM backbones, larger teams, and fact verification demonstrate consistent improvements in answer quality and communication efficiency over existing and controlled baselines.
Critical strengths
[#1] Well-motivated formulation of group-aware evidence routing.
The paper identifies an important limitation of agent-specific evidence routing: the value of sharing information with an additional agent depends not only on its individual usefulness but also on how widely that information has already been distributed. MERGE provides a principled framework for incorporating holder-dependent sharing effects into routing decisions. The paired-intervention approach is particularly interesting because it derives supervision from downstream team outcomes rather than relying solely on heuristic relevance scores.
[#2] Comprehensive empirical evaluation with meaningful controlled comparisons.
The paper provides extensive experiments across multiple benchmarks, LLM backbones, and team sizes, including transfer to fact verification. Importantly, the authors conduct controlled comparisons against Separable and Fixed-Shape variants, investigate the contributions of group-effect modeling and joint routing, and provide predictor diagnostics and sensitivity analyses. These experiments offer substantial evidence that the proposed components improve routing performance within the evaluated setting.
Critical weaknesses
[#1] Limited evidence that the learned group effect generalizes beyond the fixed aggregation mechanism.
The paper's central contribution is to model how the marginal value of sharing evidence changes with the number of existing holders. However, the supervision signal is derived from final team-level F1 under a fixed majority-based aggregation rule. Consequently, the learned group effect may reflect not only the informational value of evidence dissemination but also whether additional recipients can change the final voting outcome.
The authors partially investigate this distinction in Section 6.4 and Appendix B.12. Notably, the early-stage diminishing-return pattern persists under individual-agent F1, whereas the sharp late-stage increase largely disappears. This suggests that an important component of the learned sharing dynamics is specific to the aggregation mechanism.
Although optimizing majority-vote outcomes is legitimate, all main transfer experiments retain the same aggregation rule. Therefore, the results do not establish whether MERGE learns generalizable information-sharing dynamics or primarily exploits the structure of the chosen collective decision procedure. Evaluating routing performance under alternative aggregation mechanisms would help clarify the scope of the proposed contribution.
[#2] Insufficient validation of single-recipient marginal estimates for joint routing decisions.
MERGE learns sharing benefits from paired interventions that differ in whether one additional agent receives a selected piece of evidence. However, its inference-time procedure jointly optimizes multiple routing decisions, potentially involving different evidence items and recipients. The effects of these decisions may not be additive, as sharing one piece of evidence can change the usefulness of other evidence through complementary reasoning, redundancy, or subsequent search behavior.
The authors provide useful ablations demonstrating that joint routing and group-effect modeling are complementary, including a factorial analysis in Appendix B.9. Nevertheless, these results establish the effectiveness of the combined procedure rather than directly validating whether the estimated routing objective accurately represents the downstream effects of jointly selected actions.
This distinction is important because the proposed predictor omits cross-cluster interactions, and its supervision is based on single-recipient interventions under a fixed continuation policy. Consequently, the extent to which these estimates remain reliable for compound routing actions and states induced by the learned policy remains unclear. A direct comparison between predicted joint-routing benefits and observed outcomes under matched multi-decision interventions would strengthen the methodological justification.
Questions for authors
Q1. Distinguishing information-sharing value from aggregation-specific effects
The paper's central motivation is that the marginal value of sharing evidence depends on how many agents already possess that information. However, the intervention-based supervision measures changes in final team F1 under a fixed majority-voting mechanism. As the authors acknowledge in Section 6.4 and Appendix B.12, the sharp late-stage increase in marginal sharing value becomes substantially weaker when evaluated using individual-agent F1 instead of the final aggregated outcome.
This raises an important question about what the group-effect predictor is actually learning. Is it primarily estimating how additional evidence improves the team's collective information state, or is it learning when additional evidence holders are likely to change the majority-vote outcome?
Could the authors further disentangle these two effects? In particular:
- Would the learned group-effect predictor retain its advantage under alternative aggregation rules, such as confidence-weighted voting or evidence-based answer selection?
- If the aggregation rule changes, does the existing predictor remain effective without retraining, or must the group-effect model be refitted?
- Can the authors quantify how much of MERGE's improvement comes from better individual-agent reasoning versus changes in the final collective decision?
Addressing these questions would clarify whether MERGE captures a broadly applicable principle of group-aware evidence sharing or a more specialized, albeit useful, strategy for optimizing majority-based multi-agent systems.
Q2. Validity of marginal estimates for compound routing actions
MERGE learns the marginal benefit of sharing evidence through paired interventions that differ in whether one additional agent receives a selected piece of evidence. However, the proposed joint routing procedure simultaneously considers multiple sharing decisions, whose effects may interact.
For example, two evidence items may be individually insufficient but jointly necessary to complete a multi-hop reasoning chain. Conversely, two individually useful items may become redundant when delivered together. In such cases, the actual benefit of jointly sharing both items may differ substantially from the benefit implied by combining their independently estimated marginal effects.
The factorial analysis in Appendix B.9 demonstrates that group-effect modeling and joint routing are complementary, but it does not directly establish whether the estimated routing objective accurately predicts the outcomes of compound actions.
Could the authors provide a more direct validation of this approximation? Specifically:
- For a representative set of intermediate team states, how closely does the predicted utility of a jointly selected routing action correspond to its observed downstream improvement?
- How frequently do complementary or redundant interactions cause the combined outcome to differ substantially from the prediction based on individual marginal effects?
- Would explicitly modeling pairwise interactions between evidence-sharing decisions improve routing quality, or does the current approximation already capture most of the relevant dependencies?
These analyses would help establish whether the proposed objective is a reliable approximation to the joint routing problem, rather than merely an effective heuristic within the evaluated environment.
Q3. Reliability of intervention-based estimates under the learned routing policy
The group-effect predictor is trained using paired continuations under a fixed continuation policy, while MERGE subsequently makes adaptive routing decisions that can alter agents' future queries, retrieved evidence, and information-sharing opportunities. The paper acknowledges this discrepancy in Section 7 and provides a continuation-policy sensitivity analysis in Appendix B.8.
Nevertheless, the existing analysis does not fully establish whether the learned predictor remains calibrated on intermediate states generated by MERGE itself. This matters because an evidence-sharing decision may have a different downstream effect when subsequent communication is determined adaptively rather than replayed according to a fixed schedule.
Could the authors investigate this distributional discrepancy more directly? For example:
- How do predicted marginal sharing benefits compare with observed intervention outcomes when the intermediate states are sampled from actual MERGE trajectories?
- Does predictor calibration deteriorate as the learned router progresses through multiple search rounds?
- How sensitive is routing performance to the difference between the fixed continuation policy used for training and the adaptive continuation induced by MERGE?
- Would collecting additional intervention data from on-policy trajectories or iteratively updating the predictor improve performance?
Such experiments would strengthen the connection between the controlled intervention-based training objective and the actual sequential decision-making process at inference time.
Dear Reviewer LZzv,
Thank you again for your review. As the discussion period ends on Nov 18 (AoE), we wanted to check whether our response addressed your concerns. In brief:
Lead-agent setting: in a manager pilot (150 held-out questions), Manager + MERGE beats a local-utility relay by +1.2 F1 [+0.3, +2.1] at the same budget, and matches broadcast quality with 35% fewer tokens.
Complex tool tasks: on all 165 GAIA validation tasks with the frozen router, MERGE solves 64 vs. 59 for FIXED-SHAPE with 8.9% fewer tokens. The CI includes zero, so we report this as preliminary.
Newer models: the frozen router gives positive gains on Qwen3.6-27B, GPT-5.6 Sol and Claude Opus 5.
Gain reporting: we will replace relative numbers in Table 1 with absolute differences.
If these address your concerns, we would be grateful if you would consider updating your assessment. We are happy to clarify anything before the deadline.