acceptodds
Under review as a conference paper at ICLR 2027

MAMR-GRPO: Reinforcement Learning with Execution-Structured Credit for Multi-Agent LLMs

Abstract

LLM-based multi-agent systems (MAS) extend the capabilities of language models by decomposing complex tasks into specialized roles, enabling parallel reasoning, information gathering, and iterative refinement. In such systems, the final answer is produced collaboratively by multiple agents or roles whose outputs interact and influence subsequent executions. However, prevailing reinforcement learning methods such as Group Relative Policy Optimization (GRPO) typically assign credit at the rollout level, making it difficult to distinguish the contributions of different executions within a multi-agent trajectory. We propose Multi-Agent Multi-Round GRPO (MAMR-GRPO), which retains rollout-level comparisons while introducing local credit assignment aligned with the execution structure of MAS. Specifically, round boundaries, role dependencies, and parallel outputs provide references for comparing task assignment, subtask execution, and information aggregation. These local credits are combined with the terminal rollout advantage and applied to the corresponding generated segments of a shared LLM policy. Across seven search-augmented QA benchmarks, MAMR-GRPO reaches 46.7 average exact match, 3.1 points above trajectory-level GRPO on the same graph. Budget ablations show further gains with additional subagents and rounds.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Agent LLMs, Credit Assignment, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_h9cXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a multi-round LLM-based multi-agent framework with meta, subagent, and controller roles, and introduces MAMR-GRPO, which combines trajectory-level feedback with role-specific credits based on round values and counterfactual output deletion.

Critical strengths

None

Critical weaknesses

The architectural contribution is insufficiently justified: The paper presents the meta, subagent, and controller structure as a contribution, but similar planning, execution, and coordination structures already exist. The related work does not sufficiently clarify this distinction. More importantly, the paper provides limited intuition or theoretical justification for why this role separation is beneficial, and the experiments do not isolate its effect. Comparing the complete frameworks does not establish that the proposed three-role design explains the improvement.

The proposed credit design lacks sufficient supporting evidence: The paper introduces several credit definitions, but does not adequately justify these choices or demonstrate their benefits. Joint deletion in Eq. 9 is one example: its motivation is described, but the claimed benefit is not sufficiently validated. The concern is not simply that one particular comparison is missing. Overall, the paper proposes several design choices without enough analysis to support them. Given this complexity, the improvements in Table 1 are not sufficiently compelling to establish the significance of the work. I am not convinced that the contribution and supporting evidence are sufficient for publication at ICLR.

Questions for authors

What evidence supports the benefits of the proposed role separation and credit design, beyond the performance of the complete system?

2 comments
AuthorsOpenReview ↗

Response to Reviewer h9cX (1/2)

We sincerely thank the reviewer for the time and effort devoted to reviewing our paper, for examining the role separation and the credit design, and for asking what evidence supports them beyond the performance of the complete system. This is a fair and important question, and we appreciate the opportunity to set out the evidence clearly.

The paper contains three kinds of such evidence: a training comparison with the execution graph fixed, alternative worker-credit constructions within the same execution setting, and collaboration-budget comparisons within a fixed trained model. We clarify below what each establishes.

1. Role separation: the main gain is measured with the execution graph fixed (Weakness 1)

What we claim as the contribution. Our methodological contribution is the definition of execution-structured credit: which quantity is evaluated, which reference it is compared against, and which generated segment receives the resulting signal. The revised contribution list foregrounds these definitions and identifies the framework as their execution setting.

Same-graph training comparison. Table 1 compares MAMR-Framework + GRPO (43.6 average EM) with MAMR-GRPO (46.7) on the same execution graph and Qwen3-4B backbone; both are trained on the same data for the same 780 optimizer steps. The 3.1-point improvement therefore compares training methods while the role structure is held fixed; it is not obtained by switching between frameworks. The concern that a comparison of complete frameworks cannot explain the improvement does not apply to this result.

Execution setting before RL. Separately, the inference-only comparison uses the same backbone before any RL: MAMR-Framework attains 37.2 average EM, compared with 28.8 for SingleAgent and 32.2 for MATPO-Framework. This measures the execution setting independently of RL training.

2. Credit design: three credits defined through one value quantity (Weakness 2)

The common value. The common quantity V is the expected task reward of answers sampled from the current policy after a virtual forced stop at a collaboration boundary. The three credits use this quantity with references matched to the flow of information:

  • Assignment: V_t - V_(t-1) measures the progress made during the current round. Subtracting the previous value separates new progress from evidence that was already available; the first-round reference is zero.
  • Aggregation: with gamma = 1, the credit is V_T - V_t for a continuation and zero for a stop. A summary is consumed by later rounds, so it is credited with the progress made after it.
  • Worker execution: the task descriptions and outputs of selected workers are removed, the remaining records are retained, and the Controller prefix and forced-stop answers are regenerated. The resulting value change is split equally among the removed workers and averaged over the sampled deletion sets that contain each worker.

Joint deletion (Eq. 9). Joint deletion addresses a concrete redundancy case. If two of three workers provide interchangeable evidence, removing either alone can leave V unchanged, whereas removing both lowers it; sampling joint deletions exposes this set contribution. Appendix D.4 specifies the budgeted procedure, which includes individual deletions and samples additional proper subsets.

Terminal advantage and coefficients. The terminal group advantage is retained. The terminal mixing coefficient and the three role-credit coefficients are all set to one. Value estimation uses task-answer rewards and requires neither a learned critic nor an LLM judge.

AuthorsOpenReview ↗

Response to Reviewer h9cX (2/2)

3. Evidence beyond the complete system (Question)

(a) Alternative worker-credit constructions. Appendix E.3 replaces the worker-credit construction within the shared execution setting:

  • C3 SubMessage Lite, a C3-style leave-one-out credit over alternative worker messages, evaluated through the downstream Controller-to-answer path: 39.7 average EM.
  • SHARP-MC, a Monte Carlo approximation to Shapley-style marginal contributions over worker permutations: 40.9 average EM.

MAMR-GRPO attains 46.7, exceeding these variants by 7.0 and 5.8 points, and both variants score below the same-framework GRPO baseline (43.6). The tested alternative worker-credit constructions do not reproduce the gain of MAMR-GRPO. This compares credit constructions within one execution setting, which is precisely evidence beyond a comparison of complete systems.

(b) Breadth of the improvement. Relative to same-framework GRPO, Table 1 reports gains on 2Wiki (+6.6), Bamboogle (+5.6), HotpotQA (+3.4), MuSiQue (+4.7), PopQA (+1.2), and TriviaQA (+0.7); NQ changes by -0.6. Bamboogle, PopQA, and TriviaQA are not part of the four-dataset RL training mixture.

(c) Collaboration within a fixed trained model. The inference-budget studies vary the execution limits while keeping the trained checkpoint fixed. Increasing the worker cap from two to four at a round cap of three raises average EM from 45.43 to 46.71; increasing the round cap from one to two at a worker cap of four raises it from 44.07 to 46.21. These comparisons show that parallel and multi-round execution are used productively by the trained system.

We thank the reviewer once again for the careful review and for encouraging further component-level comparisons. The reviewer's comments prompted us to state the contribution of the paper more precisely, and we are grateful for this. We sincerely apologize that, owing to limited GPU resources, we were unable to run additional training ablations during the rebuttal period. Such ablations would add a finer-grained view, while the evidence above, namely the fixed-graph comparison and the alternative credit constructions in Appendix E.3, already supports the role-structured credit design beyond the performance of the complete system.

Reviewer_7d9hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • MAMR-Framework is a multi-round multi-agent system where three roles share one LLM. It has a Meta agent that assigns 2–4 parallel subtasks, Sub agents that search and answer their subtask, and a Controller that writes a summary and decides continue or stop. Max 3 rounds x 4 subagents in training.
  • MAMR-GRPO keeps a trajectory-level leave-one-out advantage and adds role-specific local credits built from counterfactual "round values".
  • Results on 7 search-QA benchmarks: 46.7 average EM vs 43.6 for GRPO on the same graph, above a reproduced Search-R1 (43.3), WideSeek-R1-4B (40.6), and in-framework C3 / SHARP variants. Inference budget sweeps (S = 2..6, R = 1..5), a step-50 vs step-600 behavior comparison, a case study, and the author also provided anonymized code.
Critical strengths
  • The credit idea is concrete and well motivated. Local credits are defined at the natural boundaries of the execution graph using only the task reward and no LLM judge. Joint deletion to handle redundant subagents is a sensible fix for a real weakness of leave-one-out / Shapley-style credit, and the paper explains clearly how it differs from SHARP, C3, GiGPO, TRACE and M-GRPO.
  • The main comparison is controlled: same execution graph, same backbone, same data, same shaping and boundary supervision for GRPO and MAMR-GRPO, so the +3.1 EM can be attributed to the credit design. Reproducing Search-R1, C3 and SHARP on the same 4B backbone is also the right instinct, even if the reproductions are partial.
  • The supplementary package is better than usual: anonymized code with tests, the resolved training config, and a stdlib script that checks the case-study facts (it passes). Section 5.5 is honest that the behavior statistics do not isolate which signal causes what.
Critical weaknesses
  • No component ablations and no variance. The method adds three local credits, boundary supervision, anti-redundancy shaping, joint deletion, and about ten hyper-parms, but none of them is ablated. All numbers are one training run and one evaluation sample.
  • The compute cost of the method is never reported. Training took about 7 days on 8x A100 for a 4B model. GRPO's cost on the same graph is not given, and there is no compute-matched baseline.
  • Scope and selection concerns. One task family (search QA), one backbone (4B), one retrieval setup, while recently accepted MAS-RL papers (e.g., AT-GRPO / STRONGER-MAS at ICLR 2026) show several domains and two model sizes. And the baseline called "GRPO" is actually an unnormalized leave-one-out advantage without std normalization, so the name is a bit misleading.
  • Related work misses the closest alternatives for local credit in MAS and does not compare against them: agent- and turn-wise grouping with tree sampling (AT-GRPO, ICLR 2026), agent-wise reward normalization (Dr. MAS), heterogeneous-group optimization (MHGPO), and MARTI.
Questions for authors
  • Please report generated tokens and wall-clock per training step for GRPO vs MAMR-GRPO on the same graph, and add a compute-matched GRPO run (more rollouts per question, or more steps). Does the +3.1 EM survive?
  • Can you ablate at least: (a) Meta credit only, (b) Sub credit only, with individual deletion only vs individual + joint, (c) Controller credit only, (d) no boundary supervision? And can you give 2–3 training seeds for the main row, or at least evaluation variance over sampling seeds for Table 1 and the budget sweeps?
  • How were checkpoints selected for both methods? If selection used 20% of the evaluation collection, was the same procedure applied to GRPO, and do the final scores change on the remaining 80%?
  • Why is AT-GRPO-style agent- and turn-wise grouping (which gives a local comparison group at every agent-turn) not a baseline? It targets the same problem, a rollout-level advantage shared by all segments, and it is already published. Either a comparison or a clear argument why it does not apply to your graph would help.
  • The gains are concentrated on multi-hop sets and the general-QA gain is 0.4 EM. Is there a setting where MAMR-GRPO hurts, and do you get the same ranking when final answers are decoded at temperature 0 instead of 0.7?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 7d9h (1/3)

We sincerely thank the reviewer for the detailed and constructive review. We are grateful that the reviewer found the credit idea concrete and well motivated, the joint deletion a sensible fix for a real weakness of leave-one-out and Shapley-style credit, the main comparison controlled, and our supplementary package better than usual, including the anonymized code with tests, the resolved training configuration, and the stdlib script that verifies the case-study facts. We respond to each point below.

1. Training cost and a compute-matched GRPO baseline (Weakness 2, Question 1)

Matched quantities. The two methods are matched in training examples and optimizer steps: both are trained for one epoch on the same data with the same batch configuration.

Per-step cost. According to the training logs, over the same 780 steps on the same 8x A100 80GB node:

  • Wall-clock time per step: 1,082 s for MAMR-GRPO versus 689 s for GRPO (approximately 1.57x; 234 versus 149 hours of cumulative step time).
  • Rollout generation per step, including counterfactual branches: 454 s versus 210 s.
  • Tokens in the training batch (prompt plus response tokens of trained sequences): 19.4M versus 14.5M per step, of which 7.2M versus 7.1M are response-side tokens (model generations plus retrieval results inserted between Sub-agent turns); the logs do not separate the two.
  • Credit-only generation per step (estimate): the trained rollouts of the two methods are of similar size (above), and MAMR-GRPO additionally generates credit-only outputs, which share the format of the realized ones and involve no retrieval calls: on average 8,346 deletion branches per step, each with a Controller summary and four answers (about 286 and 10 tokens), i.e. about 2.7M tokens, plus under 0.1M for forced-stop answers at nonterminal boundaries.

Total training time. Section 5.1 now reports the full training times: about 9.8 days for MAMR-GRPO and about 6.2 days for GRPO.

Compute-matched baseline. We sincerely apologize that we did not have sufficient GPUs to train a compute-matched GRPO baseline (with more rollouts or more steps) during the rebuttal period; the 3.1-point difference is therefore established under equal data and optimizer steps rather than equal compute.

2. Component ablations and variance (Weakness 1, Question 2)

Existing comparisons. Two aspects of the design have already been compared against alternatives. The same-graph GRPO row removes all local credits jointly, and Appendix E.3 replaces our worker credit with C3 SubMessage Lite or SHARP-MC on the same pipeline (39.7 and 40.9 versus 46.7).

Requested ablations and seeds. The component-wise ablations proposed by the reviewer (per-role credit only, singleton vs. joint deletion, no boundary supervision) would further disentangle the individual terms. We appreciate these suggestions, and we sincerely apologize that we did not have sufficient GPUs to train these variants or additional seeds during the rebuttal period.

Variance. The main row corresponds to a single training run (seed 42) with a single evaluation sample, and Appendix E.4 uses one sampling run per budget setting.

3. Checkpoint selection (Question 3)

Same procedure for every method. The same procedure was applied to every locally trained method, including GRPO and the reproduced Search-R1, C3 SubMessage Lite, and SHARP-MC. Checkpoints taken at 50-step intervals (16 per run) are ranked by average EM on the 20% subset (Appendix E.1), and the top-ranked candidates are evaluated on the full collection.

Results on the remaining 80%. On the 80% of questions not used for screening (n=41,293), average EM is 46.0 versus 43.8. MAMR-GRPO thus retains its advantage over GRPO on the remaining 80%.

AuthorsOpenReview ↗

Response to Reviewer 7d9h (2/3)

4. AT-GRPO and related work (Weakness 4, Question 4)

How AT-GRPO differs. AT-GRPO and our method address complementary aspects of the problem. AT-GRPO groups candidates sampled from a common observation by agent and turn, so that group-relative comparisons are made among candidates with identical prompts; its rewards combine a team reward with agent-specific local rewards defined per task (e.g., unit-test pass rates for a coder or tester), and its outcome-only variant broadcasts terminal success together with formatting checks. Our method instead defines the local credit itself, based on counterfactual changes in round value under forced stopping and joint deletion, without requiring task-specific local rewards.

Why it is not a baseline here. The two approaches are compatible: our credits could serve as the per-agent signal within agent- and turn-wise groups. Moreover, AT-GRPO branches candidates at every agent-turn, whereas our counterfactuals are evaluated within a single realized collaboration. We did not evaluate such a combination and make no empirical claim relative to AT-GRPO; Appendix E.3 includes C3 SubMessage Lite and SHARP-MC as in-framework alternative credit constructions (39.7 and 40.9 average EM vs. 46.7).

Dr. MAS, MHGPO, and MARTI. Dr. MAS normalizes each agent's advantages using reward statistics computed over that agent's own active steps, thereby stabilizing the scale of the shared outcome reward across roles. MHGPO is critic-free, propagates terminal rewards backward to upstream agents, and forms comparison groups that may span different input contexts. MARTI is a multi-agent training and inference framework that supports several optimizers together with agent-level reward allocation and history-based reward shaping. None of these methods defines credit in terms of counterfactual round-value changes or set deletion. We thank the reviewer for these references and have incorporated this discussion into the related work (Section 2 and Appendix B.1).

5. Naming of the "GRPO" baseline (Weakness 3)

Both rows use the same trajectory-level reference: an RLOO-style leave-one-out mean without standard-deviation normalization (Section 4.3, D.5); Dr. GRPO likewise omits the normalization but uses the full-group mean. The convention is therefore identical across the two main rows and is specified in the paper; to avoid ambiguity, we now state this explicitly, with the corresponding references, where the baseline is defined in Appendix D.5. The terminal weight and the three local-credit weights are all set to one.

AuthorsOpenReview ↗

Response to Reviewer 7d9h (3/3)

6. Underperformance, decoding temperature, and scope (Question 5, Weakness 3)

Where MAMR-GRPO underperforms. Table 1 reports one such case: on NQ, MAMR-GRPO attains 40.9 EM versus 41.5 for the same-graph baseline, and the mean gain on general QA is 0.4. In the budget sweep, increasing the worker cap from 4 to 6 lowers performance on MuSiQue and PopQA, while Bamboogle accounts for 1.03 of the 1.05-point gain.

Decoding temperature. MAMR-GRPO and the same-graph GRPO baseline are evaluated at the same sampling temperature of 0.7 (Appendix E.2), and the budget sweeps use the same temperature (Appendix E.4); this is also the default in the released generation configuration of the Qwen3-4B-Instruct-2507 backbone. We regard temperature 0 as a less representative operating point here: the policies are trained with sampling, so greedy decoding evaluates a deterministic policy different from the one that was optimized, and greedy decoding is prone to repetitive outputs in long multi-turn generation (2,058 to 5,136 generated tokens per question across the budget settings in Appendices E.4–E.5).

Task and model scope. Our experiments are limited to retrieval QA with a 4B backbone and a single retrieval setup, chosen because combining evidence across sub-queries plausibly favors role decomposition; broader task and model-scale coverage is left for future work.

We thank the reviewer once again for the thorough and constructive feedback. The reviewer's comments have directly improved the paper: we have added the discussion of AT-GRPO, Dr. MAS, MHGPO, and MARTI to the related work, stated the baseline's advantage convention explicitly in Appendix D.5, and corrected the reported training time in Section 5.1. We are grateful for the reviewer's engagement with our work. We sincerely apologize that, owing to limited GPU resources, we were unable to complete the suggested compute-matched GRPO baseline, component-wise ablations, and additional seeds during the rebuttal period. We nonetheless believe that the controlled same-graph comparison, its advantage on the 80% of questions not used for checkpoint screening, and the alternative credit constructions in Appendix E.3 support the contribution of the paper.

Reviewer_dMGKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MAMR-GRPO, a reinforcement learning method for a shared-policy, multi-agent system comprising task assignment, parallel subtask execution, and cross-round aggregation. It augments terminal rollout advantages with role-specific local credits: round-value improvements for task assignment, individual and joint output-deletion effects for subagents, and subsequent round-value gains for aggregation. Round values are estimated by sampling forced-stop answers and evaluating their task rewards. Using Qwen3-4B-Instruct-2507 on seven search-augmented QA benchmarks, the method achieves 46.7 average EM, compared with 43.6 for a trajectory-level GRPO baseline on the same execution graph. The paper also analyzes execution behavior and performance under different inference budgets.

Critical strengths
  1. The method addresses a consequential credit-assignment problem with an explicit execution-aware design. A single terminal advantage cannot distinguish useful intermediate work from unhelpful work within the same rollout. The proposed signals map naturally to the system’s three roles. Joint output deletion is a particularly interesting extension because individual removal can miss contributions masked by redundant evidence. The credits use task-answer rewards rather than requiring an additional LLM judge.

  2. The same-framework comparison provides meaningful empirical evidence. Holding the execution graph and backbone fixed, MAMR-GRPO improves average EM by 3.1 points over the GRPO baseline, with improvements on six of seven benchmarks. The larger gains on multi-hop QA are consistent with the motivation that intermediate contributions matter more when later searches depend on earlier findings. The appendix also provides substantial implementation detail, including sampling budgets, prompts, token masks, and auxiliary supervision.

Critical weaknesses
  1. Checkpoint selection overlaps with the final evaluation collection. Appendix E.1 states that candidate checkpoints were screened on a 20% subset of the evaluation collection before evaluation on the full collection. Consequently, part of the reported evaluation data also influenced model selection. This does not establish training-data leakage, but it means the full-collection scores are not an independent held-out estimate. The magnitude of selection bias is unclear because the number of candidate checkpoints, selection criterion, and consistency of this procedure across methods are not reported. This matters for interpreting the central 3.1-point improvement and requires results on evaluation examples excluded from checkpoint selection.

  2. The experiments do not isolate the proposed credit mechanisms from other training differences or establish their compute-adjusted benefit. The reported ablations vary inference-time agent and round limits, rather than removing the task-assignment, subagent, or aggregation credits. In addition, Appendix D.5 states that MAMR-GRPO uses answer-level leave-one-out advantages for terminal answers, whereas the GRPO baseline assigns those answers their parent trajectory’s advantage. Thus, the controlled comparison changes more than intermediate credit assignment. The method also performs eight deletion evaluations per round with four answer samples per value estimate, but the paper does not provide a sufficiently detailed accounting of the resulting training overhead. Without component ablations and explicit compute accounting, it remains unclear which proposed mechanisms explain the improvement and whether simpler alternatives offer a comparable accuracy–cost tradeoff.

Questions for authors
  1. How many candidate checkpoints were screened, using what selection criterion, and was the same procedure applied to every locally trained baseline? Could you report the main comparison on the remaining 80% of examples that were excluded from checkpoint screening, or on another independent held-out set?

  2. As for attribution experiment, could you first give the GRPO baseline the same terminal answer-level advantage estimator, then add the local credits? Within that matched setup, could you compare the full method against removing each role’s local credit and replacing joint deletion with individual deletion only? These comparisons would directly affect my assessment of the contribution.

  3. What quantities were matched under the stated training-time compute budget—training examples, optimizer steps, generated tokens, GPU-hours, or another measure? Please report total training generation and GPU-hours for GRPO and MAMR-GRPO, including counterfactual branches, and clarify whether all methods completed the same amount of training.

2 comments
AuthorsOpenReview ↗

Response to Reviewer dMGK (1/2)

We sincerely thank the reviewer for the time and effort devoted to this precise and insightful review. We appreciate the recognition that the method addresses a consequential credit-assignment problem with an execution-aware design, that joint output deletion is an interesting extension, and that the same-framework comparison provides meaningful empirical evidence. We are particularly grateful for the careful distinction between evaluation independence and training-data leakage, and for the follow-up question on the terminal-answer estimator; both reflect a close reading of the paper, and we have found them very helpful. We address the points one by one below.

1. Checkpoint selection (Weakness 1, Question 1)

Selection overlap. We agree that screening on a 20% subset of the evaluation collection (Appendix E.1) implies that the full-collection score is not a fully independent held-out estimate; the statement in Section 5.1 that no question strings are shared concerns training-data overlap, not selection overlap.

Number of candidates, criterion, and consistency across methods. The same procedure was applied to every locally trained method, including the GRPO baseline and the reproduced Search-R1, C3 SubMessage Lite, and SHARP-MC: checkpoints taken at 50-step intervals (16 per run) are ranked by average EM on the 20% subset, and the top-ranked candidates are evaluated on the full collection. The 20% subset was drawn at random with seed 42 before any evaluation began and was then kept fixed for checkpoint screening of every method; we therefore consider this a fair protocol for comparing the methods.

Results on the remaining 80%. On the 80% of questions that were not used for screening (n=41,293), the average EM is 46.0 for MAMR-GRPO versus 43.8 for the same-graph GRPO baseline. Our method thus retains its advantage over GRPO on the remaining 80%.

2. Terminal-answer estimator and attribution (Weakness 2, Question 2, follow-up comment)

Terminal-answer estimator. We thank the reviewer for this careful follow-up question. This point arises from an insufficiently clear description in Appendix D.5, which has led to a misunderstanding. In fact, both methods use the same advantage estimator for terminal answers: each terminal answer's own score is compared with the mean score of the other terminal answers for the same question in an RLOO-style leave-one-out comparison, without division by the group standard deviation (see _compute_answer_group_advantages in the submitted code).

Why D.5 says "inherit". Appendix D.5 states that baseline answers "inherit their parent trajectory's advantage" because the baseline's trajectory advantage is itself computed from the answer-level RLOO: the baseline samples a single terminal answer per rollout, so the trajectory reward is that answer's score, and the trajectory's RLOO advantage is the same number as the answer's own RLOO advantage. The code expresses this as inheritance only so that the answer tokens share a single scalar with the remaining tokens of the trajectory; it is not a different estimator.

Role-wise and singleton-deletion controls. The controls proposed by the reviewer, such as removing each role's credit and replacing joint with singleton deletion, remain valuable for separating the contributions of the individual local credits. We appreciate this suggestion, and we sincerely apologize that we did not have sufficient GPUs to train these variants during the rebuttal period. The closest existing evidence is Appendix E.3, where replacing our worker credit with C3 SubMessage Lite or SHARP-MC on the same pipeline yields 39.7 and 40.9, compared with 46.7.

AuthorsOpenReview ↗

Response to Reviewer dMGK (2/2)

3. Matched quantities in training compute (Weakness 2, Question 3)

What is matched. The matched quantities are training examples and optimizer steps: every locally trained method is trained for one epoch on the same data with the same batch configuration and thus performs the same number of optimizer updates. Generated tokens and GPU-hours are not matched.

Wall-clock time and GPU-hours. According to the training logs, over the same 780 steps on the same 8x A100 80GB node, MAMR-GRPO requires on average 1,082 s per step versus 689 s for GRPO (approximately 1.57x; 234 versus 149 hours of cumulative step time). Most of the difference arises from rollout generation, including counterfactual branches (454 s versus 210 s per step). Section 5.1 now reports the full training times: about 9.8 days for MAMR-GRPO and about 6.2 days for GRPO.

Trained tokens. Tokens in the training batch (prompt plus response tokens of trained sequences) average 19.4M versus 14.5M per step; of these, the response-side tokens (model generations plus retrieval results inserted between Sub-agent turns) are 7.2M versus 7.1M, so the trained rollouts are of similar size.

Counterfactual branches. Generations used solely for credit computation are reflected in the generation time but not in these counts. They stem from eight deletion evaluations per round and four answer samples per value estimate (Appendix D.4) and involve no retrieval calls. Since they share the format of the realized outputs, we estimate their size: on average 8,346 deletion branches per step, each generating a Controller summary and four answers (about 286 and 10 tokens, the average lengths of the final Controller output and answer in the Table 1 evaluation), i.e. roughly 2.7M generated tokens per step; forced-stop answers at nonterminal boundaries add under 0.1M.

Conclusion. We therefore present the 3.1-point difference as a comparison under equal training data and optimizer steps, rather than as a compute-adjusted benefit.

We thank the reviewer once again for the rigorous and constructive feedback and for the continued engagement in the discussion. The reviewer's questions have helped us state the training-compute comparison and the checkpoint-selection procedure more precisely, and we have corrected the reported training time in Section 5.1. We sincerely apologize that, owing to limited GPU resources, we were unable to complete the suggested role-wise ablations during the rebuttal period. We nonetheless believe that the same-graph comparison, which retains its advantage on the 80% of questions not used for screening (46.0 vs. 43.8), together with the alternative credit constructions in Appendix E.3, supports the contribution of our credit design.

Reviewer_ofw7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper claims to improve multi-agent collaboration in search-augmented QA over GRPO, with gains in performance and behavior in terms of fewer total sub-agent executions and rounds. The main algorithmic innovation comes from an additional advantage component involving role-specific local credit assignment, subagent execution, and aggregation.

Critical strengths

The methodology is well-defined and explained, especially the various components of the credit assignment mechanism. The comparisons are rigorously controlled and the reproductions are thorough, and Table 6 includes a lot of details.

Critical weaknesses

The paper claims its core contribution is “role-specific local credits using counterfactual round values and output removal”, but the credit assignment contribution is largely a combination of TRACE’s temporal credit, SHARP’s removal credit, and GRPO-MA’s MC value probe, and pointing each one to a role at round boundaries. It is interesting that nearly all of the baseline multi-agent frameworks underperform the single-agent baselines once RL-trained. It seems like this is a task where single-agent already leads, and MAMR-GRPO is rescuing the multi-agent setting with better credit assignment, which in itself is a valid claim. However, it could have been much stronger if the test bed was one where multi-agent training inherently makes more sense than single-agent training.

Questions for authors

There are several questions to resolve that would make the paper stronger: Ablation: restricting F_t to singletons -> whether the novel estimator is helping (subagent credit via joint deletion with equal split) Ablation: zeroing out λ_meta, λ_sub, λ_ctrl one at a time -> whether role-specific routing helps The paragraph starting on L355 says “the gains concentrate on multi-hop QA, which requires combining dependent results…” This seems speculative, as the gains could be explained by factors that are not a lift in the ability to combine independent results. This could also be confounded because the vast majority of the training mix is multi-hop, so another explanation is multi-hop is more in distribution with training. I suggest showing gains grouped by hop count in the training distribution. If the advantage over GRPO scales with hop count, we can be more sure that the method helps with combining dependent results

Smaller weaknesses: The mechanism has many moving parts so it is confusing many new terms were introduced without definition: forced-stop evaluation, deletion-based evaluation (L144-150) Overclaiming gains with additional subagents: abstract mentions “budget ablations show further gains with additional subagents,” and §1 mentions increasing the budget “can improve EM”, but the L378 paragraph makes clear that this is a noise-level effect on a small benchmark. I suggest softening the claim in the abstract Some paragraphs seem obviously AI-generated such as the one from line 447. Figure 1 nonsense sentence: “(C) Removing sampled SUB AGENT records and regenerating aggregation and answers estimates subagent credit.”

2 comments
AuthorsOpenReview ↗

Response to Reviewer ofw7 (1/2)

We sincerely thank the reviewer for the time and effort devoted to evaluating our paper, and for the careful reading and constructive suggestions. We are grateful that the reviewer found the methodology well defined and clearly explained, particularly the components of the credit assignment mechanism, and considered the comparisons rigorously controlled and the reproductions thorough. We also appreciate the reviewer's recognition that improving credit assignment in the multi-agent setting is a valid contribution in itself.

Several of the concerns can be addressed with results already reported in the paper, and some premises of the review differ from what Table 1 and Section 5.4 show. We address the points one by one below.

1. Relation to TRACE, SHARP, and GRPO-MA (Weaknesses)

Temporal differences, removal-based credit, and Monte Carlo value estimation are established techniques, and we do not claim them as contributions. Our contribution lies in how credit is defined for each component of a multi-round collaboration, and these definitions are not a relabeling of the cited signals:

  • TRACE measures changes in a frozen reference model's gold-answer log-ratio at tool boundaries. In contrast, our round value is the expected task reward of answers sampled from the current policy at a collaboration boundary; it involves neither a reference model nor gold-answer likelihoods.
  • SHARP (Eq. 11) uses the effect on terminal accuracy of removing a single worker, and its planner credit aggregates positive worker credits. In our formulation, assignment credit measures progress relative to the previous round, worker credit apportions the effect of removing sets of outputs on round value, and aggregation credit measures the value gained in subsequent rounds.
  • GRPO-MA provides the multi-answer value-estimation principle, which we cite and build upon; it does not define collaboration credits.

2. Is this a task in which single-agent training already leads? (Weaknesses)

What Table 1 shows. Table 1 does not support this premise. On the same execution graph, trajectory-level GRPO attains 43.6 average EM, already exceeding the locally trained single-agent Search-R1 (43.3), while MAMR-GRPO attains 46.7. Prior to RL, the framework also substantially outperforms the single-agent configuration with the same backbone (37.2 vs. 28.8). The systems that fall below Search-R1 are the alternative credit designs and frameworks (C3 39.7, SHARP-MC 40.9, WideSeek-R1 40.6).

Choice of test bed. We agree that evaluation on domains where decomposition is even more natural would further strengthen the case. We believe that multi-agent systems may hold broader promise in deep research; however, owing to resource constraints, particularly those related to retrieval APIs, it is currently difficult for us to conduct RL training for deep research on the live web.

3. The suggested ablations (Questions 1 and 2)

Restricting F_t to singletons (Q1). Joint removal addresses a specific failure mode of singleton removal: when two worker outputs are interchangeable, removing either one alone leaves the round value unchanged, so both receive zero credit even though removing the pair lowers it.

Zeroing λ_meta, λ_sub, λ_ctrl one at a time (Q2). Appendix D.5 applies λ_sub to C_sub + B; setting λ_sub = 0 therefore also removes the anti-redundancy term B, and isolating the deletion credit would require removing C_sub while retaining B.

Status. We appreciate these suggestions, and we sincerely apologize that we did not have sufficient GPUs to train these variants during the rebuttal period. These ablations would complement the evidence for the design already reported in the paper, namely the same-graph comparison with GRPO and the alternative credit constructions in Appendix E.3.

AuthorsOpenReview ↗

Response to Reviewer ofw7 (2/2)

4. Multi-hop gains and the training distribution (Question 3, L355)

Per-benchmark evidence. The per-benchmark results do not support a purely distributional explanation. NQ is included in the training mixture yet shows no gain (-0.6), whereas Bamboogle, a multi-hop benchmark absent from training, improves by +5.6; PopQA and TriviaQA, likewise absent from training, improve by +1.2 and +0.7. Since Bamboogle contains only 125 questions, we regard this as supporting evidence rather than conclusive proof, and we have revised the sentence at L355 accordingly.

Gains grouped by hop count. Following the reviewer's suggestion, we stratified the Table 1 evaluation (the same per-question outputs) by the hop or question-type labels provided with each dataset, so that every comparison stays within one benchmark and thus one training source (EM, GRPO → MAMR-GRPO):

  • MuSiQue: 2-hop 25.9 → 30.9 (n=1,252), 3-hop 10.1 → 15.8 (n=760), 4-hop 6.7 → 8.6 (n=405).
  • 2Wiki: comparison 70.8 → 76.6, compositional 37.6 → 41.4, inference 30.7 → 41.5, bridge-comparison 55.1 → 65.6.
  • HotpotQA: comparison 65.4 → 67.9, bridge 37.0 → 40.7.

Interpretation. MAMR-GRPO improves in every stratum, with the largest gains on 2Wiki inference and bridge-comparison questions. The gain does not grow monotonically with hop count (4-hop MuSiQue stays below 9 EM for both methods), so we do not claim that it scales with reasoning depth, and the L355 statement remains a hypothesis.

5. Claims about budget gains (abstract and Section 1)

Up to four subagents. Section 5.4 does not characterize the budget effect as noise. From S=2 to S=4, average EM increases by 1.28, of which Bamboogle accounts for only 0.11, so this gain is distributed across the other benchmarks; from R=1 to R=2, it increases by 2.14.

Beyond four subagents. The concentration noted by the reviewer arises here: from S=4 to S=6, Bamboogle accounts for 1.03 of the 1.05-point gain, while completion tokens increase from 3,735 to 5,136.

Revision. We have revised the abstract, introduction, and conclusion to state this range explicitly.

6. Terminology, Figure 1(C), and L447

Forced-stop and deletion-based evaluation. Both terms are defined in Section 4 (L201 and L227) but are first used in Section 3 (L147–148); we have added brief definitions at their first occurrence. Forced-stop evaluation samples answers from the current Controller context under a virtual stop, without altering the actual decision of the rollout. Deletion-based evaluation removes the task descriptions and outputs of selected subagents, retains the remaining records, and regenerates the Controller prefix and answers.

Figure 1(C). The sentence in Figure 1(C) describes this procedure; we have rephrased it as: "Selected subagent records are removed, aggregation and answers are regenerated, and subagent credit is assigned from the resulting change in round value."

Paragraph at L447. We have rewritten the paragraph at L443–447.

We thank the reviewer once again for the insightful and detailed comments. They have directly improved the paper: following the reviewer's suggestions, we have stated the range of the budget gains explicitly in the abstract, introduction, and conclusion, added definitions of forced-stop and deletion-based evaluation at their first use, rewritten the Figure 1(C) caption, and rephrased the multi-hop interpretation at L355. We are grateful for the reviewer's engagement with our work. We sincerely apologize that, owing to limited GPU resources, we were unable to complete the suggested training ablations (singleton deletion and the λ variant that isolates C_sub) during the rebuttal period. We remain confident in the contribution of the paper: on the same execution graph, MAMR-GRPO improves over trajectory-level GRPO from 43.6 to 46.7, and the alternative worker-credit constructions in Appendix E.3 reach only 39.7 and 40.9.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to Reviewer vRuy (1/2)

We sincerely thank the reviewer for the time and effort devoted to this thorough and constructive review. We appreciate the recognition of the alignment between credit and the execution graph, of the joint-removal counterfactuals for redundant worker outputs, and of the controlled same-framework baseline and broad QA evaluation. We are also grateful for the reviewer's precise specification of the controls needed to interpret the results, and for the cost and forced-stop-value questions. We address the points one by one below.

1. Component attribution (Weakness 1, Question 1)

Inference-time budget sweeps. We agree that the inference-time sweeps over workers and rounds do not identify which training signal is responsible for the gain; they are budget studies rather than training ablations.

Terminal-answer estimator. We should clarify that the statement in Appendix D.5 that MAMR-GRPO's terminal answers receive answer-level leave-one-out advantages, whereas baseline answers inherit their trajectory advantage, was not sufficiently clear and may suggest that the two methods use different terminal-answer estimators. In fact, both use the same estimator: each terminal answer's own score is compared with the mean score of the other terminal answers for the same question in an RLOO-style leave-one-out comparison, without division by the group standard deviation (see _compute_answer_group_advantages in the submitted code).

We described the baseline's answers as inheriting the trajectory advantage because that advantage is itself computed from the answer-level RLOO: the baseline samples a single terminal answer per rollout, so the trajectory reward is that answer's score and the two quantities coincide. The terminal-answer estimator is therefore not an additional difference between the two main rows, and the comparison of 46.7 vs. 43.6 evaluates the complete training method on a fixed graph.

Role-wise and deletion ablations. On this basis, the informative control would vary the local-credit terms, including removing each role's credit and comparing singleton with joint deletion. We appreciate this suggestion, and we sincerely apologize that we did not have sufficient GPUs to train these variants during the rebuttal period.

Existing component-level evidence. Appendix E.3 replaces our worker credit with C3 SubMessage Lite or SHARP-MC on the same pipeline, yielding 39.7 and 40.9, compared with 46.7.


2. Seeds and cost (Weakness 2, Question 2)

Seeds. The main row corresponds to a single training run (seed 42). We sincerely apologize that we did not have sufficient GPUs to train additional independent seeds during the rebuttal period.

Training time and tokens. According to the training logs, over the same 780 optimizer steps on the same 8x A100 80GB node, MAMR-GRPO requires on average 1,082 s per step versus 689 s for GRPO (approximately 1.57x; 234 versus 149 hours of cumulative step time). Most of the difference arises from rollout generation, including counterfactual branches: 454 s versus 210 s per step. Tokens in the training batch (prompt plus response tokens of trained sequences) average 19.4M versus 14.5M per step; generations used solely for credit computation are reflected in the generation time but not in this count. Section 5.1 now reports the full training times: about 9.8 days for MAMR-GRPO and about 6.2 days for GRPO.

Retrieval calls. Counterfactual branches and value estimates involve no retrieval calls, since Controller regeneration and answer sampling are performed without tools; the additional cost is therefore limited to generation. These branches are used only to compute training credits and incur no overhead at inference.

Credit-only generation and prompt tokens. We estimate the credit-only generation from its format, which matches the realized outputs: training performs on average 8,346 deletion branches per step, each generating a Controller summary and four answers (about 286 and 10 tokens, the average lengths of the final Controller output and answer in the Table 1 evaluation), i.e. roughly 2.7M generated tokens per step; forced-stop answers at nonterminal boundaries add under 0.1M. Prompt tokens of these branches were not logged separately.

Inference cost. At test time, in the default setting (S=4, R=3), MAMR-GRPO generates on average 3,735 completion tokens per question, with 6.72 subagent executions and 1.681 realized rounds (Appendices E.4–E.5); completion tokens exclude prompt and retrieved-document tokens. Across the budget settings, generation ranges from 2,058 to 5,136 tokens per question. Relative to GRPO, the training rollout statistics in Section 5.5 (steps 515–780) show 25.7% fewer rounds (2.006 vs. 2.699) and 7.8% fewer subagent executions (7.997 vs. 8.677) per rollout.

AuthorsOpenReview ↗

Response to Reviewer vRuy (2/2)

3. Forced-stop value, synergy, and task scope (Weakness 2, Question 3)

What the forced-stop value measures. The forced-stop value is a well-defined target rather than an approximation to an undefined quantity: it is the expected task reward of answering from the context and Controller prefix available at the current boundary. It provides a common task-reward scale for comparing round progress and output removals, without requiring a learned critic or an LLM judge.

Myopia. Its myopia is addressed by design in two ways: the terminal group advantage is always retained, and aggregation credit is defined by the value gained in subsequent rounds rather than by the current value. Information that pays off only after further search may still receive little immediate worker credit; in that case, the aggregation and terminal credits carry the signal.

Synergistic subagents. Joint deletion captures conditional set effects that singleton deletion misses in both directions: redundant outputs (removing either alone changes nothing) and complementary outputs (removing the pair costs more than the sum of the individual removals). Equal splitting of a set effect is an allocation rule rather than a unique decomposition.

Correlation with downstream gains. A direct correlation with independently measured downstream marginal gains is not reported.

Beyond retrieval QA. Our experiments are limited to retrieval QA with the specified backbone and retriever, chosen because combining evidence across sub-queries plausibly favors role decomposition; broader task coverage is left for future work.

We thank the reviewer once again for the careful, precise, and constructive feedback, which has helped us improve the paper, including the correction of the reported training time in Section 5.1. We are grateful for the reviewer's engagement with our work. We sincerely apologize that, owing to limited GPU resources, we were unable to complete the suggested component-wise ablations and additional independent seeds during the rebuttal period. We nonetheless believe that the fixed-graph comparison (46.7 vs. 43.6) and the alternative credit constructions in Appendix E.3 (39.7 and 40.9) support the contribution of execution-structured credit.

AuthorsOpenReview ↗

Common Response (1/2)

Common Response to the Area Chair and Reviewers

We sincerely thank the Area Chair for handling our submission and overseeing the review and discussion process, and all five reviewers for their time and careful reading. At the outset, we would like to sincerely apologize: several reviewers requested additional training runs (component ablations, singleton versus joint deletion, additional seeds, and a compute-matched GRPO baseline). We believe that these experiments are highly valuable, particularly those on credit assignment and joint deletion, but regrettably we no longer had sufficient GPUs to complete them within the rebuttal period. These experiments would complement, rather than replace, the evidence for our contribution already reported in the paper, namely the same-graph comparison with GRPO and the alternative credit constructions in Appendix E.3. We have therefore answered as many of them as possible from our existing training logs and the per-question evaluation outputs of the reported runs, for example the per-step training cost, the results on the 80% of questions not used for checkpoint screening, and the hop-stratified results.

We are encouraged that the reviewers found the methodology well defined and the reproductions thorough (ofw7), recognized the alignment of credit with the execution graph and the joint-deletion counterfactuals for redundant worker outputs (vRuy, dMGK, 7d9h), considered the same-framework comparison controlled and meaningful (vRuy, dMGK, 7d9h), and appreciated the supplementary code (7d9h). We also noticed that reviewer dMGK initiated a discussion with other reviewers, and we sincerely appreciate their continued interest in our work.We have uploaded a revised version and respond to each reviewer individually; below we summarize the revisions and the shared questions.

Revisions in the updated PDF (all changes relative to the submitted version are marked in red)

  • Abstract and Section 1 (Introduction): the budget statements now specify that the gains hold up to four subagents and from a second round, with uneven returns beyond that; the contribution list centers on the definition of execution-structured credit, and the framework is described as the execution setting on which these credits are defined (ofw7, h9cX).
  • Section 2 (Related Work) and Appendix B.1: new discussion of AT-GRPO, Dr. MAS, MHGPO, and MARTI (7d9h).
  • Section 3 and Figure 1(C): definitions of forced-stop and deletion-based evaluation at their first use, and a rewritten caption for panel (C) (ofw7).
  • Section 5.1 (Experimental Protocol): the training time of each method for 780 optimizer steps, about 9.8 days for MAMR-GRPO and 6.2 days for GRPO (vRuy, dMGK, 7d9h).
  • Section 5.3 (Main Results): the multi-hop interpretation (former L355) is stated as a hypothesis, together with the NQ and Bamboogle evidence (ofw7).
  • Section 5.5 (Changes in Rollout Behavior): the "Relation to training design" paragraph (former L443–447) and the closing sentences of the case study are rewritten (ofw7).
  • Sections 6 and 7 (Limitations, Conclusion): updated to match the revisions above.
  • Appendix D.5: the baseline's advantage convention is stated explicitly with references (7d9h).

Clarifications on shared questions

  1. What the main comparison controls (h9cX, vRuy, dMGK). MAMR-Framework + GRPO (43.6 average EM) and MAMR-GRPO (46.7) share the execution graph, the Qwen3-4B backbone, the training data, and the 780 optimizer steps; MAMR-GRPO improves on six of the seven benchmarks. Both methods use the same terminal-answer estimator, an RLOO-style leave-one-out comparison without standard-deviation normalization. Replacing our worker credit with C3 SubMessage Lite or SHARP-MC in the same setting yields 39.7 and 40.9 (Appendix E.3).

  2. Checkpoint selection (dMGK, 7d9h). Every locally trained method uses the same procedure: checkpoints taken at 50-step intervals are ranked on a fixed, randomly drawn 20% subset of the evaluation collection, and the top-ranked candidates are evaluated on the full collection. On the 80% of questions not used for screening (n=41,293), average EM is 46.0 for MAMR-GRPO versus 43.8 for GRPO.

  3. Multi-hop gains (ofw7). Stratified by the hop or question-type labels of each dataset, MAMR-GRPO improves in every stratum (e.g., MuSiQue 3-hop 10.1 → 15.8, 2Wiki inference 30.7 → 41.5). The gain does not grow monotonically with hop count, so the multi-hop explanation remains a hypothesis.

  4. Training cost (vRuy, dMGK, 7d9h). The methods are matched in training data and optimizer steps, not in compute: MAMR-GRPO requires on average 1,082 s per step versus 689 s for GRPO (about 1.57x), mostly due to counterfactual generation, which involves no retrieval calls and adds no inference-time cost.

We thank the Area Chair and the reviewers once again for their time and constructive feedback, which has substantially improved the clarity and precision of the paper.

AuthorsOpenReview ↗

Common Response (2/2)

Additional comment: planned component-level ablations (to the Area Chair and all reviewers)

We thank the Area Chair and the reviewers for their patience and continued engagement during the discussion period. We would like to add an update to our common response above. Several reviewers (dMGK, vRuy, ofw7, h9cX, 7d9h) asked for a finer-grained ablation than the same-execution-graph comparison with GRPO already reported in Table 1 — specifically, to isolate each local credit's contribution and to compare joint deletion against a singleton-only deletion-set family. We are currently training four additional configurations that vary only the credit weights or the deletion-set family:

  • −Meta (Meta credit weight set to 0): isolates the Meta credit's contribution.
  • −Sub (Sub/worker credit weight set to 0): isolates the worker credit's contribution.
  • −Ctrl (Controller/aggregation credit weight set to 0): isolates the Controller credit's contribution.
  • Singleton-only deletion-set family (credit weights unchanged, but every counterfactual branch removes a single worker output rather than a sampled joint subset): tests whether joint deletion, not just singleton removal, is responsible for part of the gain.

Each configuration uses the same training data, batch configuration, and 780 optimizer steps as the main runs, and the same checkpoint-selection procedure used for every other method in the paper (screening on a fixed 20% subset, then full-collection evaluation of the top-ranked checkpoint). These runs started approximately 7–10 days ago and are still in progress. Given the GPU constraints described in our other responses, we cannot commit to a specific completion date, but if any of them finish before the discussion period closes, we will post the results here as a follow-up to this comment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.