acceptodds
Under review as a conference paper at ICLR 2027

From Delegation to Coordination: Learning Efficient Multi-Agent Deep Research

Abstract

Multi-agent deep-research systems use delegation to parallelize search and keep long tool traces out of the main agent’s context. Yet delegation helps only when task allocation and subtask execution are coordinated: the main agent must issue useful subtasks, while subagents must return reliable evidence. Existing train- ing either optimizes only the main agent or broadcasts the same team advantage to all agents, which can reward misleading subagent reports and penalize use- ful ones. We introduce LEAD-RL (Learning Effective Agent Delegation with Reinforcement Learning), a joint main-agent–subagent training framework with contribution-aware credit routing. A training-time rubric evaluates the research utility of each subagent report and gates its inherited team advantage, while final- answer correctness remains the only team reward; no token, step, or speedup re- ward is used. Across five benchmarks and two model families, LEAD-RL im- proves answer quality. On BrowseComp, it improves accuracy by 2.49 percentage points and reduces critical-path cost by 24.5% over joint training with shared team credit. During validation, accuracy increases by 6.78 points at approximately con- stant critical-path cost. These results show that contribution-aware credit assign- ment improves the quality–efficiency trade-off of multi-agent deep research.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-agent systems+Deep research+Reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pXKdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LEAD-RL jointly trains a research agent and its subagents using a shared policy. A training-time judge labels subagent reports as helpful, neutral, or harmful, and these labels determine whether each report inherits the positive or negative team advantage. The paper evaluates the resulting systems on research benchmarks and reports shorter critical paths alongside improved answer accuracy.

Critical strengths
  1. The paper identifies a plausible failure of shared team credit: useful research can precede a wrong answer, while a successful team can recover from misleading evidence. Applying a simple gate at the report boundary is well matched to the delegation interface and avoids requiring counterfactual execution of an entire research trajectory.

  2. The authors compare against both main-agent-only training and joint training from the same stated initialization. They also distinguish critical-path steps from actual wall-clock speedup. The shorter research paths are interesting precisely because the method does not directly reward shorter execution.

Critical weaknesses
  1. [Major] The experiments do not establish that the gate works because its contribution judgments are correct. The judge has access to the reference answer, and its labels selectively suppress policy updates. That is substantive intermediate supervision, even if the scalar team reward remains unchanged. There is no independent audit of these labels, while the reported improvement in helpfulness is measured with the same judging approach used during training. A simple filter that suppresses the same proportion of updates could reveal whether the gain comes from meaningful credit assignment or just less aggressive subagent optimization. Without such a control, the paper demonstrates the usefulness of a training heuristic more clearly than it demonstrates better coordination through contribution assessment.

  2. [Major] Too much of the training comparison is left unspecified. The paper does not identify the SFT and RL data sources, their sizes, or how they are separated from the evaluation benchmarks. This is especially consequential because the SFT checkpoint already accounts for most of the reported improvement over the base model. Appendix C also records different training lengths for the RL variants, without clearly tying the selected test checkpoints to a matched update budget. These omissions prevent me from judging the incremental contribution of LEAD-RL with confidence. Please document the data and initialization, then report the controlled comparison at a common training budget with an explicit checkpoint-selection rule.

Questions for authors

See weaknesses.

3 comments

Thank you for your careful review and for identifying the need for a matched control and a clearer account of the training comparison.

Major 1. Count-matched random gating and intermediate supervision

We added a count-matched random-gating control to test whether the choice of subagent trajectories matters. For each rollout sampled by the control policy, we computed how many eligible subagent trajectories the contribution gate would mask. We then sampled that many eligible trajectories uniformly at random and set their inherited team advantages to zero. The main-agent advantage rule, subagent normalization rule, and configured weights were the same. Rollouts with zero team advantage required no additional masking.

BrowseComp method at step 30AccuracyCritical-path steps/question
Shared-credit RL / all-neutral71.00%323.30
Count-matched random gating71.33%349.17
LEAD-RL73.49%244.22

Relative to random gating, LEAD-RL gained 2.16 percentage points in accuracy and used 30.1% fewer critical-path steps in the reported comparison. This result favors contribution-based selection in these runs. Mask counts are matched to the contribution rule on each rollout sampled by the random-gating policy. The count-matched control is designed to match the masking budget while changing only which eligible trajectories are masked at each gating decision.

We clarify that the contribution judge provides reference-guided credit assignment rather than an additional intermediate reward. During training, it uses the reference answer to label subagent reports; these labels determine whether the outcome-derived team advantage is assigned to each subagent. Final-answer correctness remains the sole scalar team reward. Thus, the judge shapes how the outcome-derived reward is distributed across subagents, rather than adding a separate scalar reward signal. Neither the reference answers nor the judge’s outputs are used at inference.

Three annotators took part in a human audit of the contribution labels. The audit summary lists 450 reports per method and human–judge agreement of 95.1% for LEAD-RL and 93.7% for shared-credit RL. In the audited sample, human annotators rated 32.79% of LEAD-RL reports as helpful and 7.67% as harmful. The corresponding rates for shared-credit RL were 23.67% and 12.16%.

Major 2. Matched training updates and gains after SFT

The cold-start SFT stage utilized approximately 7,000 trajectories sampled from an open-source model. All three RL variants in Table 3 are initialized from the same SFT checkpoint and trained using the identical set of 500 internally synthesized RL questions under a matched configuration schedule. Each update step processes 32 questions with four rollouts per question. Table 3 evaluates all three variants at step 30, ensuring a fair comparison under equal update steps and sampled question groups. This common evaluation point was chosen by jointly balancing accuracy and critical-path cost on a fixed validation set of roughly 300 held-out synthetic questions. Appendix C reports the full training trajectories—35, 34, and 60 updates—all of which extend beyond the step-30 checkpoint used in Table 3.

BrowseComp methodCheckpointAccuracyCritical-path steps/question
SFTBefore RL72.50%403.37
Main-only RLStep 3070.39%421.76
Shared-credit RLStep 3071.00%323.30
LEAD-RLStep 3073.49%244.22

Relative to SFT, LEAD-RL improved accuracy by 0.99 percentage points and reduced critical-path cost from 403.37 to 244.22 steps per question, a 39.5% reduction. Relative to shared-credit RL, LEAD-RL improved accuracy by 2.49 percentage points and reduced critical-path cost from 323.30 to 244.22 steps per question, a 24.5% reduction.

AuthorsOpenReview ↗

Has the rebuttal addressed your concerns?

Hi,

Has our rebuttal addressed your concerns? In particular, we added a count-matched random-gating control and clarified the training data and matched-checkpoint comparisons. Would you consider updating your rating, or are there any remaining concerns we could address during this discussion period?

Thank you for your time!

AuthorsOpenReview ↗

Follow-up on the rebuttal

Dear Reviewer,

Thank you again for your review. We added experimental results and implementation details in response to your comments. If a key issue remains unresolved, could you point us to it before today’s rebuttal deadline so we have a chance to respond?

If you feel the additional results address your main concerns, we hope you’ll revisit your score in light of those findings.

Thank you for considering our response.

Reviewer_gpiPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to jointly train a main research agent and its subagents for effective delegation. LEAD-RL uses a shared policy and GSPO, with final-answer correctness as the team reward. A training-time judge classifies subagent reports as helpful, neutral, or harmful. These labels gate the inherited team advantage: helpful reports retain positive advantages, harmful reports retain negative advantages, and neutral reports retain the original signal.

The complete training pipeline improves results across five benchmarks and two model families. In the controlled BrowseComp comparison, LEAD-RL achieves 73.49% accuracy versus 71.00% for joint RL with shared credit, while reducing critical-path steps by 24.5%. The method uses no explicit token or step penalty. Its efficiency measurements describe tool-step costs and idealized parallelism, rather than measured wall-clock acceleration.

Critical strengths
  • The paper identifies a practical credit-assignment problem. A successful team can contain misleading reports, while a failed team can contain useful research. The proposed gate directly addresses these conflicting updates, and its behavior is easy to understand.

  • The controlled comparison is more informative than the broad leaderboard results. The RL variants start from the same SFT checkpoint and use the same training data, harness, and validation set. Main-only training and shared-credit joint training provide relevant reference points.

  • The efficiency improvement relative to SFT is substantial. Accuracy increases from 72.50% to 73.49%, while critical-path steps decrease from 403.37 to 244.22. I find this joint improvement more convincing than accuracy alone, especially since there is no explicit cost penalty.

  • The contribution-judging protocol is described carefully. The judge receives supporting evidence but not the main agent’s final response or correctness verdict. Appendix A also explains confidence filtering and failure handling. The authors appropriately distinguish potential evidential utility from causal contribution, and acknowledge that reduced corrective search is a hypothesis rather than a measured mechanism.

Critical weaknesses
  1. The contribution of semantic gating is not fully isolated. Equation (7) includes both normalization by the number of subagents with nonzero gated advantages and an annealed subagent weight. The paper does not explicitly establish whether the shared-credit baseline uses identical normalization and weighting. This leaves some uncertainty about whether the improvement comes from the utility labels, changes in update scaling, or their combination. An all-neutral control using same normalization and weight schedule would make the comparison more clear. Separate helpful-only and harmful-only gates would also help understand better which intervention matters.

  2. The contribution judge lacks independent validation. The helpful and harmful report rates are rubric-assessed measurements closely tied to the supervision used during training. They therefore provide limited independent evidence that report quality has improved. I did not find a human audit or cross-judge agreement study in the paper or appendices. The authors correctly describe confidence as an uncalibrated filtering signal, but the reliability of the resulting labels remains unclear. Sensitivity to the 0.8 threshold and the evidence truncation budget would be useful.

  3. Important details needed for reproduction are missing. Section 4.1 states that the variants share training data and question splits, but does not specify the source and size of the training questions, the construction of the SFT data, or the validation-set size. Appendix A provides several useful hyperparameters, but not the learning rate, GSPO clipping settings, or the concrete execution budgets underlying the matched comparisons. These omissions make it difficult to reproduce the training setup and assess separation between training and evaluation data.

  4. The robustness and scope of the controlled result remain uncertain. The accuracy improvement over SFT is 0.99 percentage points, and no confidence intervals or repeated independent RL runs are reported. The critical-path improvement is much larger, but its variation across runs is also unknown. Moreover, the controlled SFT/main-only/shared-credit/LEAD-RL comparison is reported only on BrowseComp. The broader results establish gains for the complete pipeline, but do not separately establish the contribution of the proposed RL rule across the other benchmarks and model family.

Questions for authors
  1. Does the shared-credit joint-RL baseline use the same subagent normalization and numerical weight schedule as LEAD-RL? Could you provide an all-neutral-gate control under identical settings, together with helpful-only and harmful-only variants?

  2. Could you independently audit a sample of contribution labels, using human annotation or another judge? How sensitive are the results to the confidence threshold and truncation of the supporting evidence?

  3. Could you specify the SFT and RL data sources, dataset sizes, validation split, and remaining optimization and execution settings? Appendix C records 35, 34, and 60 updates for the three RL runs. Which checkpoint from each run produces Table 3, and how was the selection criterion applied? Figure 2 already provides a useful common-horizon comparison, but the connection to the selected test checkpoints is less clear to me.

  4. Can you report paired uncertainty estimates for the BrowseComp results and variation across independent RL seeds? A controlled comparison on the second model family or another benchmark would also give more confidence on the generality of the gating rule.

  5. Equations (4)–(7) imply that a group with identical rewards has zero outcome-derived advantages, including when all four rollouts fail. Consequently, helpful reports in such a group receive no positive policy-gradient signal from that group. How frequently do all-failure groups occur, and how does this limit learning on the hardest questions?

2 comments

Thank you for your careful review. We clarify the comparisons and training setup below.

Q1. All-neutral baseline and one-sided gates

Shared-credit RL is LEAD-RL with all contribution labels set to neutral. Both use the same normalization rule, subagent weights, and schedule. Equation (7) normalizes by the number of trainable subagents with nonzero advantages after gating, so gating changes the realized denominator.

At step 30, helpful-only zeroes negative team advantages for helpful reports, while harmful-only zeroes positive team advantages for harmful reports. LEAD-RL achieves the highest accuracy and shortest critical path.

BrowseComp variantAccuracyCritical-path steps/question
Shared-credit RL / all-neutral71.00%323.30
Helpful-only72.67%259.18
Harmful-only73.00%270.13
LEAD-RL73.49%244.22

Q2. Human validation and sensitivity to judge settings

Three annotators audited 450 reports per method. Human–judge agreement was 94.3% for SFT, 93.7% for shared-credit RL, and 95.1% for LEAD-RL.

On 100 fixed reports, we varied the confidence threshold or evidence cap one at a time. A label change alters the gate only when it changes whether the team advantage is zeroed.

Confidence thresholdEvidence cap (characters)Labels changedGate decisions changedEvidence truncated
0.88,000ReferenceReference0%
0.78,0003%0%0%
0.98,0007%2%0%
0.84,00012%3%26%
0.816,0000%0%0%

Q3. Data, training settings, and checkpoint selection

Cold-start SFT used approximately 7,000 examples. The RL set contains 500 internally synthesized questions. We sampled three responses per candidate with the starting SFT model and selected 150 questions with zero correct responses and 350 with exactly one. The validation set contains approximately 300 held-out synthetic questions, separate from training and BrowseComp evaluation. We screened training and validation questions for benchmark overlap using embedding similarity and graph-based task-structure comparisons.

All three RL variants started from the same SFT checkpoint and used the same RL questions and schedule. Table 3 reports step 30 for all variants, selected by validation accuracy and critical-path cost. Appendix C reports full run lengths of 35, 34, and 60 updates.

Training used Muon, a learning rate of 7.5 × 10⁻⁶ after 10 warm-up updates, weight decay of 0.01, gradient clipping at 1.0, sample-level GSPO with lower/upper clipping of 0.20/0.28, and a KL coefficient of 0.001. Each update used 32 questions with four rollouts per question. The harness capped LLM calls at 1,000 per main agent and 400 per subagent, with up to 16 subagents per rollout and four concurrent. Both roles used temperature 1.0 and top-p 0.95. Both subagent loss-weight coefficients were 0.50.

Q4. Repeated evaluations and results on a second benchmark

We evaluated two independent training seeds per method, running each step-30 checkpoint three times on the same 300 BrowseComp questions. Averaged within each seed, LEAD-RL improved over shared-credit RL by 2.33 and 2.67 percentage points and used 126.9 and 108.9 fewer critical-path steps per question, respectively.

MethodTraining seedAccuracy runsAccuracy mean ± SDSteps/question runsSteps/question mean ± SD
Shared-credit RLA70.33% / 72.33% / 71.67%71.44% ± 1.02 pp367.34 / 396.13 / 387.05383.51 ± 14.72
LEAD-RLA73.67% / 73.00% / 74.67%73.78% ± 0.84 pp256.87 / 241.45 / 271.48256.60 ± 15.02
Shared-credit RLB70.67% / 71.67% / 71.00%71.11% ± 0.51 pp389.43 / 412.55 / 336.17379.38 ± 39.17
LEAD-RLB72.67% / 74.67% / 74.00%73.78% ± 1.02 pp248.19 / 275.22 / 287.92270.44 ± 20.29

Standard deviations summarize three evaluations of each fixed checkpoint.

On a 200-question DeepSearchQA subset, LEAD-RL achieves the best F1 score and lowest critical-path cost (Table 2).

Method / Model VariantF1 Score (%)Critical-Path Cost (Steps / Q)
SFT82.7217.3
Shared-Credit RL83.5196.5
LEAD-RL (Ours)86.3147.2

Q5. All-failure rollout groups

The first 30 training updates contain 960 question groups per method. Four-rollout all-failure rates were 48.6% for main-only RL, 32.7% for shared-credit RL, and 31.2% for LEAD-RL. Identical team rewards yield zero group-relative outcome advantage. Thus, all-failure groups provide no outcome-derived advantage to distribute: the contribution gate assigns nonzero team advantages across subagents rather than creating an additional reward. Dataset construction included the 150 questions missed by the SFT model in three screening attempts.

Thanks for your reply, I think those solve my question and concerns. I will maintain my score.

Reviewer_LWPYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper aims to improve how deep-research systems delegate and coordinate web research. A shared language-model policy acts as the main agent or as a subagent depending on its role prompt: the main agent decides what to research, can delegate subtasks, and combines and verifies the returned reports; subagents search independently in isolated contexts and return evidence-based reports. The paper keeps this shared policy jointly trained across both roles: first the Searcher is supervised-fine-tuned, then the main/subagent policy is updated with reinforcement learning. During training, a prompted DeepSeek V4 Pro judge rates each subagent report helpful, neutral, or harmful; those labels affect how the final-answer reward is passed back to subagents, while final answer correctness remains the objective.

LEAD-RL’s contribution-aware credit routing improves this particular deep-research system: on BrowseComp, it beats shared-credit joint RL by 2.49 accuracy points while reducing measured critical-path steps by 24.5%. The paper also reports gains over its standalone Searcher across five benchmarks and two model families. The useful takeaway is that how subagents receive team-level reward can matter, not just whether the main and subagents are trained together

Critical strengths

The experiments provide evidence that contribution-aware joint training improves this system’s results, especially against shared-credit joint RL on BrowseComp, with gains also reported across five benchmarks and two model families. The evidence supports a promising method-specific finding, but not yet a general conclusion that this training strategy is reliably superior across multi-agent research settings.

Critical weaknesses

The main results and training-time contribution labels rely on LLM judges, but the paper does not report human validation of their judgments; the qualitative example illustrates the workflow rather than establishing judge reliability.

Questions for authors

Could you evaluate sensitivity to paraphrased or modified role instructions and delegation briefs, and clarify whether the learned behavior depends on the exact prompts used during training?

3 comments

Thank you for your careful review and constructive feedback. In response to your questions, we conducted a human audit of the subagent contribution labels and tested sensitivity to paraphrased role instructions and delegation briefs.

W1. Human validation of subagent contribution labels

We conducted a human audit of subagent reports with three annotators. The audit summary lists 450 reports per method. The reported human–judge agreement on contribution labels was 94.3% for SFT, 93.7% for shared-credit RL, and 95.1% for LEAD-RL.

In the audited sample, human annotators rated 32.79% of LEAD-RL reports as helpful and 7.67% as harmful, compared with 23.67% helpful and 12.16% harmful for shared-credit RL. This corresponds to 9.12 percentage points more helpful reports and 4.49 percentage points fewer harmful reports for LEAD-RL.

MethodReportsRated helpful by humansRated harmful by humansHuman–judge agreement
SFT45027.55%10.11%94.3%
Shared-credit RL45023.67%12.16%93.7%
LEAD-RL45032.79%7.67%95.1%

The class-wise audit summary lists 300 sampled reports. The percentages below come from that summary.

Human labelJudge: helpfulJudge: neutralJudge: harmful
Helpful97.2%2.6%0.2%
Neutral4.3%92.6%3.1%
Harmful0.0%1.2%98.8%

A label disagreement matters for credit assignment when it changes whether a subagent receives the team advantage. When the team advantage is negative, a helpful label sets the subagent's advantage to zero, while a neutral label preserves it.

The contribution judge uses the question, reference answer, delegation brief, report, and tool evidence. The same model separately grades final-answer correctness to supply the binary team reward. The human audit covered report contributions.

Q1. Sensitivity to role instructions and delegation briefs

The main agent and subagents share policy parameters and receive role-specific instructions. The main agent generates delegation briefs from its current research state. We tested two changes separately. We paraphrased either the role instructions or the generated delegation briefs. The brief paraphrases preserved the delegated objective, facts, and constraints.

Prompt conditionShared-credit RL accuracyShared-credit RL steps/questionLEAD-RL accuracyLEAD-RL steps/question
Original prompts71.00%323.3073.49%244.22
Paraphrased role instructions70.80%344.5073.37%258.90
Paraphrased delegation briefs71.30%357.1074.13%274.10

In the reported aggregates, LEAD-RL had higher accuracy and fewer critical-path steps per question than shared-credit RL under both paraphrases. Relative to the reported original-prompt aggregates, LEAD-RL's accuracy changed by −0.12 percentage points under role paraphrasing and +0.64 percentage points under brief paraphrasing. Its critical path increased by 14.68 and 29.88 steps per question, respectively.

AuthorsOpenReview ↗

Has the rebuttal addressed your concerns?

Hi,

Has our rebuttal addressed your concerns? In particular, we provided a human audit of the contribution labels and additional prompt sensitivity experiments. Would you consider updating your rating, or are there any remaining concerns we could address during this discussion period?

Thank you for your time!

AuthorsOpenReview ↗

Follow-up on the rebuttal

Dear Reviewer,

Thank you again for your feedback. Following up on our point-by-point response and additional experiments, could you let us know before today’s rebuttal deadline which concern, if any, remains unresolved? A brief note would help us focus our reply in the time remaining.

If you find that the new results address your main concerns, we would appreciate it if you could reconsider your score.

Thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.