- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LEAD-RL jointly trains a research agent and its subagents using a shared policy. A training-time judge labels subagent reports as helpful, neutral, or harmful, and these labels determine whether each report inherits the positive or negative team advantage. The paper evaluates the resulting systems on research benchmarks and reports shorter critical paths alongside improved answer accuracy.
Critical strengths
-
The paper identifies a plausible failure of shared team credit: useful research can precede a wrong answer, while a successful team can recover from misleading evidence. Applying a simple gate at the report boundary is well matched to the delegation interface and avoids requiring counterfactual execution of an entire research trajectory.
-
The authors compare against both main-agent-only training and joint training from the same stated initialization. They also distinguish critical-path steps from actual wall-clock speedup. The shorter research paths are interesting precisely because the method does not directly reward shorter execution.
Critical weaknesses
-
[Major] The experiments do not establish that the gate works because its contribution judgments are correct. The judge has access to the reference answer, and its labels selectively suppress policy updates. That is substantive intermediate supervision, even if the scalar team reward remains unchanged. There is no independent audit of these labels, while the reported improvement in helpfulness is measured with the same judging approach used during training. A simple filter that suppresses the same proportion of updates could reveal whether the gain comes from meaningful credit assignment or just less aggressive subagent optimization. Without such a control, the paper demonstrates the usefulness of a training heuristic more clearly than it demonstrates better coordination through contribution assessment.
-
[Major] Too much of the training comparison is left unspecified. The paper does not identify the SFT and RL data sources, their sizes, or how they are separated from the evaluation benchmarks. This is especially consequential because the SFT checkpoint already accounts for most of the reported improvement over the base model. Appendix C also records different training lengths for the RL variants, without clearly tying the selected test checkpoints to a matched update budget. These omissions prevent me from judging the incremental contribution of LEAD-RL with confidence. Please document the data and initialization, then report the controlled comparison at a common training budget with an explicit checkpoint-selection rule.
Questions for authors
See weaknesses.
3 comments
Has the rebuttal addressed your concerns?
Hi,
Has our rebuttal addressed your concerns? In particular, we added a count-matched random-gating control and clarified the training data and matched-checkpoint comparisons. Would you consider updating your rating, or are there any remaining concerns we could address during this discussion period?
Thank you for your time!
Follow-up on the rebuttal
Dear Reviewer,
Thank you again for your review. We added experimental results and implementation details in response to your comments. If a key issue remains unresolved, could you point us to it before today’s rebuttal deadline so we have a chance to respond?
If you feel the additional results address your main concerns, we hope you’ll revisit your score in light of those findings.
Thank you for considering our response.
Thank you for your careful review and for identifying the need for a matched control and a clearer account of the training comparison.
Major 1. Count-matched random gating and intermediate supervision
We added a count-matched random-gating control to test whether the choice of subagent trajectories matters. For each rollout sampled by the control policy, we computed how many eligible subagent trajectories the contribution gate would mask. We then sampled that many eligible trajectories uniformly at random and set their inherited team advantages to zero. The main-agent advantage rule, subagent normalization rule, and configured weights were the same. Rollouts with zero team advantage required no additional masking.
Relative to random gating, LEAD-RL gained 2.16 percentage points in accuracy and used 30.1% fewer critical-path steps in the reported comparison. This result favors contribution-based selection in these runs. Mask counts are matched to the contribution rule on each rollout sampled by the random-gating policy. The count-matched control is designed to match the masking budget while changing only which eligible trajectories are masked at each gating decision.
We clarify that the contribution judge provides reference-guided credit assignment rather than an additional intermediate reward. During training, it uses the reference answer to label subagent reports; these labels determine whether the outcome-derived team advantage is assigned to each subagent. Final-answer correctness remains the sole scalar team reward. Thus, the judge shapes how the outcome-derived reward is distributed across subagents, rather than adding a separate scalar reward signal. Neither the reference answers nor the judge’s outputs are used at inference.
Three annotators took part in a human audit of the contribution labels. The audit summary lists 450 reports per method and human–judge agreement of 95.1% for LEAD-RL and 93.7% for shared-credit RL. In the audited sample, human annotators rated 32.79% of LEAD-RL reports as helpful and 7.67% as harmful. The corresponding rates for shared-credit RL were 23.67% and 12.16%.
Major 2. Matched training updates and gains after SFT
The cold-start SFT stage utilized approximately 7,000 trajectories sampled from an open-source model. All three RL variants in Table 3 are initialized from the same SFT checkpoint and trained using the identical set of 500 internally synthesized RL questions under a matched configuration schedule. Each update step processes 32 questions with four rollouts per question. Table 3 evaluates all three variants at step 30, ensuring a fair comparison under equal update steps and sampled question groups. This common evaluation point was chosen by jointly balancing accuracy and critical-path cost on a fixed validation set of roughly 300 held-out synthetic questions. Appendix C reports the full training trajectories—35, 34, and 60 updates—all of which extend beyond the step-30 checkpoint used in Table 3.
Relative to SFT, LEAD-RL improved accuracy by 0.99 percentage points and reduced critical-path cost from 403.37 to 244.22 steps per question, a 39.5% reduction. Relative to shared-credit RL, LEAD-RL improved accuracy by 2.49 percentage points and reduced critical-path cost from 323.30 to 244.22 steps per question, a 24.5% reduction.