- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes to train multi-agent systems where there is a large-model orchestrator and multiple small-model workers and addresses the credit assignment problem of the workers (called free-rider problem in the paper) during training. Their proposed solution to the credit assignment problem is to use an LLM judge to determine the contribution of each worker. They also included a normalization deisgn to handle the differences in the number of workers in different tasks, which would gave greater influence to episodes with more workers. Experiments shows that the proposed method has better accuracy and efficiency compared to the baselines in the paper.
Critical strengths
- The paper is easy to follow and well-written.
- The empirical results are strong, notably improve the accuracy-cost tradeoffs across mutliple orchestrator LLM and benchmarks.
Critical weaknesses
- My main concern is the lack of credit-assignment baselines under the same training and evaluation setups. While several methods is included in Table 3, they are end-to-end system comparisons, not evaluated under the same setup to demonstrate the effectiveness of the proposed training algorithm.
- The training cost of the proposed method is missing. The application of an LLM judge to determine contribution of workers could be expensive, especially given the long-context of multi-agent systems.
Questions for authors
- Could the authors compare against training algorithms with credit assignment designs under identical setups, instead of merely GRPO? This should make the paper much stronger. I am particularly interested in comparing LLM-based credit assignment with value-critic-based methods such as PPO.
- What is the cost of the Qwen3-8B workers baseline? Its accuracy is reported in table 1 but excluded in table 2.
I am willing to raise the rating if my concerns are addressed.
4 comments
Dear Reviewer 37YD,
Thank you again for your thoughtful review and questions. We have posted our response and updated the manuscript, with all revisions highlighted in blue.
We would be grateful if you could review our response and reconsider your assessment. Please let us know if any remaining questions or points would benefit from further clarification!
Thank you for the additional results. My main concern is mostly resolved. So I am raising my rating to 3. Good luck!
Thank you very much for taking the time to review our response and update your assessment. We greatly appreciate your support!
Response to Reviewer 37YD
We thank the reviewer for the constructive questions and address each of them below.
W1 & Q1 (credit-assignment baselines). As suggested, we compared OARS with PPO using a value critic, under the same setup as Table 5, where only the credit assignment changes. PPO uses the same SFT initialization, orchestrator and training settings, including data, batch size, learning rate, KL, staleness tolerance and number of training steps. In PPO, each worker trajectory receives its episode's task reward, and a critic, a second 8B model initialized from the same SFT checkpoint, assigns per-token credit through GAE.
Since a critic can be sensitive to its schedule, we also tried two other settings. One trains the critic alone for 30 steps instead of 5 before policy updates begin (a longer warm-up), so that the critic can be more accurate early on. The other also raises the critic learning rate from 1e-5 to 3e-5, so that it keeps up with the changing policy. The results are as follows:
In PPO, the critic is trained on the sparse task reward, which is shared by every worker in the same episode. The critic therefore gets indirect signals about each worker's contribution, so it can be more challenging to separate helpful from unhelpful workers.
W2 (training cost). The progress judge adds little to the cost of training: on GB300 GPUs, RL with the judge takes 275 GPU-hours versus 272 with the task reward only (both including serving the orchestrator), plus $388 in judge API cost (Gemini 3.6 Flash).
Each judge call stays relatively short despite the long multi-agent context, as it scores one worker at a time: 6.7K input and 135 output tokens on average. Moreover, a cheaper judge also works: our experiments show that RL with DeepSeek-V4-Flash as the judge reaches nearly the same reward curve, and would cut the judge cost to about $219.
Q2 (cost of the Qwen3-8B-workers row). Untrained Qwen3-8B workers cost 1.1–2.2× as much per correct answer as OARS workers, as the orchestrator needs 1.2–1.5× more turns per task to make up for the untrained workers and still solves fewer tasks. We computed this row exactly like the published rows of Table 2 (cost per correct answer, ¢):
We have added the corresponding results to the revised paper.