acceptodds
Under review as a conference paper at ICLR 2027

OARS: On Post-Training for Cost-Efficient Heterogeneous Multi-Agent Systems

Abstract

Frontier large language models (LLMs) are powerful agentic problem solvers, but deploying them at every step of a long-horizon task incurs substantial inference cost. Such tasks involve many rounds of LLM reasoning and environment interaction, yet not every step requires the same level of model capability. We introduce OARS (Orchestrator-Aware Role Specialization), which reduces reliance on expensive large models by organizing specialized small models to perform delegated work. Specifically, a frozen large-model orchestrator decomposes and delegates tasks to small-model workers through a unified work-order protocol that governs information isolation and exchange. Workers return only concise, task-relevant findings rather than their full execution histories, thereby reducing the orchestrator's computational workload and context growth. We post-train small-model workers within the complete multi-agent system. A central challenge is the *free-rider problem*: workers can be rewarded for collective success regardless of their own contribution, as a strong orchestrator or other workers may compensate for unhelpful worker reports. We address this with contribution-aware training that uses worker-level progress rewards to distinguish individual contributions from collective success. Our hierarchical credit normalization further prevents episodes with more workers or longer trajectories from receiving disproportionate training weight. On BrowseComp-Plus, OARS—with 8B workers assisting 230–550B-parameter models—improves accuracy by 6.3 percentage points on average while being up to 10.6× more cost-efficient than standalone large-model agents. On DR-256, OARS improves accuracy by 19.9 percentage points on average while achieving up to a 4.2× reduction in cost per correct answer. OARS further extends to agentic coding, cutting cost per resolved task by 66.5% on SWE-bench Verified while maintaining accuracy. We will release the code upon publication.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Efficient LLMs, LLM agents, multi-agent systems, LLM post-training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_37YDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes to train multi-agent systems where there is a large-model orchestrator and multiple small-model workers and addresses the credit assignment problem of the workers (called free-rider problem in the paper) during training. Their proposed solution to the credit assignment problem is to use an LLM judge to determine the contribution of each worker. They also included a normalization deisgn to handle the differences in the number of workers in different tasks, which would gave greater influence to episodes with more workers. Experiments shows that the proposed method has better accuracy and efficiency compared to the baselines in the paper.

Critical strengths
  • The paper is easy to follow and well-written.
  • The empirical results are strong, notably improve the accuracy-cost tradeoffs across mutliple orchestrator LLM and benchmarks.
Critical weaknesses
  • My main concern is the lack of credit-assignment baselines under the same training and evaluation setups. While several methods is included in Table 3, they are end-to-end system comparisons, not evaluated under the same setup to demonstrate the effectiveness of the proposed training algorithm.
  • The training cost of the proposed method is missing. The application of an LLM judge to determine contribution of workers could be expensive, especially given the long-context of multi-agent systems.
Questions for authors
  • Could the authors compare against training algorithms with credit assignment designs under identical setups, instead of merely GRPO? This should make the paper much stronger. I am particularly interested in comparing LLM-based credit assignment with value-critic-based methods such as PPO.
  • What is the cost of the Qwen3-8B workers baseline? Its accuracy is reported in table 1 but excluded in table 2.

I am willing to raise the rating if my concerns are addressed.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 37YD

We thank the reviewer for the constructive questions and address each of them below.

W1 & Q1 (credit-assignment baselines). As suggested, we compared OARS with PPO using a value critic, under the same setup as Table 5, where only the credit assignment changes. PPO uses the same SFT initialization, orchestrator and training settings, including data, batch size, learning rate, KL, staleness tolerance and number of training steps. In PPO, each worker trajectory receives its episode's task reward, and a critic, a second 8B model initialized from the same SFT checkpoint, assigns per-token credit through GAE.

Training signal (SFT initialization)BrowseComp-Plus (%)
No RL73.1
Task reward only (GRPO)72.0
+ progress reward75.1
OARS78.7
PPO with a value critic75.1

Since a critic can be sensitive to its schedule, we also tried two other settings. One trains the critic alone for 30 steps instead of 5 before policy updates begin (a longer warm-up), so that the critic can be more accurate early on. The other also raises the critic learning rate from 1e-5 to 3e-5, so that it keeps up with the changing policy. The results are as follows:

PPO critic settingBrowseComp-Plus (%)
Warm-up 5 steps, critic lr 1e-575.1
Warm-up 30 steps, critic lr 1e-574.9
Warm-up 30 steps, critic lr 3e-571.6

In PPO, the critic is trained on the sparse task reward, which is shared by every worker in the same episode. The critic therefore gets indirect signals about each worker's contribution, so it can be more challenging to separate helpful from unhelpful workers.

W2 (training cost). The progress judge adds little to the cost of training: on GB300 GPUs, RL with the judge takes 275 GPU-hours versus 272 with the task reward only (both including serving the orchestrator), plus $388 in judge API cost (Gemini 3.6 Flash).

RL with task reward only (GRPO)RL with progress judge (OARS)
GPU-hours272275
Time per update377 s382 s (+1.3%)
Judge API cost—$388

Each judge call stays relatively short despite the long multi-agent context, as it scores one worker at a time: 6.7K input and 135 output tokens on average. Moreover, a cheaper judge also works: our experiments show that RL with DeepSeek-V4-Flash as the judge reaches nearly the same reward curve, and would cut the judge cost to about $219.

Q2 (cost of the Qwen3-8B-workers row). Untrained Qwen3-8B workers cost 1.1–2.2× as much per correct answer as OARS workers, as the orchestrator needs 1.2–1.5× more turns per task to make up for the untrained workers and still solves fewer tasks. We computed this row exactly like the published rows of Table 2 (cost per correct answer, ¢):

OrchestratorBrowseComp-Plus: Qwen3-8B workersBrowseComp-Plus: OARSDR-256: Qwen3-8B workersDR-256: OARS
MiniMax M2.72.41.26.22.8
DeepSeek V4 Flash1.91.43.22.8
Nemotron Ultra5.94.24.54.1

We have added the corresponding results to the revised paper.

Dear Reviewer 37YD,

Thank you again for your thoughtful review and questions. We have posted our response and updated the manuscript, with all revisions highlighted in blue.

We would be grateful if you could review our response and reconsider your assessment. Please let us know if any remaining questions or points would benefit from further clarification!

Thank you for the additional results. My main concern is mostly resolved. So I am raising my rating to 3. Good luck!

Thank you very much for taking the time to review our response and update your assessment. We greatly appreciate your support!

Reviewer_eVzeOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

OARS combines a frozen large-model orchestrator with small-model workers that execute delegated tasks and return concise reports. Workers are post-trained using task-level rewards, judge-based contribution rewards, and hierarchical credit normalization. Experiments on research and coding benchmarks demonstrate improved accuracy–cost trade-offs.

Critical strengths

The paper addresses a practically important problem: reducing the inference cost of long-horizon agents.

Critical weaknesses

The evidence for addressing free-riding is incomplete. The proposed reward evaluates report usefulness and grounding, which need not reflect a worker’s marginal contribution to collective success. A correct, well-supported report can still duplicate information already available to the orchestrator. The described judge inputs do not clearly account for other workers’ reports or the orchestrator’s existing knowledge. Moreover, combining this score with the shared task reward can still assign positive advantages to non-contributing workers. Table 6 shows a negative average advantage for free-riders, but does not establish how reliably the reward identifies them or whether training reduces their prevalence.

Questions for authors

Does the contribution judge observe other workers’ reports and the orchestrator’s prior information? How does it distinguish useful verification from redundant findings?

What fraction of identified free-riders still receive positive advantages, and does their prevalence decrease during training? How well do the judge scores agree with counterfactual contribution measurements?

3 comments
AuthorsOpenReview ↗

Response to Reviewer eVze

We thank the reviewer for the detailed questions. We first clarify our reward design, which bears on several of the concerns, and then respond to each concern separately.

Our design. We would like to first clarify some aspects of our design that underlie several of the concerns.

  • Together, the judge and the final task reward provide a practical training signal aligned with a worker's marginal contribution to collective success. The task reward tells whether the team succeeded, and the judge, which knows each worker's work order and the reference answer, tells how much each report moved the team toward that answer. Counterfactual replay confirms that OARS assigns the most credit to workers with the largest contributions (Point 1).
  • Constructive vs. free-rider. A report that provides useful findings or verification for its work order is constructive, even if it repeats a previous worker's finding to verify or extend it. One that provides neither, e.g., irrelevant content, is a free-rider and receives a low score (as defined in Table 6 of the paper).
  • The goal of training is a worker that reliably carries out the sub-task it is given. Each worker sees only its own work order, which keeps its context short and its goal clean. The reward therefore credits each worker for doing its own sub-task well. Deciding what to delegate, including whether to re-check earlier findings, is the orchestrator's job (point 2).

With this design in mind, we respond to remaining concerns below.

1. Credit vs. a worker's marginal (counterfactual) contribution. Our credit assignment is consistent with marginal contribution. We measured each worker's counterfactual contribution by its Shapley value: For a set SS of workers in an episode, let v(S)=1v(S)=1 if the orchestrator, replaying its final decision with only the reports of the workers in SS, answers correctly, and v(S)=0v(S)=0 otherwise. The Shapley value ϕi\phi_i of worker ii is its marginal gain averaged over all orders π\pi in which the reports could be added:

ϕi=Eπ[v(Piπ∪{i})−v(Piπ)]\phi_i = \mathbb{E}_{\pi}\left[ v\left(P_i^{\pi} \cup \{i\}\right) - v\left(P_i^{\pi}\right) \right]

where PiπP_i^{\pi} is the set of workers before ii in π\pi. The values of an episode sum to v(N)−v(∅)v(N)-v(\emptyset), with NN the set of all its workers, so they split the credit for solving the task; workers that report the same decisive fact share it, instead of each receiving zero as under leave-one-out. The mean Shapley value by worker category (computed from 50,244 replays with different combinations of workers' reports) is consistent with our credit assignment in Table 6:

Worker categoryMean Shapley value
Pivotal0.89
Constructive0.27
Free-rider0.08

Additionally, the judge's score separates constructive from free-rider calls well: we verified that a randomly chosen constructive call receives a higher score than a randomly chosen free-rider call 87% of the time (AUROC = 0.87).

2. The judge's inputs & Redundancy. In OARS, the judge sees the work order, the worker's own observations and report, and the reference answer. The orchestrator's prior information enters through the work order, while other workers' reports are excluded. Workers rarely repeat findings without being asked: 99.3% of restating reports are aimed to answer the orchestrator's re-check request, and these reports have a mean Shapley value of 0.36.

We also demonstrate that showing the judge more does not improve the credit assignment, which is consistent with our goal of worker training. Specifically, a judge variant that also sees the orchestrator's history up to the report, including earlier workers' reports, can recognize repetition, but it does not agree better with the Shapley values. We take the pairs of workers in the same episode that the judge scores differently, and check whether the judge scores them in the same order as their Shapley values. Our judge does so for 82.4% of pairs, and this variant for 81.6%.

3. Free-riders under the shared task reward. Our reward largely neutralizes the positive rewards for free-riders: in a successful episode the shared task reward is positive for every worker, but the judge score turns most free-riders' advantage negative.

Free-rider StatisticsTask reward onlyOARS
Mean advantage+0.50−0.29
With a negative advantage3.4%75.9%
Share of the total positive advantage48.4%4.9%

While 24.1% of free-riders receive a positive advantage, they account for only 4.9% of the total positive advantage, versus 48.4% with the task reward only.

AuthorsOpenReview ↗

Response to Reviewer eVze

4. Reliability of free-rider identification and frequency of free-riding during training. The judge identifies free-riders reliably (AUROC 0.87; point 1), and RL with our reward reduces free-riding while the system solves more tasks with fewer worker calls:

BrowseComp-Plus, M2.7 orchestratorSFT (before RL)OARS (after RL)
Accuracy (%)73.178.7
Worker calls per episode6.215.82
Free-riders per solved episode1.791.40
Tasks solved per 100 worker calls11.813.5

RL further reduces free-riding on the questions both workers solve, the number of free-riders per solved episode falls from 1.79 to 1.40.

We have added the corresponding results to the revised paper.

Dear Reviewer eVze,

Thank you again for your thoughtful review and questions. We have posted our rebuttal and revised the manuscript accordingly, with all changes highlighted in blue.

We would greatly appreciate it if you could review our rebuttal and reconsider your assessment. Please let us know if any further information or clarification would be helpful!

Reviewer_BxjrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces OARS, a framework for reducing the inference cost of complex LLM agent tasks. A large, frozen model serves as an orchestrator that plans tasks and delegates work to smaller models. Workers interact with tools and return concise findings with supporting evidence, while keeping their full execution histories outside the orchestrator’s context. This reduces the amount of information the expensive model must process. In this work, the small workers are post-trained within the complete multi-agent system so they learn to provide information useful for coordination and final decision-making. The main technical contribution is a contribution-aware training method that addresses the “free-rider problem” in multi-agent reinforcement learning. When rewards depend only on the final answer, an ineffective worker can receive positive reinforcement because another worker compensates for its mistakes. OARS combines task-level success rewards with individual progress rewards that assess whether each worker’s report advances the task and is grounded in observed evidence. It also introduces hierarchical credit normalization, preventing episodes with more workers or longer trajectories from receiving disproportionate training weight. Experimental results show average accuracy improvements over standalone large models of 6.3 percentage points on BrowseComp-Plus and 19.9 points on DR-256. Ablation study shows that under the same orchestrator, supervised worker training achieves 73.1% accuracy, task-reward-only reinforcement learning achieves 72.0%, and the full OARS method achieves 78.7%, giving clear evidence for the design.

Given the author response, I will keep the score unchanged.

Critical strengths
  • The overall design of the proposed system is intuitive and reasonable to me. Workers keep their detailed tool interactions locally and return concise, evidence-backed reports (though I would say this pattern is quite common these days). This directly addresses the expense of repeatedly feeding long histories into the large model.
  • I like the ablation study and it is informative. Supervised fine-tuning raises accuracy from 56.4% to 73.1%, and the proposed reinforcement learning raises it to 78.7%, showing the effectiveness of the proposed method. The additional gain on hard questions is also interesting.
  • It's also good to see that the model trained on BrowseComp-Plus can be transferred to DR-256 (but I still need to say the question types are pretty much the same)
Critical weaknesses
  • [resolved in rebuttal] I found much of the improvement comes from SFT. The system accuracy increases from 56.4% with untrained workers to 73.1% with SFT workers, followed by a further increase to 78.7% after RL. These results leave open how much improvement reflects general search and tool-use ability versus learning to contribute within an orchestrated system. I am thinking if we directly train a 8B model from M2.7 sole trajectory as ablation experiment. Also what's the performance of OARS-trained worker alone? Is collaboration pattern helpful, or the workers are good enough?
  • I don't have other major concerns of this work.
Questions for authors
  • [provided in rebuttal] I personally would like to see use Claude and GPT as orchestrator and know the landscape of effectiveness/performance.
2 comments
AuthorsOpenReview ↗

Response to Reviewer Bxjr

We thank the reviewer for the constructive questions and address each of them below.

Q1 (Source of Gain: general search ability vs. learning to contribute). To separate general search ability from learning to contribute, we evaluate each worker both with the M2.7 orchestrator, as in the paper, and without an orchestrator, where it answers each question by itself. We also ran the ablation the reviewer suggested: we trained an 8B model from M2.7 solo trajectories, keeping the SFT recipe and the supervised-token budget fixed (3.93M tokens each), which were generated on the same 680 BrowseComp-Plus training questions. The evaluation results are as follows:

WorkerWith orchestratorWithout orchestrator
Qwen3-8B56.423.3
SFT on M2.7 solo trajectories72.246.7
SFT on our work-order trajectories73.152.7
+ contribution-aware RL (OARS)78.752.9

The SFT gain mainly reflects general search and tool-use ability, and the RL gain reflects learning to contribute within the orchestrated system. Specifically, SFT adapts the worker to the overall task setting, improving it both with and without an orchestrator, while RL further improves accuracy in the orchestration mode.

In addition, the ablation demonstrates that our work-order trajectories are better SFT data than M2.7's solo trajectories across different evaluation settings. In particular, the workers trained on M2.7 solo trajectories fail to generate answer reports more frequently: no answer on 12.4% of questions without an orchestrator (1.6% for workers trained on our SFT data), and no answer on 7.1% of work orders with the orchestrator (0.7% for workers trained on our SFT data). We attribute this to the fact that each of our trajectories covers a scoped sub-task, which is easier for a small worker to learn than M2.7's trajectory for the whole task.

Finally, the trained workers are not good enough on their own without collaboration, and this observation holds across different workers. For example, the OARS worker reaches 52.9% on BrowseComp-Plus without an orchestrator versus 78.7% with it, and the gain is largest on the Hard split (17.3% vs 51.3%).

Q2 (Claude / GPT as orchestrator). We replaced the M2.7 orchestrator with Claude Sonnet 5.5 and GPT-6 Sol, keeping the OARS workers (trained with M2.7), tasks, tools, and harness unchanged. The baseline is the same closed model solving each task alone with the same search tools (Solo).

OrchestratorBenchmarkSolo (%)+ OARS workers (%)Cost per correct answer, Solo → OARS
Claude Sonnet 5.5BrowseComp-Plus82.086.7$0.267 → $0.077 (−71%)
GPT-6 SolBrowseComp-Plus83.187.8$0.405 → $0.050 (−88%)
Claude Sonnet 5.5DR-25656.860.0$0.316 → $0.093 (−70%)
GPT-6 SolDR-25662.961.9$0.366 → $0.074 (−80%)

The OARS workers transfer to these more powerful closed-source orchestrators without retraining, cutting the cost per correct answer by 70–88% while matching or exceeding the corresponding Solo accuracy. The large model reads short worker reports instead of raw pages, so its input tokens per task drop by 93–96%, e.g., from 348K to 26K for Claude and from 834K to 31K for GPT on BrowseComp-Plus.

We have added the corresponding results to the appendix of the revised paper.

Dear Reviewer Bxjr,

We sincerely appreciate your recognition of our work and your thoughtful feedback. We have posted additional results and analyses in our rebuttal and revised the manuscript accordingly, with all changes highlighted in blue. We would be happy to provide any further clarification if helpful!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.