acceptodds
Under review as a conference paper at ICLR 2027

Learning Self-Orchestration in Decentralized Agent Swarms

Abstract

The capabilities of an agent swarm depend on both its individual members and how they coordinate. Yet existing multi-agent systems often train agents within fixed orchestration structures, separating the learning of individual capabilities from the organization of collective behavior. We introduce market-based post-training (MPT), a framework for jointly learning action policies and self-orchestration in decentralized agent swarms. Agents express their willingness to participate through bids, which determine who acts and how collective outcomes provide local training credit. Learning both action and bidding behavior allows individual capabilities and patterns of participation to adapt together without a central coordinating agent. We evaluate MPT on embodied crafting, robot navigation, and deep research. Results show gains over monolithic post-training and confirm that agent participation indeed evolves with learning. With the same post-trained action policies, learned self-orchestration outperforms alternative coordination strategies. Ablations show that both action and bidding benefit from learning, and that learned bids provide more effective action-training signals. Beyond the original training objectives, trained swarms adapt faster to related tasks and achieve better performance with larger populations at inference time. Larger populations also suffer smaller performance losses under agent dropout, demonstrating that learned self-orchestration supports reuse, scaling, and robustness in decentralized agent swarms.

Primary area:
foundation or frontier models, including LLMs
Keywords:
decentralized agent swarms, self-orchestration, credit assignment

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_sknTOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Market-based Post-Training (MPT) for jointly learning the action policies and turn allocation of specialized LLM agents. At each turn, all agents submit a discrete bid and the highest bidder acts. The winning bid also controls how much downstream value is passed to the previous actor. Actions are trained with bid-dependent credit and a per-agent baseline. Bids are trained by restoring the same state and comparing the original winner with the runner-up. Experiments cover Minecraft crafting, robot navigation, and deep research. MPT outperforms the authors’ monolithic RL baseline on all three held-out sets. The paper also includes coordination comparisons on crafting, checkpoint swaps, ablations, and preliminary studies of reuse, scaling, and dropout.

Critical strengths
  • The paper presents an interesting and appealing way to think about multi-agent coordination. Turning participation into a learned bidding process gives a simple interface for deciding who should act as agents’ capabilities change during training. Connecting the same signal to cross-agent credit assignment is an original and intuitive design choice, and the counterfactual bidding update provides a sensible way to train it. The individual tools are familiar RL techniques, but the overall formulation is thoughtful and nontrivial.

  • The method is tested in three different settings. Table 2 fixes the post-trained action policies when comparing coordination strategies, while the checkpoint-swap and random-credit experiments suggest that both action learning and bidding matter. These experiments are incomplete, but they provide useful evidence that the bidding mechanism contributes beyond role specialization alone.

Critical weaknesses
  • The method assigns two different meanings to the same bid, but the learning objective does not show that they should coincide. A high bid determines both that an agent should act and that a larger share of its downstream value should be passed to the previous actor. Yet the counterfactual bidding objective only tests whether changing the acting agent improves the branch return; it does not test whether the bid is a calibrated credit-sharing coefficient. The random-credit ablation shows that learned bids are better than random shares, but it does not establish that coupling allocation and credit through one scalar is necessary or preferable to learning them separately. This matters because the coupling is the main conceptual distinction of MPT.

  • The baseline coverage is not sufficient to identify where the gains come from. The main comparison changes role specialization, model parameterization, turn allocation, credit assignment, and the bidding objective at the same time. Table 2 helps, but it is limited to crafting, and the implementation and tuning of AgentNet and GoA are unclear. The paper lacks a state-aware handcrafted router and a learned centralized router using the same action policies and inference budget. It is therefore difficult to tell whether the gains come from specialization, learned routing in general, or decentralized bidding specifically.

Questions for authors
  • Can the authors explain or empirically test why turn allocation and credit sharing should use the same scalar bid? A particularly informative ablation would compare the current coupled design with separate learned outputs for participation and credit assignment, while holding the action policies fixed.

  • Could the authors compare MPT with a state-aware handcrafted router and a learned centralized router using the same specialized action policies? Extending this comparison beyond crafting would clarify whether MPT learns coordination beyond the role and action-space structure provided by the task designer.

  • Could the authors report the number of independent RL runs and seed-level uncertainty for the main results, together with standard navigation metrics and a robustness check for the deep-research judge?

4 comments
AuthorsOpenReview ↗

[1/2] Official Comment by Authors

We thank the reviewer for recognizing our formulation as a thoughtful and nontrivial way to think about multi-agent coordination. We are also grateful that the reviewer acknowledged our evaluation in three different settings and found that the coordination comparison provides useful evidence that the bidding mechanism contributes beyond role specialization alone. We will address your concerns in the following response.

W1 & Q1

The method assigns two different meanings to the same bid, but the learning objective does not show that they should coincide. Can the authors explain or empirically test why turn allocation and credit sharing should use the same scalar bid? A particularly informative ablation would compare the current coupled design with separate learned outputs for participation and credit assignment.

Thank you for this insightful question! Using the same bid for turn allocation and credit sharing is a deliberate design that creates an economic incentive for meaningful participation.

If participation were decoupled from credit, bidding high would come at no cost: every agent could simply bid the maximum value to win more turns. In MPT, a bid is also a price. An agent that bids higher is more likely to act, but must pass a larger share of the value it obtains back to the preceding agent. An agent therefore benefits from bidding high only when it expects its action to create enough value to cover this price. This ties each agent's willingness to participate to its expected contribution, so that the bid that allocates the turn is also a natural measure of how much credit should flow to the agent. This follows the principle of the bucket brigade in economic credit assignment. We have added this explanation to Section 2.1.

Following your suggestion, we add an ablation that decouples participation from credit assignment. We note that once credit assignment is decoupled from participation, a separately learned credit value would have no training objective. If each agent learned its credit value to maximize its own credit, it would simply pass nothing back to earlier agents. Therefore, in this ablation, bidding and turn allocation remain unchanged, while credit assignment follows the common practice in multi-agent RL, where all acting agents share the common team reward. As shown by Episode-Reward Credit in Figure 7, under matched training FLOPs, this decoupled variant performs much worse than MPT, reaching a final mean reward of only about 18, compared with about 41 for MPT.

W2 & Q2

The baseline coverage is not sufficient to identify where the gains come from. The paper lacks a state-aware handcrafted router and a learned centralized router using the same action policies and inference budget. Extending this comparison beyond crafting would clarify whether MPT learns coordination beyond the role and action-space structure provided by the task designer.

Thank you for the suggestion! Following it, we add two controls on all three tasks. Both keep the same three role models, role-specific SFT initialization, and action partitions as MPT, and differ from MPT only in how turns are allocated:

(1) Learned Centralized Router: learned bidding is replaced with a centralized router that is trained with RL to select the acting agent at each turn.

(2) Fixed Router: learned bidding is replaced with a fixed Round-Robin policy.

As shown in the updated Figure 5, under matched training FLOPs, both controls perform substantially worse than MPT on all three tasks. Since these controls share the same roles and action-space structure, these results show that MPT learns coordination beyond this structure.

In addition, we compare MPT with the monolithic baseline and routed baselines under matched inference FLOPs, where all routed baselines use the same post-trained action policies as MPT. The table below reports each method's performance at the final matched inference budget on Iron Pickaxe (Appendix D):

MethodMean Reward
Single-agent RL49.4 ± 5.5
Round-robin52.0 ± 5.0
AgentNet53.9 ± 5.9
Graph-of-Agents25.4 ± 4.3
MPT68.8 ± 4.3

At the same inference budget, MPT clearly outperforms both the monolithic baseline and all routed baselines, including the state-aware routers AgentNet and Graph-of-Agents.

AuthorsOpenReview ↗

[2/2] Official Comment by Authors

Q3

Could the authors report the number of independent RL runs and seed-level uncertainty for the main results, standard navigation metrics and a robustness check for the deep-research judge?

Thank you for the questions!

(1) Each training curve in the main results comes from a single run. Multi-agent LLM post-training is very resource-intensive, especially on the long-horizon tasks we study, so under limited compute we prioritized covering a broader range of benchmarks over running multiple seeds per benchmark. Nevertheless, we post-trained both MPT and monolithic RL with two independent seeds on the principal crafting experiment (Appendix E). The variation across seeds is small. For example, at around 3.5-4.0 × 10¹⁸ training FLOPs, the two independent runs differ by about 5% for both MPT and monolithic RL, which is much smaller than MPT's roughly 2× improvement over monolithic RL.

(2) Standard navigation metrics. On the same R2R-CE val-unseen episodes, we evaluate monolithic RL and MPT, both before and after post-training.

ModelSR (%) ↑SPL (%) ↑NE (m) ↓OSR (%) ↑
Single-agent RL initialization0.00.04.610.0
Single-agent RL4.54.54.394.5
MPT initialization4.54.54.354.5
MPT9.18.34.2413.6

MPT achieves the best performance on the four metrics. Due to limited compute, these results come from relatively short post-training runs, and we expect these metrics to improve further with more training compute.

(3) Robustness of the deep-research judge. We graded 1,000 deep-research answers with four different LLM judges (Qwen3.5-Flash, Qwen3.8-Flash, gpt-oss-120b, and DeepSeek-V4-Pro), and also graded every answer five times with our judge, Qwen3.5-Flash (Appendix F).

ComparisonAgreement (%)Full Agreement (%)Fleiss' κ
Four different judges97.495.20.933
Five repeated gradings98.797.20.965

Agreement is the mean pairwise agreement rate, and Full Agreement is the fraction of answers on which all judges or all gradings reach the same verdict. Both across different judges and across repeated gradings, the judgments are highly consistent. Moreover, MPT consistently outperforms monolithic RL. This shows that our deep-research results are robust to the choice of judge and to the randomness of judging.

Thank you for the response. My questions have been answered and the points I was unsure about are now clear. I will keep my score as it is. I appreciate the time you took to reply.

Thank you very much for your positive feedback. We are glad that our responses have addressed your questions and clarified the points you were unsure about. We sincerely appreciate your support and the constructive review.

Reviewer_JhxKOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper essentially presents a solution for “orchestrating” different agents to carry out a given task. The authors evaluate the approach on the three specific tasks: embodied crafting (based on Minecraft), robot navigation and deep research. However, the results are essentially presented only on the first one.

Critical strengths
  • The problem of orchestrating LLM agents is indeed very interesting.
Critical weaknesses
  • The authors say that the experimental set-up is based on 3 environments, but in actual fact, the experimental results are only about the “embodied crafting” task. In other words, the validation of the proposed approach is limited. In the Appendix, the authors only present some of the prompts used for the other tasks. This discussion is overall quite confusing, because it appears as if the experiments have been performed in very complex environments such as “Deep Research”, but in actual fact there are no results presented in the paper. By the way, I do not know if the authors are preparing these results while the paper is under submission. I personally think that it would not be ok to show these results during the rebuttal (in fact, it would be like getting more time to run these results getting a sort of extra time to prepare this paper). The only results presented by the authors are in Figure 5 (which is not really insightful in my opinion).

  • The mechanism used for training is unclear. It is initially presented as a sort of RL from verified rewards. But then the definition of the loss suggests a preference-based method. And then, in actual fact, an SFT mechanism is used. Overall, the presentation of the training process is not completely clear. Key details are missing, such as how in practice these mechanisms are implemented (for example, how SFT is implemented in practice given the rewards). The reviewer does not understand why the mechanism based on counterfactuals is needed. Its complexity is not fully justified in the paper (theoretically and experimentally). In fact, at the end, the training is local and an agent should just understand if the bid was correct or not in that situation. In addition, if in any case there is a full roll-out, it would just sufficient to give an actual scaled reward to each of them (or just 1 and -1 to the others). In fact, by doing the full roll-out, the correct answer is actually known. I was expecting a comparison with a centralized solution based on RL, a classic mixture of experts. In fact, essentially, the proposed solution is centralized: the calculation of the “counterfactuals” reward indirectly assumes some form of aggregation. A local aggregation of the reward would actually be inefficient.

Questions for authors
  1. The authors should provide a clear motivation for the proposed solution. Why do you need a decentralized solution, when a centralized solution would be much simpler and most probably more efficient? Related to this point, can you provide an estimation of the actual advantage of having a decentralized solution vs a centralized one?

  2. Can you please describe the actual mechanism that is used for training in detail? In fact it seems to me that in Section 2.1 the authors describe a mechanism that is essentially RL from verified rewards. But then actually the loss function itself is a preference-based fine-tuning. But then in actual fact, the authors use SFT for training, which appears to be disjoint from the two above. Please provide technical details supporting your answer.

  3. As discussed above, the reviewer is not fully convinced by the actual need for having counterfactuals. In fact, a set of rewards would be sufficient. At the end, the problem is selecting an “expert” from a set of experts.

  4. Can you please provide more details about the idea of having a “decision step”? How do you calculate the best time for having it? Also, please provide details about what the behavior in the plots of Figure 6 (what is the effect of the decision step)?

9 comments
AuthorsOpenReview ↗

[1/3] Official Comment by Authors

We thank the reviewer for recognizing that orchestrating LLM agents is a very interesting problem, and for the detailed questions on our training mechanism and design choices. We will address your concerns in the following response.

W1

The authors say that the experimental set-up is based on 3 environments, but in actual fact, the experimental results are only about the "embodied crafting" task. In other words, the validation of the proposed approach is limited. In the Appendix, the authors only present some of the prompts used for the other tasks. [...] The only results presented by the authors are in Figure 5.

Thank you for the comment! We would like to clarify that the original submission already reports results and prompts for all three tasks. Figure 5 reports training curves on embodied crafting (a), robot navigation (b), and deep research (c), and Table 1 reports held-out evaluation on all three tasks, where MPT outperforms monolithic RL. These main results are all part of the original submission, and the new experiments in this rebuttal serve as additional analyses.

For each task, Appendix B provides the full setup. All prompts are provided:

TaskSingle AgentBidding PromptsAction Prompts
Embodied CraftingPrompt B.1Prompts B.2–B.4Prompts B.5–B.7
Robot NavigationPrompt B.8Prompts B.9–B.11Prompts B.12–B.14
Deep ResearchPrompt B.15Prompts B.16–B.18Prompts B.19–B.21

For deep research, the system prompt of each society agent is identical to that of the single agent, which is shown in full in Prompt B.15, and only a role-specific paragraph is appended.

Nevertheless, we appreciate the reviewer's suggestion to further strengthen the validation on robot navigation and deep research. Beyond the main results already reported in the original submission, we extend the core analyses that isolate the contribution of learned self-orchestration to all three tasks.

(1) Comparison with other coordination strategies. We compare MPT with Round-robin, AgentNet, and Graph-of-Agents on navigation and deep research. AgentNet and GoA are not applicable to navigation, since NaVILA-8B is not instruction-tuned and cannot follow the free-form prompts these methods require.

StrategyCraftingNavigationDeep Research (%)
Round-robin23.8 ± 4.20.37 ± 0.0517.5 ± 2.8
AgentNet20.9 ± 4.2–18.5 ± 3.8
GoA25.4 ± 4.6–22.0 ± 4.2
MPT41.5 ± 5.91.23 ± 0.1831.5 ± 4.9

MPT outperforms all alternative coordination strategies.

(2) Contributions of learning bidding and action. We evaluate the four configurations that use the initial or post-trained checkpoints for bidding and action on all three tasks.

BiddingActionCraftingNavigationDeep Research (%)
InitialInitial15.4 ± 4.10.62 ± 0.1118.5 ± 4.4
TrainedInitial21.5 ± 4.70.87 ± 0.2520.0 ± 3.6
InitialTrained27.2 ± 4.80.96 ± 0.1726.5 ± 4.5
TrainedTrained41.5 ± 5.91.23 ± 0.1831.5 ± 4.9

Using both trained bidding and action yields the best performance.

(3) Comparison with fixed and centralized orchestration during training. In addition, we add a Fixed Router and a Learned Centralized Router on all three tasks, both of which keep the same three role models and role-specific SFT initialization as MPT. As shown in the updated Figure 5, under matched training FLOPs, both controls perform substantially worse than MPT.

Together, these results show that the main findings on the benefits of learned self-orchestration hold consistently across embodied crafting, robot navigation, and deep research.

AuthorsOpenReview ↗

[2/3] Official Comment by Authors

W2 & Q2

The mechanism used for training is unclear. It is initially presented as a sort of RL from verified rewards. But then the definition of the loss suggests a preference-based method. And then, in actual fact, an SFT mechanism is used. Can you please describe the actual mechanism that is used for training in detail?

We would like to clarify that MPT has two learning components that optimize two different behaviors of each agent, and SFT is used only as a cold start before MPT.

(1) SFT cold start (Section 3.1 & Appendix B). Following standard practice in LLM post-training, each agent is first fine-tuned on a small set of demonstrations to learn the input and output formats of its role. The monolithic baseline receives the same SFT cold start with matched compute. SFT is not part of MPT.

(2) Learning to act: RL (Section 2.2). After each episode, the task reward is propagated backward (Figure 3), and each agent updates its action policy with REINFORCE on the turns in which it acted.

(3) Learning to bid: preference learning (Section 2.3). We compare two branches from the same state that differ only in which agent acts. The bid that leads to the higher return is preferred, and the agent updates its bidding policy with a preference objective.

During MPT, the two components (2 and 3) alternate (Algorithm 1). This is because individual capabilities and self-orchestration are coupled: which agent acts determines which agents receive experience and learning signals, while improving agents in turn changes who should participate. Alternating the two updates allows action competence and participation to co-adapt as the capabilities of the swarm evolve.

W2 & Q3

Why is the mechanism based on counterfactuals needed? If in any case there is a full roll-out, it would just be sufficient to give an actual scaled reward to each of them.

Following the suggestion, we add a new ablation in Appendix G. We remove the counterfactual learning, and instead train every bid of agent using the scaled reward. All other components remain identical to MPT. Under the same training FLOPs, Episode-Return Bidding performs substantially worse than MPT. This shows that counterfactual rollouts provide a more effective learning signal for bidding.

W2

I was expecting a comparison with a centralized solution based on RL, a classic mixture of experts.

Thank you for the suggestion! We add a Learned Centralized Router baseline (RL-trained mixture-of-experts gate) on all three tasks. As shown in the updated Figure 5, under matched training FLOPs, the centralized solution performs substantially worse than MPT on all three tasks.

W2

Essentially, the proposed solution is centralized: the calculation of the “counterfactuals” reward indirectly assumes some form of aggregation.

Thank you for the comment! Our paper studies decentralized agent swarms, not decentralized training. Here, "decentralized" refers to how the swarm orchestrates itself. At each turn, every agent decides its bid independently from its own local context, and the turn is assigned by a simple highest-bid rule rather than by a central agent that decides who should act. During training, similar to the widely used paradigm of centralized training with decentralized execution in multi-agent RL, MPT makes use of non-local information. However, this does not introduce a central coordinator.

Q1

The authors should provide a clear motivation for the proposed solution. Why do you need a decentralized solution, when a centralized solution would be much simpler and most probably more efficient? Related to this point, can you provide an estimation of the actual advantage of having a decentralized solution vs a centralized one?

Thanks for the question! A centralized orchestrator must model the capabilities of every agent and must be retrained whenever these capabilities change. As the agents improve during training, the orchestrator must also keep tracking them.

In MPT, each agent judges its own competence in the current state through its bid, which brings three practical advantages:

(1) Reusability. The trained swarm adapts faster to new objectives (Figure 8).

(2) Robustness. There is no single point of failure. At a dropout probability of 0.9, a 12-agent swarm still retains a mean reward of about 33, compared with 19 for 3 agents (Figure 10).

(3) Effectiveness. Following your suggestion, we directly compare MPT with a learned centralized orchestrator. As shown in the updated Figure 5, under matched training FLOPs, the learned centralized router performs substantially worse than MPT on all three tasks. This shows that the decentralized solution is not only more flexible, but also more effective.

AuthorsOpenReview ↗

[3/3] Official Comment by Authors

Q4

Can you please provide more details about the idea of having a “decision step”? How do you calculate the best time for having it? Also, please provide details about what the behavior in the plots of Figure 6 (what is the effect of the decision step)?

We would like to clarify that the "decision step" is not a design component. It is simply the time index of an episode: each episode consists of a sequence of turns, and at every turn (decision step) the agents bid and the highest bidder acts. In embodied crafting, each episode contains 120 decision steps. We have added this clarification to Section 3.3.1.

Figure 6 shows how often each agent acts at each decision step, aggregated over evaluation episodes, at four checkpoints during training. The x-axis is the decision step within an episode, and the y-axis is the proportion of episodes in which the Logger, Miner, or Smith acts at that step. Since the iron pickaxe requires gathering wood first, then mining stone and iron and crafting tools, an effective swarm should let the Logger act early and the Miner and Smith act later.

Before RL, the Logger dominates throughout the episode, even at late steps where its actions can no longer advance the task. As training progresses, participation at later steps increasingly shifts to the Miner and Smith. At decision step 120, the Logger's share decreases from about 80% to 45%, while the Miner's share increases from about 10% to 40% and the Smith's from about 10% to 15%.

This shows that the swarm learns when each agent should participate.

Reviewer_i9P6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Market-based Post-Training (MPT), which lets a team of LLM agents jointly learn how to act and when to participate, without a central coordinator. At each step, agents submit bids and the highest bidder acts. The same bids also decide how the final reward is shared among the agents that acted. Agents learn their actions from this shared reward and learn their bidding from counterfactual comparisons, i.e., what would have happened if another agent had acted instead. On Minecraft crafting, robot navigation and deep research, MPT outperforms a single-model RL baseline at matched training compute, and it outperforms existing coordination strategies given the same trained agents. Further analyses on Minecraft show that participation patterns change during training, that learned bids improve credit assignment, and that trained teams adapt faster to new tasks and benefit from more agents at inference time.

Critical strengths
  1. A simple, novel mechanism that couples turn allocation with credit assignment. The paper uses a single learned quantity, the bid, both to decide which agent acts and to decide how reward flows back to earlier actors. Existing LLM multi-agent methods usually fix the orchestration and train only the agents, or adapt orchestration at inference time with frozen weights. MPT trains both together through weight updates, without a central coordinator. This matters to my assessment because it is the paper’s main conceptual contribution and addresses a real gap: orchestration and individual capability are coupled, but prior work rarely learns them jointly. The idea is clean enough to be reused beyond the specific tasks studied here.

  2. Evidence that the learned bids themselves are useful, not just the extra agents. Two analyses go beyond the main comparison and directly test the bidding mechanism. In the 2×2 study (Table 2, right), training both bidding and action gives 41.5, versus 21.5 or 27.2 when only one is trained. In the random-credit ablation (Fig. 7), replacing learned bids with random values in credit assignment drops reward from about 42 to 31.

Critical weaknesses
  1. The main comparison does not isolate the benefit of learned self-orchestration. MPT and the monolithic baseline differ in several ways at the same time. MPT uses three separately fine-tuned models, so it has three times the parameters of the baseline. Each of its agents is fine-tuned only on demonstrations for its own role, so it also benefits from specialization. In navigation, the action space is additionally split across agents by hand, which gives MPT prior knowledge that the baseline does not have. Matching training FLOPs does not remove these differences, so the gains in Fig. 5 and Table 1 may come from extra capacity and role specialization rather than from the bidding mechanism itself. To isolate the paper’s main claim, the baseline should use the same three role models trained with RL, with turns assigned by a learned centralized orchestrator (e.g., an RL-trained router or manager) or by an existing multi-agent LLM RL method such as Dr. MAS or CoLLM, both of which the paper already cites. The round-robin ablation in Fig. 7 does not serve this purpose: cyclic turn allocation is clearly unsuitable for a sequential tech-tree task, which makes it a weak baseline, and it is reported only for the crafting task.

  2. The experimental evidence is not yet convincing, for four reasons.

(1) First, the navigation results are reported only as a reward defined by the authors. Without the standard VLN-CE metrics (SR, SPL, NE, OSR) and a comparison with published NaVILA results, it is hard to judge how strong these results actually are.

(2) Second, some baseline numbers look inconsistent. In deep research, the monolithic baseline reaches about 6% accuracy at the end of training but 9% on the held-out set, and in navigation its reward is about 0.3 at the end of training but 0.52 on the held-out set. A held-out score higher than the training score is unusual, and a final training accuracy of only 6% suggests that the baseline may be under-tuned or may have collapsed during RL. The authors should explain these gaps and state whether each training curve comes from one run or from several random seeds.

(3) Third, every analysis in §3.3–3.4 uses only the Minecraft Iron Pickaxe task, including participation dynamics, the coordination comparison, the ablations, adaptation, scaling and dropout. It is therefore unclear whether these findings also hold for navigation and deep research.

(4) Fourth, the claim that larger populations improve performance may simply reflect more sampling. With temperature 1, each extra copy of an agent adds one more sampled bid and one more candidate action at every step, which is similar to drawing more samples at test time. A fair comparison would give the single agent, or a fixed orchestrator, the same number of LLM calls (e.g., best-of-N or self-consistency). The authors should also report how the inference cost per episode grows as the population increases.

Questions for authors
  1. Baseline confound (W1). Can you add a baseline that uses the same three role models, trained with RL, but with turns assigned by a learned centralized orchestrator (e.g., an RL-trained router) or by Dr. MAS / CoLLM? If MPT still clearly wins under matched compute, I would raise my score.

  2. Baseline training curves (W2-2). Why are the monolithic baseline’s held-out scores higher than its final training scores (6% vs. 9% in deep research; 0.3 vs. 0.52 in navigation)? Do the training curves come from a single run or multiple seeds, and how was the baseline tuned?

  3. Navigation metrics (W2-1). What are the SR, SPL, NE and OSR of MPT and the baseline on R2R-CE val-unseen, and how do they compare with the published NaVILA results?

  4. Scaling at matched inference compute (W2-4). How does the 3→12 agent scaling compare with the single agent or a fixed orchestrator given the same number of LLM calls (e.g., best-of-N)? What is the inference cost per episode for each population size?

6 comments
AuthorsOpenReview ↗

[1/2] Official Comment by Authors

We thank the reviewer for recognizing our simple, novel mechanism that couples turn allocation and credit assignment through one bid. We further appreciate the reviewer’s constructive suggestion to compare against the same three role models under a learned centralized orchestrator, as well as the reviewer’s willingness to raise the score if MPT still clearly wins under matched compute. We address this comparison, together with the remaining concerns, in the following response.

W1 & Q1

To isolate the paper’s main claim, the baseline should use the same three role models trained with RL, with turns assigned by a learned centralized orchestrator (e.g., an RL-trained router or manager) or by an existing multi-agent LLM RL method.

Thank you for this constructive suggestion!

Following it, we added two controls on all three tasks. Both keep the same three role models and role-specific SFT initialization as MPT, and differ from MPT only in how turns are allocated:

(1) Learned Centralized Router: as the reviewer suggested, learned bidding is replaced with a centralized router that is trained with RL to select the acting agent at each turn, while the three role models are trained with RL.

(2) Fixed Router: learned bidding is replaced with a fixed Round-Robin policy, and the three role models are trained with standard return-based RL.

As shown in the updated Figure 5, under matched training FLOPs, both controls perform substantially worse than MPT on all three tasks. Because these controls have the same model capacity and role specialization as MPT, the results show that decentralized self-orchestration through learned bids is more effective than both fixed and centralized orchestration.

W2 & Q3

Add the standard VLN-CE metrics (SR, SPL, NE, OSR) and compare with the published NaVILA results.

We note that our results are not directly comparable to the published NaVILA results, which fine-tune the whole VLM on a large data blend. In contrast, our models start from the base checkpoint with fewer than 2K format-alignment SFT decisions, as our goal is to study the effect of RL post-training on this task. Nevertheless, following your suggestion, we evaluate monolithic RL and MPT on R2R-CE val-unseen with the standard VLN-CE metrics, both before and after post-training.

ModelSR (%) ↑SPL (%) ↑NE (m) ↓OSR (%) ↑
Single-agent RL initialization0.00.04.610.0
Single-agent RL4.54.54.394.5
MPT initialization4.54.54.354.5
MPT9.18.34.2413.6

MPT achieves the best performance on the four metrics. Due to limited compute, these results come from relatively short post-training runs, and we expect these metrics to improve further with more training compute.

W2 & Q2

A held-out score higher than the training score is unusual, and a final training accuracy of only 6% suggests that the baseline may be under-tuned or may have collapsed during RL.

The held-out scores are higher than the final training scores because our held-out evaluation subset happens to be easier than the training distribution. The same subset is used for all methods, so the comparison remains fair.

The monolithic baseline is neither under-tuned nor collapsed during RL. To verify this, we compare training accuracy on deep research before and after post-training, and further continue training the monolithic baseline to twice the compute budget:

ModelTraining FLOPs (×10¹⁸)Training Accuracy (%)
Single RL initialization03.1
Single RL5.86.1
Single RL (2× FLOPs)12.113.8
MPT initialization011.3
MPT6.127.2

Monolithic RL improves training accuracy from 3.1% to 6.1%, and continues to improve to 13.8% with twice the training compute, showing that the baseline learns steadily. However, it learns much less efficiently than MPT: even with twice the compute, it still reaches only about half of MPT's accuracy.

W2 & Q2

The authors should state whether each training curve comes from one run or from several random seeds. How was the baseline tuned?

(1) Each training curve comes from a single run. Multi-agent LLM post-training is very resource-intensive, especially on the long-horizon tasks we study, so under limited compute we prioritized covering a broader range of benchmarks over running multiple seeds. Nevertheless, we post-trained both MPT and monolithic RL with two independent seeds on the principal crafting experiment (Appendix E). The variation across seeds is small. At around 3.5-4.0 × 10¹⁸ training FLOPs, the two independent runs differ by about 5% for both MPT and monolithic RL, which is much smaller than MPT's 2× improvement over monolithic RL.

(2) The monolithic baseline uses the same hyperparameters as MPT, including learning rate, batch size, gradient clipping, and generation settings. Its main additional hyperparameter is the discount factor, which we set to 0.95, a common choice in prior work on RL for multi-turn agents.

AuthorsOpenReview ↗

[2/2] Official Comment by Authors

W2

Analysis uses only the Minecraft Iron Pickaxe task. It is therefore unclear whether these findings also hold for navigation and deep research.

Thank you for the suggestion! Following it, we extend the core analyses that isolate the contribution of learned self-orchestration to all three tasks.

(1) Comparison with other coordination strategies. We compare MPT with Round-robin, AgentNet, and Graph-of-Agents on navigation and deep research. AgentNet and GoA are not applicable to navigation, since NaVILA-8B is not instruction-tuned and cannot follow the free-form prompts these methods require.

StrategyCraftingNavigationDeep Research (%)
Round-robin23.8 ± 4.20.37 ± 0.0517.5 ± 2.8
AgentNet20.9 ± 4.2–18.5 ± 3.8
GoA25.4 ± 4.6–22.0 ± 4.2
MPT41.5 ± 5.91.23 ± 0.1831.5 ± 4.9

MPT outperforms all alternative coordination strategies.

(2) Contributions of learning bidding and action. We evaluate the four configurations that use the initial or post-trained checkpoints for bidding and action on all three tasks.

BiddingActionCraftingNavigationDeep Research (%)
InitialInitial15.4 ± 4.10.62 ± 0.1118.5 ± 4.4
TrainedInitial21.5 ± 4.70.87 ± 0.2520.0 ± 3.6
InitialTrained27.2 ± 4.80.96 ± 0.1726.5 ± 4.5
TrainedTrained41.5 ± 5.91.23 ± 0.1831.5 ± 4.9

Using both trained bidding and action yields the best performance.

(3) Comparison with fixed and centralized orchestration during training. In addition, we add a Fixed Router and a Learned Centralized Router on all three tasks, both of which keep the same three role models and role-specific SFT initialization as MPT. As shown in the updated Figure 5, under matched training FLOPs, both controls perform substantially worse than MPT.

Together, these results show that the main findings on the benefits of learned self-orchestration hold consistently across embodied crafting, robot navigation, and deep research.

W2 & Q4

The claim that larger populations improve performance may simply reflect more sampling. A fair comparison would give the single agent, or a fixed orchestrator, the same number of LLM calls (e.g., best-of-N or self-consistency). The authors should also report how the inference cost per episode grows as the population increases.

Thank you for the suggestion! Following it, we compare population scaling in MPT with two baselines: Single + self-consistency, where the monolithic agent samples multiple candidate actions at each step and executes the most common one, and Round-Robin + self-consistency, where the same is applied to each acting agent under a fixed Round-Robin orchestration.

We plot pass@3 on the five crafting milestones against the inference FLOPs per episode in Figure 9, which also reports how the inference cost grows with the population size. The results show that as the inference budget increases, MPT improves steadily on the milestones, raising pass@3 from about 78% to 97% on Wooden Pickaxe, from 69% to 94% on Stone Pickaxe, and from 66% to 91% on Furnace. In contrast, neither self-consistency baseline benefits from additional samples. At comparable inference budgets, MPT clearly outperforms both baselines. This shows that the gains from larger populations do not simply come from drawing more samples.

So why can self-consistency hurt? For both baselines, pass@3 decreases on some milestones as more samples are drawn. This is expected in long-horizon decision making: majority voting always selects the most common action, which suppresses exploration. When the agent is stuck, it repeatedly executes the same common action instead of trying alternatives. In contrast, additional agents in MPT bid independently based on their own assessment of the current state, so larger populations offer more diverse participation.

Thank you for the detailed response and the additional experiments. The new controls are the right ones, and I appreciate the effort. However, after reading the response and the revised paper, three concerns remain.

  1. W1. On crafting, all three baselines end at roughly 20 while MPT reaches about 40, which is convincing. On deep research, however, the Fixed Router and Learned Centralized Router end at about 0.2, versus 0.06 for Single and 0.27 for MPT, so most of MPT's gain over the monolithic baseline is already obtained by the three role models without learned bidding. On navigation, the MPT curve fluctuates widely and overlaps with the Fixed Router. With single runs, I do not think "substantially worse on all three tasks" is supported. The 2×2 table points the same way: on deep research, trained bidding gives 18.5→20.0 with initial actions and 26.5→31.5 with trained actions, both small relative to the reported standard errors (about 4–5 points).

  2. W2-Q2. Your table shows MPT starts at 11.3% training accuracy on deep research versus 3.1% for Single, and in the 2×2 table the untrained MPT team (18.5% on deep research, 0.62 on navigation) already exceeds the RL-trained monolithic baseline on the held-out sets (9.0%, 0.52). The comparison with Single therefore mainly reflects the starting point. The baseline also improves faster in the second half of the 2× run (6.1→13.8) than in the first (3.1→6.1), so it is far from converged at the matched budget, and it shares MPT's hyperparameters rather than being tuned itself.

  3. W2-Q3. The paper reports 88 val-unseen episodes for navigation, but the SR/OSR values (0.0, 4.5, 9.1, 13.6) are all multiples of 1/22, which would be an unlikely coincidence if 88 episodes were used. Could you clarify how many episodes this table is based on? Even with 88, the SR gap between MPT and Single is 8 vs. 4 successes, which is too few to draw conclusions.

Overall, the response strengthens the evidence on crafting, but the claim that learned self-orchestration drives the gains across all three tasks is not yet established. I therefore maintain my rating for now.

Reviewer_BryvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how decentralized agent swarms can jointly learn action policies and which agent should act at each decision. It proposes market-based post-training, where discrete bids allocate turns and determine backward credit assignment, while counterfactual preference comparisons train bidding behavior. The method is evaluated on Minecraft crafting, vision-language navigation, and tool-using deep research.

Critical strengths
  1. The method ties orchestration to learning credit in a concrete, testable way. Section 2 defines a bid both as the turn-allocation signal and as the backward transfer coefficient in Equation 1, while Equation 3 uses counterfactual rollouts to update bidding decisions. This is a meaningful departure from merely choosing a fixed multi-agent graph or workflow, because the participation policy and action updates can co-adapt.

  2. The paper contains useful isolation experiments rather than relying only on the monolithic baseline. Table 2 fixes post-trained action policies and compares bidding to Round-Robin, AgentNet, and Graph-of-Agents, with MPT obtaining 41.5 mean reward versus 25.4 for the strongest listed alternative. Figure 7 further shows that replacing learned credit weights with random values substantially degrades reward, providing direct evidence that the learned bids affect more than turn selection.

Critical weaknesses
  1. The headline comparison with monolithic RL does not isolate the effect of MPT from model capacity, specialization, and inference-time computation. In Section 3.1 and Appendix B, MPT consists of three independently full-fine-tuned models, each trained on role-specific demonstrations and restricted to a specialized action subset, whereas the baseline is one full model trained on the pooled demonstrations and the full action space. Although Figure 5 states that training FLOPs are matched, the society has three separately parameterized policies and evaluates bids from all agents at every decision. Thus, the improvements in Figure 5 and Table 1 can reflect specialization and a larger effective model ensemble as well as learned self-orchestration. Table 2 and Figure 7 are valuable within-society ablations, but they only establish this separation on Iron Pickaxe and do not resolve attribution of the cross-domain comparison to a single monolithic model. A stronger control would retain the three specialist models and role-specific SFT initialization while replacing learned bidding with a fixed or non-learned routing policy across all three domains. Reporting an equal-total-parameter or equal-inference-FLOP monolithic/routed baseline would further clarify what is gained by MPT itself.

  2. The statistical reporting does not establish robustness to post-training randomness. Table 1 reports mean ± SE over held-out instances, each averaged over three evaluation runs, but it does not report independent training seeds for either MPT or monolithic RL. Likewise, Figures 5, 7, and 8 show training curves but do not state whether the shaded regions reflect instance-level variability, evaluation rollouts, or independently trained models. Since both REINFORCE updates and the counterfactual preference data are stochastic, the central claims about sample efficiency, transfer, and credit assignment need variability across independent post-training runs. Even two or three seeds on the principal crafting experiment, together with an explicit definition of all bands, would materially increase confidence.

Questions for authors
  1. [Clarification] For Figures 5, 7, and 8, what exactly do the shaded regions represent, and how many independently post-trained MPT and monolithic models were used? If the main trends persist across independent training seeds, my concern about optimization variance would be substantially reduced.

  2. [Clarification] In the matched-training-FLOP comparison of Section 3.2, did you evaluate a three-model specialist control with the same role-specific SFT data and action partitions as MPT, but fixed turn allocation and standard return-based learning, on robot navigation and deep research? Such a control would determine whether the broad gains are attributable to learned self-orchestration rather than the three-model specialization design.

1 comment

We thank the reviewer for recognizing that our method ties orchestration to learning credit in a concrete and testable way, departing meaningfully from fixed multi-agent graphs or workflows by allowing participation and action policies to co-adapt. We are also grateful that the reviewer found our isolation experiments useful beyond the monolithic baseline, including the coordination comparison under the same post-trained action policies and the random-credit ablation, which provides direct evidence that learned bids affect more than turn selection. We will address your concerns in the following response.

W1 & Q2

A stronger control would retain the three specialist models and role-specific SFT initialization while replacing learned bidding with a fixed or non-learned routing policy.

Thank you for this valuable suggestion! We added two controls on all three tasks. Both keep the three specialist models and the role-specific SFT initialization of MPT, and differ from MPT only in how turns are allocated:

(1) Fixed Router: learned bidding is replaced with a fixed Round-Robin policy.

(2) Learned Centralized Router: learned bidding is replaced with a centralized routing policy that is also trained with RL.

As shown in the updated Figure 5, under matched training FLOPs, both controls perform substantially worse than MPT on all three tasks. Because these controls have the same model capacity and role specialization as MPT, the results show that decentralized self-orchestration through learned bids is more effective than both fixed and centralized orchestration.

W1

Reporting an equal-total-parameter or equal-inference-FLOP monolithic/routed baseline would further clarify what is gained by MPT itself.

Thanks for the suggestion! Following it, we compare MPT with the monolithic baseline and the routed baselines under matched inference FLOPs on Iron Pickaxe. We plot mean reward against inference FLOPs and vary inference FLOPs by increasing the number of environment interaction steps. Details are provided in Appendix D. The table below reports each method's performance at the final matched inference budget (about 5.3 × 10¹⁴ FLOPs):

MethodMean Reward
Single-agent RL49.4 ± 5.5
Round-robin52.0 ± 5.0
AgentNet53.9 ± 5.9
Graph-of-Agents25.4 ± 4.3
MPT68.8 ± 4.3

At the same inference budget, MPT clearly outperforms both the monolithic baseline and all routed baselines. This shows that its advantage does not come from additional inference-time computation.

W2 & Q1

The statistical reporting does not establish robustness to post-training randomness. For Figures 5, 7, and 8, what exactly do the shaded regions represent, and how many independently post-trained MPT and monolithic models were used? Even two or three seeds on the principal crafting experiment, together with an explicit definition of all bands, would materially increase confidence.

Thank you for the question! All training curves in Figures 5, 7, and 8 come from a single training seed per method. Multi-agent LLM post-training is very resource-intensive, especially on the long-horizon tasks we study, so under limited compute we prioritized covering a broader range of benchmarks over running multiple seeds per benchmark. In these figures, the solid line is the exponential moving average of the per-step mean reward. The shaded region is the weighted standard deviation of the per-step mean reward around this moving average, and now we state this definition explicitly in Section 3.1.

Following your suggestion, we post-trained both MPT and monolithic RL with two independent seeds on the principal crafting experiment (Appendix E). The variation across seeds is small. For example, at around 3.5-4.0 × 10¹⁸ training FLOPs, the two independent runs differ by about 5% for both MPT and monolithic RL, which is much smaller than MPT's roughly 2× improvement over monolithic RL. This further increases confidence in the effectiveness of our method.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Discussion

We sincerely thank all reviewers for their time and constructive feedback. We are grateful that the reviewers recognized the coupling of turn allocation and credit assignment through a single learned bid as a simple, novel, and nontrivial mechanism (Reviewers Bryv, i9P6, sknT) that addresses a real gap in jointly learning orchestration and individual capabilities (Reviewer i9P6). We are also glad that Reviewer sknT found all questions answered, and that Reviewer i9P6 found most concerns addressed and raised the score.

All changes in the revised paper are highlighted in blue. The main revisions are:

  1. New baselines isolating learned self-orchestration (Figure 5). We add a Fixed Router and a Learned Centralized Router on all three tasks. Both share the same three role models and role-specific SFT initialization as MPT and differ only in how turns are allocated.

  2. Analyses extended to all three tasks (Table 2). The comparison with alternative coordination strategies and the contributions of learning bidding and action are now reported on embodied crafting, robot navigation, and deep research.

  3. New ablations of the bidding design. Episode-Reward Credit decouples participation from credit assignment (Figure 7), and Episode-Return Bidding trains bids on actual rewards instead of counterfactual comparisons (Appendix G).

  4. Matched inference compute. We compare MPT with the monolithic and routed baselines under matched inference FLOPs (Appendix D), and compare population scaling with self-consistency baselines under matched inference FLOPs (Figure 9).

Thank you again for your valuable feedback, which has helped us substantially improve the paper. In the camera-ready version, we will further include the multi-seed experiments and the hyperparameter sweeps.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.