- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Market-based Post-Training (MPT) for jointly learning the action policies and turn allocation of specialized LLM agents. At each turn, all agents submit a discrete bid and the highest bidder acts. The winning bid also controls how much downstream value is passed to the previous actor. Actions are trained with bid-dependent credit and a per-agent baseline. Bids are trained by restoring the same state and comparing the original winner with the runner-up. Experiments cover Minecraft crafting, robot navigation, and deep research. MPT outperforms the authors’ monolithic RL baseline on all three held-out sets. The paper also includes coordination comparisons on crafting, checkpoint swaps, ablations, and preliminary studies of reuse, scaling, and dropout.
Critical strengths
-
The paper presents an interesting and appealing way to think about multi-agent coordination. Turning participation into a learned bidding process gives a simple interface for deciding who should act as agents’ capabilities change during training. Connecting the same signal to cross-agent credit assignment is an original and intuitive design choice, and the counterfactual bidding update provides a sensible way to train it. The individual tools are familiar RL techniques, but the overall formulation is thoughtful and nontrivial.
-
The method is tested in three different settings. Table 2 fixes the post-trained action policies when comparing coordination strategies, while the checkpoint-swap and random-credit experiments suggest that both action learning and bidding matter. These experiments are incomplete, but they provide useful evidence that the bidding mechanism contributes beyond role specialization alone.
Critical weaknesses
-
The method assigns two different meanings to the same bid, but the learning objective does not show that they should coincide. A high bid determines both that an agent should act and that a larger share of its downstream value should be passed to the previous actor. Yet the counterfactual bidding objective only tests whether changing the acting agent improves the branch return; it does not test whether the bid is a calibrated credit-sharing coefficient. The random-credit ablation shows that learned bids are better than random shares, but it does not establish that coupling allocation and credit through one scalar is necessary or preferable to learning them separately. This matters because the coupling is the main conceptual distinction of MPT.
-
The baseline coverage is not sufficient to identify where the gains come from. The main comparison changes role specialization, model parameterization, turn allocation, credit assignment, and the bidding objective at the same time. Table 2 helps, but it is limited to crafting, and the implementation and tuning of AgentNet and GoA are unclear. The paper lacks a state-aware handcrafted router and a learned centralized router using the same action policies and inference budget. It is therefore difficult to tell whether the gains come from specialization, learned routing in general, or decentralized bidding specifically.
Questions for authors
-
Can the authors explain or empirically test why turn allocation and credit sharing should use the same scalar bid? A particularly informative ablation would compare the current coupled design with separate learned outputs for participation and credit assignment, while holding the action policies fixed.
-
Could the authors compare MPT with a state-aware handcrafted router and a learned centralized router using the same specialized action policies? Extending this comparison beyond crafting would clarify whether MPT learns coordination beyond the role and action-space structure provided by the task designer.
-
Could the authors report the number of independent RL runs and seed-level uncertainty for the main results, together with standard navigation metrics and a robustness check for the deep-research judge?
4 comments
[2/2] Official Comment by Authors
Q3
Could the authors report the number of independent RL runs and seed-level uncertainty for the main results, standard navigation metrics and a robustness check for the deep-research judge?
Thank you for the questions!
(1) Each training curve in the main results comes from a single run. Multi-agent LLM post-training is very resource-intensive, especially on the long-horizon tasks we study, so under limited compute we prioritized covering a broader range of benchmarks over running multiple seeds per benchmark. Nevertheless, we post-trained both MPT and monolithic RL with two independent seeds on the principal crafting experiment (Appendix E). The variation across seeds is small. For example, at around 3.5-4.0 × 10¹⁸ training FLOPs, the two independent runs differ by about 5% for both MPT and monolithic RL, which is much smaller than MPT's roughly 2× improvement over monolithic RL.
(2) Standard navigation metrics. On the same R2R-CE val-unseen episodes, we evaluate monolithic RL and MPT, both before and after post-training.
| Model | SR (%) ↑ | SPL (%) ↑ | NE (m) ↓ | OSR (%) ↑ |
|---|---|---|---|---|
| Single-agent RL initialization | 0.0 | 0.0 | 4.61 | 0.0 |
| Single-agent RL | 4.5 | 4.5 | 4.39 | 4.5 |
| MPT initialization | 4.5 | 4.5 | 4.35 | 4.5 |
| MPT | 9.1 | 8.3 | 4.24 | 13.6 |
MPT achieves the best performance on the four metrics. Due to limited compute, these results come from relatively short post-training runs, and we expect these metrics to improve further with more training compute.
(3) Robustness of the deep-research judge. We graded 1,000 deep-research answers with four different LLM judges (Qwen3.5-Flash, Qwen3.8-Flash, gpt-oss-120b, and DeepSeek-V4-Pro), and also graded every answer five times with our judge, Qwen3.5-Flash (Appendix F).
| Comparison | Agreement (%) | Full Agreement (%) | Fleiss' κ |
|---|---|---|---|
| Four different judges | 97.4 | 95.2 | 0.933 |
| Five repeated gradings | 98.7 | 97.2 | 0.965 |
Agreement is the mean pairwise agreement rate, and Full Agreement is the fraction of answers on which all judges or all gradings reach the same verdict. Both across different judges and across repeated gradings, the judgments are highly consistent. Moreover, MPT consistently outperforms monolithic RL. This shows that our deep-research results are robust to the choice of judge and to the randomness of judging.
Thank you for the response. My questions have been answered and the points I was unsure about are now clear. I will keep my score as it is. I appreciate the time you took to reply.
Thank you very much for your positive feedback. We are glad that our responses have addressed your questions and clarified the points you were unsure about. We sincerely appreciate your support and the constructive review.
[1/2] Official Comment by Authors
We thank the reviewer for recognizing our formulation as a thoughtful and nontrivial way to think about multi-agent coordination. We are also grateful that the reviewer acknowledged our evaluation in three different settings and found that the coordination comparison provides useful evidence that the bidding mechanism contributes beyond role specialization alone. We will address your concerns in the following response.
W1 & Q1
Thank you for this insightful question! Using the same bid for turn allocation and credit sharing is a deliberate design that creates an economic incentive for meaningful participation.
If participation were decoupled from credit, bidding high would come at no cost: every agent could simply bid the maximum value to win more turns. In MPT, a bid is also a price. An agent that bids higher is more likely to act, but must pass a larger share of the value it obtains back to the preceding agent. An agent therefore benefits from bidding high only when it expects its action to create enough value to cover this price. This ties each agent's willingness to participate to its expected contribution, so that the bid that allocates the turn is also a natural measure of how much credit should flow to the agent. This follows the principle of the bucket brigade in economic credit assignment. We have added this explanation to Section 2.1.
Following your suggestion, we add an ablation that decouples participation from credit assignment. We note that once credit assignment is decoupled from participation, a separately learned credit value would have no training objective. If each agent learned its credit value to maximize its own credit, it would simply pass nothing back to earlier agents. Therefore, in this ablation, bidding and turn allocation remain unchanged, while credit assignment follows the common practice in multi-agent RL, where all acting agents share the common team reward. As shown by Episode-Reward Credit in Figure 7, under matched training FLOPs, this decoupled variant performs much worse than MPT, reaching a final mean reward of only about 18, compared with about 41 for MPT.
W2 & Q2
Thank you for the suggestion! Following it, we add two controls on all three tasks. Both keep the same three role models, role-specific SFT initialization, and action partitions as MPT, and differ from MPT only in how turns are allocated:
(1) Learned Centralized Router: learned bidding is replaced with a centralized router that is trained with RL to select the acting agent at each turn.
(2) Fixed Router: learned bidding is replaced with a fixed Round-Robin policy.
As shown in the updated Figure 5, under matched training FLOPs, both controls perform substantially worse than MPT on all three tasks. Since these controls share the same roles and action-space structure, these results show that MPT learns coordination beyond this structure.
In addition, we compare MPT with the monolithic baseline and routed baselines under matched inference FLOPs, where all routed baselines use the same post-trained action policies as MPT. The table below reports each method's performance at the final matched inference budget on Iron Pickaxe (Appendix D):
At the same inference budget, MPT clearly outperforms both the monolithic baseline and all routed baselines, including the state-aware routers AgentNet and Graph-of-Agents.