acceptodds
Under review as a conference paper at ICLR 2027

When Is Further Collaboration Worthwhile? Cost-Aware Multi-Agent Reasoning via Counterfactual Branching

Abstract

Multi-Agent Debate (MAD) can enhance the reasoning capabilities of Large Language Models (LLMs), but additional collaboration rounds introduce substantial token overhead. Existing approaches typically make decisions at the instance level to determine whether collaboration should be initiated. However, they do not continuously reassess the need for further interaction as the reasoning state evolves. Consequently, they could incur redundant interactions on tasks that have already been resolved while providing insufficient refinement for those that still require further reasoning. To address this limitation, we introduce RLMAS, a cost-sensitive reinforcement learning framework for adaptive multi-agent reasoning. RLMAS augments the collaboration policy state with capability-aware confidence signals and trains a state-dependent policy using a reward that balances improvements in answer quality against incremental token costs. We introduce an offline counterfactual branching scheme to improve per-state action coverage beyond single-path trajectories by expanding alternative actions from shared reasoning states. The resulting feedback for different actions helps the policy learn when to stop and which refinement action to select. During deployment, the learned collaboration policy selects actions based on the current reasoning state without constructing counterfactual branches. Across diverse reasoning benchmarks and model scales, RLMAS reduces token usage by up to 92.6% while improving accuracy by up to 8.8% relative to baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Multi-agent reasoning, Efficiency and scalability, Reinforcement Learning, Counterfactual Branching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3wjCOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper formulates multi-agent collaboration as a cost-sensitive sequential decision problem. At each reasoning state, the controller chooses among exit, lightweight self-refinement, and high-cost team review. It uses capability-aware confidence features and offline counterfactual branching trees to train a fitted Q-network, while online deployment executes only the selected action. Experiments on MATH, MMLU-Pro, and SuperGPQA with three LLM scales show large token reductions and competitive accuracy.

Critical strengths
  1. The cost-sensitive sequential formulation is well motivated and technically coherent. Counterfactual branching provides state-action supervision that single-path trajectories cannot provide, which is a meaningful contribution to credit assignment for adaptive collaboration.

  2. The online results and ablations support the practical value of the approach: RLMAS substantially reduces token use relative to MAD and other multi-agent baselines while often preserving or improving accuracy.

Critical weaknesses
  1. The training horizon is only H=2, and the action space is limited to E/L/H. The theoretical guarantees are for fixed counterfactual trees, not the underlying stochastic collaboration process, so the practical optimality claim remains narrow.

  2. The offline counterfactual tree construction cost is not sufficiently integrated into the cost-benefit analysis. It is also unclear whether all baselines use identical token accounting, judge calls, and failure retries, which matters for the claimed efficiency gains.

Questions for authors
  1. What is the total offline cost of constructing counterfactual trees, and at what deployment volume does it amortize?

  2. How does the method behave when the decision horizon is extended beyond H=2?

  3. Are token counts for all methods computed under the same protocol, including initial responses, judge/evaluator calls, and retries?

  4. What is the practical implication of Theorem 3 given that it applies to fixed trees rather than the stochastic deployment environment?

Reviewer_bzHVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes cost-sensitive framework RLMAS for better accuracy and token cost trade-off. RLMAS applies an additional small policy to decide when to collaborate. The authors design the feature and reward for decision policy. The experiments demonstrate higher accuracy and lower token usage.

Critical strengths
  1. This paper considers a critical issue in multi-agent: overhead cost, which many multi-agent papers dodge.
  2. The proposed method demonstrates a favorable accuracy–efficiency trade-off, achieving substantial token savings while maintaining competitive accuracy across different models and reasoning benchmarks.
Critical weaknesses
  1. The counterfactual branching mechanism with fixed actions and an additional decision policy are very well studied since the beginning of LLM. Although saying "lack of novelty" is discouraged, it is still my biggest concern.
  2. The state feature in equation (5) contains many notations, some only have brief description, which are not enough for reproduction. For example, ctrevc_t^\text{rev}: what is review response? hth_t: how the history is represented? e(x)e(x): what is this? ρt\rho_t: how is it normalized?
  3. The ablation study only studies branching and confidence representation. There are many components in the state feature, it is good to know which ones are important.
Questions for authors
  1. In equation (6), zt+1z_{t+1} should depend on the next action. How to you decide the value of zt+1z_{t+1} with the branching mechanism?
  2. Different models may have different confidence values. Does this framework also apply to different model debate or transfer to other models?
  3. Could you justify why you introduce the coefficient η\eta? Since you set it to 11, and the two indicators in equation (6) seem to carry the same meaningful weight, it might not be necessary to introduce one more tunable parameter.
Reviewer_YoWfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to reduce the token cost of multi-agent debate (MAD) while preserving its accuracy gains. Specifically, the authors argue that existing efficient MAD methods decide only at the instance level whether to start collaboration, and do not reassess at each step whether further collaboration is worth its cost. To address this, they propose RLMAS, which trains a Q-network to choose among exiting, self-refine, and team review at each step, using confidence features from token log-probabilities and a reward that trades off answer correctness against token cost. The Q-targets are computed exactly by enumerating all actions on offline counterfactual branching trees. Experiments on MATH, MMLU-Pro, and SuperGPQA with three LLMs show that RLMAS reduces tokens by 38–93% relative to MAD while matching or exceeding its accuracy in six of nine settings.

Critical strengths
  1. The problem is practical and the idea is straightforward and easy to understand.
  2. The experimental results demonstrate its effectiveness.
Critical weaknesses
  1. The decision problem has only three actions and a horizon of H = 2 (Appendix B.1), so RLMAS makes at most two decisions, which is only marginally different from instance-level methods such as iMAD. Since the full branching tree is enumerated offline, training essentially reduces to supervised learning of the best action, and the theoretical results (Theorems 2 and 3) are standard backward-induction arguments. The “capability-aware” confidence is also simply the mean token log-probability. Thus, I am concerned that the contribution of this paper over existing adaptive collaboration methods is limited.
  2. Key baselines are missing. Since the action space consists of self-refine and team review, the natural baselines are simple strategies on the same actions: always self-refine, a confidence-threshold rule for exit, and self-consistency at a matched token budget. Without them, it is unclear whether RL is needed.
  3. The experiments are not convincing enough. The ablation gains are small and limited in scope: the 0.8–1.6 pp gains (Table 2) correspond to only 4–8 questions, no variance across seeds is reported, and the ablation uses only one model. In addition, the offline cost is not reported. Building the branching trees runs team review at every node, but this cost is never given. More results are needed to convincingly demonstrate the effectiveness of the method.
Questions for authors

Please see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.