acceptodds
Withdrawn from ICLR 2027

Stop, Continue, or Escalate: Adaptive Control of Multi-Agent Debate

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-agent debate can improve the reasoning of compact language models, but applying the same amount of debate to every query is inefficient. Easy queries are often resolved within one or two rounds, some disagreements benefit from further debate, and others persist or converge to shared errors. When local debate remains unreliable, the query can be escalated to a stronger external model, although access to it may be limited by cost or usage constraints. This raises a practical question of whether, after each round of local debate, the system should stop, continue debating, or escalate the query. We formulate this problem as budget-aware, round-by-round control of multi-agent debate. Our framework learns to resolve the current disagreement from the evolving debate trajectory and chooses among the three actions by estimating the value of the current local answer, further local debate, and escalation, while an online mechanism keeps escalation within a prescribed budget. Across four reasoning benchmarks and four agent configurations (three homogeneous and one heterogeneous), our method achieves a better accuracy-computation trade-off than fixed-depth debate as well as routing, cascading, and adaptive-control baselines. At a 10% escalation rate, it improves accuracy by 2.3 percentage points over the strongest baseline while using 18% fewer local model calls than full-length debate. Further analysis shows that our answer resolver improves answer selection under disagreement, whereas the debate controller avoids unnecessary debate rounds through selective continuation and prioritizes escalation when predicted local outcomes are weak.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, multi-agent debate

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_rymXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates a round-by-round control mechanism for multi-agent debate, operating under a shared budget constraint for external model calls. The approach employs a learned "resolver" to select a local answer. Using two correctness evaluators, it compares the outcomes of "terminating the debate" versus "executing a fixed strategy". A threshold-based policy is then used to decide whether to STOP, CONTINUE, or ESCALATE the debate. Furthermore, the system can effectively regulate the usage of external models by updating these thresholds online. Experiments cover four reasoning benchmarks and four distinct agent configurations.

Critical strengths
  1. The paper presents a clear, well-motivated framework for jointly controlling answer selection, debate continuation, and model escalation under a shared call budget.
  2. Shared trajectories, question-level splits, resolver-matched comparisons, and detailed cost analyses strengthen the evaluation, while task-specific analyses clarify when further debate is beneficial.
Critical weaknesses
  1. The methodological novelty is limited, as the approach primarily augments an existing MAD framework with a few lightweight modules. The formalization of the proposed method also lacks sufficient clarity.
  2. The reasoning process for GSM8K requires further clarification. Appendix B.3 notes that if the gold answer for a query is not present in the candidate set, that query is excluded from the training of the "resolver" and from the analysis of answer selection. However, in end-to-end evaluation, such queries are handled via "confidence-weighted voting" to arrive at a final answer. During the inference phase, since labels are unavailable, it is impossible to determine whether the gold answer is missing. Furthermore, ascertaining whether the gold answer ever appeared in the candidate set might necessitate additional rounds of debate.
  3. The computational metrics in the paper account for local costs but overlook the aspect of the trade-off that involves external models. Table 3 and Figure 2 present data on the number of local calls, local token consumption, and local processing time, whereas the external inference component is represented solely by a "call frequency budget." The routing mechanism employed prior to the debate appears less costly at the local level precisely because it bypasses the local debate phase in favor of external inference.
Questions for authors
  1. How is the confidence-weighted voting fallback triggered for GSM8K without access to gold labels or future debate rounds? Does this fallback affect stopping or escalation decisions, and how are training prefixes without the gold candidate handled?
  2. Do the reported efficiency advantages persist when external inference costs are included? Please report each method's realized external-call rate and external token usage alongside its local computational costs.
Reviewer_Vip2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an adaptive control framework for multi-agent debate, deciding whether to stop, continue debating, or escalate to a stronger model. The method combines a learned answer resolver, correctness estimators for stopping and continuation, and an online mechanism for managing the escalation budget. Experiments across four reasoning benchmarks and four agent configurations show improved accuracy and reduced local computation compared with several fixed-depth and adaptive baselines. At a 10% escalation budget, the method reports 78.7% accuracy with 4.06 local model calls per query. Additional ablations examine the contributions of the resolver and controller, while transfer experiments reveal limitations when adapting to unseen tasks.

Critical strengths
  1. The paper addresses a practical problem of balancing answer quality, debate computation, and external-model usage. Table 1 provides controlled comparisons showing that adaptive control can improve accuracy while reducing local model calls.

  2. The ablations and analyses (Table 4 and Figure 1) help explain why different tasks benefit from different debate depths.

Critical weaknesses
  1. Limited methodological novelty.

The framework mainly combines existing techniques for answer selection, correctness estimation, and budget control. Despite meaningful accuracy–efficiency gains (Table 1), the greedy controller offers limited methodological novelty and does not explicitly account for the cost of further debate.

  1. Restrictive consensus-based termination.

The controller cannot intervene once agents reach unanimous agreement (Sec. 2, Algorithm 1), despite substantial wrong-convergence rates reported in Table 14. Although acknowledged in Appendix J, this restriction leaves an important source of errors outside the proposed adaptive control mechanism.

Questions for authors
  1. What is the key methodological contribution beyond combining existing answer aggregation, correctness prediction, and budget-control techniques, particularly compared with Budgeted Act-or-Defer?

  2. Given the wrong-convergence rates in Table 14, how much does unconditional consensus termination limit the method's performance? Could selective verification of consensus states improve accuracy under the same escalation budget?

Reviewer_AJ6BOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how to allocate local debate rounds and calls to a stronger external model. At each non-converged round, a learned resolver selects an answer from the debate history. The controller compares predicted correctness for stopping and completing a local rollout with the external model’s task-level accuracy, then stops, continues, or escalates. An online threshold regulates external calls. Experiments use MMLU, ECQA, AQuA, and GSM8K with four agent configurations. At a 10% escalation budget, the method reports 78.7% accuracy and 4.06 local calls per query, outperforming the evaluated baselines in accuracy.

Critical strengths

1. Savings remain when the answer resolver is held fixed

Table 1 provides the clearest positive result. At a 10% budget, using the same resolver with full debate gives 77.9% accuracy at 4.94 local calls; the proposed controller gives 78.7% at 4.06 calls. Tables 3 and 9 also account for resolver overhead in the processing-time comparison. This is a useful result for applications with a limited external-model quota.

2. The experiments help explain where the gains arise

The factorial ablation in Table 4 separates answer selection from control. Figure 1 shows why extra debate can help one task and hurt another, and Tables 17–18 distinguish transfer across agent configurations from transfer across tasks. Sharing stored trajectories and preserving question-level splits across configurations also makes the comparisons easier to interpret.

Critical weaknesses

W1 The methodological addition is too limited

Budgeted Act-or-Defer already makes per-round stopping and deferral decisions, with a risk-control objective; CascadeDebate (Chang et al., 2026) couples debate with model escalation. The present paper adds learned intermediate answer selection and greedy control under an external-call budget (Section 1; Appendix A). Appendix B.3 uses an MLP to score candidates and two logistic regressors to predict local outcomes. The continuation target is a fixed no-escalation rollout, not the outcome of following the adaptive policy; the external-model estimate is a task average (Section 3.3; Eqs. (3), (7)–(8)). In my view, the main addition is the feature design and how these predictors are combined with the answer resolver.

W2 The results do not establish enough additional value

Once the resolver is shared, the accuracy lead over the best controlled comparison is 0.3–0.8 points across budgets (Table 1). At a 30% budget, the no-debate variant obtains 82.9% with 3.00 calls, versus 83.2% with 3.31 calls for the full method. Figure 1 also suggests a simpler explanation for part of the gain: ECQA often benefits from stopping early, whereas the other tasks benefit from more debate. The fixed-depth controls use one depth across tasks. Selecting depth separately for each task and agent configuration on validation data, while retaining the resolver and learned escalation under the shared budget, would show what is gained by deciding again for each individual query. Tables 10 and 12 do not resolve this distinction. All experiments use three agents and three rounds of ReConcile. In Table 18, transferring both learned components to an unseen task gives 75.7%, below Fixed-3 at 76.8%, though with fewer calls. The strongest results rely on workload adaptation.

Questions for authors

Q1. At the same realized escalation rate, how does the method compare with a validation-selected depth for each task and agent configuration, using the same resolver and learned escalation? Please report both accuracy and local calls. Q2. How much does the state-dependent continuation estimate contribute? A comparison replacing it with mean rollout correctness for the task and round, while keeping the other components fixed, would help isolate its role. Q3. In Appendix B.3, is the GSM8K fallback only a reporting convention, or does it alter inference? Please explain what happens to training prefixes whose gold answer is absent or excluded by the five-candidate cap, and whether all test queries pass through the same controller without access to gold labels or future rounds. If the fallback only changes an already-wrong local answer after stopping, please make that explicit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.