- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates a round-by-round control mechanism for multi-agent debate, operating under a shared budget constraint for external model calls. The approach employs a learned "resolver" to select a local answer. Using two correctness evaluators, it compares the outcomes of "terminating the debate" versus "executing a fixed strategy". A threshold-based policy is then used to decide whether to STOP, CONTINUE, or ESCALATE the debate. Furthermore, the system can effectively regulate the usage of external models by updating these thresholds online. Experiments cover four reasoning benchmarks and four distinct agent configurations.
Critical strengths
- The paper presents a clear, well-motivated framework for jointly controlling answer selection, debate continuation, and model escalation under a shared call budget.
- Shared trajectories, question-level splits, resolver-matched comparisons, and detailed cost analyses strengthen the evaluation, while task-specific analyses clarify when further debate is beneficial.
Critical weaknesses
- The methodological novelty is limited, as the approach primarily augments an existing MAD framework with a few lightweight modules. The formalization of the proposed method also lacks sufficient clarity.
- The reasoning process for GSM8K requires further clarification. Appendix B.3 notes that if the gold answer for a query is not present in the candidate set, that query is excluded from the training of the "resolver" and from the analysis of answer selection. However, in end-to-end evaluation, such queries are handled via "confidence-weighted voting" to arrive at a final answer. During the inference phase, since labels are unavailable, it is impossible to determine whether the gold answer is missing. Furthermore, ascertaining whether the gold answer ever appeared in the candidate set might necessitate additional rounds of debate.
- The computational metrics in the paper account for local costs but overlook the aspect of the trade-off that involves external models. Table 3 and Figure 2 present data on the number of local calls, local token consumption, and local processing time, whereas the external inference component is represented solely by a "call frequency budget." The routing mechanism employed prior to the debate appears less costly at the local level precisely because it bypasses the local debate phase in favor of external inference.
Questions for authors
- How is the confidence-weighted voting fallback triggered for GSM8K without access to gold labels or future debate rounds? Does this fallback affect stopping or escalation decisions, and how are training prefixes without the gold candidate handled?
- Do the reported efficiency advantages persist when external inference costs are included? Please report each method's realized external-call rate and external token usage alongside its local computational costs.