- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Adaptive Argument Scheduling for Multi-Agent Debate (AAS-MAD), a framework designed to improve the accuracy and computational efficiency of multi-agent reasoning. Instead of performing synchronous debate rounds, AAS-MAD organizes agents' reasoning into a shared argument graph and selectively schedules revisions targeting specific claims or reasoning dependencies. The framework trains two ExtraTrees predictors to estimate the potential rescue and damage of candidate revisions, which are then used for cost-aware action selection and adaptive stopping. Experiments on eight reasoning benchmarks and a FIFA 2026 forecasting dataset show that AAS-MAD achieves 70.76% pooled accuracy on the reasoning benchmarks, outperforming the strongest evaluated debate baseline by 1.03 percentage points while reducing token consumption by 28.84–44.34%. The paper further analyzes opening-relative improvements to distinguish successful corrections from potentially harmful revisions.
Critical strengths
- The paper introduces an interesting perspective on multi-agent debate by shifting the unit of computation from full debate rounds to individual argument-focused revisions. The proposed framework explicitly considers the expected benefits, risks, and costs of candidate actions, providing a systematic approach to allocating computational resources during multi-agent reasoning.
- The experiments cover eight reasoning benchmarks spanning multiple domains, along with an additional forecasting task. The paper compares AAS-MAD against several established multi-agent debate methods and reports both accuracy and token consumption. The results demonstrate substantial computational savings while maintaining competitive accuracy across the evaluated tasks.
Critical weaknesses
My main concern is that the paper does not sufficiently establish why the learned scheduler should generalize across diverse reasoning tasks, or to what extent it is responsible for the observed improvements. The scheduler relies on two ExtraTrees predictors trained on 5,101 one-step branches, yet the paper does not directly evaluate their predictive performance on unseen domains, such as rescue/damage prediction accuracy, calibration, or action-ranking quality; consequently, it remains unclear whether these models learn transferable patterns of revision utility.
The ablation results also provide limited evidence for the necessity of learned argument-level scheduling: in Table 3, the full method improves accuracy over random action selection with the same learned STOP gate by only 1.00 percentage point on 300 examples (74.67% vs. 73.67%), while consuming more tokens (18.65k vs. 14.45k), and removing graph-derived predictor features results in a similarly small accuracy drop. Furthermore, since most evaluation cases stop without any revision and AAS-MAD already has a stronger opening than the competing methods, it is difficult to disentangle the contribution of the learned scheduler from that of the initial reasoning protocol and adaptive stopping. More convincing evidence would require direct cross-domain evaluation of the learned predictors, together with controlled comparisons against simpler heuristic scheduling and stopping strategies under matched initial states and computational budgets.
Questions for authors
N/A
5 comments
Response to Reviewer Zpy4 — Part 2/3
The paper does not directly evaluate their predictive performance on unseen domains, such as rescue/damage prediction accuracy, calibration, or action-ranking quality; consequently, it remains unclear whether these models learn transferable patterns of revision utility.
Re: The frozen rescue/damage predictors learn transferable revision signals. We distinguish (1) predictor discrimination, (2) calibration, and (3) action selection.
We evaluate all observed legal actions in complete initial-state menus, including actions the controller would not select.
The six unseen domains are FLDx2, ForecastQA, TriviaQA, GPQA, FEVEROUS, and HOVER. Predictors and thresholds remain frozen, without refitting.
Table R3. Frozen predictors on unseen domains.
| Metric | 20260906 | 20260907 | 20260908 |
|---|---|---|---|
| Unseen-domain problems / actions | 1,124 / 5,273 | 1,123 / 5,235 | 1,123 / 5,181 |
| Rescue AUROC ↑ | 0.857 | 0.867 | 0.853 |
| Damage AUROC ↑ | 0.823 | 0.797 | 0.826 |
| Rescue AUROC, initially incorrect ↑ | 0.777 | 0.794 | 0.772 |
| Damage AUROC, initially correct ↑ | 0.893 | 0.873 | 0.884 |
| Within-state pairwise ranking accuracy ↑ | 0.567 | 0.499 | 0.558 |
| Rescue recall@1, same-cost rescue available ↑ | 0.632 | 0.667 | 0.736 |
| Rescue Brier: head / constant baseline ↓ | 0.0418 / 0.0349 | 0.0358 / 0.0298 | 0.0369 / 0.0257 |
| Damage Brier: head / constant baseline ↓ | 0.0292 / 0.0257 | 0.0274 / 0.0233 | 0.0273 / 0.0254 |
| One-action net-correction gain, pp [95% CI] | +0.53 [−0.36, 1.47] | −0.04 [−0.93, 0.85] | +1.16 [0.29, 2.03] |
AUROC ranges across the three generation seeds, by unseen domain:
Table R4. Unseen-domain AUROC ranges across seeds.
| Domain | Rescue AUROC range ↑ | Damage AUROC range ↑ |
|---|---|---|
| FLDx2 | 0.839–0.891 | 0.779–0.829 |
| ForecastQA | 0.920–0.937 | 0.855–0.893 |
| TriviaQA | 0.824–0.886 | 0.648–0.846 |
| GPQA | 0.792–0.878 | 0.808–0.838 |
| FEVEROUS | 0.883–0.969 | 0.799–0.915 |
| HOVER | 0.838–0.888 | 0.840–0.904 |
(1) Transferable discrimination.
- Both heads retain discrimination in all six unseen domains, with similar pooled AUROCs across seeds.
- Conditional AUROC tests rescue only in initially incorrect states and damage only in initially correct states. This rules out initial correctness alone as the explanation.
- Rescue recall@1 shows that the selector can identify rescuing actions when available at the same nominal call cost.
(2) Calibration. Each Brier pair compares a learned predictor with a constant predictor of that event's training-set frequency. The heads retain cross-domain discrimination, but probability prediction error is higher than the constant baseline's.
(3) Action selection. We force exactly one action from the same state, comparing learned and uniform random selection at the same nominal call cost. The row reports learned-minus-random gains. Local selection gains vary across seeds, with a positive three-seed average.
Response to Reviewer Zpy4 — Part 3/3
The ablation results also provide limited evidence for the necessity of learned argument-level scheduling: in Table 3, the full method improves accuracy over random action selection with the same learned STOP gate by only 1.00 percentage point on 300 examples (74.67% vs. 73.67%), while consuming more tokens (18.65k vs. 14.45k), and removing graph-derived predictor features results in a similarly small accuracy drop.
It is difficult to disentangle the contribution of the learned scheduler from that of the initial reasoning protocol and adaptive stopping.
Re: Learned selection improves on random + STOP, and adaptive stopping reduces computation. We examine (1) overall effects, (2) opening-disagreement cases, and (3) the STOP counterfactual.
(1) Overall ablations. All 1,500 problems and three seeds. Opening only is an unranked reference.
Three-seed means.
Table R5. Component ablations with shared openings.
| Configuration (shared AAS-MAD opening) | Accuracy (%) ↑ | Mean total tokens ↓ | Full minus control, pp [95% CI] | Full token savings ↑ |
|---|---|---|---|---|
| Opening only (reference) | 71.91 | 10.75k | +1.11 [0.33, 1.89] | −119.44% |
| AAS-MAD (Full) | 73.02 | 23.58k | — | — |
| Random action + same learned STOP | 72.20 | 16.70k | +0.82 [0.20, 1.47] | −41.21% |
| Same learned ranker, fixed six-action continuation | 72.53 | 49.82k | +0.49 [−0.11, 1.11] | 52.67% |
| Graph predictor inputs removed | 72.60 | 22.34k | +0.42 [−0.13, 1.00] | −5.55% |
| Damage penalty removed from STOP | 72.78 | 24.40k | +0.24 [−0.04, 0.56] | 3.35% |
| Reflect-only + learned STOP | 72.98 | 23.32k | +0.04 [−0.09, 0.18] | −1.12% |
- Random selection retains the same gate, although trajectories and stopping times can change. The one-action test isolates local selection.
- No graph inputs refits both heads on the same FIT set because inputs change. Graph actions and context remain available.
Each component has a positive pooled point estimate. The separate accuracy gains from graph inputs, the damage penalty, and Resolve remain uncertain: their paired intervals include zero.
Many MAD problems are already correct at the opening and need no debate. This limits overall accuracy gains. Revising valuable cases and stopping elsewhere is central to our accuracy–cost trade-off. We therefore examine opening-disagreement cases to clarify the role of action scheduling.
(2) Opening-disagreement ablations. Before revision, without gold labels, we select valid opening answers that disagree.
- Seeds 06/07/08 contribute 472/472/452 observations: 1,396 total, covering 672 distinct problems. Means weight observations.
- Paired intervals resample problems within datasets, retaining all seeds and subgroup membership.
Table R6. Ablations on opening-disagreement cases.
| Configuration (shared AAS-MAD opening) | Acc. (%) ↑ | Tokens (k) ↓ | Full minus control, pp [95% CI] | Full token savings ↑ |
|---|---|---|---|---|
| Opening only (reference) | 50.14 | 15.80 | +3.58 [1.15, 6.02] | −261.25% |
| Full | 53.72 | 57.07 | — | — |
| Random + STOP | 51.07 | 34.94 | +2.65 [0.60, 4.72] | −63.37% |
| Fixed six | 52.22 | 72.99 | +1.50 [−0.42, 3.37] | 21.81% |
| No graph inputs | 52.36 | 53.15 | +1.36 [−0.43, 3.20] | −7.39% |
| No damage penalty | 52.94 | 59.71 | +0.79 [−0.15, 1.76] | 4.42% |
| Reflect-only | 53.58 | 56.23 | +0.14 [−0.29, 0.58] | −1.49% |
The original six configurations are unchanged. Opening only is an unranked reference. All rows share the original opening. Positive token savings mean Full uses fewer tokens than the control. Negative savings mean Full uses more tokens.
Component effects remain relatively consistent and are more pronounced in this subgroup:
- Adaptive STOP substantially reduces tokens versus fixed six, with a positive accuracy point estimate.
- Learned selection improves on random + STOP, with a paired interval above zero.
(3) STOP counterfactual. Execute the same first-ranked action in each state.
Table R7. STOP counterfactual: the same first-ranked action.
| Gate decision | Valid states | Rescues | Damages | Net corrections |
|---|---|---|---|---|
| Stop | 3,510 | 29 | 29 | 0 |
| Admit | 980 | 118 | 76 | +42 |
Forcing skipped actions costs 19.42M extra tokens for zero net corrections. This supports stopping low-yield revisions.
Response to Reviewer Zpy4 — Part 1/3
Thank you for your thorough analysis and for recognizing argument-focused revision and the substantial computational savings. We address shared-opening and budget controls here, predictor transfer in Part 2, and component contributions in Part 3.
Re: Substantial computational savings remain after matching the opening. We separate (1) the shared-opening comparison from (2) where revision helps.
(1) Matched initialization. All methods receive the same four complete AAS-MAD opening responses, majority-vote readout, and tie-breaking.
Three-seed means on the same 1,500 problems, sampled randomly within each of the eight datasets. Tokens are mean total usage per problem–seed run. Opening only is an unranked reference.
Table R1. Shared-opening comparison.
Bold italics/bold mark the best/second-best distinct values among continuation methods (ties retained). Opening only is an unranked reference.
With all methods sharing AAS-MAD's opening advantage, AAS-MAD retains competitive accuracy and substantial computational savings.
(2) Why revision gains concentrate in opening-disagreement cases.
Using tiny local models to avoid unnecessary expensive LLM generation is an intended design benefit.
Re: AAS-MAD trades a small accuracy gap to Sparse for substantially lower cost, and gains accuracy over disagreement stopping at comparable token use. We compare (1) budget controls and (2) operating points.
(1) Budget controls. The 5k/15k/30k values are post-opening revision-token admission thresholds. Each policy retains its own trajectory, action rules, and STOP.
Both heuristics use fixed round-robin reflection with complete peer responses, without graph focus or learned ranking.
Both obey the revision limit and budget threshold. Checks occur at complete-action boundaries. We report actual total tokens, including overshoot.
Each cell is accuracy (%) / actual total tokens (k), including the opening. Bold italics/bold rank each metric within a threshold. Opening only is unranked. Three-seed means.
Table R2. Budget sweeps: accuracy (%) / total tokens (k).
(2) Accuracy–cost analysis. At 5k:
A similar pattern holds at the other thresholds.
At 30k:
The practical choice is substantial compute savings relative to Sparse, or higher accuracy at nearly the same computation as disagreement stopping.