acceptodds
Under review as a conference paper at ICLR 2027

Learning What to Debate: Adaptive Argument Scheduling for Multi-Agent Reasoning

Abstract

Multi-agent debate (MAD) lets Large Language Models (LLMs) read one another's responses and iteratively revise their reasoning. Yet MAD is judged by collective decision quality, while computation is allocated through coarse, full-response rounds. This can repeatedly revisit low-value reasoning, waste computation, and propagate erroneous arguments. We instead formulate MAD as selective computation over a **public argument state**, organizing independently generated rationales into a shared representation of claims, candidate decisions, evidence links, and support and conflict relations. Our method predicts the rescue and damage of candidate revisions, ranks focused revision actions by predicted rescue and call cost, and stops through a damage-aware gate. Opening-relative **promotion**, **acquisition**, and **damage** distinguish revision gains from the starting ensemble. Across eight reasoning benchmarks, AAS-MAD achieves 70.76% pooled accuracy, exceeding the strongest evaluated debate baseline by 1.03 percentage points with 28.84–44.34% fewer tokens. Measured from its own opening rather than from the best baseline, its 1.45-percentage-point gain is the largest among the five MAD methods; at the evaluated policy endpoints, all successful corrections promote existing candidates. On our curated FIFA 2026 dataset, AAS-MAD reaches 68.59% accuracy, improving on the strongest evaluated debate baseline by 1.92 percentage points. These results support allocating deliberation to arguments that can still improve the shared decision, limiting expenditure on unproductive revisions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent debate, large language model agents, argument graphs, collaborative reasoning, multi-agent reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Zpy4OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Adaptive Argument Scheduling for Multi-Agent Debate (AAS-MAD), a framework designed to improve the accuracy and computational efficiency of multi-agent reasoning. Instead of performing synchronous debate rounds, AAS-MAD organizes agents' reasoning into a shared argument graph and selectively schedules revisions targeting specific claims or reasoning dependencies. The framework trains two ExtraTrees predictors to estimate the potential rescue and damage of candidate revisions, which are then used for cost-aware action selection and adaptive stopping. Experiments on eight reasoning benchmarks and a FIFA 2026 forecasting dataset show that AAS-MAD achieves 70.76% pooled accuracy on the reasoning benchmarks, outperforming the strongest evaluated debate baseline by 1.03 percentage points while reducing token consumption by 28.84–44.34%. The paper further analyzes opening-relative improvements to distinguish successful corrections from potentially harmful revisions.

Critical strengths
  1. The paper introduces an interesting perspective on multi-agent debate by shifting the unit of computation from full debate rounds to individual argument-focused revisions. The proposed framework explicitly considers the expected benefits, risks, and costs of candidate actions, providing a systematic approach to allocating computational resources during multi-agent reasoning.
  2. The experiments cover eight reasoning benchmarks spanning multiple domains, along with an additional forecasting task. The paper compares AAS-MAD against several established multi-agent debate methods and reports both accuracy and token consumption. The results demonstrate substantial computational savings while maintaining competitive accuracy across the evaluated tasks.
Critical weaknesses

My main concern is that the paper does not sufficiently establish why the learned scheduler should generalize across diverse reasoning tasks, or to what extent it is responsible for the observed improvements. The scheduler relies on two ExtraTrees predictors trained on 5,101 one-step branches, yet the paper does not directly evaluate their predictive performance on unseen domains, such as rescue/damage prediction accuracy, calibration, or action-ranking quality; consequently, it remains unclear whether these models learn transferable patterns of revision utility.

The ablation results also provide limited evidence for the necessity of learned argument-level scheduling: in Table 3, the full method improves accuracy over random action selection with the same learned STOP gate by only 1.00 percentage point on 300 examples (74.67% vs. 73.67%), while consuming more tokens (18.65k vs. 14.45k), and removing graph-derived predictor features results in a similarly small accuracy drop. Furthermore, since most evaluation cases stop without any revision and AAS-MAD already has a stronger opening than the competing methods, it is difficult to disentangle the contribution of the learned scheduler from that of the initial reasoning protocol and adaptive stopping. More convincing evidence would require direct cross-domain evaluation of the learned predictors, together with controlled comparisons against simpler heuristic scheduling and stopping strategies under matched initial states and computational budgets.

Questions for authors

N/A

5 comments
AuthorsOpenReview ↗

Response to Reviewer Zpy4 — Part 1/3

Thank you for your thorough analysis and for recognizing argument-focused revision and the substantial computational savings. We address shared-opening and budget controls here, predictor transfer in Part 2, and component contributions in Part 3.

(...)AAS-MAD already has a stronger opening than the competing methods, it is difficult to disentangle the contribution of the learned scheduler from that of the initial reasoning protocol(...)

Re: Substantial computational savings remain after matching the opening. We separate (1) the shared-opening comparison from (2) where revision helps.

(1) Matched initialization. All methods receive the same four complete AAS-MAD opening responses, majority-vote readout, and tie-breaking.

  • MAD-M2, Vanilla, and Sparse retain their native continuations.
  • Scheduler ablations also share the argument state, legal actions, and revision interface.

Three-seed means on the same 1,500 problems, sampled randomly within each of the eight datasets. Tokens are mean total usage per problem–seed run. Opening only is an unranked reference.

Table R1. Shared-opening comparison.

Method (shared AAS-MAD opening)Accuracy (%) ↑Mean total tokens ↓AAS-MAD − row, pp [95% CI]
Opening only71.9110.75k+1.11 [0.33, 1.89]
AAS-MAD73.0223.58k—
MAD-M273.0239.03k0.00 [−0.82, 0.84]
Vanilla73.1135.92k−0.09 [−0.98, 0.82]
Sparse73.6231.80k−0.60 [−1.56, 0.36]

Bold italics/bold mark the best/second-best distinct values among continuation methods (ties retained). Opening only is an unranked reference.

With all methods sharing AAS-MAD's opening advantage, AAS-MAD retains competitive accuracy and substantial computational savings.

(2) Why revision gains concentrate in opening-disagreement cases.

  • Opening disagreement: identified before revision without gold labels, it covers 1,396/4,500 observations (31.02%). Accuracy gains concentrate here.
  • High consensus: stopping at the opening avoids unnecessary debate and is a main source of token savings.

Using tiny local models to avoid unnecessary expensive LLM generation is an intended design benefit.

(...)controlled comparisons against simpler heuristic scheduling and stopping strategies under matched initial states and computational budgets.

Re: AAS-MAD trades a small accuracy gap to Sparse for substantially lower cost, and gains accuracy over disagreement stopping at comparable token use. We compare (1) budget controls and (2) operating points.

(1) Budget controls. The 5k/15k/30k values are post-opening revision-token admission thresholds. Each policy retains its own trajectory, action rules, and STOP.

Both heuristics use fixed round-robin reflection with complete peer responses, without graph focus or learned ranking.

  • Confidence reflection: stops when all agents' mean inverse perplexity reaches the fixed calibration-set 75th-percentile cutoff.
  • Disagreement reflection: stops when all current answers agree.

Both obey the revision limit and budget threshold. Checks occur at complete-action boundaries. We report actual total tokens, including overshoot.

Each cell is accuracy (%) / actual total tokens (k), including the opening. Bold italics/bold rank each metric within a threshold. Opening only is unranked. Three-seed means.

Table R2. Budget sweeps: accuracy (%) / total tokens (k).

Method (shared AAS-MAD opening)5k (↑ / ↓)15k (↑ / ↓)30k (↑ / ↓)
Opening only (reference)71.91 / 10.7571.91 / 10.7571.91 / 10.75
AAS-MAD73.07 / 14.5873.07 / 15.5473.22 / 16.63
MAD-M272.11 / 21.8672.87 / 25.8072.96 / 31.79
Vanilla72.82 / 19.2173.00 / 25.4773.11 / 29.42
Sparse73.13 / 18.4473.38 / 24.3073.60 / 27.55
Confidence reflection72.40 / 18.8572.31 / 24.7772.27 / 28.09
Disagreement reflection72.40 / 13.9472.29 / 15.2872.24 / 16.06
Random + learned STOP72.36 / 13.5472.33 / 13.7972.22 / 14.03

(2) Accuracy–cost analysis. At 5k:

  • MAD-M2 and confidence reflection: AAS-MAD has higher accuracy point estimates with fewer tokens. Paired gains are +0.96 [0.24, 1.67] and +0.67 [−0.02, 1.38] pp.
  • Random + learned STOP: AAS-MAD gains +0.71 [0.13, 1.31] pp at comparable token use.

A similar pattern holds at the other thresholds.

At 30k:

  • Against Sparse: AAS-MAD substantially reduces cost, with an accuracy difference of −0.38 [−1.31, 0.56] pp.
  • Against disagreement reflection: AAS-MAD gains +0.98 [0.27, 1.69] pp at comparable token use.

The practical choice is substantial compute savings relative to Sparse, or higher accuracy at nearly the same computation as disagreement stopping.

AuthorsOpenReview ↗

Response to Reviewer Zpy4 — Part 2/3

The paper does not directly evaluate their predictive performance on unseen domains, such as rescue/damage prediction accuracy, calibration, or action-ranking quality; consequently, it remains unclear whether these models learn transferable patterns of revision utility.

Re: The frozen rescue/damage predictors learn transferable revision signals. We distinguish (1) predictor discrimination, (2) calibration, and (3) action selection.

We evaluate all observed legal actions in complete initial-state menus, including actions the controller would not select.

The six unseen domains are FLDx2, ForecastQA, TriviaQA, GPQA, FEVEROUS, and HOVER. Predictors and thresholds remain frozen, without refitting.

Table R3. Frozen predictors on unseen domains.

Metric202609062026090720260908
Unseen-domain problems / actions1,124 / 5,2731,123 / 5,2351,123 / 5,181
Rescue AUROC ↑0.8570.8670.853
Damage AUROC ↑0.8230.7970.826
Rescue AUROC, initially incorrect ↑0.7770.7940.772
Damage AUROC, initially correct ↑0.8930.8730.884
Within-state pairwise ranking accuracy ↑0.5670.4990.558
Rescue recall@1, same-cost rescue available ↑0.6320.6670.736
Rescue Brier: head / constant baseline ↓0.0418 / 0.03490.0358 / 0.02980.0369 / 0.0257
Damage Brier: head / constant baseline ↓0.0292 / 0.02570.0274 / 0.02330.0273 / 0.0254
One-action net-correction gain, pp [95% CI]+0.53 [−0.36, 1.47]−0.04 [−0.93, 0.85]+1.16 [0.29, 2.03]

AUROC ranges across the three generation seeds, by unseen domain:

Table R4. Unseen-domain AUROC ranges across seeds.

DomainRescue AUROC range ↑Damage AUROC range ↑
FLDx20.839–0.8910.779–0.829
ForecastQA0.920–0.9370.855–0.893
TriviaQA0.824–0.8860.648–0.846
GPQA0.792–0.8780.808–0.838
FEVEROUS0.883–0.9690.799–0.915
HOVER0.838–0.8880.840–0.904

(1) Transferable discrimination.

  • Both heads retain discrimination in all six unseen domains, with similar pooled AUROCs across seeds.
  • Conditional AUROC tests rescue only in initially incorrect states and damage only in initially correct states. This rules out initial correctness alone as the explanation.
  • Rescue recall@1 shows that the selector can identify rescuing actions when available at the same nominal call cost.

(2) Calibration. Each Brier pair compares a learned predictor with a constant predictor of that event's training-set frequency. The heads retain cross-domain discrimination, but probability prediction error is higher than the constant baseline's.

(3) Action selection. We force exactly one action from the same state, comparing learned and uniform random selection at the same nominal call cost. The row reports learned-minus-random gains. Local selection gains vary across seeds, with a positive three-seed average.

AuthorsOpenReview ↗

Response to Reviewer Zpy4 — Part 3/3

The ablation results also provide limited evidence for the necessity of learned argument-level scheduling: in Table 3, the full method improves accuracy over random action selection with the same learned STOP gate by only 1.00 percentage point on 300 examples (74.67% vs. 73.67%), while consuming more tokens (18.65k vs. 14.45k), and removing graph-derived predictor features results in a similarly small accuracy drop.

It is difficult to disentangle the contribution of the learned scheduler from that of the initial reasoning protocol and adaptive stopping.

Re: Learned selection improves on random + STOP, and adaptive stopping reduces computation. We examine (1) overall effects, (2) opening-disagreement cases, and (3) the STOP counterfactual.

(1) Overall ablations. All 1,500 problems and three seeds. Opening only is an unranked reference.

Three-seed means.

Table R5. Component ablations with shared openings.

Configuration (shared AAS-MAD opening)Accuracy (%) ↑Mean total tokens ↓Full minus control, pp [95% CI]Full token savings ↑
Opening only (reference)71.9110.75k+1.11 [0.33, 1.89]−119.44%
AAS-MAD (Full)73.0223.58k——
Random action + same learned STOP72.2016.70k+0.82 [0.20, 1.47]−41.21%
Same learned ranker, fixed six-action continuation72.5349.82k+0.49 [−0.11, 1.11]52.67%
Graph predictor inputs removed72.6022.34k+0.42 [−0.13, 1.00]−5.55%
Damage penalty removed from STOP72.7824.40k+0.24 [−0.04, 0.56]3.35%
Reflect-only + learned STOP72.9823.32k+0.04 [−0.09, 0.18]−1.12%
  • Random selection retains the same gate, although trajectories and stopping times can change. The one-action test isolates local selection.
  • No graph inputs refits both heads on the same FIT set because inputs change. Graph actions and context remain available.

Each component has a positive pooled point estimate. The separate accuracy gains from graph inputs, the damage penalty, and Resolve remain uncertain: their paired intervals include zero.

Many MAD problems are already correct at the opening and need no debate. This limits overall accuracy gains. Revising valuable cases and stopping elsewhere is central to our accuracy–cost trade-off. We therefore examine opening-disagreement cases to clarify the role of action scheduling.

(2) Opening-disagreement ablations. Before revision, without gold labels, we select valid opening answers that disagree.

  • Seeds 06/07/08 contribute 472/472/452 observations: 1,396 total, covering 672 distinct problems. Means weight observations.
  • Paired intervals resample problems within datasets, retaining all seeds and subgroup membership.

Table R6. Ablations on opening-disagreement cases.

Configuration (shared AAS-MAD opening)Acc. (%) ↑Tokens (k) ↓Full minus control, pp [95% CI]Full token savings ↑
Opening only (reference)50.1415.80+3.58 [1.15, 6.02]−261.25%
Full53.7257.07——
Random + STOP51.0734.94+2.65 [0.60, 4.72]−63.37%
Fixed six52.2272.99+1.50 [−0.42, 3.37]21.81%
No graph inputs52.3653.15+1.36 [−0.43, 3.20]−7.39%
No damage penalty52.9459.71+0.79 [−0.15, 1.76]4.42%
Reflect-only53.5856.23+0.14 [−0.29, 0.58]−1.49%

The original six configurations are unchanged. Opening only is an unranked reference. All rows share the original opening. Positive token savings mean Full uses fewer tokens than the control. Negative savings mean Full uses more tokens.

Component effects remain relatively consistent and are more pronounced in this subgroup:

  • Adaptive STOP substantially reduces tokens versus fixed six, with a positive accuracy point estimate.
  • Learned selection improves on random + STOP, with a paired interval above zero.

(3) STOP counterfactual. Execute the same first-ranked action in each state.

Table R7. STOP counterfactual: the same first-ranked action.

Gate decisionValid statesRescuesDamagesNet corrections
Stop3,51029290
Admit98011876+42

Forcing skipped actions costs 19.42M extra tokens for zero net corrections. This supports stopping low-yield revisions.

Reviewer_1tK7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AAS-MAD, an adaptive multi-agent debate framework that attempts to allocate deliberation more intelligently than conventional full-response debate protocols.

The system constructs a shared argument graph from independently generated reasoning chains, identifies candidate revision actions, and uses two extremely randomized tree predictors to estimate the probability that an action will rescue an incorrect collective decision or damage a correct one. A cost-sensitive scheduler selects revisions, while a separate stopping gate determines whether further deliberation is worthwhile.

Across eight reasoning benchmarks, AAS-MAD achieves 70.76% pooled accuracy compared with 69.73% for MAD-M2, while consuming substantially fewer tokens. The paper also evaluates the approach on a retrospective FIFA 2026 forecasting dataset.

The principal empirical result is interesting: most debate rounds appear unnecessary, and selectively allocating revisions can preserve or modestly improve accuracy while substantially reducing computation.

My concern is that the paper bundles several mechanisms together and does not convincingly establish which components are responsible for its reported advantage. In particular, the evidence supporting learned argument-level scheduling is considerably weaker than the evidence supporting early stopping and selective reflection.

Critical strengths
  1. The paper asks an important question about the actual value of multi-agent deliberation.

A recurring problem with multi-agent debate research is the assumption that more interaction constitutes better reasoning. The authors correctly recognize that exchanging additional reasoning chains can reinforce errors, introduce new ones, and consume computation without improving the final decision.

Treating deliberation as a sequential resource-allocation problem is sensible. The explicit separation between rescue, damage, and computation cost is a useful way to formulate the problem.

  1. The opening-relative evaluation is a good methodological improvement.

I appreciate that the authors distinguish accuracy attributable to the initial ensemble from accuracy gained through subsequent interactions.

The decomposition into promotion, acquisition, and damage is particularly useful. It provides a much clearer account of what debate accomplishes than reporting final accuracy alone.

The paper is also reasonably transparent about the limitations of these measurements. For example, the authors explicitly distinguish their diagnostic six-action trajectories from the online stopping policy, rather than presenting them as equivalent experiments.

  1. The computation savings are substantial.

AAS-MAD reduces pooled token consumption from 37.89k to 21.09k relative to MAD-M2, while improving reported accuracy. Its revision-stage token consumption is also substantially lower.

Even if the accuracy improvement turns out to be statistically indistinguishable from a strong baseline, achieving comparable accuracy with substantially less computation is independently valuable.

The implementation details, dataset split accounting, and matched FIFA stopping control are additional strengths.

Critical weaknesses

Before anything else, the authors should consider renaming their method. Unfortunately, saying it out-loud makes it sound obscene. I do not let this quirk impact my decision.

  1. Most of the reported accuracy advantage over MAD-M2 is already present before debate.

This is the central issue with the experimental framing.

AAS-MAD finishes with 2,546 correct decisions across 3,598 cases, compared with 2,509 for MAD-M2. That is a difference of 37 correct decisions, or approximately 1.03 percentage points.

However, AAS-MAD begins with 2,494 correct decisions, whereas MAD-M2 begins with 2,465.

Thus, 29 of the 37 additional correct endpoint decisions are accounted for by the difference in opening accuracy.

The incremental improvement from subsequent interaction is 52 cases for AAS-MAD versus 44 for MAD-M2, a difference of eight cases across 3,598 examples.

The paper does report these opening-relative results, which I appreciate. But the overall framing still risks attributing too much of the final performance advantage to the adaptive scheduler.

This does not mean the scheduler is ineffective. Different methods have different opening protocols, so simply subtracting these quantities does not provide a causal estimate of the scheduler's contribution.

That is why a stronger matched comparison is needed.

I would like to see the competing schedulers operate on identical initial agent responses, using the same answer aggregation and revision interfaces. Without this comparison, improvements from initialization, structured prompting, action selection, and stopping remain entangled.

  1. The proposed argument-level debate mechanism appears to reduce mostly to selective reflection.

The paper motivates its approach through argument graphs, conflict relations, reasoning dependencies, and focused dispute resolution.

However, Appendix C.2 reports that the final policy executes 3,009 Reflect actions and only 90 Resolve actions.

That means approximately 97% of executed revisions are reflections!!!

This is not necessarily a defect. If reflection is the economically preferable operation, the scheduler should select it. But it substantially changes the interpretation of the contribution.

The system is presented as learning which arguments should be debated. In practice, the dominant behavior is deciding whether an agent should reconsider its answer using its peers' reasoning.

Moreover, the component ablation in Table 3 shows that Reflect-only achieves 74.33% accuracy compared with 74.67% for the full system, with almost identical token consumption.

On the 300-case ablation set, that is a difference of one correct decision.

Similarly, removing graph features decreases accuracy from 74.67% to 73.67%, equivalent to three decisions, while retaining the graph-derived action menu and deliberation context.

These results suggest that much of the benefit comes from the selective revision and stopping protocol instead of the richer argument-level machinery.

The authors should demonstrate that explicit graph structure and dispute resolution provide benefits that simpler selective-reflection policies cannot recover.

Otherwise, the paper is adding considerable architectural complexity to obtain a relatively small and poorly isolated improvement.

  1. The promotion-only outcome raises an important question about whether debate is necessary.

One of the more revealing results appears in Section 5.2.

The final adaptive policy produces 136 promotions, zero acquisitions, and 84 opening-relative damages.

In other words, every successful endpoint correction selects an answer already present among the opening agents' commitments.

The authors correctly note that this does not imply revised reasoning is useless. A model may need additional argumentation to recognize that an initially available answer is correct.

Nevertheless, this result makes a much simpler competing hypothesis plausible:

Perhaps the essential task is not multi-agent debate, but selecting the correct answer from an existing ensemble.

If so, why not train a comparatively inexpensive verifier or reranker over the opening candidates?

The paper needs a stronger comparison against decision selection from the existing candidate pool, ideally using the same initial reasoning chains and a comparable inference budget.

This is particularly important because the method never demonstrates an endpoint correction that requires introducing a previously absent answer.

I would also like to see the empirical oracle coverage of the opening candidate set. That would tell us how much of the remaining error is even theoretically recoverable through candidate promotion alone.

The promotion/acquisition accounting is a valuable contribution. But the empirical results it reveals arguably motivate a simpler method than the one proposed.

  1. The learned controller has not been sufficiently validated independently of the overall pipeline.

The method relies on two learned predictors of rescue and damage. These predictions directly determine the ranking and stopping decisions.

Yet the empirical analysis focuses primarily on downstream accuracy and token consumption.

There is limited direct evidence that the predictors reliably estimate the quantities they claim to model, particularly under distribution shift.

This matters because the two regressors are fitted using 5,101 one-step branches from 1,139 training examples, with scheduler and stopping parameters tuned on 341 validation examples. The resulting policy is then applied to several domains absent from the fitting data.

The transfer performance is encouraging, but it does not establish predictive calibration.

For example, do actions assigned a 20% rescue probability actually succeed approximately 20% of the time? How well are damaging actions identified? How reliably does the ranker distinguish candidate actions within the same state?

The paper would benefit from calibration plots, ranking metrics, and outcome analysis stratified by predicted rescue and damage.

I am also concerned about the cost model. The scheduler uses nominal generation-call counts rather than predicted token expenditure, despite the paper's primary efficiency claims being expressed in tokens.

A two-call dispute is not guaranteed to cost twice as many tokens as a one-call reflection. Indeed, Appendix C.2 shows substantial task-dependent variation in observed action costs.

An explicit token-cost predictor would be a more direct implementation of the proposed resource-allocation principle.

  1. The ablations do not sufficiently isolate the value of learned scheduling and stopping.

Table 3 contains useful ablations, but they cover only 300 cases.

Random action selection with the same stopping gate reaches 73.67% accuracy versus 74.67% for the full method, while consuming fewer tokens: 14.45k versus 18.65k.

Removing graph features produces the same 73.67% accuracy at 17.19k tokens.

These are not negligible differences, but they are small enough that uncertainty matters. A three-decision difference in a 300-case evaluation is insufficient to establish a reliable advantage for a comparatively complicated learned scheduler.

The results also do not demonstrate that the complete method dominates simpler alternatives along a matched accuracy–cost frontier.

The stopping experiments provide somewhat stronger evidence. In particular, the matched FIFA trajectories isolate considerable computational savings from avoiding additional revisions.

However, the corresponding reasoning-domain trajectories are explicitly acknowledged not to be matched stopping interventions.

Consequently, the evidence that adaptive stopping saves computation is persuasive, but the evidence that the particular learned ranking mechanism is superior to simpler heuristics remains limited.

I would like the authors to evaluate random scheduling, confidence-based stopping, disagreement-based stopping, and simple reflection-only controllers across the full reasoning suite, ideally with token-budget sweeps.

  1. Statistical uncertainty and the FIFA evaluation need more careful treatment.

The primary reasoning evaluation reports one seed per registered test case, and the reported 1.03-percentage-point pooled advantage amounts to 37 decisions across 3,598 examples.

Without paired uncertainty estimates or repeated runs, it is difficult to determine how much confidence to place in that difference.

The situation is more pronounced for FIFA.

The evaluation consists of 104 fixtures with three seeds, producing 312 fixture–seed observations. Those observations are not equivalent to 312 independent football matches.

AAS-MAD achieves 214/312 correct predictions compared with 208/312 for Sparse MAD, a difference of six fixture–seed predictions.

More importantly, the matched opening already achieves 213/312 correct predictions. The entire adaptive deliberation process therefore adds one correct prediction across the 312 evaluations, while increasing token consumption from 22.64k to 27.10k.

The opening also achieves a slightly better Brier score (0.505 versus 0.506).

The FIFA result consequently provides convincing evidence that additional debate can be avoided, but almost no evidence that adaptive argument scheduling improves forecasting accuracy beyond its own initialization.

I would like fixture-clustered confidence intervals, paired significance testing, and a clearer accounting of uncertainty due to stochastic generation.

Given that the FIFA evaluation is retrospective, the authors should also explicitly discuss how temporal integrity is maintained across the evidence snapshots and the language model's knowledge. This is a methodological concern, not an allegation of leakage.

Questions for authors

Questions for the Authors

Q1. Can the authors compare the learned scheduler against alternative scheduling policies operating on the exact same opening responses, argument state, and answer readout? How much of the endpoint advantage remains when initialization is held fixed?

Q2. Given that all successful endpoint corrections are promotions, how does AAS-MAD compare against an inexpensive reranker or verifier selecting among the four opening candidates? What is the empirical oracle coverage of the opening candidate set?

Q3. Why does the final policy select Resolve in only 90 of 3,099 revisions? Can the authors demonstrate a statistically reliable benefit from graph-directed dispute resolution over a reflection-only scheduler?

Q4. How well calibrated are the rescue and damage predictors on held-out tasks? What are their within-state action-ranking performance and false-positive rates for damaging revisions?

Q5. Can the authors report a token-budget/accuracy Pareto frontier for the full controller and simpler alternatives, with uncertainty estimates across the entire reasoning suite?

Q6. For FIFA, how much benefit remains over the matched zero-revision opening under fixture-clustered uncertainty estimates? Can the authors more clearly separate the value of initialization, the terminal forecasting head, and the learned revision controller?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1tK7 — Part 1/3 (Q1–Q2)

Thank you for your detailed analysis and for recognizing our problem formulation and substantial token savings. We address Q1–Q2 here, Q3–Q4 in Part 2, and Q5–Q6 in Part 3.

Experiments use 1,500 problems sampled randomly within each of the eight datasets, three generation seeds, and frozen models and thresholds.

Thank you also for pointing out the naming issue, which we overlooked. We will use ArgSched in a revision. These responses retain the submitted name.

[W1 & Q1] Can the authors compare the learned scheduler against alternative scheduling policies operating on the exact same opening responses, argument state, and answer readout? How much of the endpoint advantage remains when initialization is held fixed?

Re: Savings remain after all methods share AAS-MAD's opening advantage. We compare two groups under the same opening and readout:

  • Alternative controllers: random, fixed-step, reflection-only, and two stopping heuristics share the complete opening responses and initial argument state. Random/fixed-step controls retain our revision interface. Heuristics use round-robin reflection on complete peer responses, stopping on confidence or unanimity.
  • MAD baselines: MAD-M2, Vanilla, and Sparse retain their native continuations.

Three-seed endpoint means. Heuristics use the 30k revision threshold. Tokens include openings. Bold italics/bold mark best/second-best values. Opening only is unranked. Q5 gives budget sweeps.

Table R1. Alternative policies with shared openings.

Policy / protocol (shared AAS-MAD opening)Accuracy (%) ↑Tokens (k) ↓Full − row, pp [95% CI]Full token savings ↑
Opening only (reference)71.9110.75+1.11 [0.33, 1.89]−119.44%
AAS-MAD (Full)73.0223.58——
Random + learned STOP72.2016.70+0.82 [0.20, 1.47]−41.21%
Same ranker, fixed six72.5349.82+0.49 [−0.11, 1.11]52.67%
Reflect-only + learned STOP72.9823.32+0.04 [−0.09, 0.18]−1.12%
Confidence reflection72.2728.09+0.76 [0.04, 1.47]16.06%
Disagreement reflection72.2416.06+0.78 [0.07, 1.49]−46.84%
MAD-M273.0239.030.00 [−0.82, 0.84]39.59%
Vanilla73.1135.92−0.09 [−0.98, 0.82]34.36%
Sparse73.6231.80−0.60 [−1.56, 0.36]25.84%

With initialization fixed, substantial computational savings remain alongside competitive accuracy.

  • Against heuristics: Full is more accurate and cheaper than confidence reflection. Disagreement stopping is cheaper but less accurate.
  • Against MAD baselines: Full retains competitive accuracy at substantially lower cost. MAD-M2/Vanilla have similar point accuracy. Sparse trades more tokens for higher accuracy.

Q5 compares budgets. Random and fixed-step controls separate selection from stopping.

[W3 & Q2] Given that all successful endpoint corrections are promotions, how does AAS-MAD compare against an inexpensive reranker or verifier selecting among the four opening candidates? What is the empirical oracle coverage of the opening candidate set?

Re: Budgeted AAS-MAD improves on the reranker at comparable token use.

(1) Matched reranker comparison.

Low-cost reranker. The same frozen LLM selects an existing candidate using the task and complete rationales.

  • Unanimous openings return directly.
  • Otherwise, one graph-free LLM call selects an answer. All usage counts.

AAS-MAD, 5k revision threshold. The same frozen controller and STOP admit a complete action only below 5k revision tokens. Total tokens include the opening and final-action overshoot. Opening only is unranked.

Table R2. Candidate reranker and budgeted revision.

Method (same opening)Accuracy (%) ↑Tokens (k) ↓Method − reranker, pp [95% CI]
Opening only (reference)71.9110.75−0.40 [−1.22, 0.44]
Candidate reranker72.3113.44—
AAS-MAD, 5k revision threshold73.0714.58+0.76 [0.02, 1.53]
AAS-MAD, default73.0223.58+0.71 [−0.04, 1.51]

Results suggest:

  • Revision adds value: the reranker is cheaper than Full, with lower point accuracy and a small gain over opening voting. Budgeted AAS-MAD has a larger gain at comparable token use.
  • Possible mechanism: our unified state representation may help promote correct answers. Stopping may still need optimization.
  • A useful low-cost option: the reranker is substantially cheaper than default AAS-MAD, a natural extension of the paper's experiments worth further study.

(2) Opening-candidate coverage.

Table R3. Opening-candidate coverage and rescues.

Opening-candidate diagnosticResult
At least one correct candidate3,680 / 4,496 (81.85%)
Full rescues: wrong aggregate, correct candidate available157 / 444 (35.36%)
Full rescues: all candidates wrong0 / 816

All rescues had a correct candidate. This motivates the reranker comparison.

AuthorsOpenReview ↗

Response to Reviewer 1tK7 — Part 2/3 (Q3–Q4)

[W2 & Q3] Why does the final policy select Resolve in only 90 of 3,099 revisions? Can the authors demonstrate a statistically reliable benefit from graph-directed dispute resolution over a reflection-only scheduler?

Re: Resolve produces local rescues, but the expanded evaluation does not establish a statistically reliable average accuracy gain.

Table R4. Resolve ablation and observed use.

EvaluationFull − Reflect-only, pp [95% CI]Exact McNemar pResolve use; local rescues / damages
Original 300 problems+0.33 [0.00, 1.00]1.00—
1,500 problems × 3 seeds+0.04 [−0.09, 0.18]—106 / 4,161 (2.55%); 5 / 3
  • Why rare: Resolve uses two nominal generation calls versus one for Reflect. It therefore requires a larger predicted rescue benefit under cost-aware scheduling.
  • Why retained: we designed Resolve to study targeted interaction beyond reflection. It produces local rescues, while overall accuracy and token use remain similar to reflection-only. Its endpoint accuracy benefit is not yet established. Since the results do not establish a negative net effect, we retain it for mechanism studies.
  • What the experiments show: more complex information exchange is not always necessary.

[W4 & Q4] How well calibrated are the rescue and damage predictors on held-out tasks? What are their within-state action-ranking performance and false-positive rates for damaging revisions?

Re: We separately test predictor discrimination, calibration, action ranking, and damaging-action errors.

We evaluate all observed legal actions in complete initial-state menus, including actions the controller would not select.

The six unseen domains are FLDx2, ForecastQA, TriviaQA, GPQA, FEVEROUS, and HOVER. Predictors and thresholds remain frozen, without refitting.

Table R5. Frozen predictors on unseen domains.

Metric202609062026090720260908
Unseen-domain problems / actions1,124 / 5,2731,123 / 5,2351,123 / 5,181
Rescue AUROC ↑0.8570.8670.853
Damage AUROC ↑0.8230.7970.826
Within-state pairwise ranking accuracy ↑0.5670.4990.558
Rescue Brier: head / constant baseline ↓0.0418 / 0.03490.0358 / 0.02980.0369 / 0.0257
Damage Brier: head / constant baseline ↓0.0292 / 0.02570.0274 / 0.02330.0273 / 0.0254
One-action net-correction gain, pp [95% CI]+0.53 [−0.36, 1.47]−0.04 [−0.93, 0.85]+1.16 [0.29, 2.03]
Rescue AUROC, initially incorrect ↑0.7770.7940.772
Damage AUROC, initially correct ↑0.8930.8730.884
Rescue recall@1, same-cost rescue available ↑0.6320.6670.736
Damage FPR / FNR (%), cutoff 0.05 ↓6.52 / 78.266.28 / 78.235.65 / 81.34
Admitted actions / damaging actions (all eight domains)328 / 23337 / 31315 / 22
Damage among admitted actions (%) ↓7.019.206.98

AUROC ranges across the three generation seeds, by unseen domain:

Table R6. Unseen-domain AUROC ranges across seeds.

DomainRescue AUROC range ↑Damage AUROC range ↑
FLDx20.839–0.8910.779–0.829
ForecastQA0.920–0.9370.855–0.893
TriviaQA0.824–0.8860.648–0.846
GPQA0.792–0.8780.808–0.838
FEVEROUS0.883–0.9690.799–0.915
HOVER0.838–0.8880.840–0.904

(1) Transferable discrimination.

  • Both heads retain discrimination in all six unseen domains, with similar pooled AUROCs across seeds.
  • Conditional AUROC tests rescue only in initially incorrect states and damage only in initially correct states. This rules out initial correctness alone as the explanation.
  • Rescue recall@1 shows that the selector can identify rescuing actions when available at the same nominal call cost.

(2) Calibration. Each Brier pair compares a learned predictor with a constant predictor of that event's training-set frequency. The heads retain cross-domain discrimination, but probability prediction error is higher than the constant baseline's.

(3) Action selection. We force exactly one action from the same state, comparing learned and uniform random selection at the same nominal call cost. The row reports learned-minus-random gains. Local selection gains vary across seeds, with a positive three-seed average.

(4) Damage errors. The diagnostic cutoff yields low FPR but high FNR. A positive means “predicted damaging.” The deployed gate jointly uses rescue, damage, and cost, not this cutoff alone.

Reasoning-domain confidence intervals use dataset-stratified, paired resampling of problems with their three seeds kept together.

AuthorsOpenReview ↗

Response to Reviewer 1tK7 — Part 3/3 (Q5–Q6)

[W5 & Q5] Can the authors report a token-budget/accuracy Pareto frontier for the full controller and simpler alternatives, with uncertainty estimates across the entire reasoning suite?

Re: AAS-MAD trades a small accuracy gap to Sparse for substantially lower cost, and gains accuracy over disagreement stopping at comparable token use.

(1) Budget experiments. We apply 5k/15k/30k post-opening thresholds at complete-action boundaries, retaining each policy's actions, STOP, and trajectory.

Both heuristics use fixed round-robin reflection with complete peer responses, without graph focus or learned ranking.

  • Confidence reflection: stops when all agents' mean inverse perplexity reaches the fixed calibration-set 75th-percentile cutoff.
  • Disagreement reflection: stops when all current answers agree.

Both obey the revision limit and budget threshold. Reported actual total tokens include overshoot.

Three-seed means. Tokens include the opening. Bold italics/bold mark best/second-best per threshold. Opening only is unranked.

Table R7. Budgets: accuracy (%) / tokens (k).

Method (shared AAS-MAD opening)5k (↑ / ↓)15k (↑ / ↓)30k (↑ / ↓)
Opening only (reference)71.91 / 10.7571.91 / 10.7571.91 / 10.75
AAS-MAD73.07 / 14.5873.07 / 15.5473.22 / 16.63
MAD-M272.11 / 21.8672.87 / 25.8072.96 / 31.79
Vanilla72.82 / 19.2173.00 / 25.4773.11 / 29.42
Sparse73.13 / 18.4473.38 / 24.3073.60 / 27.55
Confidence reflection72.40 / 18.8572.31 / 24.7772.27 / 28.09
Disagreement reflection72.40 / 13.9472.29 / 15.2872.24 / 16.06
Random + learned STOP72.36 / 13.5472.33 / 13.7972.22 / 14.03

Table R8. Paired comparisons.

Full operating pointComparatorΔ accuracy, pp [95% CI]Full token savings ↑
5kMAD-M2, 5k+0.96 [0.24, 1.67]33.30%
5kConfidence reflection, 5k+0.67 [−0.02, 1.38]22.67%
5kRandom, 5k+0.71 [0.13, 1.31]−7.66%
15kRandom, 15k+0.73 [0.09, 1.38]−12.69%
30kRandom, 30k+1.00 [0.36, 1.64]−18.55%
30kSparse, 30k−0.38 [−1.31, 0.56]39.64%
30kDisagreement reflection, 30k+0.98 [0.27, 1.69]−3.56%

(2) Accuracy–cost analysis. At 30k:

  • Against Sparse: AAS-MAD substantially reduces cost, with an accuracy difference of −0.38 [−1.31, 0.56] pp.
  • Against disagreement reflection: AAS-MAD gains +0.98 [0.27, 1.69] pp at comparable token use.

The practical choice is substantial compute savings relative to Sparse, or higher accuracy at nearly the same computation as disagreement stopping. A similar pattern holds at the other thresholds.

[W6 & Q6] For FIFA, how much benefit remains over the matched zero-revision opening under fixture-clustered uncertainty estimates? Can the authors more clearly separate the value of initialization, the terminal forecasting head, and the learned revision controller?

Re: Revision chiefly saves computation versus fixed six. Its accuracy gain over the matched opening is not statistically established. We separate initialization, the forecasting head, and revision below.

Shared opening, 104 fixtures × three seeds, clustered by fixture. Tokens are per-run means. Zero revisions is the matched reference.

Table R9. FIFA component controls.

ContinuationNative readout: correct / 312 ↑With head / 312 ↑Tokens (k), with head ↓Brier, with head ↓
Zero revisions20421322.640.5047
Adaptive revision20121427.100.5065
Fixed six19921384.720.5086

Table R10. FIFA contrasts by fixture.

Paired contrastΔ accuracy, pp [95% CI]Cluster sign-flip pAdaptive token savings ↑
Adaptive − zero revisions, same head+0.32 [0.00, 0.96]1.00−19.72%
Adaptive − fixed six, same head+0.32 [−0.64, 1.60]1.0068.01%
Head − native readout, zero revisions+2.88 [−1.28, 7.37]0.258—
Head − native readout, adaptive+4.17 [0.00, 8.97]0.100—

Results suggest:

  • Initialization: the strong opening already captures most forecasting accuracy.
  • Terminal forecasting head: native-versus-head comparisons at opening and revised endpoints show higher accuracy point estimates with the head. When it helps most remains unclear.
  • Revision controller: its main benefit is avoiding prolonged debate and its damage risk, with substantially fewer tokens than fixed six. With the head fixed, its accuracy point estimate is slightly positive.
  • Temporal integrity: frozen evidence is checked for post-event results. No runtime retrieval is used, and Qwen-Flash terminal readouts disable thinking. This retrospective test cannot exclude model-parameter knowledge. Under matched settings and head, Full is more accurate than every listed competing baseline.
Reviewer_iKvpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces AAS-MAD (Adaptive Argument Scheduling for Multi-Agent Debate), a multi-agent debate framework that (1) predicts the rescue and damage of a set of candidate revisions, (2) ranks those revisions by predicted rescue and cost, and (3) stops via a damage-aware gate. Across eight benchmarks, the method outperforms the strongest baseline by 1.03 percentage points while using approximately 28–44% fewer tokens.

Critical strengths
  • The problem is well motivated and addresses the inefficient allocation of computation across reasoning steps.

  • The approach of using an argument graph provides a convincing way to select promising revisions.

  • The paper's strongest contribution is its reduction in token usage while performing slightly better than the baselines.

Critical weaknesses
  • Performance gains are not consistent across benchmarks. On FLDx2 and HOVER, the method trails the strongest baselines by approximately 4.4 and 9.2 percentage points, respectively. It would be beneficial to provide a deeper analysis of why the method performs worse on these datasets and how practitioners can account for these performance differences.

  • Restricting the method to reflection only results in a relatively small performance drop of 0.33 percentage points. It would be useful to better understand the significance of this difference, for example, by reporting confidence intervals.

  • Can the authors clarify how much labeled data is needed for the predictors to perform well? While the token savings are substantial, practitioners would need a sufficiently large labeled dataset to train the classifiers.

Questions for authors

see above

2 comments
AuthorsOpenReview ↗

Response to Reviewer iKvp — Part 1/2 (W1)

Thank you for recognizing our motivation and the central contribution of token savings at competitive accuracy. We analyze dataset differences here. Part 2 continues that analysis with practical guidance, then addresses Resolve and training-data requirements.

Unless noted otherwise, results use the same dataset-stratified random slice of 1,500 problems from the eight reasoning benchmarks, evaluated with three generation seeds. Accuracy and tokens are three-seed means.

[W1] Performance gains are not consistent across benchmarks.(...)It would be beneficial to provide a deeper analysis of why the method performs worse on these datasets and how practitioners can account for these performance differences.

Re: Correction opportunities and revision risk differ by task. We use (1) matched baselines and (2) ablations to explain the differences.

(1) Matched baselines. We examine FLDx2, HOVER, and FEVEROUS, three datasets that depart markedly from the aggregate pattern. All four methods share AAS-MAD's opening and readout. Rescue/loss counts pool seeds. Loss includes incorrect or failed endpoints after a correct opening.

Table R1. Shared-opening results by dataset.

DatasetMethodAccuracy (%) ↑Tokens (k) ↓Rescues / losses
FLDx2Opening only81.567.03—
FLDx2AAS-MAD84.5717.6936 / 19
FLDx2MAD-M285.4638.2546 / 24
FLDx2Vanilla84.9332.2944 / 25
FLDx2Sparse86.5228.0249 / 21
HOVEROpening only54.5519.22—
HOVERAAS-MAD58.1127.5938 / 18
HOVERMAD-M260.2557.2565 / 33
HOVERVanilla60.6148.0754 / 20
HOVERSparse62.3945.3264 / 20
FEVEROUSOpening only63.1218.31—
FEVEROUSAAS-MAD62.9427.7911 / 12
FEVEROUSMAD-M259.0460.7514 / 37
FEVEROUSVanilla56.9147.5113 / 48
FEVEROUSSparse56.3844.3815 / 53

Bold italics/bold rank continuations within each dataset. Opening only is unranked.

Task properties alone do not explain the gap.

  • FLDx2 tests formal deduction.
  • HOVER requires multi-hop verification.
  • FEVEROUS combines text and tables.

HOVER and FEVEROUS both require evidence-based reasoning, yet show opposite relative results. We therefore examine differences in correction opportunities and revision risk, rather than explain the gap by task properties alone.

(2) Ablations identify different design trade-offs. Values are Full minus control in accuracy points, not additive contributions. All controls share the opening. One-Reflect also restricts action type.

Table R2. Dataset-wise ablations: Full minus control (pp).

ControlFLDx2HOVERFEVEROUS
Random + STOP+2.30+3.03−0.71
Fixed six+2.13−1.78+1.95
No graph inputs+0.89−1.07+1.24
No damage penalty+0.35−0.36+0.53
Reflect-only−0.350.000.00
AAS-MAD opening STOP + one Reflect−0.89+1.070.00

Among admitted states with complete action menus, the first action and the first-action-to-endpoint transitions are:

Table R3. Correctness transitions by revision stage.

DatasetAdmitted statesFirst action: rescues / lossesAfter first action: rescues / losses
FLDx217129 / 1311 / 10
HOVER12728 / 1512 / 5
FEVEROUS1049 / 83 / 4

FLDx2: the gap lies after revision starts.

  • Random selection, removing graph inputs, and fixed six lower accuracy. One Reflect improves it.
  • The first action provides most net benefit. Later continuation adds little.
  • Implication: improving stopping is more targeted than simply extending the same path. Part 2 tests this with the initial gate fixed.

HOVER: continued reasoning has greater correction value.

  • Learned selection beats random. Later transitions add net corrections, and fixed six beats Full. Unlike FLDx2, this supports later stopping.
  • Sparse rescues more errors with few extra losses. Its advantage comes from cases Full initially stops.
  • Implication: these results support improving both admission and stopping.

FEVEROUS: protecting correct openings dominates.

  • All four methods lose accuracy after debate. Full is slightly below opening-only but clearly above the other baselines.
  • Fixed six and removing graph inputs add rescues at a greater increase in losses. The damage penalty helps. Later transitions can undo first-action gains.
  • Implication: Full's advantage mainly comes from limiting harmful revision, rather than uniformly stronger correction.

Part 2 fixes the initial gate and exchanges only the continuation to test this explanation, then gives practical guidance.

AuthorsOpenReview ↗

Response to Reviewer iKvp — Part 2/2 (W1-W3)

[W1, continued] Separate admission from continuation. Each baseline applies AAS-MAD's frozen STOP once after the shared opening:

  • STOP: return the opening answer directly.
  • CONTINUE: hand control to MAD-M2, Vanilla, or Sparse, replaying its native continuation and later stopping rules.

AAS-MAD does not schedule later actions. All methods share the opening and readout.

Cells show accuracy (%) / total tokens (k) on matched states with recorded initial decisions: 564/560/559 observations for FLDx2/HOVER/FEVEROUS. Missing-gate observations (0/1/5) are excluded pairwise here. Bold italics/bold rank each metric within a column.

Table R4. Opening STOP controls: accuracy (%) / tokens (k).

Shared-opening continuationFLDx2 (↑ / ↓)HOVER (↑ / ↓)FEVEROUS (↑ / ↓)
AAS-MAD (Full)84.57 / 17.6958.21 / 27.1763.51 / 27.67
MAD-M285.46 / 38.2560.36 / 56.8859.57 / 60.15
MAD-M2 + AAS-MAD opening STOP86.52 / 19.1359.46 / 28.1761.54 / 28.58
Vanilla84.93 / 32.2960.71 / 47.4057.25 / 46.59
Vanilla + AAS-MAD opening STOP85.99 / 16.9457.86 / 26.6561.36 / 26.40
Sparse86.52 / 28.0262.50 / 44.7056.71 / 43.62
Sparse + AAS-MAD opening STOP87.06 / 15.2657.68 / 25.7261.00 / 25.15

The gate improves all three baselines on FLDx2 and FEVEROUS but removes much of Sparse's advantage on HOVER. Stopping value depends on the continuation, not just the problem.

For practitioners: use representative labeled validation data to measure rescues/losses by initial STOP decision and revision depth under the intended protocol.

  • FLDx2-like: retain selective admission. Test shorter continuations before increasing depth.
  • HOVER-like: if stopped cases yield net rescues, validate looser admission jointly with depth. More post-admission budget may not reach those cases.
  • FEVEROUS-like: retain protective stopping. Confirm net corrections before adding debate.

Choose the lowest-token configuration meeting the accuracy target.

[W2] Restricting the method to reflection only results in a relatively small performance drop of 0.33 percentage points. It would be useful to better understand the significance of this difference, for example, by reporting confidence intervals.

Re: The paired interval includes zero, so the expanded evaluation does not establish an average accuracy gain from Resolve. The control retains learned scheduling and STOP.

Table R5. Paired accuracy effects of Resolve.

EvaluationFull − Reflect-only, pp [95% CI]Exact McNemar p
Original 300 problems+0.33 [0.00, 1.00]1.00
1,500 problems × 3 seeds+0.04 [−0.09, 0.18]—
  • Why rare: Resolve uses two nominal generation calls versus one for Reflect. It therefore requires a larger predicted rescue benefit under cost-aware scheduling.
  • Why retained: we designed Resolve to study targeted interaction beyond reflection. It produces local rescues, while overall accuracy and token use remain similar to reflection-only. Its endpoint accuracy benefit is not yet established. Since the results do not establish a negative net effect, we retain it for mechanism studies.
  • What the experiments show: more complex information exchange is not always necessary.

[W3] Can the authors clarify how much labeled data is needed for the predictors to perform well?

Re: Training uses 1,139 FIT problems and 6,161 action observations, with automatically derived action labels.

(1) Corrected count. We apologize for the paper's incorrect 5,101 count. It omitted:

  • 480 later-state branches.
  • 580 earlier training observations from 56 problems, disjoint from current test and validation sets.

We will correct this in the revision. Main and supplementary experiments use the same frozen checkpoint.

(2) Training-size sensitivity. For each smaller size:

  • Refit only the two heads on three problem-grouped, dataset-stratified subsets. Evaluate each fit on all three generation-seed test sets.
  • Keep the feature pipeline fixed, including the scalar ranker fitted on full FIT. The 1,139-problem row uses the original heads.

Table R6. Predictor training size and unseen-domain AUROC.

Training (FIT) problemsRescue AUROC range ↑Damage AUROC range ↑
1110.780–0.8430.615–0.851
2830.793–0.8500.714–0.839
5680.837–0.8690.753–0.839
1,1390.853–0.8670.797–0.826

Smaller subsets retain useful rescue discrimination, while damage prediction varies more with limited data.

Action labels come from comparing pre/post-action decisions with gold answers. No manual annotation of individual action examples is needed.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Thank you for your follow-up

Thank you for your comprehensive review and constructive feedback. We especially appreciate your willingness to reconsider your assessment and your recognition of our additional matched-opening, reranker, and budget-controlled experiments. Your feedback has been invaluable, and we will incorporate these results and clarify the contributions and limitations in our revision.

Best regards,
The Authors

AuthorsOpenReview ↗

Thank you for your follow-up

Thank you for your encouraging response and continued support of our work. We are grateful for your confidence in our manuscript and your thoughtful engagement throughout the review process. Your encouragement means a lot to us, and we sincerely appreciate the time and consideration you have given this submission.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.