acceptodds
Under review as a conference paper at ICLR 2027

Beyond Solo and Consistency: Vindicating Multi-Agent Debate through Conditional Progressive Pruning

Abstract

Large Language Model (LLM) based Multi-Agent Debate (MAD) is one of the most effective test time scaling techniques. Through multi-round communication, agents complement each other in knowledge and reasoning and solve tasks that no single member can solve. However, existing MAD frameworks fail to beat strong Single Agent and Consistency-based baselines under the same strict cost limit, which shakes the foundation of the MAD field. We propose Conditional Progressive Pruning (CPP), a lightweight pruning framework that fully exploits multi-round MAD. CPP outperforms all existing MAD frameworks on multiple dominated benchmarks. It is also the first to fully outperform consistency methods. Our code, detailed Agent interaction records will be released soon.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Agent Debate, Reinforcement Learning, Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_AdJfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Conditional Progressive Pruning (CPP), a learned controller for heterogeneous multi-agent debate. CPP sequentially decides which earlier responses each agent should receive, conditioning later pruning choices on selected responses and prior debate rounds. A FuseGLU architecture combines the current query, candidate responses, and Agent Behavior Embeddings extracted from responses. The controller is trained with policy gradients using correctness-transition rewards and discounted returns over related actions. On sampled MATH-Hard, GPQA-Diamond, and MMLU-Pro tasks, experiments compare CPP with debate, self-improvement, and scaled consistency baselines using model-dependent inference costs. Additional experiments examine reward design, behavior embeddings, adversarial conditions, output truncation, and alternative model configurations. Reported results show modest accuracy improvements with adjustable accuracy–cost trade-offs.

Critical strengths
  1. Conditioning pruning choices on actual intermediate responses and previously selected messages is a reasonable way to exploit information arising during debate. The approach goes beyond static topology built solely from model characteristics.

  2. Empirical comparisons include closely related learned methods such as ARG-Designer and AnyMAC, as well as scaled consistency baselines. Accounting for heterogeneous model costs is more informative than comparing raw token counts alone.

Critical weaknesses
  1. The paper is difficult to read. The layout is overly dense, with long blocks of text, insufficient structure, and a lack of clear logical progression.

  2. The claim of outperforming consistency methods comprehensively “at strictly identical cost” exceeds the evidence. On MMLU-Pro, CPP achieves 66.33% at 99.43% normalized cost, while Consistency-Fuse achieves 66.17% at 78.94% cost. Budget–performance curves and paired significance analysis would better show whether such a small accuracy gain warrants the extra computation.

  3. Each main test set contains only 100 questions, and the principal configuration uses three fixed models and three debate rounds. Scaling experiments rearrange these models rather than testing substantially larger teams. Although variability is reported, several gains are small relative to the stated standard deviations, limiting claims of broad superiority and scalability.

  4. Important training and cost-accounting details remain insufficiently specified. For correctness-transition rewards, the paper should clarify how intermediate answers are generated and rewards assigned after each edge decision, and report rollout costs, model pricing, encoder configuration, and final hyperparameters. The dependency table in Appendix A does not fully describe the training procedure.

  5. The key autoregressive conditioning mechanism is not cleanly isolated. Table 6 studies embeddings, fusion structure, and rewards, but neither removes dependence on previously selected messages under fixed architecture and training budgets nor distinguishes within-round from cross-round conditioning.

Questions for authors

See Weaknesses.

1 comment
AuthorsOpenReview ↗

First, we really appreciate your valuable comments, and we apologize for the poor quality of the writing in current version. We consider all our experimental results very solid, but the writing does have problems. The responses below are meant for productive academic discussion rather than as a rebuttal.

Response 1:

  1. We strictly controlled variables in all experiments. FuseGLU, the debate reward function, and Agent Behavior Embedding are all proposed in our paper, yet when comparing CPP with ARG-Designer and AnyMAC, we applied the same feature extraction, reward function, and model architecture to ARG-Designer and AnyMAC as well. The original ARG-Designer and AnyMAC therefore do not perform as well as the results reported in our paper; those results already include the gains from our Agent Behavior Embedding, debate reward, and FuseGLU architecture.
  2. ARG-Designer performs conditional pruning, but it does not consider intermediate results; it only models interactions between agent embeddings. AnyMAC considers intermediate results (progressive pruning), but it selects each edge independently, without conditional pruning.
  3. Therefore, the main table already validates the contribution of CPP with all other variables strictly controlled. More precisely, it shows CPP's contribution over enhanced versions of ARG-Designer and AnyMAC.

Response 2:

  1. We applied the same cost cap to all baselines and to CPP. The remaining cost differences come from where each algorithm converges. Note that some methods that look cheaper have already reached their optimum, so raising their cost to the shared cap would not improve their performance. This is especially clear for the consistency-based baselines. We also gave the consistency baselines an advantage, which makes our conclusion stronger: for consistency, we searched over different agent types and numbers and kept the best, while all debate methods use a fixed number of agent types and a fixed topology. Moreover, CPP's actual gain is larger than the table shows, because ARG-Designer, AnyMAC, and the other baselines already include the gains from our reward, features, and architecture.
  2. Every experiment was repeated six times (random seeds: 2025, 2026, 27, 42, 58, 60).
  3. Normalized cost means we do not simply count tokens, because different token types have different prices. We convert every token type (e.g., input, output, cached) of every model to the equivalent price of GPT input tokens, so the numbers reflect real cost savings. The normalized cost excludes the pruner and the encoder model, since both can be lightweight open-source models.
Reviewer_9q2COpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the difficulty of Multi-Agent Debate (MAD) in consistently outperforming single-agent and consistency-based methods under the same computational budget.

The authors propose Conditional Progressive Pruning (CPP), a framework that uses Agent Behavior Embeddings to model agent characteristics and FuseGLU to autoregressively select communication edges based on intermediate responses and previous communication decisions. In addition, the authors design a debate-specific reward mechanism and optimize the communication topology through reinforcement learning.

Experiments are conducted on MATH-Hard, GPQA-Diamond, and MMLU-Pro. The authors claim that CPP is the first method to comprehensively outperform existing MAD and consistency-based methods under strictly equal computational budgets, thereby demonstrating the effectiveness of MAD.

Critical strengths

S1. The research problem is meaningful. Evaluating whether MAD can outperform consistency-based methods under the same computational budget is an important question in multi-agent collaboration.

S2. CPP autoregressively selects communication edges based on intermediate responses and previous communication decisions. Compared with methods relying solely on static agent profiles, this content-aware dynamic communication strategy offers some technical value.

S3. The experiments cover single-agent, consistency-based, heuristic MAD, and learning-based MAD baselines. The paper also provides several ablation studies on the model architecture and reward design.

Critical weaknesses

W1. Inconsistencies in Data Splitting and Model Initialization

Inconsistent dataset split sizes. Section 4.1 states that GPQA-Diamond is divided into training, validation, and test sets containing 98/50/100 samples, totaling 248 samples. However, the standard GPQA-Diamond dataset contains only 198 questions. The paper does not explain where these 248 samples come from or whether there is any overlap among the training, validation, and test sets.

Inconsistencies in Behavior Embedding initialization. Section 3.1.1 states that Behavior Embeddings are initialized using agent responses from the validation set, while Section 4.3.3 states that the main experiments use 60 samples for initialization. However, the GPQA-Diamond validation set contains only 50 samples. Furthermore, Table 6 includes a 150-sample ablation setting, while the MATH-Hard validation set contains only 100 samples. The authors should clarify the actual sources of these initialization samples.

W2. Overstated Claims That Are Not Sufficiently Supported by the Experimental Results

The paper repeatedly claims in the Abstract, Introduction, and Conclusion that CPP is the first method to fully outperform consistency-based methods under the same computational budget, thereby establishing the value of MAD and settling the debate over its effectiveness. However:

The claim of being the first is insufficiently justified. Prior work, Efficient Multi-Agent Reasoning via Confidence-Guided Adaptive Debate (ICML 2026), proposed LASE, which uses selective interaction and an asymmetric Leader–Supporter communication structure. It reported higher accuracy than Majority Voting on multiple benchmarks while substantially reducing token consumption. Although the experimental settings differ, this work has already demonstrated accuracy and efficiency advantages of selective debate-based collaboration over voting. CPP does not discuss this important prior work, making its claim of being the first to establish the advantages of MAD questionable.

Table 2 does not sufficiently support the claim that CPP consistently outperforms existing MAD methods. On MATH-Hard, CPP achieves 68.17% accuracy compared with 68.00% for ARG-Designer, an improvement of only 0.17 percentage points. On GPQA-Diamond, CPP achieves 68.67% compared with 68.17% for IoE-Debate, an improvement of only 0.50 percentage points. These differences are much smaller than the reported standard deviations, and no statistical significance tests are provided. Therefore, the results do not convincingly support the claimed superiority over existing MAD methods.

W3. Missing Ablation Study on the Core Autoregressive Conditional Pruning Mechanism

The central technical contribution of this paper is Autoregressive Conditional Pruning, which conditions subsequent communication decisions on previous decisions and selected intermediate responses. However, the ablation studies in Table 6 mainly examine Behavior Embeddings, the FuseGLU architecture, and reward design, without directly evaluating the necessity of autoregressive conditioning. The authors should conduct an ablation that removes the conditional dependencies between communication decisions, allowing each edge to be selected independently, while keeping the model architecture, training data, reward design, and parameter count unchanged. This would help isolate the contribution of the proposed autoregressive mechanism.

W4. Insufficient Experimental Details, Incomplete Appendices, and Presentation Issues

Missing implementation details. The paper does not adequately specify the complete cost calculation formula, token pricing and cost normalization across different LLMs, or the detailed hyperparameters used for policy gradient training. It also lacks a clear description of how the final answers from multiple agents are aggregated. In addition, the training, validation, and test configurations used for different baselines are not sufficiently explained.

Incomplete appendices. Appendices B, C, and D consist almost entirely of tables, without sufficient explanations of the experimental settings or analysis of the results. These sections appear incomplete.

Poor figure quality. Some text and legends in Figures 1 and 2 are blurry and difficult to read. Several figures also have noticeable border artifacts, which negatively affect readability.

Questions for authors

Q1. Could the authors explain why the reported GPQA-Diamond splits exceed the size of the standard dataset, and clarify the inconsistencies between the Behavior Embedding initialization sample sizes and the validation set sizes?

Q2. Could the authors provide an ablation that removes autoregressive conditioning and makes communication edge decisions independent, to verify the contribution of this core mechanism?

1 comment
AuthorsOpenReview ↗

First, we really appreciate your valuable comments, and we apologize for the poor quality of the writing in current version. We consider all our experimental results very solid, but the writing does have problems. The responses below are meant for productive academic discussion rather than as a rebuttal.

Q1:

  1. For GPQA, the training set has 98 samples and the test set has 100, so 98 + 100 = 198 samples, which make up the full GPQA-Diamond set. All validation samples are drawn from the part of GPQA outside GPQA-Diamond.

  2. Agent behavior embeddings are initialized using training samples only; no validation samples are used. The training set has 150 samples for MATH-Hard and MMLU-Pro and 98 for GPQA. In the main experiments, the agent embeddings for all datasets are initialized with 60 training samples.

  3. The initialization experiment with 150 samples is part of the ablation study, which we ran only on MATH-Hard. There we tried initializing agent embeddings with various amounts of data, up to 150 samples, which is the entire training set.

Q2:

  1. We strictly controlled variables in all experiments. FuseGLU, the debate reward function, and Agent Behavior Embedding are all proposed in our paper, yet when comparing CPP with ARG-Designer and AnyMAC, we applied the same feature extraction, reward function, and model architecture to ARG-Designer and AnyMAC as well. The original ARG-Designer and AnyMAC therefore do not perform as well as the results reported in our paper; those results already include the gains from our Agent Behavior Embedding, debate reward, and FuseGLU architecture.
  2. ARG-Designer performs conditional pruning, but it does not consider intermediate results; it only models interactions between agent embeddings. AnyMAC considers intermediate results (progressive pruning), but it selects each edge independently, without conditional pruning.
  3. Therefore, the main table already validates the contribution of CPP with all other variables strictly controlled. More precisely, it shows CPP's contribution over enhanced versions of ARG-Designer and AnyMAC.

Reply to W2:

  1. LASE's main contribution is that it flexibly rearranges the MAD further orchestration round by round, essentiall a great early stopping mechanism. However, its experimental claim that MAD is superior to consistency is flawed, for the following reasons:

1.1 LASE uses a homogeneous MAD team. Under this setting, the Google DeepMind paper Large Language Models Cannot Self-Correct Reasoning Yet has already shown that MAD cannot outperform homogeneous consistency under the same compute budget. MAD only appears better than consistency in LASE because a query-level dynamic method (different from ours, LASE adapting on agent numbers and MAD structure) is compared against benchmark-level static consistency. This comparison is not fair; to show that MAD is superior to consistency, this setup is simply wrong. Consistency should be given the same query-level dynamics, which brings us back to the scenario already refuted by the DeepMind paper.

1.2 Showing that MAD is superior to consistency at equal cost takes two steps, i.e. homogeneous and heterogeneous consistency. First, since homogeneous MAD cannot beat consistency, a heterogeneous MAD setting should be used. At the same time, one must show that it beats homogeneous consistency methods too, not only heterogeneous ones. This is why our consistency experiments are so detailed.

1.3 To show that MAD beats a given consistency setting, one should first follow the DeepMind paper and flatten the MAD topology. For example, a 3×33 \times 3 MAD flattens into 3×3 \times GPT + 3×3 \times Grok + 3×3 \times Mistral. This is still not enough, because communication in MAD adds cost, so the comparison remains unfair to consistency. Consistency must be given more agents until it reaches the same budget cap. Agents can be added either by adding only the strongest agent (best adding) or by adding agents proportionally (fuse adding).

1.4 After fixing the maximum number of agents for consistency, we search the agent counts in order of agent strength.For example, assuming a 3×33 \times 3 MAD corresponds to 7×7 \times Grok + 3×3 \times GPT + 3×3 \times Mistral, with Grok > Mistral > GPT. When searching over the number of agents nn, we first add Grok, then Mistral, and finally GPT. This is the greedy order within a heuristic search space.

1.5 Using the example above, showing that MAD is superior to consistency requires showing, at a minimum, that MAD beats consistency for every consistency setting (homogeneous and heterogeneous, best adding and fuse adding) and for all 9≤n≤139 \le n \le 13. LASE only shows this for n=13n = 13, not for 9≤n≤139 \le n \le 13. LASE also never goes beyond the homogeneous setting, so it cannot refute the argument of the DeepMind paper.

1.6 Our paper is indeed the first work to rigorously establish this conclusion, and we prove a stronger version of it: our results cover 1≤n≤131 \le n \le 13, while the MAD agent types and numbers stay fixed. In effect, we gave consistency an advantage in agent types and numbers and did not change the MAD topology, leaving a large optimization space unused.

Reviewer_pGhJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Conditional Progressive Pruning (CPP), a framework for heterogeneous multi-agent debate (MAD). It combines Agent Behavior Embedding, FuseGLU, autoregressive pruning decisions, and reinforcement learning with debate-tailored rewards. Experiments on MATH-Hard, GPQA-Diamond, and MMLU-Pro report accuracy improvements over MAD and consistency baselines, alongside analyses of normalized cost, adversarial robustness, and output truncation.

Critical strengths
  1. The paper addresses an important practical question: how to reduce unnecessary communication in multi-agent debate while preserving the benefits of collaboration.
  2. Conditioning pruning decisions on previously selected responses is well motivated. The usefulness of a response depends on the information already available to the receiving agent, so conditional selection may help avoid redundant or misleading messages.
  3. The experiments cover MAD methods, consistency methods, adversarial settings, component ablations, and output truncation. These comparisons provide useful insights into CPP’s behavior within the evaluated settings.
Critical weaknesses
  1. Some reported gains need stronger budget-matched comparisons and statistical support. On MMLU-Pro, CPP achieves 66.33% accuracy at 99.43% normalized cost, whereas Consistency-Fuse achieves 66.17% at 78.94% cost. Thus, a marginal accuracy improvement comes with a substantially higher cost. Other improvements are also small, such as the 0.17-percentage-point gain over ARG-Designer on MATH-Hard. The MMLU-Pro comparison would benefit from closer budget matching. Given that each benchmark uses only 100 test questions, paired statistical analyses are also needed to assess the reliability of these small gains.
  2. The implementation of action-level rewards is insufficiently explained. The reward depends on answer correctness before and after each pruning action, but the paper does not clearly explain how these intermediate answers are generated or whether additional LLM calls are required. This makes the training procedure and its actual cost difficult to assess.
  3. The contribution of conditional pruning is not directly isolated. Existing ablations examine Agent Behavior Embedding, FuseGLU, and reward variants, but do not directly compare conditional and independent edge selection under otherwise identical settings. Consequently, the extent to which the improvements arise from modeling dependencies between pruning decisions remains unclear.
  4. The evaluation is limited to a small, fixed debate setting. The main experiments use three agents and three rounds, and no transfer results across tasks or model teams are reported. It remains unclear how CPP performs beyond these settings. The scalability and generalization claims should be qualified accordingly.
Questions for authors
  1. How are intermediate answers generated and evaluated for the action-level rewards in Section 3.2.1? Does each pruning action require an additional LLM call? How are ambiguous, malformed, or unevaluable responses handled, and what training cost do these evaluations introduce?
  2. Can you directly compare conditional and independent edge selection while keeping Agent Behavior Embedding, FuseGLU, the reward function, and the training budget unchanged, to isolate the contribution of conditional pruning?
  3. Do you train a new pruning policy and new behavior embeddings for each benchmark and model team? Can CPP be used on new tasks or with different LLMs without retraining?
  4. Do CPP’s gains remain reliable when inference budgets are matched? Please report the number of independent runs, paired confidence intervals or significance tests for the performance differences, and whether normalized cost includes text encoding and pruning-controller overhead.
  5. The main experiments use three agents and three rounds. How do CPP’s accuracy and cost change with more agents or more debate rounds?
1 comment
AuthorsOpenReview ↗

First, we really appreciate your valuable comments, and we apologize for the poor quality of the writing in current version. We consider all our experimental results very solid, but the writing does have problems. The responses below are meant for productive academic discussion rather than as a rebuttal.

Q1:

  1. Intermediate results are the outputs of the previous round of LLMs. For example, for a second-round LLM, the lightweight pruner uses the first-round outputs as input to decide whether to prune.
  2. Once the action for an edge is set, there are two cases. If the action is pruning, we make no extra LLM call and treat the change as correct →\rightarrow correct or incorrect →\rightarrow incorrect. If the action is not pruning, we make one extra LLM call to find out how the action changes the model's correctness.
  3. We strictly control the LLM output format so that every output can be parsed. Corner cases, such as exceeding max tokens or hitting API policy limits, count as incorrect answers. We evaluate an action (prune or not prune) by how the target LLM's correctness changes from before the action to after it.
  4. In our setup of three rounds with three LLMs per round, the most LLM calls happen when every action is "not prune": 3+3×4+3×4=273 + 3 \times 4 + 3 \times 4 = 27, which is 3×3\times the cost at test time. However, sparse sampling keeps the actual training cost (e.g. 2 epochs, 150 training samples per epoch) fully manageable. And we conduct training without CoT, which reduces the number of input tokens. Unlike inference, this training cost is incurred only once. Consequently, when comparing two methods A and B, if A has a lower test-time cost than B, A will eventually pay off in the long run, even if B requires no training at all.

Q2:

  1. We strictly controlled variables in all experiments. FuseGLU, the debate reward function, and Agent Behavior Embedding are all proposed in our paper, yet when comparing CPP with ARG-Designer and AnyMAC, we applied the same feature extraction, reward function, and model architecture to ARG-Designer and AnyMAC as well. The original ARG-Designer and AnyMAC therefore do not perform as well as the results reported in our paper; those results already include the gains from our Agent Behavior Embedding, debate reward, and FuseGLU architecture.
  2. ARG-Designer performs conditional pruning, but it does not consider intermediate results; it only models interactions between agent embeddings. AnyMAC considers intermediate results (progressive pruning), but it selects each edge independently, without conditional pruning.
  3. Therefore, the main table already validates the contribution of CPP with all other variables strictly controlled. More precisely, it shows CPP's contribution over enhanced versions of ARG-Designer and AnyMAC.

Q3:

  1. Currently, the lightweight pruner model is trained separately for each task. However, it can be well-trained on small training data and the cost is not high.
  2. This is a great question. It concerns the scalability of the pruner in MAD, including changes in agent positions (which our paper covers), changes in topology, and changes in the types and number of agents, and across different tasks. Scalability is not the focus of this paper, but we agree it is a very valuable research direction.

Q4:

  1. We applied the same cost cap to all baselines and to CPP. The remaining cost differences come from where each algorithm converges. Note that some methods that look cheaper have already reached their optimum, so raising their cost to the shared cap would not improve their performance. This is especially clear for the consistency-based baselines. We also gave the consistency baselines an advantage, which makes our conclusion stronger: for consistency, we searched over different agent types and numbers and kept the best, while all debate methods use a fixed number of agent types and a fixed topology. Moreover, CPP's actual gain is larger than the table shows, because ARG-Designer, AnyMAC, and the other baselines already include the gains from our reward, features, and architecture.
  2. Every experiment was repeated six times (random seeds: 2025, 2026, 27, 42, 58, 60).
  3. Normalized cost means we do not simply count tokens, because different token types have different prices. We convert every token type (e.g., input, output, cached) of every model to the equivalent price of GPT input tokens, so the numbers reflect real cost savings. The normalized cost excludes the pruner and the encoder model, since both can be lightweight open-source models.

Q5:

  1. This is a great question, but scalability is not the focus of this paper.
  2. In MAD, once the types, number, and arrangement of agents are fixed, the agents' output lengths do not change much across rounds. Cost differences therefore come mainly from the input side, driven by which edges are selected.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.