acceptodds
Withdrawn from ICLR 2027

Self-Improvement of Language Models by Post-Training on Multi-Agent Debate

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Self-improvement, where models improve beyond their current performance without external supervision, remains a challenge. The core difficulty is sourcing a training signal stronger than what the model itself can currently produce. Majority voting has been shown to provide such a signal by aggregating over multiple samples, helping mitigate some of the inconsistencies in LM reasoning. In this work, we study multi-agent debate—where models collaborate and exchange reasoning over multiple rounds—as an alternative to single-round majority voting for constructing self-improvement signals. We introduce Multi-Agent Consensus Alignment (MACA), which uses reinforcement learning (RL) to post-train models on debate-derived signals. We study scalar consensus rewards and preference learning over majority and minority reasoning traces, comparing them with SFT on consensus-supporting traces. MACA improves (1) multi-agent debate accuracy (up to +26.87% on MATH), (2) single-agent accuracy (up to +21.51% on MathQA), and (3) self-consistency (up to +27.6% on GSM8K). We also observe gains on unseen benchmarks, including +16.3% on GPQA and +11.6% on CommonsenseQA. In a separate rollout-matched study, we evaluate online self-debate GRPO, in which one policy debates among its own rollouts during RL. The resulting policy improves held-out GPQA accuracy by 3.39% over the zero-shot base model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-agent Debate, Self-improvement, LLMs, Self-Consistency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qVPqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the reliance on human labels in language model self-improvement and proposes MACA, a framework that uses consensus from multi-round multi-agent debate as a label-free training signal. It post-trains models with MV-SFT, MV-GRPO, MV-DPO, and MV-KTO, and introduces online self-debate GRPO for single-agent self-improvement, improving multi-agent collaboration and single-agent reasoning on math, science, and commonsense tasks.

Critical strengths

1.The paper systematically turns debate consensus into a training signal and compares SFT, GRPO, DPO, and KTO in one framework, offering useful evidence on whether multi-round debate consensus beats single-round majority voting. 2.Online self-debate is compared with TTRL, ScPO, and others under matched rollout budgets and training settings, with added debate-protocol ablations, making the conclusions reasonably convincing.

Critical weaknesses

1.Online self-debate is currently validated mainly on Qwen3-4B and GPQA, where the test set is relatively limited and the gain is close to its standard deviation; additional results on more models and benchmarks would strengthen the conclusion. 2.The paper reports that the debate majority is incorrect on about 37.9% of training examples; clarifying how performance changes on originally correct examples would help better delineate the method's applicability.

Questions for authors

1.Are there preliminary results for online self-debate on MATH or GSM8K, and if so, are they consistent with the GPQA findings? 2.For examples with an incorrect debate majority, could the authors briefly report how accuracy changes on examples that were originally correct?

Reviewer_eAnNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper propose MACA- where 3 copies of the same small model debate for 2 rounds and the final majority answer is used as a label to post-train the model with GRPO/DPO/KTO, and compare it with SFT on majority traces. Experiments are on qwen 1.5b llama 3b/8b and phi 3 mini trained on MATH, GSM8K and MathQA with 256 max tokens and temperature 1.0... paper report better debate accuracy, single agent accuracy and self-consistency, and transfer to SVAMP, GPQA and CSQA. They also add an online self-debate GRPO on qwen3 4b which improves GPQA by 3.39 over the base model.

Critical strengths

1- Wide set of experiments... 4 model families, 3 training objectives + SFT, 3 seeds in main tables, transfer, iterative training and an online version. Training on round 2 responses with peer context is simple, cheap and needs no labels, and Table 14 shows it helps final round debate accuracy in most settings.

2- Authors report results that go against their main story in the appendix (truncation analysis, no improvement on AMC when the base model gets nothing right, and the wrong majority analysis in Sec 5.3). This is rare to see and it makes the claims easier to check.

Critical weaknesses

1- Novelty is limited.. MV-GRPO is basically TTRL with the label taken from the debate final round, and MV-DPO is close to ScPO on round 2 traces. DTE (Srivastava et al., EMNLP 2025, arXiv 2505.15734) already uses 3 agent debate consensus as a GRPO reward without ground truth, on similar models (Qwen2.5-1.5B, Llama-3B/8B) and similar datasets (GSM8K, MATH, GPQA, CSQA), with SFT and DPO baselines. SDRL (arXiv 2601.22297) already trains one policy on its initial and debate conditioned rollouts with GRPO, which is very close to Sec 5.3. CoMAS (ICLR 2026) and MAPoRL (ACL 2025) also train models from multi-agent interaction. None of these are discussed/mentioned in the paper. Please clarify what is new here in your paper.

2- Most of the improvement looks like learning to finish inside 256 tokens. Base models are far below their known numbers... Qwen2.5-1.5B gets 23.0 GSM8K / 7.67 MATH vs 73.2 / 55.2 in the official Qwen2.5 report, and Llama-3.1-8B gets 57.93 / 22.93 vs 84.5 / 51.9 on the Meta model card. Even after MACA, Llama-8B (77.36 / 46.00) is still below the official base. When the limit goes to 512 tokens, Llama-3B GSM8K improvement goes from +10.8 to -0.2. GPQA stays below random chance (25%) everywhere (0.5% to 16.8% at 256 tokens, 23.7% at 512), so the GPQA number in abstract dont say anything about reasoning. Also a cut off answer can never match the majority, so it always becomes the rejected sample.. many DPO pairs are just finished vs unfinished answers. So, the claims are misleading.. Please clarify. Also, the debate vs single round majority vote claim is not shown fairly. Main tables compare only with base and MV-SFT, TTRL and ScPO are only in App P on 2 datasets, and TTRL drops well below base there (Llama-3B GSM8K 72.8 to 50.5) so it looks not tuned. Debate also uses 6 samples per question (3 + 3) vs fewer in single round. In Sec 5.3 the test set is 128 questions, so 3.39 is about 4 questions with std 2.39, and even supervised GRPO with gold labels is below the base model.. very hard to conclude anything from this.

Questions for authors
  1. Please clarify the novelty compared to DTE, SDRL, TTRL and ScPO - a clear distinguish b/w what was already known and what your paper brings..

  2. Rerun the main tables with 1024 to 2048 tokens, no quantization, and the base model at greedy or temperature 0.6? How much improvement is left?

  3. Can you add a format only baseline.. DPO where chosen is any finished answer and rejected is a cut off one? If this gets close to MACA, the debate signal is not the reason.

  4. Qwen2B debate accuracy on MATH is 32.4 so most majority labels must be wrong, but MV and GT give almost same result (21.33 vs 23.00, Table 11). How? Also several GT runs are far below base (Llama-3B KTO-GT GSM8K 27.27 vs 57.33).. why?

  5. Base numbers change across tables (Llama-3B MATH is 27.87 / 32.0 / 41.2 / 40.2 in Tables 2, 3, 21, 23). Which base is each improvement measured from?

Reviewer_3sN6OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce a framework, MACA, for multi-agent LLM post-training that uses labels derived from majority voting over multi-agent debate. The authors evaluate this framework on a number of simple reasoning tasks.

Critical strengths

The paper is well-written, and the MACA framework is well-motivated.

Critical weaknesses

The authors omit [1] from their related work section, and do not conduct any empirical comparisons to it. I regard this omission as being potentially significant for the paper's stated novelty of conceptual contribution, as well as it empirical claims.

In particular, the authors should clarify how their approach introduces novelty beyond replacing labels used by [1] with labels derived from multi-agent debate. Without clear extra demonstration of novelty beyond this, I do not see the paper as sufficiently novel to warrant acceptance.

Lastly, empirical comparisons are restricted to simple benchmarks and small models; however, this is forgivable given the compute requirements of larger-scale evaluations. It limits, however, the degree to which the authors' results can be assumed to transfer to larger models and harder benchmarks.

[1] Motwani et al., MALT: Improving Reasoning with Multi-Agent LLM Training, COLM 2025.

Questions for authors
  1. How is your method novel compared to [1]?
  2. Can you give an analysis of finite-sample estimator properties (in particular, bias and variance) of Equ. 1?
  3. Table 6 outperformance does not seem statistically significant, is my reading correct?
  4. You state that "we treat answers that persist through debate as a heuristic signal of stable reasoning." (line 145) - do you have any independent evidence that this is justified under e.g. model correlations or other debate pathologies?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.