acceptodds
Under review as a conference paper at ICLR 2027

Alignment by Confident Debate: Preference Learning from Multi-Agent Debate

Abstract

Post-training language models for reasoning typically rely on gold answers, task-specific verifiers, or learned reward models, limiting scalability when such signals are scarce. Multi-agent debate offers a self-supervised alternative, but converting debate into reliable training supervision remains challenging: majority voting weights agreeing responses equally, lacks a principled tie-breaking rule, and is ill-defined for open-ended tasks without a canonical answer; likelihood-based scoring can conflate fluency with correctness; and homogeneous committees may reinforce correlated errors. We introduce **Alignment by Confident Debate** (AbCD), which converts debate traces into preference data without gold answer labels. A sampling-diverse committee debates each problem, after which final answers are clustered by task-level semantic equivalence. We use verifiable task’s (mathematical reasoning, multiple-choice quantitative QA, and executable code-generation) as a controlled test bed: their answer keys or executable checkers let us evaluate the quality of debate-derived supervision against ground truth, even though the oracle itself never uses gold labels. Post-training on the resulting preferences outperforms inference-only ensembling and debate, as well as SFT on the same traces, across five model families spanning 1B to 8B parameters and four datasets, improving consensus accuracy, with a maximum gain of . Ablations show that the approach transfers to a larger model and to code generation, while also revealing limitations of heterogeneous committees and agreement-based weighting. Overall, the results suggest that debate traces can yield useful preference supervision when answer equivalence is combined with confidence-weighted disagreement.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
alignment, multi-agent debate, preference learning, self-training, confidence estimation, mathematical reasoning, code generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6gfzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors tackle a scenario where reliable supervision signals for post-training are scarce. In such a scenario, the authors argue that multi-agent debate could serve as a solution, and propose Alignment by Confident Debate (AbCD), where they convert debate traces into preference data without gold answer labels. The procedure is to have multiple agents generate solutions and revise them after seeing each other’s responses. The final responses are grouped based on answer equivalence, and each group is scored by summing the agents’ answer-span confidence scores. Responses from the highest-scoring group are treated as preferred, while the remaining responses are treated as dispreferred. The authors evaluate this supervision using SFT, DPO, KTO, and GRPO across mathematical reasoning, quantitative QA, and code generation, and report improvements over inference-only ensembling and debate.

Critical strengths

The paper studies a useful problem: obtaining post-training supervision when gold answers are unavailable/sparsely available. The authors propose a practical way to construct preferences without gold answer labels. The approach uses debate traces and token probabilities to identify preferred and dispreferred responses.

The authors evaluate whether these preferences actually help post-training. They consider multiple models and training objectives, and report improvements in both consensus accuracy and average agent accuracy. This provides evidence that the generated preferences can improve model performance beyond inference-time answer selection.

Critical weaknesses
  1. The contribution of the confidence oracle is not clear. Table 1 mainly compares post-training against inference-only baselines and SFT. These comparisons do not establish the benefit of confidence-based labels over majority-based labels. Table 3 addresses this partially only for Qwen on GSM8K, where the default span oracle improves over full-trace confidence only by 0.5 points. Given prior debate preference training (MACA), whose approach is very very similar to the setup of the authors, except for the confidence based clustering as a first step. and confidence-weighted voting (CISC), the authors need broader comparisons against majority-based DPO and DPO on independent samples using the proposed approach.

2).The agents use temperatures from 0.01 to 1.0, but Section 3.2 does not clarify whether confidence uses probabilities before or after temperature scaling and top-p filtering. If the transformed probabilities are used, low-temperature agents can receive higher confidence without being more accurate. The authors should specify the scoring distribution and test whether the advantage persists when all candidates are scored under a common distribution.

  1. Non-code generation is capped at 256 tokens, and in the approach the authors are counting unparseable responses as incorrect and label them as dispreferred. Appendix J acknowledges truncation problems, particularly for Llama on GSM8K. Training could therefore improve measured accuracy by producing shorter, more consistently parseable answers. The authors should report truncation and parseability rates and repeat the comparison with a longer generation budget. Table 13 measures agent accuracy after debate, so it also does not establish broad improvements in standalone reasoning.

  2. Given that reasoning models are becoming increasingly the de facto standard across, various standards, the utility of these supervision signals derived from debate is not very clear if it will work these thinking models. The claim made by the authors about the signal being useful for the models they currently evaluated might be acceptable, but if they will generalize to reasoning models is questionable.

[1] Self-Improvement of Language Models by Post-Training on Multi-Agent Debate [2] Confidence Improves Self-Consistency in LLMs

Questions for authors

Can the authors compare AbCD against [1] and DPO trained on independent samples using the same confidence-based selection, under matched training budgets? Do the gains from the proposed oracle persist beyond Qwen on GSM8K? Also, can the authors perform this analysis on harder tasks, where the even after debating it is hard to arrive at the correct answer? This will help in proving the usefulness of the paper and make it stronger.

Are confidence scores computed before or after temperature scaling and top-p filtering? Does the advantage persist when all candidates are scored under a common probability distribution, removing the effect of different decoding temperatures?

What are the truncation and parseability rates before and after training? Do the improvements persist with a longer generation budget, and does the trained model improve when answering independently without debate, on harder tasks like AIME, OlympiadBench, GPQA-D? These results would help distinguish reasoning improvements from better formatting or shorter responses and also the generalizability of the proposed approach.

Can the authors evaluate AbCD on reasoning/thinking models with sufficient generation budgets? Does debate-derived supervision provide additional gains for these models, or is its usefulness primarily limited to the instruction-tuned models currently evaluated?

Reviewer_ebq3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper studies the problem of training LLMs with multi-agent debate. The authors systematically test and find that when debate happens for >= 3 rounds, we see representational collapse (unanimous agreement between agents) go up, but accuracy plateaus.
  • The authors cap debate at one round and also propose to score token confidence only using the final answer span, rather than grading the whole output. This avoids being fooled by flawed reasoning. Answers are then grouped into buckets and the bucket with the highest combined answer confidence becomes the "preferred" solution, while the others become the "dispreferred" one.
Critical strengths
  • The paper makes a strong empirical finding that multi-agent debate's benefits plateau after just one round. This seem to be an important result that previous papers in the area did not find.
  • The authors propose a practical recipe to make multi-agent debate work, which involves capping at 2 rounds, temperature diversity, and using answer confidence.
  • Fluency bias and the answer-span solution were interesting and impactful.
Critical weaknesses
  • My primary takeaway from this paper is that there are a set of engineering choices that can improve MAD. This is useful, but for a stronger paper, it would have been nice to have more scientific / theoretical grounding.
  • Are there ways that we can engineer MAD to allow the improvement to scale to more than 2 rounds rather than capping it? In the current paper, it feels like this direction was not explored, but it was my natural inclination to ask after seeing the initial results of plateauing.
Questions for authors
  • Does it make sense to sum token probabilities across different temperatures?
  • How sensitive are your results to various forms of setting up the MAD? I can imagine so many ways that agents can debate, and I'd expect that some of them are better than others and may even allow more rounds.
Reviewer_3YWROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AbCD, which uses multi-agent debate and answer-span confidence to construct preference data for post-training. A strength of the approach is that it constructs preference data without gold-answer labels and uses answer-span confidence to weight the votes of multiple agents. The analysis also identifies limitations of additional debate rounds and agreement-based weighting, which helps clarify the method’s practical design choices.

Critical strengths

The research idea is interesting.

Critical weaknesses

(1) the experiments do not isolate the contribution of debate beyond independent sampling and confidence-based preference construction, (2) the results in Table 4 are internally inconsistent and require clarification, and (3) it is unclear whether the training responses remain self-contained after the debate context is removed.

The comparisons among majority voting, full-trace confidence, and answer-span confidence evaluate preference-labeling strategies applied to debate responses, rather than the benefit of debate itself. To isolate this benefit, please add a DPO baseline trained on preference pairs constructed from independently sampled responses.

The Phi-4-mini results in Table 4 on page 9 are internally inconsistent. The row reports overall accuracies of 87.6% for Span and 88.4% for MV, putting Span 0.8 percentage points below MV. However, the methods select different answers on only 1.2% of examples, where their respective accuracies are 33.3% and 0%.

Under the table’s definitions, both methods select the same answer on all remaining examples. These numbers imply that Span should outperform MV by approximately 0.4 percentage points, contradicting the reported overall results. Please check and clarify these values.

The treatment of debate context in the training data also needs clarification. Appendix G states that training examples exclude previous-round peer context, even though the final responses were generated using it. A response may refer to another agent’s conclusion or skip reasoning steps already covered in the discussion. Removing that context could leave the response incomplete.

The prompt asks agents to restate their reasoning, but the paper does not explain whether responses are checked for dependence on that context. Please clarify this process.

Additional suggestion that did not impact the score:

The experiments primarily use older model families and benchmarks, and do not yet show whether the gains extend to newer, stronger models and more challenging tasks. I suggest evaluating the method on stronger models and more challenging benchmarks.

Questions for authors

Please refer to the above part.

Comments on the paper

1 comment

Dear Chair, I wanted to bring to your attention that we have provided detailed responses to all reviewer comments, conducted additional experiments addressing the concerns, and updated the manuscript accordingly. We have also explicitly highlighted the corresponding changes and new results in our rebuttal.

As the author-reviewer discussion period is approaching its conclusion, we wanted to ensure that the reviewers have an opportunity to consider these additional results and raise any remaining questions. We would appreciate your help in facilitating the discussion, particularly if any further clarification is needed from our side.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.