- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors tackle a scenario where reliable supervision signals for post-training are scarce. In such a scenario, the authors argue that multi-agent debate could serve as a solution, and propose Alignment by Confident Debate (AbCD), where they convert debate traces into preference data without gold answer labels. The procedure is to have multiple agents generate solutions and revise them after seeing each other’s responses. The final responses are grouped based on answer equivalence, and each group is scored by summing the agents’ answer-span confidence scores. Responses from the highest-scoring group are treated as preferred, while the remaining responses are treated as dispreferred. The authors evaluate this supervision using SFT, DPO, KTO, and GRPO across mathematical reasoning, quantitative QA, and code generation, and report improvements over inference-only ensembling and debate.
Critical strengths
The paper studies a useful problem: obtaining post-training supervision when gold answers are unavailable/sparsely available. The authors propose a practical way to construct preferences without gold answer labels. The approach uses debate traces and token probabilities to identify preferred and dispreferred responses.
The authors evaluate whether these preferences actually help post-training. They consider multiple models and training objectives, and report improvements in both consensus accuracy and average agent accuracy. This provides evidence that the generated preferences can improve model performance beyond inference-time answer selection.
Critical weaknesses
- The contribution of the confidence oracle is not clear. Table 1 mainly compares post-training against inference-only baselines and SFT. These comparisons do not establish the benefit of confidence-based labels over majority-based labels. Table 3 addresses this partially only for Qwen on GSM8K, where the default span oracle improves over full-trace confidence only by 0.5 points. Given prior debate preference training (MACA), whose approach is very very similar to the setup of the authors, except for the confidence based clustering as a first step. and confidence-weighted voting (CISC), the authors need broader comparisons against majority-based DPO and DPO on independent samples using the proposed approach.
2).The agents use temperatures from 0.01 to 1.0, but Section 3.2 does not clarify whether confidence uses probabilities before or after temperature scaling and top-p filtering. If the transformed probabilities are used, low-temperature agents can receive higher confidence without being more accurate. The authors should specify the scoring distribution and test whether the advantage persists when all candidates are scored under a common distribution.
-
Non-code generation is capped at 256 tokens, and in the approach the authors are counting unparseable responses as incorrect and label them as dispreferred. Appendix J acknowledges truncation problems, particularly for Llama on GSM8K. Training could therefore improve measured accuracy by producing shorter, more consistently parseable answers. The authors should report truncation and parseability rates and repeat the comparison with a longer generation budget. Table 13 measures agent accuracy after debate, so it also does not establish broad improvements in standalone reasoning.
-
Given that reasoning models are becoming increasingly the de facto standard across, various standards, the utility of these supervision signals derived from debate is not very clear if it will work these thinking models. The claim made by the authors about the signal being useful for the models they currently evaluated might be acceptable, but if they will generalize to reasoning models is questionable.
[1] Self-Improvement of Language Models by Post-Training on Multi-Agent Debate [2] Confidence Improves Self-Consistency in LLMs
Questions for authors
Can the authors compare AbCD against [1] and DPO trained on independent samples using the same confidence-based selection, under matched training budgets? Do the gains from the proposed oracle persist beyond Qwen on GSM8K? Also, can the authors perform this analysis on harder tasks, where the even after debating it is hard to arrive at the correct answer? This will help in proving the usefulness of the paper and make it stronger.
Are confidence scores computed before or after temperature scaling and top-p filtering? Does the advantage persist when all candidates are scored under a common probability distribution, removing the effect of different decoding temperatures?
What are the truncation and parseability rates before and after training? Do the improvements persist with a longer generation budget, and does the trained model improve when answering independently without debate, on harder tasks like AIME, OlympiadBench, GPQA-D? These results would help distinguish reasoning improvements from better formatting or shorter responses and also the generalizability of the proposed approach.
Can the authors evaluate AbCD on reasoning/thinking models with sufficient generation budgets? Does debate-derived supervision provide additional gains for these models, or is its usefulness primarily limited to the instruction-tuned models currently evaluated?
Dear Chair, I wanted to bring to your attention that we have provided detailed responses to all reviewer comments, conducted additional experiments addressing the concerns, and updated the manuscript accordingly. We have also explicitly highlighted the corresponding changes and new results in our rebuttal.
As the author-reviewer discussion period is approaching its conclusion, we wanted to ensure that the reviewers have an opportunity to consider these additional results and raise any remaining questions. We would appreciate your help in facilitating the discussion, particularly if any further clarification is needed from our side.