acceptodds
Under review as a conference paper at ICLR 2027

Probabilistic Consensus for Weighted Multi-Agent Debate

Abstract

Multi-agent debate (MAD) offers a promising approach to improving the reasoning capabilities of large language models (LLMs), but reaching a correct consensus requires accounting for uncertainty in both proposed answers and the feedback used to evaluate them. Agents may produce incorrect solutions, and their assessments of other agents’ solutions may themselves be mistaken. We introduce Weighted Iterative Society-of-Experts (WISE), a zero-shot framework for weighted multi-agent debate with probabilistic consensus. WISE augments textual critiques with discrete correctness weights assigned to candidate answers, providing structured evidence for modeling uncertainty in agents’ evaluations. At its core is WISE-Dawid–Skene, an unsupervised probabilistic model that jointly estimates solver and scorer error patterns and infers answer correctness from weighted responses across debate rounds. This formulation accounts for uncertainty in both answer generation and scoring, enabling consensus to reflect the accumulated evidence and inferred capabilities of participating agents. Iterative feedback guides solution refinement, while the joint probabilistic model integrates the resulting answers and assessments to infer a collective decision. Experiments on standard multimodal reasoning benchmarks demonstrate that WISE consistently improves zero-shot reasoning, outperforming leading models by 6–12% on average. These results highlight weighted debate and joint probabilistic modeling of answer and feedback uncertainty as an effective approach to improving collective reasoning.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
probabilistic inference, multi-agent reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ABG1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focused on the consensus and response aggregation of a multi-agent debate system. The authors claimed that “agreement alone does not establish correctness”, and besides the uncertainty of problem solver agents, the critique agents introduce a second source of uncertainty in their feedback. Therefore, using a direct aggregation of multiple agents like in previous methods is unreliable. To solve this, they proposed to fit a multi-agent system into a probabilistic framework, where both solvers and critiques are treated as noisy generators, and the final answer is derived by maximizing a posterior inference. Experiments demonstrated the effectiveness to some extent.

Critical strengths
  1. The idea of utilizing a probabilistic model to fit two-layers of uncertainty is a good point, as the uncertainty of critique agents is less focused in the literature.

  2. Experiments on several benchmarks showed that the proposed method outperformed the selected multi-agent baseline without adding too much compute cost.

Critical weaknesses
  1. Only one multi-agent reasoning method is compared, which is also old. More recently published methods should be compared, especially those focusing on the uncertainty of multi-agent systems.

  2. When quantifying the uncertainty of agents, the authors claimed that using a discrete manner (three uncertainty levels) is a good choice. However, there are no experiments or metrics demonstrating the superiority of such a design compared to other choices (e.g., a continuous variable).

Questions for authors
  1. Can you either provide comparative experiments with more multi-agent baselines or explain why it is not necessary to do so?

  2. Can you provide more explanation or analysis about why using a discrete uncertainty space is the best choice?

Reviewer_2xKUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes WISE, a zero-shot multi-agent debate framework in which solver models generate candidate answers and independent reviewer models provide diagnostic feedback and discrete correctness weights. Its core contribution, WISE-DS, models debate as a coupled two-channel process and uses expectation–maximization (EM) to jointly estimate solver and reviewer error parameters and the posterior distribution over the true answer without supervision. A consensus answer is then obtained through maximum a posteriori (MAP) inference.

Critical strengths
  1. The paper is well motivated and addresses an important research problem.
  2. The qualitative analysis provides valuable insights and further strengthens the paper’s claims.
  3. SMART-840++ is a useful dataset that lays a foundation for future research.
Critical weaknesses
  1. The paper’s writing requires substantial improvement, particularly in the organization of its arguments, logical flow, and precision of language. These issues hinder readers’ understanding of the method and its contributions, and the current manuscript falls short of the expected standards for academic writing. Although the rebuttal may clarify some technical issues, it cannot substitute for a systematic revision of the paper. Given the current writing quality, I do not support acceptance.

  2. Structurally, the proposed method is essentially a Dawid–Skene (DS) model with an auxiliary observation channel. The paper does not adequately explain what substantive methodological innovations it offers beyond applying DS to aggregate solver answers and introducing an observation model for the correctness weights assigned by reviewers. The authors should clearly distinguish extensions to a new application setting from innovations in the model itself and define the core contribution more precisely.

  3. The paper cites aggregation models such as MACE, GLAD, and BCC but does not include them as baselines. Comparisons under a consistent experimental setup would help establish the advantages of the proposed model over alternative aggregation approaches and provide more direct, convincing empirical support for the choice of this model class.

Questions for authors
  1. In Table 4, why was the strongest WISE configuration evaluated only on a subset?

  2. How were the recency weights chosen? Does recency weighting affect the results reported in Table 2?

  3. Why is reviewer reliability assumed to remain constant across rounds, while solver reliability is modeled as round-dependent? Reviewers also observe evolving candidate answers and the surrounding feedback context. Please justify this assumption or provide an ablation study.

Reviewer_87vpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes WISE, a debate system in which solvers generate answers, critics provide textual feedback and three-level correctness scores, and a distiller prepares feedback for the next round. WISE-DS extends Dawid-Skene to combine these outputs and estimate solver and critic reliability without ground-truth labels. Experiments cover several multimodal benchmarks.

Critical strengths
  1. Separating solver ability from critic reliability makes sense, since a model may be better at answering questions than evaluating other answers. The three-level scoring scheme also allows critics to abstain when the evidence is unclear.

  2. Comparing aggregation methods on the same debate outputs helps isolate their contribution.

  3. The paper provides a useful test of generalization through controlled changes in puzzle instances and difficulty.

Critical weaknesses
  1. The probabilistic model does not explicitly account for dependencies created by debate. Later answers use earlier feedback, and the same model can serve as both solver and critic, yet their likelihoods are multiplied as separate evidence.

  2. Several implementation details remain unclear: whether the 100 problems used to fit WISE-DS overlap with evaluation, how EM is initialized, and which recency weights are used. Results across different fitting subsets would also help establish stability.

  3. The handling of free-form answers is underspecified. The answer values and equivalent expressions vary across questions, so it is unclear how candidate answers are grouped and confusion matrices are shared.

Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.