- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper uses Blackwell’s informativeness framework to motivate posterior aggregation in multi-LLM systems. It estimates calibrated option probabilities from generated responses and combines them using PoE or learned LOP. Experiments on six multiple-choice benchmarks report competitive accuracy against voting and debate baselines.
Critical strengths
The modeling is clear and well motivated, and the theoretical framework is presented coherently.
The experiments cover a broad range of baselines, with useful ablations and evaluations across different model configurations.
Critical weaknesses
The connection between the Blackwell perspective and the proposed aggregation rules seems indirect, since the tractable Bayesian factorization is justified under conditional independence instead of any Blackwell structure family. Also, the results concerning voting and debate follow almost directly from the definition of garbling, which is interesting but somewhat not a substantial theoretical contribution.
The learned weights provide an intuitive adjustment, but dependence among agents’ evidence may vary across questions. If one set of weights is shared across all questions, it cannot explicitly adapt the correction to this variation. The paper should clarify whether the weights depend on the question and better justify their interpretation as a correction for dependence.
The evaluation seems using only several hundred test questions per benchmark. It is unclear whether this reflects an experimental design choice or computational constraints. If computation is limiting, please clarify whether the proposed pipeline or the debate baselines account for most of the cost. Voting itself has negligible aggregation overhead once responses are available. Given the modest gains over some baselines, the tradeoff between accuracy and computational cost warrants closer examination.
Questions for authors
For relatively easy questions, are the agents’ choices nearly deterministic, and do the estimated posteriors reflect this through concentrated probability mass? Are the improvements primarily on difficult questions, or do easy questions also benefit? Is there any statistics related to performance gains and posterior concentration by question difficulty?
6 comments
Question-dependent weights
Question-dependent weights.
Instead of one scalar per agent, we learn a generator that outputs a weight vector for each question (new App. M, Tab. 11, p. 27). The generator does not read the question itself; its input is a vector of statistics of the agents' calibrated posteriors on that question. For agent let be its calibrated posterior from Eq. 9, its confidence and its normalised entropy. The feature vector is $ \phi(x)=\big[c_1,\dots,c_M,;h_1,\dots,h_M\big]\in\mathbb R^{10}\quad(M=5). $ The generator is a one-hidden-layer MLP . The question-dependent weights are:
-- MLP-: is trained from scratch.
-- MLP-Finetune: predicts a multiplicative correction to the global LOP weights and is initialised exactly at LOP, so any change from the global rule is learned from data.
So in both cases , and the pooled posterior is Eq. 8 with replaced by . None of the features uses the label or the question text, so is available at test time from the same five responses per agent that PoE and LOP already use. Only is trained; and the calibration head are fixed. The networks are trained per case with the negative log-likelihood on the small set in App. D.3. Neither gives a consistent gain: only 1 points over LOP on Professional Medicine, while worse than PoE on HellaSwag and CSQA, and the variant initialised at LOP moves in the same directions, so the labelled split pulls it away from weights that generalised better. The difficulty analysis (App. P) explains why: posterior concentration and agreement separate contested from uncontested items, but they do not distinguish a unanimous correct answer from a unanimous wrong one, which is the case a per-question dependence correction would need to recognise.
Test-set size and compute
Test-set size and compute. The sizes follow Choi et al. (2025a), so that the debate baselines match their published setting; Formal Logic and Professional Medicine use the full MMLU test splits. Cost is dominated by debate (App. S, Tab. 17): on MedMCQA, five-round debate uses 14.6k--23.1k tokens per item. PoE and LOP use the same five samples per agent as self-consistency (about 4.9k tokens) plus under one second of NLI scoring, so self-consistency is the cost-matched voting baseline; single-sample majority voting is cheaper (about 1k tokens). The average gains are modest partly because most items are easy: 67--76% are answered correctly by at least four agents and solved by nearly every rule. On contested items the differences are large (next answer).
Clarification of Connection
Clarification of Connection. The connection is through the target of approximation: Blackwell’s ordering establishes the joint private information as a protocol-independent information upper bound, and Bayesian decision theory identifies the corresponding pooled posterior as sufficient for optimal decisions. The result identifies a shared information upper bound, and what we did are derived from the theoretic results. Our method directly targets this posterior by estimating and combining per-agent option distributions. Conditional independence serves a separate purpose: it makes the ideal pooled posterior tractable through a product factorization; it is not required to establish the upper bound. With dependent LLMs and imperfect posterior estimates, PoE/LOP provide practical approximations to this target. Thus, although the garbling results follow directly from classical theory, they establish the reference point that guides our method.