acceptodds
Withdrawn from ICLR 2027

HOW MUCH INDEPENDENT INFORMATION DOES AN LLM COMMITTEE PROVIDE?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Adding agents to an LLM committee can improve accuracy without providing proportionally more independent information. Member accuracy, error dependence, and aggregation quality change together, obscuring what a larger or more communicative committee contributes. We study this problem with a committee information audit. End-task accuracy alone cannot tell whether a gain comes from better individual answers, less redundant errors, or better selection of existing answers. We therefore separate three quantities on frozen output records within each protocol arm: dependence among member outputs, complementarity available in that arm’s candidate answers, and the gain recovered by an aggregator. This separation distinguishes information that is present in the committee from information that a deployable policy can actually use. The audit combines explicit evidence allocation, matched communication contrasts, and held-out evaluation of fixed-candidate selectors. Across controlled domains, held-out template families, and two external benchmarks, variance-equivalent effective size shows saturation. In a paired Qwen3-4B full-topology control, the observed decay is steeper than a fixed-correlation null. Five related diagnostics show the same directional pattern on the shared output matrix. Conditioning on input structure explains much of the unconditional dependence, while quality-matched model diversity does not consistently reduce it. In the original output-budget-matched Qwen comparisons, peer discussion increased plurality accuracy over self-revision by 5.4 percentage points on HotpotQA and 7.4 on MuSiQue, while increasing error correlation and reducing candidate-oracle coverage. In the stricter input-token-matched Qwen validation, the peer–self plurality changes were +0.4 pp (95% CI [−2.4, 3.0]) on HotpotQA and +4.0 pp (95% CI [2.0, 6.0]) on MuSiQue; fixed-candidate oracle coverage fell by 15.2 and 10.6 pp, respectively. Grouped validation further shows that weighting and learned selection recover useful but uneven fractions of the available complementarity. Thus, communication can improve an endpoint score while increasing error correlation and reducing the fraction of questions with at least one correct candidate. Committee evaluation should therefore report dependence, candidate coverage, and realized gain together rather than use accuracy as a proxy for all three.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Multi-Agent Systems, Ensemble Learning, Error Dependence, Information Aggregation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_FhJjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Committee Information Audit that separates dependence among member errors, complementary information present in fixed candidate answers, and the gain recovered by an aggregation policy. The experiments cover three synthetic evidence-integration domains, held-out template families, HotpotQA, and MuSiQue. They vary committee size, evidence allocation, model-family composition, one-round communication, and fixed-candidate aggregation. The reported results are that variance-equivalent effective information decreases as committees grow; model-family diversity does not consistently reduce error dependence; peer discussion can improve endpoint accuracy while increasing error correlation and reducing fixed-candidate coverage; and weighted or learned selectors recover only part of the available complementarity.

Critical strengths

The paper provides a useful decomposition of committee performance. Separating dependence, candidate coverage, and realized selection gain makes it possible to distinguish redundant errors, missing correct candidates, and ineffective selection. The matched initial traces, frozen-candidate evaluation, grouped nested validation, and explicit reporting of failed pre-committed predictions make the empirical analysis more informative than an accuracy-only comparison.

Critical weaknesses
  1. The experimental scale and model coverage limit the practical significance of the conclusions. The main experiments use 4B-14B open-weight models and only one optional discussion round, without frontier models, multi-round interaction, or tool-using protocols. The reliable strict communication comparison is effectively driven by Qwen, while Mistral has invalid-format rates of roughly 30%?55% across the reported arms and is not used for reliable communication accuracy conclusions. The evidence therefore supports claims about the tested small open models and one-round protocol, but not broad claims about LLM committees in general. Please add a format-robust communication evaluation for another model family and a second communication schedule, or narrow the title, abstract, and conclusion to the tested scope.
  2. The heterogeneity comparison may confound model-family diversity with evidence assignment. Appendix F states that heterogeneous committees place one model from each family in fixed order, while the controlled tasks assign explicit evidence views and, in some domains, different reports to different seats. The paper does not report the family-to-seat mapping or any permutation analysis. A heterogeneous result may therefore partly reflect which model receives which evidence view or report position, rather than model-family diversity. Please report family-to-seat permutations, or restrict the heterogeneity conclusion to the fixed seat assignment used in the experiments.
Questions for authors
  1. What is the exact model-family-to-seat mapping in heterogeneous committees? Were all relevant permutations evaluated while keeping evidence views fixed? If not, how do the authors rule out a model-evidence assignment effect?

  2. Can the authors provide reliable communication results for at least one additional model family and a second discussion schedule, or explicitly restrict the communication conclusions to Qwen under the one-round protocol?

Reviewer_KD7EOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper asks a practical question: when a group of language models votes or talks, does the group actually bring new independent information, or does it mostly repeat the same mistakes in a way that a scoreboard hides? The authors separate three things that usually get grouped together.

First, how alike the member errors are, summarized as an “effective” group size that is often much smaller than the number of seats. Second, whether the set of answers already on the table contains a correct one. Third, whether a rule you could actually deploy can pick that correct answer on fresh questions.

They study this on handmade reasoning tasks, on reworded versions of those tasks, and on two public question-answering benchmarks, HotpotQA and MuSiQue. The effective group size levels off as more members are added. Letting models read each other’s answers can raise the voted score while making their errors more alike and destroying correct answers that only one member had found. Simple weighting helps on the handmade tasks and recovers only part of what a perfect picker of existing answers could get. The bookkeeping behind these measurements is internally consistent. The write-up still overstates what the main comparisons show.

Critical strengths

In my opinion, the three-way split is the right way to read a committee. A higher final score can mean better individual answers, less duplicated error, or a better rule for choosing among answers that were already there. Those are different conclusions, and the paper refuses to treat them as one number. They include a control that asks whether the effective size only shrinks because of the formula. On one controlled run, the shrinkage is steeper than that formula alone would produce. The communication test reuses the same first-round answers, so a gain from reading peers can be distinguished from the mere chance to answer a second time. They also say plainly when one model family mostly fails by producing unreadable answers, and they do not lean on those runs.

Authors check on where the similarity comes from is the most informative analysis in the paper. Much of the shared error is tied to the structure of the question and the evidence, rather than to the model brand. They also report predictions that failed, including the prediction that mixing model families would make errors more independent.

Critical weaknesses
  • The opening summary leads with the larger accuracy gains. Those gains come from a comparison the authors themselves later call secondary in Line 361, because the prompts were not matched in length. In the comparison they call primary, the HotpotQA gain is small and statistically compatible with no gain. The figure 3d still highlights the larger, secondary numbers.
  • The main phenomenon is already reported in nearby works like [1][2][3]: effective committee size saturates near two independent votes, mixing models does not reliably fix that, and smarter aggregation closes little of the gap. Saturation near two independent votes is already in [1] and [2]. [3] already shows discussion producing agreement while information is lost. The new pieces are the evidence-allocation audit and the fixed-candidate gap. The conflicts with [1] and [2] are not discussed.
  • Minor but meaningful weakness: The effective-size formula appears in Section 3 before the right/wrong mark is defined, and the oracle is used even earlier, in the introduction. The correlation itself is never given a formula. Names such as the full-topology sweep, SHAM, DEC, and RANDI show up in the results with no definition. The setup, the results, and the appendix also keep mixing what was pre-committed, what was decided after seeing the data.

[1] Bertalanič, Blaž, and Carolina Fortuna. "The Ringelmann Effect in Multi-Agent LLM Systems: A Scaling Law for Effective Team Size." arXiv preprint arXiv:2606.02646 (2026). [2] Kohli, Guneet. "Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels." arXiv preprint arXiv:2605.29800 (2026). [3] Wan, Herun, et al. "The Deliberative Illusion: Diagnosing Factual Attrition and Stance Homogenization in Multi-Agent LLM Deliberation." arXiv preprint arXiv:2606.03032 (2026).

Questions for authors
  1. In the runs, what was the actual difference between the shared-evidence condition and the separate-evidence condition? Where are the head-to-head numbers for accuracy, error similarity, effective group size, and the chance that some member is correct?
  2. You treat the length-matched peer-versus-self comparison as the one that counts, and on HotpotQA that accuracy change is compatible with no change. Why do the abstract and Figure 3(d) lead with the larger unmatched gains? What are the uncertainty ranges for the coverage drops on the traces you already have?
  3. What do the two full-document single-model numbers, 0.552 and 0.602, refer to? These appendix N numbers are far above the committee vote. Why?
  4. Ref [1] provided in weakness section, report that mixing model families reduces duplicated error. Your mixed committees go the other way. What difference in the setup explains that?
Reviewer_QkZCOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks how much independent information an LLM committee actually provides. The authors argue that accuracy alone can’t answer this, and propose reporting three quantities separately: how correlated members’ errors are (summarized by n_eff), how many correct answers the fixed candidate set contains beyond the best member (the oracle gap), and how much of that gap a deployable aggregator recovers. They test this on three synthetic domains with controlled evidence allocation and on HotpotQA and MuSiQue with paragraphs split across agents, using 4B–14B models and committees of up to eight members. Their main findings are that n_eff/n drops as committees grow, that error correlation is largely explained by input structure rather than model family, that discussion can raise accuracy while increasing correlation and reducing oracle coverage, and that aggregators recover only about a third of the available oracle gap.

Critical strengths
  1. I agree with the starting point. Many multi-agent papers report a single accuracy number, and it really is hard to tell where the gains come from. Separating dependence, available complementarity, and realized gain is a sensible idea.
  2. Some of the experimental design is careful. The matched comparisons keep round-1 answers and justifications identical and change only whether discussion happens. Dependence is calibrated against permutation nulls, including ones stratified by input structure. The held-out template families check that the results don’t hinge on surface wording.
Critical weaknesses
  1. Readability is the biggest problem. The main text and appendix are full of provenance notes and caveats: “historical,” “post-hoc reconstruction,” “S1 folds remain unavailable,” “S3 is not strict family-disjoint.” There are also many abbreviations that are never defined in the main text: DEC, RANDI, SHAM, bulletin, E-FULL s21, OIPG F1. The abstract is packed with numbers and confidence intervals. It took me two passes to work out which results are the main claims, which are secondary, and which the authors have since walked back. Right now the paper reads more like an audit log than a paper. I’d suggest a substantial rewrite: keep three or four main findings and the experiments behind them in the main text, and condense the provenance discussion into one paragraph in the appendix.
  2. Limited new insight. Discussion leading to conformity and more correlated errors, model diversity not necessarily reducing correlation, and a gap between aggregators and the oracle have all been discussed in the ensemble and multi-agent debate literature. The main contribution is really a recommendation to report three quantities together. That recommendation is reasonable, but as currently written, it’s hard to see what new understanding it gives us.
  3. The saturation result may mostly reflect item difficulty, not committee redundancy. n_eff is computed from unconditional error correlation. But Appendix R shows that conditioning on input structure brings the mean excess correlation from 0.276 down to 0.065, so most of the “dependence” comes from some items being hard for everyone. So I’m not sure whether they show redundancy among members or just that hard questions are hard for everyone.
Questions for authors
  1. How does each agent’s evidence change as n goes from 2 to 8? How do you separate the effect of n from the effect of evidence allocation?
  2. Under strict matching, discussion barely improves accuracy. Why does the abstract lead with the output-budget-matched numbers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.