- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a Committee Information Audit that separates dependence among member errors, complementary information present in fixed candidate answers, and the gain recovered by an aggregation policy. The experiments cover three synthetic evidence-integration domains, held-out template families, HotpotQA, and MuSiQue. They vary committee size, evidence allocation, model-family composition, one-round communication, and fixed-candidate aggregation. The reported results are that variance-equivalent effective information decreases as committees grow; model-family diversity does not consistently reduce error dependence; peer discussion can improve endpoint accuracy while increasing error correlation and reducing fixed-candidate coverage; and weighted or learned selectors recover only part of the available complementarity.
Critical strengths
The paper provides a useful decomposition of committee performance. Separating dependence, candidate coverage, and realized selection gain makes it possible to distinguish redundant errors, missing correct candidates, and ineffective selection. The matched initial traces, frozen-candidate evaluation, grouped nested validation, and explicit reporting of failed pre-committed predictions make the empirical analysis more informative than an accuracy-only comparison.
Critical weaknesses
- The experimental scale and model coverage limit the practical significance of the conclusions. The main experiments use 4B-14B open-weight models and only one optional discussion round, without frontier models, multi-round interaction, or tool-using protocols. The reliable strict communication comparison is effectively driven by Qwen, while Mistral has invalid-format rates of roughly 30%?55% across the reported arms and is not used for reliable communication accuracy conclusions. The evidence therefore supports claims about the tested small open models and one-round protocol, but not broad claims about LLM committees in general. Please add a format-robust communication evaluation for another model family and a second communication schedule, or narrow the title, abstract, and conclusion to the tested scope.
- The heterogeneity comparison may confound model-family diversity with evidence assignment. Appendix F states that heterogeneous committees place one model from each family in fixed order, while the controlled tasks assign explicit evidence views and, in some domains, different reports to different seats. The paper does not report the family-to-seat mapping or any permutation analysis. A heterogeneous result may therefore partly reflect which model receives which evidence view or report position, rather than model-family diversity. Please report family-to-seat permutations, or restrict the heterogeneity conclusion to the fixed seat assignment used in the experiments.
Questions for authors
-
What is the exact model-family-to-seat mapping in heterogeneous committees? Were all relevant permutations evaluated while keeping evidence views fixed? If not, how do the authors rule out a model-evidence assignment effect?
-
Can the authors provide reliable communication results for at least one additional model family and a second discussion schedule, or explicitly restrict the communication conclusions to Qwen under the one-round protocol?