acceptodds
Under review as a conference paper at ICLR 2027

Auditing the Auditors: Validating Automatic Evaluation in Multi-Model LLM Consensus Systems

Abstract

Multi-model review aims to improve language-model answers through critique and revision, but evaluating its benefits requires reliable correctness judgments. We examine this problem through TriGuard, a three-model generate–review–revise pipeline using quantized open-weight models. Our evaluation combines controlled replay, which holds primary answers and reviewer feedback fixed across synthesis policies, with human annotation and documented adjudication. The first annotator received recorded AI assistance; a second annotator independently assessed the answers while blinded to prior labels and study results. On two 100-question TruthfulQA slices, we audit all 24 questions affected by a static gate that bypasses synthesis when reviewer feedback is net-negative and the primary model has higher historical accuracy. Automatic scoring indicates a 7.5-percentage-point accuracy reduction relative to unconditional synthesis. Adjudicated labels instead imply a reduction of 5–6 percentage points across the 200 questions. This range reflects unresolved-label assignments, not sampling uncertainty. A separate, outcome-enriched 20-question audit indicates a positive primary-to-synthesis difference within that selected sample, without establishing a full-sample benefit. We also document limitations of two evaluation components: a semantic-equivalence mechanism agrees with human labels on 15 of 30 answer pairs from ten questions, and a HaluEval judge continues to approve reference-contradicting answers after a prompt-and-parser repair. These findings show how evaluation choices can materially change conclusions about multi-model systems. We present a reproducible audit workflow that distinguishes implementation errors, substantive judgment failures, and unresolved labels, while bounding claims to the comparisons actually validated.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
large language models, LLM-as-a-judge, multi-agent systems, automatic evaluation, human annotation, model consensus, TruthfulQA, hallucination detection

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_tVXeOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors study a multi-model LLM pipeline in which a primary model answers, two reviewer models score the answer, and a judge model grades the result. They focus on a gate that skips the synthesis step when reviewers reject the primary answer, and ask whether the automatic judge can be trusted to evaluate this gate. On two 100-question TruthfulQA slices the gate changes 24 answers. Human annotation of these 24 questions changes the estimated effect from −0.075 (automatic judge) to between −0.060 and −0.050 (adjudicated), after an AI-assisted first pass gave −0.035 to −0.025.

Critical strengths

The audit is careful. Because only 24 answers change, the authors can label all of them by hand, and the paper keeps label ambiguity separate from sampling error. The main statistics check out, and the supplement reproduces the automatic −0.075.

Critical weaknesses
  • The main finding is narrower than the framing. In the supplement the gate weights are constants (0.66 for the primary vs. 0.41 and 0.385 for the reviewers), so the gate simply keeps the primary answer whenever the mean reviewer score is below 0.5.
  • The result then mostly says that reviewer rejections are informative, on 24 questions from one model trio and one benchmark. This is a case study, and the title and conclusion claim much more.
  • The human labels are less independent than described. The introduction says "two independent human annotators", but annotator 1 had a verdict suggested by AI for 22 of 54 records in l.256. At the same time, the AI Disclosure says AI was used "only to improve the language and grammar".
Questions for authors

see weaknesses

Reviewer_PvHkOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper establishes TriGuard, a generate–review–revise pipeline with three quantized open-weight models, to evaluate the "multi-model review system“ . The authors introduce a controlled-replay protocol that freezes primary answers and reviewer feedback while varying only the synthesis policy, isolating a static reviewer gate's causal effect from resampling and prompt-wording confounds. They conduct a complete two-annotator, documented-adjudication human audit of all 24 questions the gate altered across two 100-question TruthfulQA slices, finding a 5–6 percentage-point accuracy reduction relative to unconditional synthesis (Sec. 3.7.1; Eq. 6). Additional contributions include a prospectively specified confirmatory test that did not reach significance (Appendix 9), a documented HaluEval judge failure leading to withheld accuracy claims (Sec. 3.6.2), and a semantic-equivalence voting mechanism reported as evaluated and rejected (Appendix 8).

Critical strengths

Transparent reporting of failed validation and non-confirmatory results. The authors report that the prospective primary-versus-synthesis test fails to confirm the exploratory result, reject their semantic-equivalence voting mechanism after human validation, and withhold HaluEval accuracy claims after identifying a substantive judge failure. This restraint is consequential: it prevents the paper from presenting an unvalidated evaluator output as evidence of system performance and makes the methodological limitations visible to readers.

Critical weaknesses
  1. Terminological clarity and consistency. The paper contains a large number of self-created terms, and authors did not clarify most of terms clearly when they first mention them. Examples include “ static reviewer gate,” “unresolved-label bounds,” and “primary-vs-v1”, etc. Some terms describe relatively standard experimental procedures or data subsets, yet are presented as distinct named concepts. This creates unnecessary cognitive overhead and makes the methodological contributions harder to identify.

Also, there are inconsistent terminological of the same for the same concept within the context of the paper. For example, the overall pipeline is described as both “generate–review–revise” and “propose-critique-revise.” Similarly, the gate-related terminology shifts among “v3,” “static reviewer gate,” “isolated gate,” and “audited gate,” making it difficult to track the exact intervention being discussed.

  1. Limited research significance. The paper audits a custom multi-model pipeline and a specific static gating strategy, but does not provide enough evidence to demonstrate the significance, influence and representativeness of such pipeline and strategy. Consequently, the broader significance of the findings remains unclear, particularly given existing work on the limitations of multi-agent debate.

  2. Limited empirical scope and generalizability. The evaluation covers only TruthfulQA and HaluEval, with HaluEval used solely to investigate evaluator reliability rather than to report system performance. Moreover, the experiments use relatively small, 4-bit quantized versions of 2024-era models (Llama 3.1 8B, Qwen 2.5 7B, and Gemma 2 9B). The limited benchmark coverage and model configurations make it difficult to assess whether the findings generalize to contemporary models and other multi-agent systems.

  3. Terminological ambiguity: “multi-model” vs. “multi-agent.” The term “multi-model” may be confused with “multimodal” in the broader ML literature. Since the paper focuses on multiple models collaborating in distinct agent roles, “multi-agent” would be a clearer and more conventional term, provided the authors consistently define the agents and their roles.

Questions for authors

Please see Critical Weaknesses.

Reviewer_KmT6OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies a multi-model question-answering pipeline in which a primary model answers, two reviewers provide feedback, and the primary model revises its answer. A static gate retains the original answer when reviewers reject it but the primary model has higher historical accuracy. Using cached outputs to isolate this decision, the authors audit all 24 affected questions across 200 TruthfulQA questions and find that the gate reduces accuracy by 5–6 percentage points relative to unconditional revision. The paper also reports evaluator failures and several smaller validation studies.

Critical strengths

The paper examines a relevant risk in multi-model collaboration: using historical model performance to decide whether to accept reviewer feedback can backfire. Its negative case cautions against assuming that a more elaborate decision process necessarily produces more reliable answers.

Critical weaknesses
  1. The central experiment establishes that one particular gate hurts performance, but does not sufficiently explain why this finding matters beyond that rule. The rationale in Section 3.1 is to trust the historically more accurate primary model when reviewers disagree. However, overall answering accuracy does not directly measure a reviewer’s ability to identify errors, nor does it establish that the primary model remains more reliable on the disputed questions. The paper does not adequately test this assumption or establish that the rule represents a consequential existing practice. The PDF also leaves key calibration details to the supplementary materials rather than presenting the evidence needed to assess this rationale. Human adjudication confirms the losses, but the analysis does not systematically explain which useful feedback was rejected or when historical accuracy becomes misleading. My concern is not that the result is negative: it is that the rule is weakly motivated and its failure yields limited insight.

  2. The paper presents controlled replay, human adjudication, and evaluator checks as major contributions, but I find their added value beyond validating this particular experiment insufficiently established. Auditing every affected question is useful, yet the disagreement between automatic and adjudicated labels also raises concerns about results that remain dependent on automatic scoring. The estimated gate loss changes from 7.5 percentage points to 5–6 points after adjudication. This does not invalidate the adjudicated gate result, but it shows that automatic labels materially affect the estimated effect size. The primary-versus-v1 comparisons still rely on automatic labels that the gate audit does not validate in full. The outcome-enriched 20-question pilot cannot establish the full-sample benefit, while the targeted HaluEval retests demonstrate specific failures rather than characterize their prevalence. The authors acknowledge these limits, but the resulting evidence remains a narrow case study rather than a convincing new evaluation methodology or a systematic account of evaluator error. The presentation makes this harder to assess: Section 3 announces v1, v3, and v4, yet provides no corresponding v4 results, while the gate audit, collaboration comparison, and evaluator checks are scattered across the main text and appendices without a clear consolidated account of what each establishes.

Questions for authors

Please address the concerns in Critical Weaknesses, particularly the research motivation for the gate, the reliability of comparisons that still depend on automatic scoring, and the contribution beyond routine experimental validation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.