acceptodds
Under review as a conference paper at ICLR 2027

RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detection

Abstract

Automatic evaluation of faithfulness increasingly relies on a large language model acting as a judge, yet the most reliable judges are proprietary frontier models. Their per-call cost, rate limits, and off-premises inference, however, make them ill-suited to the continuous, high-throughput monitoring that benchmarking and production use demand. In this work we investigate three fundamental questions: whether a panel of cheap open-weight judges (4–9B parameters) can be aggregated to stand in for a frontier one, what the substitution sacrifices in agreement (Cohen's ) and accuracy, and when it is worth making. To address them, we propose RAIM, an aggregation scheme robust to the members' correlated errors, which couples: () a cross-fitted stacked logistic regression that fits the members' weights jointly, therefore penalising weaker judges echoing a stronger one; and () an admissibility test that, read from the members' own outputs, identifies a regime in which aggregating them improves on their best member and stays within reach of the frontier judge. We instantiate RAIM under ten judges drawn from disjoint families and we measure the substitution across eight faithfulness benchmarks. Read as paired differences against Claude Sonnet, the substitution clearly improves on one benchmark and clearly worsens on three (only two by a non-negligible margin), leaving the remaining four unresolved. The panel retains a median 93% of the frontier judge's agreement and gives up only 2.9 points of balanced accuracy on average. Inference runs at a sixty-fourth of the frontier's price at conventional cloud rates, or at the electricity cost on premises, so the operative expense is a one-time in-domain calibration of some fifty to a hundred labelled records, amortised over a few tens of thousands of evaluated items. Against the literature, the panel is competitive with purpose-trained detectors on their home benchmarks, losing 1.3 points of accuracy to GPT-4o and 1.9 to the best LLM-AggreFact leaderboard model; yet the strongest detector we reran falls 6 points behind on our grounded sets, and cannot score the ungrounded ones. In conclusion, whether pursuing the aggregation is worthwhile depends on the members themselves, making the gain auditable from how widely competence is spread across them and how far their errors decorrelate — both read at no further cost off the aggregator's calibration set. If several capable members err on different items, the panel improves on its best judge and approaches the frontier; conversely, where one dominates, the stacker recovers the leader, yet only there does the frontier remain materially ahead. A panel of cheap judges can therefore stand in for a frontier one at a small fraction of the cost wherever this audit admits it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-judge, hallucination detection, faithfulness evaluation, judge ensembles, error correlation, open-weight models, cost-efficient evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_9YRNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Automatic evaluation of faithfulness increasingly relies on a large language model acting as a judge, yet the most reliable judges are proprietary frontier models. Their per-call cost, rate limits, and off-premises inference, however, make them ill-suited to the continuous, high-throughput monitoring that benchmarking and production use demand. In this work we investigate three fundamental questions: whether a panel of cheap open-weight judges (4–9B parameters) can be aggregated to stand in for a frontier one, what the substitution sacrifices in agreement (Cohen's ) and accuracy, and when it is worth making. To address them, we propose RAIM, an aggregation scheme robust to the members' correlated errors, which couples: ( ) a cross-fitted stacked logistic regression that fits the members' weights jointly, therefore penalising weaker judges echoing a stronger one; and ( ) an admissibility test that, read from the members' own outputs, identifies a regime in which aggregating them improves on their best member and stays within reach of the frontier judge. We instantiate RAIM under ten judges drawn from disjoint families and we measure the substitution across eight faithfulness benchmarks. Read as paired differences against Claude Sonnet, the substitution clearly improves on one benchmark and clearly worsens on three (only two by a non-negligible margin), leaving the remaining four unresolved. The panel retains a median 93% of the frontier judge's agreement and gives up only 2.9 points of balanced accuracy on average. Inference runs at a sixty-fourth of the frontier's price at conventional cloud rates, or at the electricity cost on premises, so the operative expense is a one-time in-domain calibration of some fifty to a hundred labelled records, amortised over a few tens of thousands of evaluated items. Against the literature, the panel is competitive with purpose-trained detectors on their home benchmarks, losing 1.3 points of accuracy to GPT-4o and 1.9 to the best LLM-AggreFact leaderboard model; yet the strongest detector we reran falls 6 points behind on our grounded sets, and cannot score the ungrounded ones. In conclusion, whether pursuing the aggregation is worthwhile depends on the members themselves, making the gain auditable from how widely competence is spread across them and how far their errors decorrelate — both read at no further cost off the aggregator's calibration set. If several capable members err on different items, the panel improves on its best judge and approaches the frontier; conversely, where one dominates, the stacker recovers the leader, yet only there does the frontier remain materially ahead. A panel of cheap judges can therefore stand in for a frontier one at a small fraction of the cost wherever this audit admits it.

Critical strengths

How to use AI models to answer a question and produce a very large report in a very short time.

Critical weaknesses

There is nothing new to advance AI or machine learning.

Questions for authors

How many of the papers listed in the reference section that the authors have read? How much of the results reported in the paper is produced from hallucination? Author relied on AI tools for the result, but they have only API access. Currently, there is no method to know how much of the model's output is because of hallucinations. How much confidence the author has that the reported result is hallucination free?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 9YRN (1/2): on paper contribution, and on the references read

We updated the response as we noticed we did not published to "Everyone". No content was changed.

We thank the reviewer for their time in evaluating our work.

The critical weakness the reviewer attributes to our work, There is nothing new to advance AI or machine learning, shows that we did not convey clearly enough what our work contributes and how it advances on prior work, a positioning that, for space, sat almost entirely in Appendix L. In the revision, we will move part of it into the introduction (new paragraph Relation to prior work, §1), a point Reviewer uHii also raises. We claim no novelty in the aggregator structure itself: the stacker is a standard logistic regression, chosen so that any gain is attributable to the judges, not to the estimator.
Our main contribution is, instead, a rigorous study of how a panel of cheap LLMs can be employed for hallucination detection in place of a single frontier LLM judge, and of how this route compares with the related ones (escalation and specialisation). Indeed, the provided systematization made other reviewers find the paper valuable: "asks a practical question and puts a price on the answer" (tuNS); "evaluation protocol is rigorous" (tuNS); "a trustworthy mixed result" (p13k); and, note that it reports "negative results instead of presenting aggregation as universally effective" (uHii). In detail, our main contributions are the following:

  • A priced substitution. We measure what we lose when a paid frontier judge (Claude Sonnet) is replaced by a panel of 4–9B open-weight judges run locally (tested on eight hallucination benchmarks), and we price the replacement including the labels the aggregator consumes. Prior panel studies, such as PoLL (Verga et al., 2024), price inference alone and count no labels.
  • A decision rule. We give an admissibility test, read from the same 50–100 labelled records, which combines how widely competence is spread across the judges with how far their errors decorrelate. The closest diagnostics read error dependence alone (Zhou and Lin, 2026) or choose an aggregator once a panel is in use (Zhu and Rao, 2026).

How many of the papers listed in the reference section that the authors have read?

The paper cites 88 works, and we understand the scepticism expressed by the reviewer on the matter, in an era where LLMs are ubiquitous. However, twenty references identify what we run: the reports of the eleven models (the ten judges and the frontier judge) and the nine sources of the eight benchmarks. Fourteen are the classical statistical and aggregation literature: five we use or compare against (Cohen's κ, Holm's correction, Youden's index, stacking, and Dawid–Skene), and nine on which our account rests, from Condorcet's jury theorem to the descendants of Dawid–Skene and the study of ensemble diversity. The other 54 are related work (LLM judges and panels, correlated errors, cascades, trained detectors), each discussed in Appendix L. As the AI use statement says, every reference was retrieved and read in the original, and every published value beside ours checked against its source.

AuthorsOpenReview ↗

Response to Reviewer 9YRN (2/2): on how hallucinations enter the work, on where the gold labels come from, and on the role of API access

We updated the response as we noticed we did not published to "Everyone". No content was changed.

How much of the results reported in the paper is produced from hallucination?

Currently, there is no method to know how much of the model's output is because of hallucinations.

How much confidence the author has that the reported result is hallucination free?

We pair these three comments as they all concern the same matter: hallucinations, their detection, and how they enter our work. Again, we should have communicated better. We want to clarify upfront that no reported number is produced by a language model. Indeed, the judges are the object of the study: each is asked whether a candidate text is hallucinated and their verdicts are the data; every number in the paper is then computed from those verdicts by deterministic code, against a ground truth the benchmarks themselves publish.
Hallucinations therefore enter the work in two roles, both as data under analysis: as the candidate texts the benchmarks supply (which the judges must classify) and as the judges' own wrong verdicts (which we measure against the published labels and which the panel we introduce is designed to reduce). In neither role can they alter the measurement itself.
As to the confidence that the reported results are hallucination free: hallucination is a property of text that a model generates, and no reported number is generated by a model, so none of them can be hallucinated by definition. What can be wrong in a deterministic pipeline is the code, in which errors remain possible, as in any other empirical work and despite our best efforts and continuous committment; this is why the code and all verdicts are released, for every number to be regenerated and checked by the community (Reproducibility statement).

As to how hallucination can be measured at all, likely the details might have been exposed in a more clear manner: every item carries the label published by the benchmark's own authors, and every verdict, from the panel and from the frontier judge alike, is scored against it. No label is produced by any model we evaluate. "Grounded" and "ungrounded" describe what the judge is shown, not whether a label exists:

  • on the six grounded sets, the judge sees the reference, and the label is the benchmark's own annotation of whether the candidate is supported by it: human span annotation on RAGTruth, the human support labels of LLM-AggreFact on WiCE, XSum, CNN, and ExpertQA, and, on MedHallu, the labels fixed by its construction (a published answer against a hallucinated one the benchmark generated);
  • on the two ungrounded sets, the judge sees no evidence and must rely on its own knowledge, whereas the label remains the benchmark's own: human atomic-fact annotation against Wikipedia on FActScore, and the authors' correct and incorrect reference answers on TruthfulQA.

Indeed, measuring how often a model's output is unsupported, against benchmarks labelled in this way, is precisely what the hallucination-detection literature does. The misuderstanding that raised the criticism is likely due to the fact that the paper states the provenance of the labels only in Appendix J and in the Ethics statement, and that the task definition in §2 can be read as defining the label by what the judge sees. The revision will therefore state it in §2 and will add the label source of each dataset to Table 14.


Author relied on AI tools for the result, but they have only API access.

This point rests on another misunderstanding, below clarified.
As above reported, AI tools do not enter our work as producers of our results: their outputs are the very data we study, and all of them (except the frontier reference's) are generated under a reproducible (up to the small nondeterminism of GPU serving) procedure that the reviewer has access to and can freely run (following raim-verdicts/QUICKSTART.md or raim-verdicts/FRESH_BOX.md in the supplementary material). The frontier reference (i.e. the judge the proposed panel aims to replace) is the only part of the data whose generation required paying for a commercial API, and the only part that is not exactly reproducible since the provider might change the model behind it and, even at the temperature of 0 we use, does not hence guarantee identical outputs over time; its inference code is nonetheless released as well.
In any case, with the data fixed (as in the snapshot provided in the supplementary material), all the results are deterministically generated by the released Python code, with no LLM or API entering the loop.


We are confident to have clarified all the points raised by the reviewer, and we trust the work to be reassessed on that basis, while remaining available for any further question.

Reviewer_uHiiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether several small, inexpensive language-model judges can replace a more costly frontier model for hallucination detection. RAIM combines the binary decisions of ten open-weight judges (4B–9B parameters) using cross-fitted logistic regression. It also proposes an admissibility test based on individual judge quality and error correlation to estimate when an ensemble would help detection.

The method is evaluated on eight faithfulness benchmarks, comparing it with individual judges, majority voting, other aggregation methods, specialist detectors, and Claude Sonnet. The panel performs competitively at much lower estimated inference cost, although it does not consistently match the frontier judge and needs labeled examples from each deployment domain.

Critical strengths
  1. The paper tests ten judges across eight datasets and compares supervised and label-free aggregation, the best individual judge, and a frontier judge. Paired confidence intervals help distinguish meaningful gains from uncertain differences. The authors also report negative results instead of presenting aggregation as universally effective.

  2. RAIM explicitly considers model cost, calibration labels, correlated errors, and the option of running judges locally. This makes the study relevant to high-volume evaluation settings where privacy or API access is a concern.

Critical weaknesses

RAIM's methodological novelty is limited, as judge aggregation and correlated errors have already been studied in PoLL (Verga et al., 2024), CARE (Zhao et al., 2026), and Balasubramanian et al. (2026). Calibration-limited panel decisions are also explored by Zhu and Rao (2026) and Zhou and Lin (2026). RAIM mainly relies on standard logistic stacking, with statistically significant improvement over the best single judge on only one of eight benchmarks. Moreover, Qwen2.5-32B achieves comparable accuracy (72.6% vs. 73.0%) at a lower reported inference cost, weakening the practical advantage of the ensemble.

The admissibility test lacks independent validation, as its thresholds were developed and evaluated on the same eight benchmarks. The additional 1,888 sub-panel experiments reuse the same data, and the dataset-level relationships are not statistically significant. The lack of transfer gains on the four core grounded datasets and evaluation on artificially balanced datasets further raise concerns about generalization to unseen, real-world tasks. This is particularly important since identifying when the ensemble should replace a frontier judge is a central contribution of the paper.

Questions for authors

Can authors please provide additional details beyond the established stacking and prior work on correlated judge panels? Can the authors report paired, per-dataset comparisons against the single Qwen2.5-32B judge, including calibration and deployment costs?

Can the proposed rule be tested on genuinely unseen tasks or different judge rosters, with its thresholds fixed beforehand? How often would it incorrectly recommend replacing the frontier judge?

How do the results change under realistic, non-50/50 hallucination rates and when calibration labels are costly or unavailable? Does the estimated cost advantage remain after accounting for serving ten models and the need to recalibrate each new domain?

4 comments
AuthorsOpenReview ↗

Response to Reviewer uHii: novelty (1/4)

We thank the reviewer for their careful review, and in particular for naming the five works closest to ours, which we use below to sharpen the positioning. Every number below comes from the corrected release, aggregated over the new grounded core of eight datasets (four of them new AggreFact sets) or over all fourteen. The general update we posted earlier this week sets out the experiments and the IDs (E1–E13, A1–A5) used below, and the reviewer might wish to read it first.

Please note that the sections, appendices, and tables cited below refer to the revised manuscript, which we will upload with the corrected code release shortly, once our replies to all reviewers are posted, and announce in a general update. Each answer states its results in full, so this reply can be read before then.


Q1a. What does the work add beyond stacking and prior work on correlated panels?

We agree that neither ingredient is new: PoLL already argues that a cross-family panel can replace a stronger judge at a lower price, CARE and Balasubramanian et al. already identify correlated judge errors as the failure of naive voting, and our aggregator is a standard stacked logistic regression, chosen so that any gain is attributable to the members rather than to the estimator. What we claim is the question we pose and the decision rule we introduce, which, to the best of our knowledge, none of the five addresses:

  • Pricing a frontier replacement with its labels counted.
    PoLL prices inference against GPT-4 at list rates; none of the five counts the labels the aggregator consumes or derives a break-even volume, and Zhu and Rao report no monetary cost.
  • Deciding per task whether the panel can replace the frontier judge, from competence and correlation together.
    CARE and Balasubramanian et al. are label-free aggregators: they improve how a panel is read, not whether it should be run, and, being label-free, they rest on structural assumptions (a latent-confounder model; a parametric Ising law) that need not hold where the panel's consensus is its shared error (App. C and App. L). Zhu and Rao spend a label budget on choosing the aggregator family and panel size once a panel is to be run. JuryProbe is the nearest analogue: it flags a three-judge panel from the dependence of its errors alone, yet with no competence coordinate, and escalates to the same panel given a trusted reference rather than to a frontier judge.
  • Setting.
    None of the five evaluates a panel of 4–9B open-weight judges against a frontier judge on hallucination detection; only Balasubramanian et al. include a faithfulness set (synthetic CNN/DM inconsistencies), with large and API judges. The others, instead, evaluate reference-based question answering and Arena-Hard with three larger or API judges (PoLL), scoring, classification, and preference benchmarks with open judges of 0.6–14B (CARE), preference and summarisation benchmarks with six 7–12B open judges and one API model (Zhu and Rao), and binary factuality with three 7–12B open judges and no frontier comparator (JuryProbe).

Revision: §1, Relation to prior work, and App. L (A1).

AuthorsOpenReview ↗

Response to Reviewer uHii: the single 32B judge (2/4)

Q1b. How does the panel compare with the single Qwen2.5-32B judge, per dataset and with costs?

We agree that the 32B is a close rival on average, offering inference at a lower cost per item (zero-shot, it runs at USD 0.008 per 1,000 items against the panel's USD 0.039, since on the same card it needs less GPU time per item than the ten members run in sequence). As a consequence, no volume of evaluated items repays the panel's one-time calibration of 50 to 100 labelled records against it. However, the two are not rivals: we propose a method rather than a roster, and a larger open judge can enter the panel as one more member instead of replacing it entirely. Added as an eleventh member (E9), the 72B keeps the panel's clean wins on MedHallu and RAGTruth (though not LFQA's win, on which it becomes the strongest single member), clearly improves the ten-member stack on five sets and clearly worsens it on none, and turns two of its four clear deficits against Sonnet (TruthfulQA and CNN) into unresolved ones, with the protocol unchanged and the stacker refitted on the same records; the 32B, added the same way, costs RAGTruth's win, where it becomes the strongest single member. As stronger open models appear, the panel might thus absorb them at the price of a refit, whereas a single judge can only be replaced by another.

Alone, the panel also holds the larger clear margins: its largest clear lead over the 32B is 0.140 κ (MedHallu) and its largest clear deficit 0.110 (TofuEval-MeetB), it keeps its MedHallu lead when the 32B is given the same labels as demonstrations, and, at 4 bits, it runs on a 16 GB consumer card on which the 32B does not fit.

To answer the question in full, we compare the two set-by-set in what follows.

Per dataset analysis

The mean balanced accuracies are close (panel 75.8 and 32B 75.6 over the fourteen datasets; 79.8 and 78.1 over the grounded core; E7). Per dataset, the panel is a clear improvement on the 32B on four sets (MedHallu +0.140 κ, FactCheck-GPT +0.134, ExpertQA +0.049, and Reveal +0.046), a clear worsening on three (TofuEval-MeetB −0.110, ClaimVerify −0.063, and RAGTruth −0.050), and unresolved on the other seven.

With the same labels

When the 32B receives the stacker's own 50 or 100 records as 4 or 8 demonstrations (four configurations; E8), it still trails the panel on MedHallu in every configuration (on 60–90% of draws), and on FactCheck-GPT in one. Elsewhere the 32B pulls ahead: clearly on RAGTruth, TruthfulQA, and CNN, in every configuration and on most draws (70–100%), and less consistently (20–80% of draws) on ExpertQA (in every configuration) and on LFQA and Reveal (in three of the four).

The demonstrations end in a bare verdict, whereas the zero-shot prompt asks for a brief reason first, so they change the 32B's answer format as well. That format alone helps the 32B most on MedHallu, yet the panel's lead there holds (App. H).

Used instead to calibrate the 32B's decision threshold, the same records leave the panel clearly ahead on FactCheck-GPT (and on MedHallu at 50 records) and clearly behind on RAGTruth, TruthfulQA, ClaimVerify, and TofuEval-MeetB (and on CNN at 50).

Panel against the 32B, over the fourteen datasetsClear worseningUnresolved worseningUnresolved improvementClear improvement
Zero-shot3344
Few-shot, 4 of 50 records6431
Few-shot, 8 of 50 records6521
Few-shot, 4 of 100 records4622
Few-shot, 8 of 100 records6341
Threshold on 50 records5342
Threshold on 100 records4541

Few-shot and threshold rows: the difference averaged over ten calibration draws.

Deployment

Quantised to 4 bits, the whole panel runs on one 16 GB consumer card, on which neither the bf16 panel nor the 32B fits, and it does so at USD 0.014 per 1,000 items (E12). Over the fourteen datasets, the quantised panel keeps the full-precision panel's mean balanced accuracy (75.9 against 75.8), differs clearly from it on at most four sets per card, and keeps the clean wins on MedHallu and RAGTruth (though not LFQA's win); at matched precision, its clear improvements over the 32B on MedHallu and FactCheck-GPT hold at both bf16 and 4 bits. The revision states this trade-off as such:

[…] the case for the panel over a single open judge rests on its quality set by set, and on the hardware each needs […], not on price. (App. H)

Revision: Table 2 and App. H, Scaling, The same labels for the single judge, Open judges inside the stack, and Rented and owned compute.

AuthorsOpenReview ↗

Response to Reviewer uHii: the admissibility test and significance (3/4)

Q2. Does the rule hold on unseen tasks, with its thresholds fixed beforehand, and how often would it wrongly recommend replacing the frontier judge?

We agree that the thresholds were formed on the submission's eight datasets, and that the sub-panel check reuses the same data. Checking fixed thresholds on held-out data is what JuryProbe, among the works the reviewer cites, already does, and what our test lacked; we therefore now check the submission's thresholds, unchanged, on six held-out sets, and also choose the thresholds out of sample.

The submission's thresholds on new data

On the six sets added in E1, with roster, prompt, and thresholds unchanged, the test rejected all six, each on the correlation bar (φ₀ = 0.40). We recall, however, that this bar was chosen to be conservative: as stated in the original submission, at a share of one half every bar from φ₀ = 0.335 upwards gave the same calls on our eight sets (App. C of the submission), so the data bounded it from below only.

How often would it wrongly recommend replacing the frontier judge? On these six, never, since it recommended no replacement; over all fourteen datasets, it admits none on which any of the five reference judges is clearly ahead of the panel.

The cost of that choice, however, shows on three of the five sets that φ₀ = 0.40 rejects on correlation alone while no reference judge is clearly ahead of the panel. On two of the six added sets, FactCheck-GPT and LFQA, the panel clearly beats both its CV-selected best member and Sonnet. On XSum, one of our original eight, six members are competent, but the mean pairwise error correlation is 0.485 on the corrected verdicts.

Moving the correlation bar

Since the data bounded the bar from below only, we chose it by the paper's procedure on the submission's eight sets alone, leaving one out at a time: seven of the eight folds choose φ₀ = 0.56 against each reference judge (XSum's fold, 0.42–0.47). On the six added sets (unused by this choice), φ₀ = 0.56 admits FactCheck-GPT, LFQA, and Reveal, while it rejects ClaimVerify and both TofuEval sets. Four of these six calls are right against Sonnet (where 0.40 had one), and neither 0.40 nor 0.56 admits a set on which any reference judge is clearly ahead. The revision then tunes the bar the same way on the grounded core (E2): seven of its eight folds choose φ₀ = 0.57–0.59 (the fold without ClaimVerify, 0.67), and their median, φ₀ = 0.59, is adopted; indeed, it makes the same calls as 0.56 on all fourteen datasets, as does every bar from Reveal's φ̄ (0.553) up to TofuEval-MeetB's (0.600).

Thresholds chosen out of sample

Finally, we tested the whole rule out of sample, re-choosing both thresholds by the paper's procedure with each of the fourteen datasets left out in turn, and calling the one left out. The rule calls nine of the fourteen right, one fewer than admitting every dataset, but its errors lie mostly on the safe side. Admitting every dataset would in fact pass the four on which Sonnet leads by more than 0.10 κ (RAGTruth, ClaimVerify, CNN, and TruthfulQA), while the test admits only ClaimVerify (and it does so only in the fold that leaves it out). Its four other errors (XSum, ExpertQA, FActScore, and TofuEval-MediaS) actually reject datasets on which the panel stays within 0.10 κ of Sonnet, forgoing a saving but never costing quality.

The test is therefore a conservative screen for frontier substitution and not a classifier of tasks. For the gain over the panel's own best member, by contrast, the rule is at chance out of sample (J −0.05 on the core), and the revision no longer uses the test for that comparison; our fourth comment describes where that gain lies.

To wrap up and answer directly: with thresholds set without the six held-out sets, the rule wrongly recommends replacing the frontier judge on none of them, whether at the submission's φ₀ = 0.40 or at the 0.56 chosen on the original eight; with both thresholds chosen out of sample, it does so once in fourteen calls (ClaimVerify), against four for always admitting. A second roster is beyond the discussion window, and the revision states it as a limitation.

Revision: §3.3 and App. C; the roster in App. M, Scope of the panel and of its competitors.


The improvement over the best single judge is significant on only one benchmark

Under the corrected verdicts, the clear wins over the best member on the submission's four core sets go from one to two (MedHallu and RAGTruth). With the sets added in E1, they are four of the eight core sets (MedHallu, RAGTruth, FactCheck-GPT, and LFQA), all surviving Holm's correction over the eight at a one-sided 0.05 (all but LFQA at 0.025), and each a clear win on 90–100% of 100 re-drawn fold splits (E6).

Revision: abstract, §3.2, and Table 1; the fold splits in App. K.

AuthorsOpenReview ↗

Response to Reviewer uHii: prevalence, labels, serving, and the remaining weaknesses (4/4)

Q3a. How do the results change at realistic hallucination rates?

The sets are balanced by construction, so that chance agreement is fixed and κ and balanced accuracy are comparable across sets (App. J). Under a change of prevalence alone, each class's items, and hence each judge's and the stacker's per-class recall, are unchanged, so balanced accuracy does not move: every balanced-accuracy result, the paired differences against Sonnet and the 32B included, holds at any hallucination rate. κ, by contrast, does depend on prevalence, and a deployer minimising plain error would shift the stacker's intercept by the log-ratio of the priors, or draw the calibration sample from the deployment stream. We leave an evaluation re-weighted to each benchmark's native rate to future work. Read per class, the panel flags fewer items as hallucinated than Sonnet on most sets, trading recall on that class for specificity at about the same precision; at a low hallucination rate, where false alarms weigh more, this lean works in the panel's favour, and the stacker's threshold can shift it if a deployer's costs call for it.

Revision: the restriction stated in the main text (A5); App. M, Class balance.


Q3b. What if calibration labels are costly or unavailable?

Where labels are costly, the relevant question is what the panel gives at the small budgets of the costing, and we now report it paired (E5). With the stacker and the best member both selected on 50 labelled records, the panel stays clearly ahead of its best member on all eight grounded-core sets with the interval averaged over draws, and on most single draws on MedHallu and RAGTruth (91% and 66%; LFQA 47%). Against Sonnet, it is clearly behind on three core sets at that budget (RAGTruth, ClaimVerify, and MedHallu, the last on 66% of draws) against two with the full calibration pool. Over the sets large enough for each budget, it retains a median 96% of Sonnet's κ at 50 records and 97% at 100 (99% on the same sets at the full pool), and gives up 3.6 and 2.3 points of balanced accuracy on average (1.7 and 1.0). Where no labels are available, the panel offers no guarantee over its best member; the label-free aggregators we compare do not clearly beat the stacked panel on any of the fourteen datasets. The labels' cost, moreover, already enters our account (Q3c).

Revision: the cost claim stated with these budgets in the abstract and §4; the paired comparisons in App. C.


Q3c. Does the cost advantage survive serving ten models and recalibrating each domain?

Recalibration is already in the price: the break-even charges a one-time calibration per domain, with the labels priced at USD 1 each (two to a record on the paired sets), an outlay of USD 100 to USD 200 that the panel's cheaper inference repays against Sonnet between some 40,000 and 81,000 evaluated items; a new domain pays it again, and the break-even restarts. On serving: the 4-bit panel runs in sequence on a single 16 GB consumer card (Q1b), at USD 0.014 per 1,000 items.

Revision: App. H, Rented and owned compute and Amortised cost and break-even.


The dataset-level relationships are not significant

We agree that the published coordinates do not predict the gain over the best member out of sample, and the revision no longer uses them for it. In exploratory analysis (E13) we found that the gain follows whether the leader has a partner: a member nearly as competent that errs differently from it, max_i (κ_i/κ_L)(1 − φ_Li). Within datasets it tracks the stacked sub-panels' gain over their own best member on all fourteen (mean Spearman ρ +0.38 [+0.27, +0.49]; on the grounded core +0.31 [+0.20, +0.42], eight of eight; outside it +0.46 [+0.27, +0.63], six of six); across datasets, the five on which the stack clearly beats its best member are five of the six with the strongest partner, the sixth being WiCE. The test's own coordinate answers the other question: within datasets, the number of competent members tracks the panel's standing against Sonnet (ρ +0.40 [+0.29, +0.51]) but barely its gain over the best member (+0.11 [−0.06, +0.28]), and the partner does the reverse (+0.06 against Sonnet). Found on these data, the partner is reported as a description of where the gain lies, not as a decision rule.

Revision: §3.3 and App. B.


There are no transfer gains on the core

We agree, and indeed the paper reports it as a negative result; on the corrected release, a stacker fitted on the other core sets transfers no win to any of the eight grounded-core sets, because the most reliable judge changes by dataset and a fixed weight vector cannot follow it. We read it as a scope condition and not a defect to hide: calibration is per domain, and our costing charges it per domain, as Q3c sets out.

Revision: App. E.

Reviewer_p13kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The=is work assesses a group of ten open-weight judges to see if they can replace a frontier judge for hallucination detection. It combines a cross-fitted stacked logistic regression over the members’ binary votes with an admissibility test that admits a task when at least half the members reach a certain agreement and mean pairwise error correlation is at most .4. Validation covers a few benchmarks using paired cluster-bootstrap differences. The panel clearly beats the cross-validated best single member on only 1 set. Against Sonnet it is worse on 6 of 8. It keeps approx. 93% of Sonnet’s k and loses 3 points of balanced accuracy. Inference costs a fraction of Sonnet’s.

Critical strengths

The question has practical utility. The paired cluster bootstrap, a comparator ladder ordered by labels consumed, explicit win and tie rules, and honest reporting of mostly unresolved results make this a trustworthy mixed result. It also shows in item-level analysis of the some 2k cases where the leading judge errs links panel gains to how competence is spread and how errors decorrelate. Both can be read from the calibration set, which gives deployers something actionable.

Critical weaknesses

The ideas is not that novel,. The main result is weak. Personally, I blieve even negative result or weak result like this is also important when it is a top of mind question for several of us. The threshold robustness check uses sub-panels drawn from the same ten members on the same datasets, which is not independent. The panel clearly beats its own best member on one benchmark and trails the frontier on six of eight. As validated, the test is a post-hoc description, not a predictor. Table 2 shows a single Qwen nearly matching the panel. It costs about a fifth as much & needs no in-domain labels. There are no cheap API judges. Members contribute only hard verdicts, not verdict probabilities. Cascades are discussed but not measured. The cost analysis omits the operational cost of hosting ten models.

Questions for authors

Does a ten-model panel clearly beat a single Qwen2.5-32B or 72B, with matched labels, on any benchmark? Does adding those models to the stack change the results? How does the panel compare with Claude Haiku or GPT 4o mini under the same protocol and cost accounting? Does stacking verdict probabilities instead of binary votes change the number of wins over the best single member?

3 comments
AuthorsOpenReview ↗

Response to Reviewer p13k: the first two questions (1/3)

We thank the reviewer for the insights provided. It is worth noticing that our results are conditional, rather than merely negative. In particular, on the grounded core of eight, we have four clear wins over the panel's best member, and two clear improvements and two clear worsenings against Sonnet, each characterised and analysed. Every number below comes from the corrected release, aggregated over the new grounded core of eight datasets (four of them new AggreFact sets) or over all fourteen. The general update we posted earlier this week sets out the experiments and the IDs (E1–E13, A1–A5) used below, and the reviewer might wish to read it first.

Please note that the sections, appendices, and tables cited below refer to the revised manuscript, which we will upload with the corrected code release shortly, once our replies to all reviewers are posted, and announce in a general update. Each answer states its results in full, so this reply can be read before then.


Q1. Does the panel clearly beat a single Qwen2.5-32B or 72B with matched labels?

Over the fourteen datasets, the above three approaches are close in mean balanced accuracy (panel 75.8, 32B 75.6, 72B 76.1; E7). Zero-shot: the panel is a clear improvement on the 72B on five sets (MedHallu, RAGTruth, ExpertQA, FActScore, and FactCheck-GPT) and a clear worsening on three (CNN, TruthfulQA, and TofuEval-MeetB); on the 32B, a clear improvement on four (MedHallu, ExpertQA, FactCheck-GPT, and Reveal) and a clear worsening on three (RAGTruth, ClaimVerify, and TofuEval-MeetB).

When the 32B receives the stacker's own 50 or 100 records as 4 or 8 demonstrations (four configurations; E8; the 72B ran zero-shot only), it still trails the panel on MedHallu in every configuration (on 60–90% of draws), and on FactCheck-GPT in one. Elsewhere the 32B pulls ahead: clearly on RAGTruth, TruthfulQA, and CNN, in every configuration and on most draws (70–100%), and less consistently (20–80% of draws) on ExpertQA (in every configuration) and on LFQA and Reveal (in three of the four).
The demonstrations end in a bare verdict, whereas the zero-shot prompt asks for a brief reason first, so they change the 32B's answer format as well. That format alone helps the 32B most on MedHallu, yet the panel's lead there holds (App. H).

Used instead to calibrate the 32B's decision threshold, the same records leave the panel clearly ahead on FactCheck-GPT (and on MedHallu at 50 records) and clearly behind on RAGTruth, TruthfulQA, ClaimVerify, and TofuEval-MeetB (and on CNN at 50).

So the answer is yes, the panel is still ahead on a few benchmarks that depend on how the labels are used (MedHallu with demonstrations, FactCheck-GPT with a calibrated threshold); although the zero-shot 32B is cheaper per item, the panel's further case is hardware (see below).

Panel against, over the fourteen datasetsClear worseningUnresolved worseningUnresolved improvementClear improvement
72B, zero-shot3335
32B, zero-shot3344
32B, few-shot, 4 of 50 records6431
32B, few-shot, 8 of 50 records6521
32B, few-shot, 4 of 100 records4622
32B, few-shot, 8 of 100 records6341
32B, threshold on 50 records5342
32B, threshold on 100 records4541

Few-shot and threshold rows: the difference averaged over ten calibration draws.

Revision: App. H, Scaling and The same labels for the single judge.


Q2. Does adding the 32B or the 72B to the stack change the results?

Adding the 72B as an eleventh vote (E9) keeps MedHallu and RAGTruth as clean wins and FactCheck-GPT as a win, all Holm-significant at 0.025 over the core, but not LFQA, where the 72B becomes the strongest single; over the fourteen datasets it leaves two clear worsenings against Sonnet (RAGTruth, −0.142; ClaimVerify, −0.140) where the ten members leave four, TruthfulQA's and CNN's clear worsenings becoming unresolved (−0.146 to −0.010; −0.259 to −0.138). The 72B clearly improves the ten-member stack on WiCE, RAGTruth, CNN, TruthfulQA, and TofuEval-MeetB, and clearly worsens it nowhere. Adding the 32B clearly improves the stack on RAGTruth and TruthfulQA and clearly worsens it on ExpertQA, and it costs RAGTruth's win, the 32B itself becoming the strongest single there.

Revision: App. H, Open judges inside the stack.

AuthorsOpenReview ↗

Response to Reviewer p13k: the weaknesses (2/3)

The main result is weak

Under the corrected verdicts, the clear wins over the best member on the submission's four core sets go from one to two (MedHallu and RAGTruth); with the sets added in E1, they are four of the eight core sets and five of all fourteen. Each of the four core wins is a clear win on 90–100% of 100 re-drawn fold splits (E6). At a margin of 0.10 κ, the panel is also non-inferior to Sonnet on six of the eight core sets (E4), and over all fourteen datasets it retains a median 99% of Sonnet's κ and gives up 1.7 points of balanced accuracy on average, where the submission's eight sets alone still give 93% and 2.9 (E1).

Indeed, the paper's main result is the substitution rather than the count of wins: what replacing Sonnet costs, and a screen (read from the same calibration records) for where that cost stays small. Beyond this roster, the paper also offers a protocol that can be applied to other panels and tasks: every comparison is paired and reported in one four-way form, against both the best member and Sonnet; the calibration labels are counted in the price; and the whole chain, from the verdicts to every table, accompanies the submission and will be released publicly.

Revision: abstract, §3.2, §4, and Table 1; non-inferiority in App. C; the fold splits in App. K.


The test is post hoc, and its robustness check is not independent

We agree that the sub-panel check is not independent, since sub-panels share members and items. We therefore answer both points with data the test had not seen, and with thresholds chosen out of sample.

The submission's thresholds on new data

On the six sets added in E1, with roster, prompt, and thresholds unchanged, the test rejected all six, each on the correlation bar (φ₀ = 0.40). We recall, however, that this bar was chosen to be conservative: as stated in the original submission, at a share of one half every bar from φ₀ = 0.335 upwards gave the same calls on our eight sets (App. C of the submission), so the data bounded it from below only. At φ₀ = 0.40, the test errs only on the safe side: over all fourteen datasets, it admits none on which the panel is clearly behind Sonnet.

The cost of that choice, however, shows on three of the five sets that φ₀ = 0.40 rejects on correlation alone while no reference judge is clearly ahead of the panel. On two of the six added sets, FactCheck-GPT and LFQA, the panel clearly beats both its CV-selected best member and Sonnet. On XSum, one of our original eight, six members are competent, but the mean pairwise error correlation is 0.485 on the corrected verdicts.

Moving the correlation bar

Since the data bounded the bar from below only, we chose it by the paper's procedure on the submission's eight sets alone, leaving one out at a time: seven of the eight folds choose φ₀ = 0.56 against each reference judge (XSum's fold, 0.42–0.47). On the six added sets (unused by this choice), φ₀ = 0.56 admits FactCheck-GPT, LFQA, and Reveal, while it rejects ClaimVerify and both TofuEval sets. Four of these six calls are right against Sonnet (where 0.40 had one), and neither 0.40 nor 0.56 admits a set on which any reference judge is clearly ahead. The revision then tunes the bar the same way on the grounded core (E2): seven of its eight folds choose φ₀ = 0.57–0.59 (the fold without ClaimVerify, 0.67), and their median, φ₀ = 0.59, is adopted; it makes the same calls as 0.56 on all fourteen datasets, as does every bar from Reveal's φ̄ (0.553) up to TofuEval-MeetB's (0.600).

The test as a predictor, out of sample

Finally, we tested the whole rule as a predictor, re-choosing both thresholds by the paper's procedure with each of the fourteen datasets left out in turn, and calling the one left out. The rule calls nine of the fourteen right, one fewer than admitting every dataset, but its errors lie mostly on the safe side. Admitting every dataset would in fact pass the four on which Sonnet leads by more than 0.10 κ (RAGTruth, ClaimVerify, CNN, and TruthfulQA), while the test admits only ClaimVerify (and it does so only in the fold that leaves it out). Its four other misjudgements (XSum, ExpertQA, FActScore, and TofuEval-MediaS) actually reject datasets on which the panel stays within 0.10 κ of Sonnet, forgoing a saving but never costing quality.

As a predictor, the test is therefore a conservative screen for frontier substitution and not a classifier of tasks. For the gain over the panel's own best member, by contrast, the rule is at chance out of sample (J −0.05 on the core), and the revision no longer uses the test for that comparison; our third comment describes where that gain lies.

Revision: §3.3 and App. C.

AuthorsOpenReview ↗

Response to Reviewer p13k: the last two questions, and the weaknesses continued (3/3)

Q3. How does the panel compare with Claude Haiku and GPT-4o mini?

Haiku 4.5 sits between the panel and Sonnet (mean balanced accuracy over the fourteen datasets 77.0, against 75.8 and 77.5; E10): the panel is a clear improvement on it on MedHallu, ExpertQA, and FactCheck-GPT and a clear worsening on RAGTruth, CNN, TruthfulQA, ClaimVerify, and TofuEval-MeetB, the four on which Sonnet is clearly ahead of the panel and one more. GPT-4o mini (74.4) is clearly worse than the panel on MedHallu, ExpertQA, and FactCheck-GPT and clearly better only on ClaimVerify. At their list token prices, on the paper's cost profile, Haiku costs USD 0.84 and GPT-4o mini USD 0.12 per 1,000 items, against USD 0.039 for the panel and USD 2.52 for Sonnet; against GPT-4o mini, at about three times the panel's price, the panel's case rests on on-premises inference and on quality.

Revision: Table 2 and App. H, Five reference judges.


Q4. Does stacking verdict probabilities instead of votes change the wins?

We stacked the members' verdict probabilities instead of their votes, with the same cross-fitting, on all fourteen datasets (E11). No set's soft-minus-hard difference resolves (at most 0.037 κ), the sets on which the panel beats its best member are unchanged, and against Sonnet only LFQA changes form, its clear improvement becoming unresolved (+0.044 against +0.047).

Revision: App. H, Probabilities instead of verdicts.


On novelty

We agree that neither ingredient is new (cross-family panels are present in PoLL, correlated judge errors in CARE and Balasubramanian et al.), and our stacker is kept standard, so that any gain is attributable to the members. What we claim is the question and the decision rule: pricing a frontier replacement (including the calibration labels it requires), and deciding per task, from competence and correlation together, whether the panel can replace the frontier judge.

Our reply to Reviewer uHii sets this against the five closest works.

Revision: §1, Relation to prior work, and App. L (A1).


Where the gain over the best member comes from

In exploratory analysis (E13) we found that the gain follows whether the leader has a partner: a member nearly as competent that errs differently from it, max_i (κ_i/κ_L)(1 − φ_Li). Within datasets it tracks the stacked sub-panels' gain over their own best member on all fourteen (mean Spearman ρ +0.38 [+0.27, +0.49]; on the grounded core +0.31 [+0.20, +0.42], eight of eight; outside it +0.46 [+0.27, +0.63], six of six); across datasets, the five on which the stack clearly beats its best member are five of the six with the strongest partner, the sixth being WiCE. Found on these data, it is reported as a description of where the gain lies, not as an admission rule.

Revision: §3.3 and App. B.


The operational cost of hosting ten models

Quantised to 4 bits, the whole panel runs on one 16 GB consumer card, on which neither the bf16 panel nor the 32B fits, at USD 0.014 per 1,000 items (E12). Over the fourteen datasets, the quantised version keeps the full-precision panel's mean balanced accuracy (75.9 against 75.8), the clean wins on MedHallu and RAGTruth (though not LFQA's win), and every clear result against Sonnet.

Revision: App. H, Rented and owned compute.


Cascades are discussed but not measured

Indeed, not measuring one is a choice of scope: a cascade still sends its hardest items off-premises to a rate-limited, versioned endpoint, the constraint the panel exists to remove (App. L). We read a panel-first cascade as complementary, and leave open what share of frontier calls it would retain at matched quality.

A cascade answers a different question from ours: how many frontier calls can be avoided, not whether the frontier call can be removed. (App. M)

Revision: App. L and App. M state this scope (A2).

Reviewer_tuNSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper examines whether a panel of small open-weight LLM judges can replace a proprietary frontier judge in binary hallucination detection: each judge classifies a candidate answer or claim as supported or hallucinated, and agreement with gold labels is measured by Cohen’s κ on class-balanced data. Frontier judges are the most reliable option, but usage costs, rate limits, and the need to send text off-premises make them ill-suited to continuous, high-throughput monitoring. The authors argue that naive panels suffer from correlated errors among judges with shared pretraining data, allowing weak, redundant members to outvote a capable one. To address this, they propose RAIM. RAIM fits a cross-fitted stacked logistic regression to member verdicts using a small labelled calibration set. It pairs this with an admissibility test that accepts a task when at least half the members exceed a κ threshold, and their mean pairwise error correlation is below a threshold. Across eight faithfulness benchmarks using ten 4–9B judges, the panel significantly outperforms its cross-validated best member on one dataset and is statistically indistinguishable from it on the other seven. Against its cross-validated best member, the panel’s κ is higher on seven of eight datasets, but the difference is significant only on one. Compared with Claude Sonnet, it retains a median 93% of κ at about 1/64 of the inference cost, following a one-time calibration on 50–100 labelled records.

Critical strengths
  1. The paper asks a practical question and puts a price on the answer. Rather than asking only whether a panel matches a frontier judge, it accounts for the labels the aggregator consumes and derives a break-even volume, the quantity a deployer needs.
  2. The evaluation protocol is rigorous. Every comparison uses paired cluster-bootstrap differences, and results are reported in four categories that distinguish direction from resolvability. The headline MedHallu result is also rechecked on the baseline’s committed items.
  3. The comparator set is broad and well organized. Baselines are grouped by the labelled information they use: label-free aggregators, single-judge selections, purpose-trained detectors, larger open judges, and a frontier judge. This shows what each additional level of supervision contributes.
  4. Several empirical findings are notable in their own right. Unweighted majority voting performs significantly worse than the best member on four datasets, but supervised weighting closes that gap. The member with the lowest individual accuracy can still receive one of the largest stacker weights, supporting the inclusion of weak but non-redundant judges.
Critical weaknesses

[W1] The claim that the panel “can stand in for a frontier one wherever this audit admits it” (Abstract, ll. 042–043) overstates the evidence. Even on the three admitted datasets, matching the frontier is not established: the panel is slightly but significantly worse than Sonnet on MedHallu (−0.036), and on WiCE and FActScore the paired intervals reach −0.140 and −0.152, beyond the paper’s own 0.10-κ tolerance (Appendix C). For unseen downstream tasks, there is no evidence that the current setting and thresholds still identify where the panel matches the frontier, since they were set on these same eight datasets and never evaluated on a held-out task, as the authors acknowledge in Appendix M. The claim should be limited to the observed datasets (e.g., “on the three datasets this audit admits”), ideally with a non-inferiority analysis at a stated margin.

[W2] A single Qwen2.5-32B nearly matches the panel at a fifth of its inference cost. The 32B judge reaches 72.6 suite BACC against the panel’s 73.0 (Table 2) and has a higher κ on four of the eight datasets (Table 13), while needing no labels and offering the same on-premises properties, so the case for running ten models plus a stacker is unclear. The paper argues that a deployer cannot know in advance where the 32B judge holds (§4), but the same applies to the panel, whose admissibility thresholds were set on these eight datasets; moreover, the 50–100 labels the panel already requires would suffice to check the 32B judge directly on a new task.

[W3] Quality and cost are measured at different label budgets. The headline comparisons (Table 1) fit the stacker and select the CV-best using full training pools of 87–1,088 records per fold, while the cost analysis assumes 50–100 records. At 100 records, the stacker’s κ on ExpertQA drops from 0.235 to 0.177, and paired comparisons with the CV-best and Sonnet at this budget are not reported. Recomputing the right-hand columns of Table 1 at 50 and 100 records is necessary to make the cost claim self-consistent.

[W4] The regime coordinates depend strongly on the prompt. Changing only the prompt framing increases the number of competent members from 0/10 to 9/10 on WiCE and from 0/10 to 4/10 on XSum (Table 11). The admissibility call therefore depends jointly on the roster, prompt, and task, rather than on the task alone. Reporting the coordinates and call for two or three prompt variants per dataset would show deployers how stable the call is.

Questions for authors
  1. With the thresholds fixed in advance at (0.30, ½, 0.40), what are the admissibility decisions and the panel’s paired differences relative to Sonnet and Qwen2.5-32B on two or three held-out benchmarks? This will help demonstrate the proposed method’s generalizability and advantages.

  2. How do Qwen2.5-32B calibrated on the same 50–100 labels (threshold or few-shot), and Sonnet with those labels as few-shot examples, compare with the panel? The panel uses 50–100 in-domain labels to fit its stacker, whereas both single-judge alternatives are evaluated zero-shot. A single Qwen2.5-32B already nearly matches the panel without labels (72.6 vs. 73.0 suite BACC; Table 2), so with the same labels it may close or reverse the gap. Similarly, giving Sonnet the labels as few-shot examples, a comparison the authors note they did not run (Appendix M), would show whether the reported substitution cost (a median 93% of Sonnet’s κ) is understated.

  3. Does the MedHallu win survive Holm correction at a one-sided α of 0.025, and how often does it hold across, say, 10 random fold-split seeds? The main decision rule declares a win when the 95% paired interval excludes zero (§2), corresponding to a one-sided α of 0.025. That threshold is also used to classify XSum (one-sided p = 0.045) as unresolved (§3.2). However, the Holm-adjusted MedHallu p-value of 0.048 is declared significant against 0.05 (§3.2; Appendix K). These standards are inconsistent: under the stricter threshold, the suite’s only clean win would not survive correction. Because the win’s lower bound is only +0.004 and the 5-fold split uses a single seed, testing its stability across splits would also clarify how robust the result is.

5 comments
AuthorsOpenReview ↗

Response to Reviewer tuNS: the test on held-out benchmarks (1/4)

Edited: XSum's mean error correlation is evaluated on all ten members; we anchor the range of equivalent bars to the two datasets that bound it; the pointer for φ₀ = 0.335 now names the submission's App. C, which the revised manuscript no longer carries.

We thank the reviewer for a precise and constructive review; all its requests but two are now measured: Sonnet with the labels as examples (part of Q2), left out for its cost, and the prompt variants, run on one dataset rather than on each (W4, explained in our fourth comment).

Every number below comes from the corrected release, aggregated over the new grounded core of eight datasets (four of them new AggreFact sets) or over all fourteen. The general update we posted earlier this week sets out the experiments and the IDs (E1–E13, A1–A5) used below, and the reviewer might wish to read it first.

Please note that the sections, appendices, and tables cited below refer to the revised manuscript, which we will upload with the corrected code release shortly, once our replies to all reviewers are posted. Each answer states its results in full, so this reply can be read before then.


Q1. What does the test decide on held-out benchmarks, with its thresholds fixed in advance?

The submission's thresholds on new data

We ran six LLM-AggreFact sets outside the submission's (ClaimVerify, FactCheck-GPT, LFQA, Reveal, and the two TofuEval sets; E1), with roster, prompt frame, and thresholds unchanged. All six were rejected, each on the correlation bar φ₀ = 0.40 (mean error correlation 0.455–0.675). We recall, however, that this bar was chosen to be conservative: as stated in the original submission, at a share of one half every bar from φ₀ = 0.335 upwards gave the same calls on our eight sets (App. C of the submission), so the data bounded it from below only.

Against Sonnet, the panel is a clear worsening on ClaimVerify (−0.150 κ), a clear improvement on FactCheck-GPT (+0.143) and LFQA (+0.047), and unresolved on the other three; against the 32B, a clear improvement on FactCheck-GPT (+0.134) and Reveal (+0.046), a clear worsening on ClaimVerify (−0.063) and TofuEval-MeetB (−0.110), and unresolved on the other two. Indeed, the test errs only on the safe side: over all fourteen datasets it admits none on which the panel is clearly behind Sonnet.

The cost of that choice shows on two of the six (FactCheck-GPT and LFQA), on which the panel clearly beats both its CV-selected best member and Sonnet, and on XSum, whose ten members, six of them competent, have a mean error correlation of 0.485 on the corrected verdicts: φ₀ = 0.40 rejects all three on correlation alone, although no reference judge is clearly ahead of the panel on any of them.

Moving the correlation bar

Since the data bounded the bar from below only, we chose it by the paper's procedure on the submission's eight sets alone, leaving one out at a time: seven of the eight folds choose φ₀ = 0.56 against each reference judge (XSum's fold, 0.42–0.47). On the six added sets (unused by this choice), φ₀ = 0.56 admits FactCheck-GPT, LFQA, and Reveal, while it rejects ClaimVerify and both TofuEval sets. Four of these six calls are right against Sonnet (where 0.40 had one), and neither 0.40 nor 0.56 admits a set on which any reference judge is clearly ahead. The revision then tunes the bar the same way on the grounded core (E2): seven of its eight folds choose φ₀ = 0.57–0.59 (the fold without ClaimVerify, 0.67), and their median, φ₀ = 0.59, is adopted; it makes the same calls as 0.56 on all fourteen datasets, as does every bar from Reveal's φ̄ (0.553) up to TofuEval-MeetB's (0.600). The fixed-threshold result above remains the answer to the test as the reviewer posed it.

Thresholds chosen out of sample

Finally, we tested the whole rule out of sample, re-choosing both thresholds by the paper's procedure with each of the fourteen datasets left out in turn, and calling the one left out. The rule calls nine of the fourteen right, one fewer than admitting every dataset, but, in this case, it errs in the safe direction. Admitting every dataset would in fact pass the four on which Sonnet leads by more than 0.10 κ (RAGTruth, ClaimVerify, CNN, and TruthfulQA), while the test admits only ClaimVerify (and it does so only in the fold that leaves it out). Its four other errors (XSum, ExpertQA, FActScore, and TofuEval-MediaS) actually reject datasets on which the panel stays within 0.10 κ of Sonnet, causing a saving loss but never costing quality.

The test is therefore a conservative screen for frontier substitution rather than a classifier of tasks. For the gain over the panel's own best member, by contrast, the rule is at chance out of sample (J −0.05 on the core), and the revision no longer uses the test for that comparison.

Revision: §3.3 and App. C.

AuthorsOpenReview ↗

Response to Reviewer tuNS: the single 32B judge (2/4)

Edited: the costs per 1,000 items (USD 0.008 for the 32B, USD 0.039 for the panel), lost in rendering, restored; the quotation of §4 matched to the revised manuscript as uploaded; the 72B's effect on the clean wins stated exactly (LFQA's passes to it).

Q2 (and W2). How does Qwen2.5-32B given the same labels, and Sonnet given them as examples, compare with the panel?

Over the fourteen datasets, the panel and the two open judges are close in mean balanced accuracy (panel 75.8, 32B 75.6, 72B 76.1; E7). Zero-shot, the panel is a clear improvement on the 32B on MedHallu (+0.140 κ), FactCheck-GPT (+0.134), ExpertQA (+0.049), and Reveal (+0.046), a clear worsening on TofuEval-MeetB (−0.110), ClaimVerify (−0.063), and RAGTruth (−0.050), and unresolved on the other seven.

With the same labels

When the 32B receives the stacker's own 50 or 100 records as 4 or 8 demonstrations (four configurations; E8), it still trails the panel on MedHallu in every configuration (on 60–90% of draws), and on FactCheck-GPT in one. Elsewhere the 32B pulls ahead: clearly on RAGTruth, TruthfulQA, and CNN, in every configuration and on most draws (70–100%), and less consistently (20–80% of draws) on ExpertQA (in every configuration) and on LFQA and Reveal (in three of the four).

The demonstrations, however, also end in a bare verdict, and that format alone (i.e. with no demonstrations) already raises the 32B's κ on MedHallu, FactCheck-GPT, LFQA, and TruthfulQA and lowers it on ClaimVerify, as the demonstrations do (App. H); indeed, the panel's MedHallu lead holds although the format helps the 32B most there.

Used instead to calibrate a decision threshold on the 32B's own zero-shot replies (its probability of a hallucination verdict, with nothing regenerated), 100 records leave the panel clearly ahead on FactCheck-GPT (+0.086 κ, 90% of draws) and clearly behind on RAGTruth (−0.158), TofuEval-MeetB (−0.109), TruthfulQA (−0.076), and ClaimVerify (−0.069), with the other nine unresolved. At 50 records, MedHallu (+0.028) joins the clear improvements and CNN (−0.211) the clear worsenings.

Panel against the 32B, over the fourteen datasetsClear worseningUnresolved worseningUnresolved improvementClear improvement
Zero-shot3344
Few-shot, 4 of 50 records6431
Few-shot, 8 of 50 records6521
Few-shot, 4 of 100 records4622
Few-shot, 8 of 100 records6341
Threshold on 50 records5342
Threshold on 100 records4541

Few-shot and threshold rows: the difference averaged over ten calibration draws.

Price, hardware, and what labels allow

Zero-shot, the 32B is cheaper per item, at USD 0.008 per 1,000 items against the panel's USD 0.039 (Table 2). However, the panel's case lies elsewhere: in its quality where it holds, in a smaller hardware envelope, and in being a general methodology which, in principle, can take a larger open judge in as one more member rather than competing with it; indeed, added as an eleventh member, the 72B keeps the clean wins on MedHallu and RAGTruth (LFQA's win passes to the 72B, which becomes its strongest single member) and clearly worsens the stack on no set (E9).

As to deployment, quantised to 4 bits, the whole panel runs on one 16 GB consumer card, on which neither the bf16 panel nor the 32B fits; and it does so while keeping the full-precision panel's mean balanced accuracy (75.9 against 75.8) and its clean wins on MedHallu and RAGTruth (E12).

Also, we agree that 50–100 labels would let a deployer check the 32B directly: comparing the two on those records picks the better judge on 71–77% of the draws where one is clearly better (50% by chance), and the revision no longer claims that this cannot be known in advance. The one exception is ExpertQA, where the stacker keeps gaining up to its full pool (App. C): fitted on 50–100 records, it is still well short of its final κ, so the check prefers the 32B on about two draws in three, although the fully calibrated panel is clearly ahead (+0.049).

A single Qwen2.5-32B is nonetheless a close rival on average (75.6 against 75.8 BAcc over the full suite, Table 2), yet where it holds must be checked on labels (the same ones the panel's regime is read off). (§4)

Sonnet with the labels as few-shot examples was not run, for cost; the revision states that the zero-shot comparison may understate Sonnet's lead.

Revision: Table 2 and App. H, Scaling, The same labels for the single judge, and Rented and owned compute; the unrun Sonnet arm in App. M.

AuthorsOpenReview ↗

Response to Reviewer tuNS: significance and the claim (3/4)

Edited: the description and quotation of the abstract matched to the revised manuscript as uploaded.

Q3. Does the MedHallu win survive Holm at a one-sided 0.025, and across fold-split seeds?

Over the grounded core, the panel wins over its CV-best member on MedHallu (+0.079 [+0.053, +0.104]), RAGTruth (+0.065 [+0.029, +0.100]), FactCheck-GPT (+0.076 [+0.030, +0.120]), and LFQA (+0.032 [+0.007, +0.057]), all but FactCheck-GPT also beating the oracle member. No bootstrap resample falls at or below zero on the first three, so their Holm-adjusted p over the eight is below 0.004 (2,000 resamples), and all three survive at 0.025; LFQA's is 0.035.

On the submission's four core sets, like for like, MedHallu and RAGTruth both win (Holm-adjusted p < 0.002). Over 100 split seeds (E6), each moving both the folds and the bootstrap, MedHallu wins on all 100, clean and Holm-significant at 0.025 over the core on every one; RAGTruth wins on 90, is clean on 84, and survives Holm at 0.025 on 53; FactCheck-GPT wins on 96 (Holm at 0.025 on 70), LFQA on 90 (on 9).

Pooled over the seeds by median aggregation, the four wins stand with smaller margins (MedHallu +0.050, RAGTruth +0.051, FactCheck-GPT +0.072, LFQA +0.028), the reported seed lying on the favourable side for each; LFQA then falls short of Holm at 0.05 (adjusted p 0.064), and its clear improvement over Sonnet, which holds on 27 of the 100 seeds, becomes unresolved.

We now report Holm at the one-sided 0.025 the reviewer suggests, beside 0.05, so that the win rule and the correction share one standard: three wins survive at 0.025, LFQA only at 0.05.

Revision: §3.2 and App. K.


W1. The substitution claim overstates the evidence

We agree that "stand in for a frontier one wherever this audit admits it" could be read as a claim of parity, which it was not meant to be: it summarised the sentence before it, in which, where several capable members err on different items, the panel "approaches the frontier", at a fraction of its cost. The revision makes that reading explicit, with a margin-qualified claim limited to the observed datasets (A4). The abstract now states that, on six of the seven benchmarks the test admits, the panel falls no more than 5 points of balanced accuracy behind Sonnet (one-sided 97.5% bound), the same six sets on which it is non-inferior within 0.10 κ (§4), and closes:

On all (but one) of the benchmarks the test admits, a cheap local panel can therefore stand in for a frontier judge at a bounded price: a small loss in accuracy, against a large saving in cost and with nothing leaving the deployer's premises. (Abstract)

As suggested, we tested non-inferiority to Sonnet at κ margins of 0.05, 0.10, and 0.15 (one-sided α = 0.025; E4). At 0.10 the panel is non-inferior on six of the grounded core's eight sets (MedHallu, WiCE, XSum, FactCheck-GPT, LFQA, and Reveal), on eight of all fourteen, and on six of the seven the revised test admits; FActScore, the exception, is not "non-inferior" even at 0.15, its interval being wide at n = 330 (−0.067 [−0.182, +0.042]). MedHallu, cited originally as a small clear worsening, is an unresolved one on the corrected release (−0.014 [−0.046, +0.019]). At either correlation bar (the submission's and the revision's), the test admits no dataset on which the panel is clearly behind Sonnet, but it is conservative: it rejects seven on which the panel is within 0.10 κ of Sonnet by point estimate at the submission's bar, three at the revision's (Q1).

Revision: abstract, §4, and §5; non-inferiority in App. C.

Comments on the paper

7 comments
AuthorsOpenReview ↗

On what counts as a contribution, and on the observations and explanations offered by the paper

We updated the response as we noticed we did not published to "Everyone". No content was changed.

We thank the reviewer for the reply, but we respectfully disagree with his reading on the following three grounds.

First, the empirical understanding of LLM judges is a contribution the top venues recognise as such.
When an LLM judge can be trusted, what drives its errors, and at what cost it can be replaced, are questions studied in their own right at top-tier conferences like ICLR itself, ICML, or NeurIPS, by works that measure and explain judge behaviour without necessarily proposing a new architecture:

Our work answers the question these works leave open (Appendix L, and a new paragraph the revision will add to §1): whether a panel of cheap local judges can replace a paid frontier judge, at what price once the calibration labels are counted, and how a deployer can tell in advance.

Second, the paper offers observations and explanations the reviewer asks for.

  • An "unintuitive observation". One would expect the results to split between the grounded sets (where the judge is shown a reference) and the ungrounded ones (where it is not). We show it does not: the two ungrounded sets fall into opposite regimes, one of them being the most panel-favourable set in the suite (§3.3, p. 7, l. 348; Appendix B, p. 19, ll. 1022–1024).
  • A logical consequence. Two binary verdicts scored against a common label agree exactly when both are right or both are wrong, so the correlation of the members' errors is readable without labels, whereas their competence is not; this is why label-free aggregators fail on correlated panels, whose latent consensus is their shared error (§3.3; Appendix L).
  • A prediction checked against the frontier. The frontier judge stays materially ahead only on the two sets dominated by one judge, which is the regime in which the admissibility test rejects the panel in advance (§3.3, p. 7, l. 366; §4).

Third, a note on the ICLR reviewer guidelines.
They ask reviewers to bear in mind that "not all work needs to compete on an established leaderboard" and that significance "does not necessarily require state-of-the-art results", as submissions bring value "when they convincingly demonstrate new, relevant, impactful knowledge (incl., empirical, theoretical, for practitioners, etc)". Furthermore, the FAQ section states that "a lack of state-of-the-art results does not by itself constitute grounds for rejection". The other three reviewers read the work in these terms: Reviewer p13k writes that "even negative result or weak result like this is also important when it is a top of mind question for several of us", and that reading competence and error correlation off the calibration set "gives deployers something actionable"; Reviewer tuNS that the paper "asks a practical question and puts a price on the answer"; Reviewer uHii that it is "relevant to high-volume evaluation settings where privacy or API access is a concern". On a side note, we cannot but notice that other reviewers' concerns are specific (the comparison with a single larger open judge, the out-of-sample validation of the admissibility test, the label budget) and grounded on a factual basis, not opinions. Indeed, we are answering them with new experiments, currently running.

AuthorsOpenReview ↗

No AI model judges our results, and detection quality is what is measured in the paper

We updated the response as we noticed we did not published to "Everyone". No content was changed.

This comment rests on a misreading of our previous reply, which hence requires a further clarification.

We did not state that we had only API access; we stated the opposite.
The ten panel judges are open-weight models, downloaded and run on our own GPUs (Appendix K, Table 21; Reproducibility statement), with the code to rerun them in the supplementary material. The only model accessed through an API is Claude Sonnet, which is not part of the method: it is the comparator the panel is meant to replace.

No AI model, accessed through an API or otherwise, judges our results.
Every verdict, from the panel and from the frontier judge alike, is scored by deterministic code against the label each benchmark's own authors published, as detailed in our previous reply; no language model enters that step.

The quality of the detection is the quantity we measure in the paper.
Each judge, the panel, and the frontier judge are scored against those labels by Cohen's κ and balanced accuracy, each difference with a 95% paired cluster-bootstrap interval (§2 and §3). The other three reviewers single out this protocol as a strength: Reviewer tuNS finds it "rigorous", Reviewer p13k reads its outcome as "a trustworthy mixed result", and Reviewer uHii notes that the "paired confidence intervals help distinguish meaningful gains from uncertain differences".

The task is consistently and fully defined.
On the six grounded sets the judge is shown a reference and a candidate, and asked whether the candidate is supported by the reference (§2, p. 3, ll. 140–143; p. 4, ll. 206–208). On the two ungrounded sets the evidence block is dropped, so the judge must decide from its own knowledge whether the candidate answer is incorrect or fabricated, which is the closest to the hallucination detection the reviewer refers to. The prompt is reproduced verbatim in Appendix K (p. 40, ll. 2139–2157).

Our contribution is a systematic study of whether, and when, a panel of cheap judges improves on the current practice, i.e. a single frontier model used as a judge.
Said otherwise, our contribution is not a mere compute task of running experiments and judging them with AI models.

As per the title of this paragraph, worth repeating, the contribution is the systematic study of whether, and when, a panel of cheap judges improves on the current practice, a single frontier model used as a judge. We share some of the reservations about that practice (Appendix L), yet it is the one the community relies on, notably to annotate large volumes of data; precisely because it is widespread and its reliability open to question, measuring it against human labels, and establishing when a cheaper and local alternative holds, is the advance we claim.

In light of the above points, we respectfully ask the reviewer to ground the assessment in the content of the submission, as the ICLR reviewer guidelines require ("read the paper carefully, check arguments in sufficient detail to convince yourself of the paper's main claims"), and to point to the specific claims they find unsupported.

AuthorsOpenReview ↗

Corrections applied after the submission, the re-run release (v0.1.1), and what changed (1/4)

Edited: 94.2 → 94.8 (stacked agreement, recounted on the released files).

We thank the reviewers again for their careful reading.

Before replying to each of them, we report something that concerns all four reviews.
While preparing the new experiments, we audited every reported run against the employed software stack. We found that the serving libraries were updated, with consequential changes on how some panel members read their prompts. Under release 5.x5.x of the transformers library, which served the four question-answering sets, one member's chat template was tokenised incorrectly; under the older engine and library used for the four claim-verification sets, another member produced degenerate text on long inputs. Serving every member correctly therefore required a single stack on which all ten read their prompts as intended, and with it the re-generation of every verdict. The re-generation also corrects an error of our own, which doubled one token in five members' prompts. None of the defects was in the aggregation or the statistics.
We re-generated everything. In detail: we regenerated every verdict of the panel and of the open reference judges on one pinned stack, re-ran the frontier judge, and re-derived every number of the paper from the corrected release. The individual replies use the corrected numbers throughout.


Defects found

  1. A tokeniser change in the library. Under transformers 5.x5.x, Yi-1.5-9B loads a different tokeniser class, which splits its chat role markers into plain text; Yi thus saw a malformed conversation and often wrote several verdict lines.
  2. Degenerate output on an older engine. The submission's four LLM-AggreFact sets had been served by older releases (vLLM 0.7.3, transformers 4.49), under which Phi-4-mini produced degenerate text on many of the long WiCE and ExpertQA documents; it does not under the current releases.
  3. A doubled beginning-of-sequence token. Our code, set up on the first judge we integrated, passed each templated prompt to vLLM as text, to which vLLM adds its own beginning-of-sequence (BOS) token; the five members whose chat templates already write one (Llama-3.1-8B, Mistral-7B, Gemma-2-9B, Command-R7B, GLM-4-9B) thus received it twice in every run.
  4. Parser edge cases. Non-standard verdict labels (e.g. "UNSUPPORTED"), replies continuing into a new user turn, and replies echoing the prompt without a verdict were read inconsistently.
  5. Unrecorded model revisions. The two halves of the suite had been served by different library versions, and the model revisions were not recorded.
  6. An input-length asymmetry. On 24 of the suite's 7,670 items (0.3%; WiCE 8, ExpertQA 16), the open judges read documents cut to their 8,192-token window while the frontier judge read them whole, unlike what the paper states.

Applied solutions

  • Prompts now reach the engine as token ids with exactly one BOS; model revisions are pinned; one stack serves every open judge (vLLM 0.23.0, which accepts transformers from 4.56.0; transformers 4.57.6, the last 4.x4.x release which tokenises Yi correctly while also serving the other nine members as 5.x5.x does); the parser rules are fixed, tested, and audited again.
  • We re-generated all ten members on all eight sets (76,700 verdicts), and the open reference judges (Qwen2.5-32B and -72B, Prometheus-2, JudgeLM, Auto-J) on the same stack; MiniCheck runs through its own package and is unchanged.
  • We re-ran the frontier judge (Claude Sonnet 4.6) on the whole suite, now reading on the 24 long items the prompts cut to the 8,192-token window, as the Qwen reference judges' tokeniser cuts them; under their own tokenisers, the panel members cut 8–11 of the WiCE items and 14–22 of the ExpertQA ones, which Sonnet still reads whole when its prompt is not cut. Alongside, we measured how stable Sonnet is over time: on 400 suite items (50 per set) scored twice on the same day, its verdicts agree on 98.3% [96.4, 99.1] (Wilson interval); against the June run they agree on 96.3% and 96.0%, and its accuracy is unchanged (75.8% in June, 76.0% and 76.3% now; McNemar p = 1.00).
  • We measured how reproducible the re-generation is. Two separate runs of the same configuration, on the same machine and stack, agree on 97.3–99.3% of member verdicts and 94.8–99.5% of stacked verdicts on the four question-answering sets, and MedHallu and RAGTruth are clean wins in both; on two different machines and GPU models (test on CNN), member verdicts agree on 99.7% and the stacked verdicts are identical. The differences that remain are the numerical non-determinism of GPU inference: the raw replies of the two runs start identically and diverge mid-text.
  • The corrected verdicts are to be released as v0.1.1, with each generative judge's model revision and serving stack recorded; the supplementary snapshot will be updated with it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.