acceptodds
Under review as a conference paper at ICLR 2027

Can We Trust This Evaluation? Instance-Level Reliability of LLM-as-a-Judge

Abstract

LLM-as-a-judge evaluation commonly aggregates multiple judges' decisions into a majority vote, yet this panel decision process does not guarantee that the evaluation of an individual item is reliable. We propose Structured Cross-fitted Uncertainty Risk (SCUR), a label-free approach for estimating instance-level evaluation reliability based on the judge panel's behavior. A latent measurement model serves as the backbone for measurement system analysis, from which SCUR derives three complementary signals capturing panel uncertainty, measurement cross-entropy, and perturbation sensitivity. These three sources of evidence are combined into a risk score. We evaluate the SCUR approach on five real-world benchmarks using majority-vote error ranking, selective trust, and risk stratification. Furthermore, we show our approach to be more effective at detecting high-consensus risk items, which are the blind spots for many other LLM-as-a-judge evaluation methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-judge, evaluation reliability, uncertainty quantification, risk estimation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wBX8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SCUR, a label-free score for ranking individual LLM-judge-panel decisions by risk of error. A 2PL-IRT-style latent measurement model decomposes judges’ verdict-token log-odds into item preference, judge discrimination/bias, and shared/judge-specific perturbation effects. Using leave-one-judge-out (“cross-fitted”) item margins, three signals are computed: entropy of the mean cross-fitted probability (U), binary cross-entropy between each judge’s native probability and its cross-fitted prediction (C), and a model-implied perturbation sensitivity (S). Each is normalized by an ECDF on an unlabeled reference split and fused via a complement-geometric-mean. On five benchmarks, SCUR beats four uncertainty baselines (PE, VE, DS entropy, MSP) on AUROC / 1−AURC / Err@50 for majority-vote error, with a high-consensus (HC) subset analysis and a transfer analysis to DS/MACE/GLAD/CARE aggregation.

Critical strengths
  • Clear and useful problem framing. The paper separates "is the aggregate right on average" from "can I trust this particular verdict." This distinction is practically important for human-review routing, and the selective-trust use case is concrete and well-motivated.
  • The high-consensus analysis is the right stress test. Section 4.2 and Appendix D directly target the regime where disagreement-based scores fail. The exact-matched vote-pattern AUROC is a careful design choice: it holds both the vote split and the identity of the supporting judges fixed, so it isolates what the score adds beyond vote counting.
  • Sound evaluation protocol. Item-level 50/30/20 splits, ECDF normalization frozen before evaluation, no labels used in fitting or normalization, tie handling in ranking metrics, five independent seeds with reported standard deviations, and an anonymized code release.
Critical weaknesses
  • No ablation of the three channels. The paper never reports U, C, or S alone, any pairwise combination, SCUR without ECDF normalization, an alternative fusion rule, or sensitivity to the fixed penalties (λq=λa=λb=1\lambda_q=\lambda_a=\lambda_b=1, λδ=5\lambda_\delta=5, λκ=10\lambda_\kappa=10). The channels are not obviously independent: U is PE on cross-fitted probabilities; C decomposes into per-judge entropy plus a mismatch term (Sec. 3.2 acknowledges this), overlapping with MSP; and since δc,κjc\delta_c, \kappa_{jc} are per-judge/condition constants, S varies across items only through ηij0\eta_{ij0} and may be close to a rescaling of U. This is the most important missing experiment.

  • Modest gains on some benchmarks, high variance on the small ones, no paired tests. On HaluEval ({U,C} only) AUROC is .732±.006 vs .728±.006 for PE and Err@50 .079 vs .081, so the LMM adds little over raw PE there. MMLU is clearer (Err@50 .062 vs .079). On LLMBar (83 test items) and LLMEval² (40 test items, ~8 MV errors per split) SDs of .03–.12 exceed the gaps between methods. Since all methods share the same item splits, marginal SDs are not the right comparison; paired tests over seeds or item-level bootstraps are needed. "Consistently outperforms" is overclaimed.

  • Missing baselines. (a) Bidirectional Preference Entropy (BPE) from SCOPE: Selective Conformal Optimized Pairwise LLM Judging (Badshah, Emami & Sajjad, ICML 2026; arXiv:2602.13110) is a published, label-free, order-aware entropy built for exactly the swap condition available on Arena/LLMBar/LLMEval²: it queries the judge under both answer orders, averages the order-corrected preference probabilities, and takes the entropy. The paper adapts PE from the same group's Judge, Retrieve, or Abstain (Badshah et al., 2026a; arXiv:2608.17994) but omits BPE. An observed order-flip rate, as measured in Large Language Models are not Fair Evaluators (Wang et al., ACL 2024), is the simpler version. Both use the item's actual perturbed responses, unlike SCUR's model-implied S. (b) Sec. 4.3 already fits MACE, GLAD, and CARE (CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation, Zhao et al., ICML 2026; arXiv:2603.00039), yet only DS posterior entropy is used as a risk baseline. (c) A single-score LMM residual (mean ∣mijc−fitted∣|m_{ijc} - \text{fitted}|) would show whether the three-channel construction beats the fitted model itself. (d) Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory (Choi et al., ICML 2026; arXiv:2602.00521) uses an IRT Graded Response Model to assess judge-level reliability (intrinsic consistency under prompt variation and human alignment), not instance-level risk, so it need not be a competitor, but the paper should state how its LMM differs from that framework.

  • Under-specified benchmarks; "correlation collapse" addressed only at the symptom level. For Arena/HaluEval/MMLU the paper uses "existing judge outputs" without naming the judges, the logprob source, or how 4-way MMLU and HaluEval labels map to a binary "A wins" event. Arena ties and human-label noise are not discussed. The LLMBar/LLMEval² panel is five cross-family open models (7B–70B, one a tool-use fine-tune), the easy regime for shared bias. The HC analysis shows errors under agreement are detectable, but the LMM has no shared-confounder term, no experiment manipulates judge correlation (e.g., a same-family panel), and the ethics statement concedes low SCUR is unreliable under shared judge bias. CARE (Zhao et al., 2026, ICML) models exactly this and is used only as an aggregation rule.

Questions for authors
  1. Definition of S. Does S_i use any observed perturbed responses p_ijc (c != 0) beyond their contribution to q_i^(-j)? If not, what per-item information does S carry that is not already in U, given both are functions of the same cross-fitted margins? Please report the correlation between z^U and z^S on the three benchmarks where S is available.

  2. Channel ablations and an order-consistency baseline. Please report U alone, C alone, S alone, each pair, and SCUR without ECDF normalization on all five benchmarks. On Arena, LLMBar, and LLMEval2, please also add an observed flip-rate or mean |p_ij0 - p_ij,swap| baseline and compare it to SCUR and to S alone.

  3. Judge panels and binarization. Which models constitute the 17-judge Arena and MMLU panels and the 6-judge HaluEval panel, and where were the verdict-token logprobs obtained? How were 4-way MMLU and binary HaluEval labels mapped to the "canonical A wins" event, and how were Arena ties handled?

4 comments

Thanks for the reviewer's feedback. Please check our response below.

Q1: Definition of S

When perturbations are available, UU and SS may share some information because both depend on the fitted latent measurement model. However, this overlap does not necessarily imply that the perturbation information is redundant. As suggested, we report the Pearson correlations among UU, CC, and SS in Table 10. This comparison covers all items and two subsets: high consensus by vote agreement (HC-V) and high consensus by panel-mean probability (HC-P). A summary is shown here:

BenchmarkAll Items ρ(U,C)All Items ρ(U,S)All Items ρ(C,S)HC-V ρ(U,C)HC-V ρ(U,S)HC-V ρ(C,S)HC-P ρ(U,C)HC-P ρ(U,S)HC-P ρ(C,S)
Chatbot Arena.6445.2924.2961.5885.4967.5786.4532.2289.5335
HaluEval.6602——.4211——.2161——
MMLU.7941——.7047——.5848——
LLMBar.6861.9351.5681.6761.9561.5936.5930.9836.5682
LLMEval².7586.8029.6321.7164.7258.5911.6138.6380.5619

The correlation between UU and SS is relatively low on Chatbot Arena (.2924) and higher on the two smaller benchmarks, LLMBar (.9351) and LLMEval$^2$ (.8029). More importantly, the ablation analysis in Table 9. It suggests that incorporating perturbation observations generally improves SCUR's performance. The results are summarized below (each entry reports SCUR averaged across benchmarks over 5 seeds).

Obs. (Avg. 5 benchmarks)All Items AUROC ↑All Items 1−AURC ↑All Items Err@50 ↓HC-V AUROC ↑HC-V 1−AURC ↑HC-V Err@50 ↓HC-P AUROC ↑HC-P 1−AURC ↑HC-P Err@50 ↓
SCUR w/o perturbations.6815.8302.1567.6661.8614.1381.6338.8745.1416
SCUR w/o S.6996.8379.1455.6882.8696.1265.6639.8837.1201
SCUR.7018.8409.1468.6956.8737.1231.6740.8885.1104

A similar pattern is observed in Table 3, where adding SS improves AUROC on all benchmarks except LLMEval$^2$. (Please see the risk signal ablation analysis table under our response to Q2 for eviewer 7fmh.) These results demonstrate that perturbation information provides additional value for identifying judge-panel errors.

Q2: Channel ablations and an order-consistency baseline

We provide the suggested ablation analyses in Section 5 and Appendix E. Here, we briefly summarize the main findings. The ablation regarding risk signals in Table 3 shows that SCUR improves AUROC over UU on four of the five benchmarks, with LLMEval$^2$ as the exception. The more detailed results can be seen in the table under our response to Q2 for eviewer 7fmh.

For the missing baseline, we added three additional baselines mentioned by the reviewer: bidirectional preference entropy (BPE), the observed order-flip rate (Filp), and absolute probability change (Δp\Delta p). Their performance is evaluated on three answer-order swap benchmarks: Chatbot Arena / LLMBar / LLMEval$^2$. We provide the averaged results over 5 seeds below (these baselines use the same available observations as SCUR):

Answer-order swap (Avg. 3 benchmarks)All Items AUROC ↑All Items 1−AURC ↑All Items Err@50 ↓HC-V AUROC ↑HC-V 1−AURC ↑HC-V Err@50 ↓HC-P AUROC ↑HC-P 1−AURC ↑HC-P Err@50 ↓
PE.6670.7943.2004.6506.8200.1797.6211.8392.1785
VE.6525.7841.2057.6362.8086.1785.6108.8273.1775
DS entropy.6504.7719.1978.6225.7889.1904.5632.7952.2036
MSP.6348.7697.2214.6338.8077.1925.5824.8185.1949
BPE.6570.7907.2043.6543.8220.1871.6310.8409.1699
Flip.6273.7709.2153.6255.8012.1829.6118.8271.1817
Δp.6410.7871.2127.6337.8174.1909.6242.8415.1590
SCUR.6609.7962.1977.6568.8241.1769.6352.8442.1577

We also update these three new baselines in Tables 1, 4, and 7 and Figures 2, 3, 4, 6, 7, 8, and 9. In these comparisons, SCUR achieves the best performance in most evaluation settings.

In addition, We've compared risk scores derived from all aggregation methods evaluated in the paper, DS, MACE, GLAD, and CARE, in Table 7. The results are shown here. SCUR still leads the average performance on various evaluation settings.

Aggregation posteriorsAll Items AUROC ↑All Items 1−AURC ↑All Items Err@50 ↓HC-V AUROC ↑HC-V 1−AURC ↑HC-V Err@50 ↓HC-P AUROC ↑HC-P 1−AURC ↑HC-P Err@50 ↓
DS entropy.6527.8061.1805.5804.8290.1690.5059.8365.1677
MACE entropy.6510.8049.1825.5778.8271.1690.5055.8352.1677
GLAD entropy.6318.7920.1963.5643.8118.1760.4939.8216.1801
CARE margin.5681.7546.2295.5301.8040.1849.4973.8298.1682
SCUR.7018.8409.1468.6956.8737.1231.6740.8885.1104

Q3: Judge panels and binarization

For the Chatbot Arena, HaluEval, and MMLU benchmarks, we use publicly available judge outputs from Hugging Face, and for LLMBar and LLMEval$^2$, we conduct the judgments ourselves. We have added the dataset links and detailed judge composition to Appendix C.

The verdict-token logprobs are the log-probabilities assigned to A/B in Chatbot Arena and Yes/No in MMLU and HaluEval during next-token prediction. We exclude human label ties for Chatbot Arena.

Item-level bootstraps are needed. "Consistently outperforms" is overclaimed.

We have removed the term "consistently" from the sentence "...consistently outperforms...". But our empirical results show that SCUR outperforms alternative baselines in most settings. We also added 1000 bootstrap resamples to show intervals in Figures 7, 8, and 9.

Thanks to the reviewer for their time and effort in reviewing our work. All revisions are highlighted in blue in the updated manuscript.

Thank you for the response. However, I would still like to maintain my score.

Thanks for your reply. However, just stating that you would like to maintain the score without providing any reasoning is difficult to understand. If you believe our responses did not resolve your concerns, please specify what remains unresolved and why. We are happy to provide further clarification or discussion.

Reviewer_7fmhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SCUR, a label-free score for identifying unreliable LLM-judge decisions. It combines uncertainty, disagreement, and sensitivity to perturbations to prioritize cases for review. Across five tasks, average AUROC improves from 0.673 to 0.702 over PE, while the error rate among the lowest-risk half of the data drops from 17.4% to 14.7%.

Critical strengths

The paper addresses a practical problem: deciding which judge outputs need closer inspection. Producing these priorities without correctness labels is a useful feature. The experiments cover several tasks and aggregation methods, and evaluate both error ranking and selective acceptance.

Critical weaknesses

The methods do not always receive the same information. Appendix B states that the baselines use only judgments from the original condition, while SCUR also uses perturbed judgments on some tasks. On LLMBar and LLMEval2, this means 30 judgments per item for SCUR versus 5 for the baselines. It is therefore unclear how much of the gain comes from the proposed modeling rather than the additional information. Comparisons with baselines using the same observations are needed.

The main design choices lack ablation studies. I could not find direct comparisons of individual U/C/S signals and their combinations, removing cross-fitting, or using simpler fusion rules such as averaging or taking the maximum. Without these checks, it is hard to tell which components matter and whether the added complexity is necessary.

Questions for authors
  1. Does SCUR still outperform simple probability- and perturbation-based scores when all methods use the same observations?
  2. How much does each component contribute, and can a simpler configuration achieve similar results?
1 comment

We appreciate reviewer 7fmh's thoughtful comments and suggestions. We provide our responses below.

  1. Performance for simple probability- and perturbation-based scores when all methods use the same observations?

Thank you for the suggestion. We have added an ablation analysis on Page 9, and additional details are provided in Appendix E. Table 9 presents a comprehensive comparison in which SCUR and the baselines are evaluated both with and without perturbation observations. The comparison covers all items and two subsets: high consensus by vote agreement (HC-V) and high consensus by panel-mean probability (HC-P). The results are summarized below (each entry reports SCUR/strongest baselines averaged over 5 seeds).

Obs. (5 benchmarks on averge)MetricAll ItemsHC-VHC-P
w/o perturbationAUROC ↑.6815 / .6730.6661 / .6313.6338 / .5897
1−AURC ↑.8302 / .8184.8614 / .8453.8745 / .8590
Err@50 ↓.1567 / .1741.1381 / .1538.1416 / .1298
w/ perturbationAUROC ↑.7018 / .7032.6956 / .6701.6740 / .6272
1−AURC ↑.8409 / .8362.8737 / .8657.8885 / .8793
Err@50 ↓.1468 / .1524.1231 / .1304.1104 / .1301

It shows that SCUR outperforms the baselines in most evaluation settings, with two exceptions: Err@50 on the HC-P subset without perturbations and AUROC on all items with perturbations.

  1. Contribution for each component, and can a simpler configuration achieve similar results?

We present the suggested ablation analyses in Section 5 and Appendix E. Here, we summarize the main findings; more detailed results are provided in the revised manuscript (All results are averaged over 5 seeds). The ablation regarding risk signals in Table 3 shows that SCUR improves AUROC over UU on four of the five benchmarks, with LLMEval$^2$ as the exception.

Risk Signals (AUROC)Chatbot ArenaHaluEvalMMLULLMBarLLMEval²Avg.HC-VHC-P
U.6835.7143.7824.6501.7288.7118.6970.6661
C.6657.7210.7720.6121.5715.6685.6519.6180
S.6174——.6406.6531———
U+C.6901.7320.7942.6421.6395.6996.6882.6639
U+S.6892——.6470.6947———
C+S.6732——.6493.5776———
SCUR.6961.7320.7942.6544.6321.7018.6956.6740

We also examine the contribution of the latent item factor qiq_i in Table 8, where including qiq_i reduces the MSE of the predictions by 65.86% on average.

Latent factorMSE (w/ qᵢq_ᵢ) ↓MSE (w/o qᵢq_ᵢ) ↓Reduction (%)
Average32.254591.652265.86

For the combination rules for risk signals, Table 9 shows that no single combination rule performs best across all settings. This indicates a trade-off among different ways of combining the signals, while SCUR performs best on the HC-P subset. Table 9 also shows that incorporating perturbations generally improves SCUR's performance.

Aggregation rule (Avg. 5 benchmarks)All Items AUROC ↑All Items 1−AURC ↑All Items Err@50 ↓HC-V AUROC ↑HC-V 1−AURC ↑HC-V Err@50 ↓HC-P AUROC ↑HC-P 1−AURC ↑HC-P Err@50 ↓
Mean.7042.8409.1492.6946.8728.1216.6707.8873.1122
Max.6996.8400.1433.6970.8738.1197.6778.8882.1111
Product.7030.8402.1518.6930.8719.1294.6677.8862.1134
SCUR.7018.8409.1468.6956.8737.1231.6740.8885.1104
Obs. (Avg. 5 benchmarks)All Items AUROC ↑All Items 1−AURC ↑All Items Err@50 ↓HC-V AUROC ↑HC-V 1−AURC ↑HC-V Err@50 ↓HC-P AUROC ↑HC-P 1−AURC ↑HC-P Err@50 ↓
SCUR w/o perturbations.6815.8302.1567.6661.8614.1381.6338.8745.1416
SCUR w/o S.6996.8379.1455.6882.8696.1265.6639.8837.1201
SCUR.7018.8409.1468.6956.8737.1231.6740.8885.1104

Finally, Table 11 shows that cross-fitting is particularly effective when at least one high-confidence judge disagrees among the remaining judges, improving AUROC from .5702 to .5906.

Cross-fitting (AUROC, Avg. 5 benchmarks)All itemsHC-VHC-PHigh-confidence disagreement
w/o CF.7011.6928.6759.5702
w/ CF (SCUR).7018.6956.6740.5906

Finally, we appreciate the reviewer's time and effort in reviewing our work. All revisions are highlighted in blue in the updated manuscript.

Reviewer_qTeAOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

LLM-based evaluation increasingly relies on panels of judges. Prior work (e.g., DS, MACE, GLAD, CARE) studies how to aggregate judge decisions beyond majority vote, but not how confident a collective decision is. This paper addresses that gap with SCUR, a label-free risk score built on a latent measurement model. The model splits each judge's verdict log-odds into item preference, judge discrimination and bias, and perturbation effects (e.g., swapped answer order or a changed rubric), either shared across judges or specific to one. Using leave-one-judge-out predictions, SCUR computes three signals: preference uncertainty, mismatch between each judge's confidence and what the other judges imply, and perturbation sensitivity. It converts these to percentiles on an unlabeled reference split and fuses them into a single score. Across five benchmarks, SCUR outperforms baselines at ranking majority-vote errors, including on high-consensus items. It also works with other aggregation methods and shows why each item was flagged.

Critical strengths

I liked the problem statement and its motivation. Automated evaluation increasingly relies on groups of models to make collective decisions. Prior work has studied how to aggregate multiple judgments into a final verdict, but it usually doesn't report how confident that verdict is. This paper fills that gap, and I think the problem is interesting and important to the field. The experiments cover diverse benchmarks and task types (pairwise preference, hallucination detection, multiple-choice correctness, and instruction following), which is diverse to support the main claims. The authors also show that SCUR works with existing aggregation methods (DS, MACE, GLAD, and CARE), which makes it a practical add-on for flagging unreliable verdicts for human review rather than another competing aggregator.

Critical weaknesses
  1. The experiments use the panel that comes with each benchmark, so it's less clear how SCUR depends on panel composition. It would be helpful to see how performance changes with fewer judges, or when all judges come from the same model family. The second case seems especially relevant, since that is where correlated errors would likely be strongest.
  2. More detail on the gold labels would be helpful. SCUR itself is label-free, but checking whether it works still requires ground truth. In practice, how would one validate SCUR, or choose a review threshold, on a new task where gold labels aren't readily available?
  3. The current framework focuses on pairwise A/B judgments. The LMM models the log-odds that A wins, and U, C and S are defined for binary outcomes. Many LLM-as-a-judge setups instead use Likert scores, rank more than two candidates, or grade on several rubric dimensions. A short discussion of how SCUR might extend to these settings would broaden its appeal.
Questions for authors

Listed in the weaknesses.

1 comment

We thank reviewer qTeA for the appreciation of our work and the thoughtful feedback. We provide our responses below.

  1. The experiments use the panel that comes with each benchmark, so it's less clear how SCUR depends on panel composition.

Thanks for pointing this out. For the Chatbot Arena, HaluEval, and MMLU benchmarks, we use publicly available judge outputs from Hugging Face, and for LLMBar and LLMEval$^2$, we conduct the judgments ourselves. We have added the dataset links and detailed judge composition to Appendix C.

As suggested, we added experiments using judge panels from the same model family. We use three Llama-3 judges on Chatbot Arena and three Qwen3 judges on LLMBar. The results are reported in Table 4. SCUR outperforms the baselines across both panels. Additionally, we extend this experiment on Chatbot Arena and MMLU to broader Llama, Llama-3, and Mistral families, as well as cross-family panels composed of one judge from each model family (Figure 7). SCUR still achieves the highest AUROC among the baselines in this setting. The results are summarized below (each entry reports SCUR / strongest baseline averaged over 5 seeds).

BenchmarkPanel (3 judges)All-items AUROC
Chatbot ArenaLlama-3.6826 / .6651
Llama family.6606 / .6583
Mistral family.6419 / .6403
Cross-family.6687 / .6604
MMLULlama-3.8082 / .7686
Llama family.7659 / .7573
Mistral family.6898 / .6700
Cross-family.7590 / .7568
LLMBarQwen3.6818 / .6705

We also tested SCUR by varying the number of judges from 3 to 5, 9, and 17 on Chatbot Arena and MMLU (Figure 8). We present the results here: the table below reports AUROC for SCUR versus the strongest baseline across panel sizes. HC-V denotes high consensus based on vote agreement (≥.8\ge .8), while HC-P denotes high consensus based on panel-mean probability (≥.8\ge .8). The baselines include predictive entropy (PE), bidirectional preference entropy (BPE), and vote entropy (VE). Each entry reports SCUR / strongest baseline averaged over 5 seeds.

BenchmarkSubset3 judges5 judges9 judges17 judges
Chatbot ArenaAll items.6465 / .6452.6786 / .6597.6893 / .6680.6961 / .6742
HC-V.6434 / .6644.6893 / .6691.6987 / .6751.7129 / .6799
HC-P.6366 / .6524.6633 / .6516.6783 / .6507.6759 / .6453
MMLUAll items.7241 / .7291.7549 / .7447.7755 / .7690.7942 / .7872
HC-V.7333 / .6993.7709 / .7282.7954 / .7321.8228 / .7511
HC-P.7044 / .6427.7375 / .6459.7636 / .6532.7972 / .6644

SCUR generally improves as panel size increases and leads the strongest baseline in 21 of 24 settings, with a more pronounced advantage on high-consensus subsets. The distinction becomes more evident with five or more judges.

  1. Gold labels and threshold in practice.

We use the published ground-truth provided by each benchmark. For our evaluation metrics, such as AUROC, the evaluation label is defined as whether the majority vote of the judge panel disagrees with the ground truth.

In Appendix F.2, we explore three approaches to select review thresholds: a budget-based approach that prioritizes the highest risky items within a given review capacity, a reference-quantile approach that detects unusually high-risk items relative to the reference set (90th percentile of the SCUR distribution on the reference set), and a density-based approach that identifies a separation at the minimum between the two rightmost density peaks of the reference distribution. We report their performance in terms of errors captured per 10% of items reviewed in Table 12 (the baseline for random review is expected to be 10%).

Review threshold (average on 5 benchmarks)Items reviewed (%)Errors captured (%)Errors captured per 10% reviewed (%)
Budget top 10%10.1716.1315.84
Reference Q9010.5716.9316.33
Density-based35.0557.4816.68
  1. A discussion of how SCUR extends beyond binary cases.

Thank you for the suggestion. We have added this discussion to Section 5 and Appendix F.1. A brief description is provided here: the latent measurement model can be adapted to different judgment types by changing the link function and likelihood. For example, multiclass judgments can use a softmax link function, and ordinal ratings can use a cumulative logit link function. The risk signals need to be adjusted correspondingly, but the normalization and aggregation rule stays the same.

Finally, we appreciate the reviewer's time and effort in reviewing our work. All revisions are highlighted in blue in the updated manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.