- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SCUR, a label-free score for ranking individual LLM-judge-panel decisions by risk of error. A 2PL-IRT-style latent measurement model decomposes judges’ verdict-token log-odds into item preference, judge discrimination/bias, and shared/judge-specific perturbation effects. Using leave-one-judge-out (“cross-fitted”) item margins, three signals are computed: entropy of the mean cross-fitted probability (U), binary cross-entropy between each judge’s native probability and its cross-fitted prediction (C), and a model-implied perturbation sensitivity (S). Each is normalized by an ECDF on an unlabeled reference split and fused via a complement-geometric-mean. On five benchmarks, SCUR beats four uncertainty baselines (PE, VE, DS entropy, MSP) on AUROC / 1−AURC / Err@50 for majority-vote error, with a high-consensus (HC) subset analysis and a transfer analysis to DS/MACE/GLAD/CARE aggregation.
Critical strengths
- Clear and useful problem framing. The paper separates "is the aggregate right on average" from "can I trust this particular verdict." This distinction is practically important for human-review routing, and the selective-trust use case is concrete and well-motivated.
- The high-consensus analysis is the right stress test. Section 4.2 and Appendix D directly target the regime where disagreement-based scores fail. The exact-matched vote-pattern AUROC is a careful design choice: it holds both the vote split and the identity of the supporting judges fixed, so it isolates what the score adds beyond vote counting.
- Sound evaluation protocol. Item-level 50/30/20 splits, ECDF normalization frozen before evaluation, no labels used in fitting or normalization, tie handling in ranking metrics, five independent seeds with reported standard deviations, and an anonymized code release.
Critical weaknesses
-
No ablation of the three channels. The paper never reports U, C, or S alone, any pairwise combination, SCUR without ECDF normalization, an alternative fusion rule, or sensitivity to the fixed penalties (, , ). The channels are not obviously independent: U is PE on cross-fitted probabilities; C decomposes into per-judge entropy plus a mismatch term (Sec. 3.2 acknowledges this), overlapping with MSP; and since are per-judge/condition constants, S varies across items only through and may be close to a rescaling of U. This is the most important missing experiment.
-
Modest gains on some benchmarks, high variance on the small ones, no paired tests. On HaluEval ({U,C} only) AUROC is .732±.006 vs .728±.006 for PE and Err@50 .079 vs .081, so the LMM adds little over raw PE there. MMLU is clearer (Err@50 .062 vs .079). On LLMBar (83 test items) and LLMEval² (40 test items, ~8 MV errors per split) SDs of .03–.12 exceed the gaps between methods. Since all methods share the same item splits, marginal SDs are not the right comparison; paired tests over seeds or item-level bootstraps are needed. "Consistently outperforms" is overclaimed.
-
Missing baselines. (a) Bidirectional Preference Entropy (BPE) from SCOPE: Selective Conformal Optimized Pairwise LLM Judging (Badshah, Emami & Sajjad, ICML 2026; arXiv:2602.13110) is a published, label-free, order-aware entropy built for exactly the swap condition available on Arena/LLMBar/LLMEval²: it queries the judge under both answer orders, averages the order-corrected preference probabilities, and takes the entropy. The paper adapts PE from the same group's Judge, Retrieve, or Abstain (Badshah et al., 2026a; arXiv:2608.17994) but omits BPE. An observed order-flip rate, as measured in Large Language Models are not Fair Evaluators (Wang et al., ACL 2024), is the simpler version. Both use the item's actual perturbed responses, unlike SCUR's model-implied S. (b) Sec. 4.3 already fits MACE, GLAD, and CARE (CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation, Zhao et al., ICML 2026; arXiv:2603.00039), yet only DS posterior entropy is used as a risk baseline. (c) A single-score LMM residual (mean ) would show whether the three-channel construction beats the fitted model itself. (d) Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory (Choi et al., ICML 2026; arXiv:2602.00521) uses an IRT Graded Response Model to assess judge-level reliability (intrinsic consistency under prompt variation and human alignment), not instance-level risk, so it need not be a competitor, but the paper should state how its LMM differs from that framework.
-
Under-specified benchmarks; "correlation collapse" addressed only at the symptom level. For Arena/HaluEval/MMLU the paper uses "existing judge outputs" without naming the judges, the logprob source, or how 4-way MMLU and HaluEval labels map to a binary "A wins" event. Arena ties and human-label noise are not discussed. The LLMBar/LLMEval² panel is five cross-family open models (7B–70B, one a tool-use fine-tune), the easy regime for shared bias. The HC analysis shows errors under agreement are detectable, but the LMM has no shared-confounder term, no experiment manipulates judge correlation (e.g., a same-family panel), and the ethics statement concedes low SCUR is unreliable under shared judge bias. CARE (Zhao et al., 2026, ICML) models exactly this and is used only as an aggregation rule.
Questions for authors
-
Definition of S. Does S_i use any observed perturbed responses p_ijc (c != 0) beyond their contribution to q_i^(-j)? If not, what per-item information does S carry that is not already in U, given both are functions of the same cross-fitted margins? Please report the correlation between z^U and z^S on the three benchmarks where S is available.
-
Channel ablations and an order-consistency baseline. Please report U alone, C alone, S alone, each pair, and SCUR without ECDF normalization on all five benchmarks. On Arena, LLMBar, and LLMEval2, please also add an observed flip-rate or mean |p_ij0 - p_ij,swap| baseline and compare it to SCUR and to S alone.
-
Judge panels and binarization. Which models constitute the 17-judge Arena and MMLU panels and the 6-judge HaluEval panel, and where were the verdict-token logprobs obtained? How were 4-way MMLU and binary HaluEval labels mapped to the "canonical A wins" event, and how were Arena ties handled?
4 comments
In addition, We've compared risk scores derived from all aggregation methods evaluated in the paper, DS, MACE, GLAD, and CARE, in Table 7. The results are shown here. SCUR still leads the average performance on various evaluation settings.
| Aggregation posteriors | All Items AUROC ↑ | All Items 1−AURC ↑ | All Items Err@50 ↓ | HC-V AUROC ↑ | HC-V 1−AURC ↑ | HC-V Err@50 ↓ | HC-P AUROC ↑ | HC-P 1−AURC ↑ | HC-P Err@50 ↓ |
|---|---|---|---|---|---|---|---|---|---|
| DS entropy | .6527 | .8061 | .1805 | .5804 | .8290 | .1690 | .5059 | .8365 | .1677 |
| MACE entropy | .6510 | .8049 | .1825 | .5778 | .8271 | .1690 | .5055 | .8352 | .1677 |
| GLAD entropy | .6318 | .7920 | .1963 | .5643 | .8118 | .1760 | .4939 | .8216 | .1801 |
| CARE margin | .5681 | .7546 | .2295 | .5301 | .8040 | .1849 | .4973 | .8298 | .1682 |
| SCUR | .7018 | .8409 | .1468 | .6956 | .8737 | .1231 | .6740 | .8885 | .1104 |
Q3: Judge panels and binarization
For the Chatbot Arena, HaluEval, and MMLU benchmarks, we use publicly available judge outputs from Hugging Face, and for LLMBar and LLMEval$^2$, we conduct the judgments ourselves. We have added the dataset links and detailed judge composition to Appendix C.
The verdict-token logprobs are the log-probabilities assigned to A/B in Chatbot Arena and Yes/No in MMLU and HaluEval during next-token prediction. We exclude human label ties for Chatbot Arena.
Item-level bootstraps are needed. "Consistently outperforms" is overclaimed.
We have removed the term "consistently" from the sentence "...consistently outperforms...". But our empirical results show that SCUR outperforms alternative baselines in most settings. We also added 1000 bootstrap resamples to show intervals in Figures 7, 8, and 9.
Thanks to the reviewer for their time and effort in reviewing our work. All revisions are highlighted in blue in the updated manuscript.
Thank you for the response. However, I would still like to maintain my score.
Thanks for your reply. However, just stating that you would like to maintain the score without providing any reasoning is difficult to understand. If you believe our responses did not resolve your concerns, please specify what remains unresolved and why. We are happy to provide further clarification or discussion.
Thanks for the reviewer's feedback. Please check our response below.
When perturbations are available, U and S may share some information because both depend on the fitted latent measurement model. However, this overlap does not necessarily imply that the perturbation information is redundant. As suggested, we report the Pearson correlations among U, C, and S in Table 10. This comparison covers all items and two subsets: high consensus by vote agreement (HC-V) and high consensus by panel-mean probability (HC-P). A summary is shown here:
The correlation between U and S is relatively low on Chatbot Arena (.2924) and higher on the two smaller benchmarks, LLMBar (.9351) and LLMEval$^2$ (.8029). More importantly, the ablation analysis in Table 9. It suggests that incorporating perturbation observations generally improves SCUR's performance. The results are summarized below (each entry reports SCUR averaged across benchmarks over 5 seeds).
A similar pattern is observed in Table 3, where adding S improves AUROC on all benchmarks except LLMEval$^2$. (Please see the risk signal ablation analysis table under our response to Q2 for eviewer 7fmh.) These results demonstrate that perturbation information provides additional value for identifying judge-panel errors.
We provide the suggested ablation analyses in Section 5 and Appendix E. Here, we briefly summarize the main findings. The ablation regarding risk signals in Table 3 shows that SCUR improves AUROC over U on four of the five benchmarks, with LLMEval$^2$ as the exception. The more detailed results can be seen in the table under our response to Q2 for eviewer 7fmh.
For the missing baseline, we added three additional baselines mentioned by the reviewer: bidirectional preference entropy (BPE), the observed order-flip rate (Filp), and absolute probability change (Δp). Their performance is evaluated on three answer-order swap benchmarks: Chatbot Arena / LLMBar / LLMEval$^2$. We provide the averaged results over 5 seeds below (these baselines use the same available observations as SCUR):
We also update these three new baselines in Tables 1, 4, and 7 and Figures 2, 3, 4, 6, 7, 8, and 9. In these comparisons, SCUR achieves the best performance in most evaluation settings.