- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies LLM-as-Judge, which is now the dominant scalable evaluation tool but is well-known to exhibit systematic biases, whereas human verification is realistic only for a small fraction of judgments. The paper proposes PUAUDIT, which reframes LLM-evaluation bias correction as a positive–unlabeled (PU) learning problem whose aim is auditing an existing frozen judge
Each pairwise comparison is encoded as a unit-norm difference embedding using a frozen reward-model encoder (Skywork-RM by default). Within each question type, verified positives are filtered to produce a clean anchor set D_P'. A single Partial Optimal Transport problem is then solved; the resulting normalized alignment score is used to flip any unlabeled judgment. The score is explicitly described as a transport-alignment measure, not a calibrated probability. The whole audit is one linear program.
In controlled simulation, four combinations of noise (Class-Dependent vs Dependent-on-Difficulty) and verification (SCAR vs SAR) are tested; PUAUDIT raises corrupted-judge accuracy by ~25 pp in three of four combinations. On Chatbot Arena, across 9 configurations at 5% verification, PUAUDIT improves judge–human agreement by +0.4 to +34.5 pp (mean +18.9 pp); at 10% the mean margin to the best PU baseline is +4.5 pp (Table 5). On MT-Bench, however, the method loses 2.7 pp at 5% verification and 2.7 pp at 10%, because 263 inconsistent-comparison corrections are offset by 329 false flips of originally-consistent ones.
Critical strengths
1 The PU + POT formulation fits the operational constraints precisely. The positive-only verification pattern of selective review, the ability to leave decisions unflipped, and the absence of model training all map cleanly onto the POT constraint with the sub-threshold flip rule.
2 The experiments support a conditional claim rather than a universal one: PUAUDIT is a cheap, label-efficient audit that delivers large gains when the embedding space geometrically separates judge errors from correct decisions (simulation CD/DD + SCAR, 9 Arena configurations) but can mildly degrade an already-strong judge when that separation does not hold (MT-Bench).
Critical weaknesses
The whole pipeline is anchored on one load-bearing design choice,Skywork-RM as F_θ, justified in a single sentence (section 3.3: "Skywork-RM separate them more clearly than general-purpose encoders such as Gemma-300M, so we use Skywork-RM throughout"). The paper provides no cross-RM comparison (e.g., Skywork-Reward-LLaMA-3.1-8B v0.2 vs an earlier Skywork, vs LLaMA-RM, vs Qwen-RM), and no discussion of the fact that Skywork-RM is itself trained on Chatbot Arena preferences,a potential representational overlap that could inflate Arena numbers while contributing nothing on MT-Bench.
Questions for authors
Please refer to the weakness section.
2 comments
We additionally evaluated three reward models: Skywork v0.2, GRM Gemma 2, and Tulu 3. The comparison covers all ten main-paper configurations at verification budgets of 1%, 5%, 10%, and 20%. Below are mean accuracies (%) for one Arena configuration and one MT-Bench configuration at 5% verification, each evaluated over ten random seeds.
| Method / encoder | Arena: Llemma QTA | MT-Bench: GPT-5.4-mini |
|---|---|---|
| Original judge | 52.53 | 77.87 |
| PUAudit / Skywork, 1024 tokens | 73.74 | 75.60 |
| PUAudit / Skywork, 2048 tokens | 74.16 | 78.59 |
| PUAudit / GRM Gemma 2 | 71.39 | 73.00 |
| PUAudit / Tulu 3 | 73.54 | 73.78 |
We will include results for all configurations and budgets, together with the input-length comparison using the same Skywork checkpoint, in the revision.
We also checked the released Skywork preference dataset of 77,016 pairs for overlap with the evaluation data, and reran the audit after excluding matched evaluation examples.
| Check | Result |
|---|---|
| Exact or near-duplicate prompts | Matches found |
| Exact match of the prompt and both responses | No matches found |
| Mean accuracy change after excluding matches and resplitting, at 5% verification | +1.04 percentage points |
The check covers the released preference data, and the rerun changes both the retained examples and the data splits. We will include the matching criteria and rerun results in the revision.
Thank you for your advise and review.
Table 3 in Appendix B.1 already compares four frozen encoders: BGE-M3, Gemma-300M, Qwen3-0.6B, and Skywork-RM. On MT-Bench, averaged across three local judges and GPT-5.4-mini, PUAudit with Skywork-RM improves accuracy from 69.4% to 73.9%. Replacing Skywork-RM with the general-purpose encoder BGE-M3 also improves accuracy, from 69.4% to 70.8%. Performance varies across encoders, with Skywork-RM achieving the strongest results among those evaluated, which motivates its use as the default encoder.
Regarding potential training-data overlap, we have not found public evidence confirming overlap between Skywork-RM’s training data and Chatbot Arena, although we cannot guarantee that no overlap exists. We will state this limitation explicitly. Finally, the PU baselines in our comparison use the same frozen Skywork-RM embeddings. This holds the encoder fixed when comparing the auditing methods, allowing us to assess their respective contributions.