acceptodds
Under review as a conference paper at ICLR 2027

Quantifying and Auditing LLM Evaluation via Positive–Unlabeled Learning

Abstract

Large Language Models (LLMs) are increasingly used as judges for scalable evaluation, yet such LLM-as-a-Judge systems exhibit systematic biases that are decoupled from semantic quality, most notably verbosity bias. Meanwhile, human supervision is costly and typically selective, yielding reliable positive judgments but leaving most outputs unlabelled and potentially mixed in quality. We formulate LLM evaluation under selective human supervision as a positive–unlabelled learning problem and propose a geometric auditing framework based on Partial Optimal Transport. By aligning a small set of human-verified positives with a reliable subset of unlabelled outputs in a fixed embedding space, our method identifies human-consistent preferences and corrects biased judges without retraining. Experiments on simulated data and Chatbot Arena show that the method can improve judge–human agreement with few verified comparisons. For each unlabelled comparison, it also returns a normalized transport-alignment score that measures relative alignment with the verified-positive set; this score is not a calibrated probability.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM-as-a-judge, PU learning, LLM evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iU6WOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies LLM-as-Judge, which is now the dominant scalable evaluation tool but is well-known to exhibit systematic biases, whereas human verification is realistic only for a small fraction of judgments. The paper proposes PUAUDIT, which reframes LLM-evaluation bias correction as a positive–unlabeled (PU) learning problem whose aim is auditing an existing frozen judge

Each pairwise comparison is encoded as a unit-norm difference embedding using a frozen reward-model encoder (Skywork-RM by default). Within each question type, verified positives are filtered to produce a clean anchor set D_P'. A single Partial Optimal Transport problem is then solved; the resulting normalized alignment score is used to flip any unlabeled judgment. The score is explicitly described as a transport-alignment measure, not a calibrated probability. The whole audit is one linear program.

In controlled simulation, four combinations of noise (Class-Dependent vs Dependent-on-Difficulty) and verification (SCAR vs SAR) are tested; PUAUDIT raises corrupted-judge accuracy by ~25 pp in three of four combinations. On Chatbot Arena, across 9 configurations at 5% verification, PUAUDIT improves judge–human agreement by +0.4 to +34.5 pp (mean +18.9 pp); at 10% the mean margin to the best PU baseline is +4.5 pp (Table 5). On MT-Bench, however, the method loses 2.7 pp at 5% verification and 2.7 pp at 10%, because 263 inconsistent-comparison corrections are offset by 329 false flips of originally-consistent ones.

Critical strengths

1 The PU + POT formulation fits the operational constraints precisely. The positive-only verification pattern of selective review, the ability to leave decisions unflipped, and the absence of model training all map cleanly onto the POT constraint with the sub-threshold flip rule.

2 The experiments support a conditional claim rather than a universal one: PUAUDIT is a cheap, label-efficient audit that delivers large gains when the embedding space geometrically separates judge errors from correct decisions (simulation CD/DD + SCAR, 9 Arena configurations) but can mildly degrade an already-strong judge when that separation does not hold (MT-Bench).

Critical weaknesses

The whole pipeline is anchored on one load-bearing design choice,Skywork-RM as F_θ, justified in a single sentence (section 3.3: "Skywork-RM separate them more clearly than general-purpose encoders such as Gemma-300M, so we use Skywork-RM throughout"). The paper provides no cross-RM comparison (e.g., Skywork-Reward-LLaMA-3.1-8B v0.2 vs an earlier Skywork, vs LLaMA-RM, vs Qwen-RM), and no discussion of the fact that Skywork-RM is itself trained on Chatbot Arena preferences,a potential representational overlap that could inflate Arena numbers while contributing nothing on MT-Bench.

Questions for authors

Please refer to the weakness section.

2 comments

Table 3 in Appendix B.1 already compares four frozen encoders: BGE-M3, Gemma-300M, Qwen3-0.6B, and Skywork-RM. On MT-Bench, averaged across three local judges and GPT-5.4-mini, PUAudit with Skywork-RM improves accuracy from 69.4% to 73.9%. Replacing Skywork-RM with the general-purpose encoder BGE-M3 also improves accuracy, from 69.4% to 70.8%. Performance varies across encoders, with Skywork-RM achieving the strongest results among those evaluated, which motivates its use as the default encoder.

Regarding potential training-data overlap, we have not found public evidence confirming overlap between Skywork-RM’s training data and Chatbot Arena, although we cannot guarantee that no overlap exists. We will state this limitation explicitly. Finally, the PU baselines in our comparison use the same frozen Skywork-RM embeddings. This holds the encoder fixed when comparing the auditing methods, allowing us to assess their respective contributions.

We additionally evaluated three reward models: Skywork v0.2, GRM Gemma 2, and Tulu 3. The comparison covers all ten main-paper configurations at verification budgets of 1%, 5%, 10%, and 20%. Below are mean accuracies (%) for one Arena configuration and one MT-Bench configuration at 5% verification, each evaluated over ten random seeds.

Method / encoderArena: Llemma QTAMT-Bench: GPT-5.4-mini
Original judge52.5377.87
PUAudit / Skywork, 1024 tokens73.7475.60
PUAudit / Skywork, 2048 tokens74.1678.59
PUAudit / GRM Gemma 271.3973.00
PUAudit / Tulu 373.5473.78

We will include results for all configurations and budgets, together with the input-length comparison using the same Skywork checkpoint, in the revision.

We also checked the released Skywork preference dataset of 77,016 pairs for overlap with the evaluation data, and reran the audit after excluding matched evaluation examples.

CheckResult
Exact or near-duplicate promptsMatches found
Exact match of the prompt and both responsesNo matches found
Mean accuracy change after excluding matches and resplitting, at 5% verification+1.04 percentage points

The check covers the released preference data, and the rerun changes both the retained examples and the data splits. We will include the matching criteria and rerun results in the revision.

Thank you for your advise and review.

Reviewer_tTRpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to detect and correct disagreements between LLM-as-a-Judge evaluations and human preferences. The authors argue that only a limited set of correct judgments is verified by human in practice, while the remaining unlabeled samples may contain both correct and incorrect judgments. They therefore formulate the problem as positive–unlabeled learning. Rhe authors propose PUAUDIT. The method is based on partial optimal transport and does not require finetuning. It aligns LLM-labeled data with human-verified positive examples and directly reverses those with low alignment scores, aiming to improve agreement with human preferences. The authors provide a theoretical analysis of the method's effectiveness. Empirically, they evaluate PUAUDIT on simulated data, Chatbot Arena, and MT-Bench, showing that it can improve agreement with human preferences.

Critical strengths
  1. The proposed method provides a simple, training-free auditing framework that corrects existing LLM judgments using frozen embeddings and partial optimal transport.

  2. The paper provides comprehensive empirical evaluation, including simulations and real-world datasets, SCAR and SAR sampling settings, sensitivity analyses of transport mass and denoising thresholds, and comparisons with multiple PU learning baselines.

Critical weaknesses
  1. I find the motivation for this work is unconvincing. The authors argue that only a limited number of human-verified positive samples are available in practice due to selection bias in human annotation. However, in crowdsource human annotation, annotators generally do not decide which samples to annotate, and they directly provide preference judgments. Therefore, obtaining negative samples does not require a separate annotation process or additional annotation costs. In the experiments, the authors artificially construct the PU learning setting by discarding disagree judgements from datasets that already contain both positive and negative labels, further undermining the practical relevance of the proposed formulation.

  2. The proposed method offers limited improvements over existing alternatives. In the setting reported in Table 2, PUAUDIT relies on embeddings extracted from a pretrained reward model, yet directly using the same reward model achieves nearly identical performance. This further raises questions about the necessity of the proposed PU-learning-based auditing approach.

Overall, I consider these to be concerns regarding the motivation and practical significance of the proposed approach, rather than minor issues that can be addressed through clarification or additional experiments.

Questions for authors
  1. The paper emphasizes selective human annotation as a central motivation, arguing that human annotations tend to favor certain types of responses. However, the proposed method does not explicitly model or correct for this selection bias. The experiments also do not adequately consider settings with such selection bias, such as where the verification probability depends on factors such as response quality or ambiguity/disagreement between judges.

  2. Theorem 4.3 establishes that transported samples are positive (transported => positive). However, the proposed algorithm flips all judgments with alignment scores below a certain threshold, implicitly assuming that untransported samples are negative (not transported => negative). This implication is not supported by the theoretical guarantee.

  3. The paper sets the transport mass mm to the agreement rate among human-verified samples, which requires knowing the total number of verified samples, including disagreements. However, the proposed PU formulation assumes that only verified positive examples are available. How can mm be computed in such a setting?

2 comments

1.PUAudit uses an existing set of human preferences to audit LLM judgments on comparisons without human labels. For each labeled comparison, we construct the positive example using the human’s choice, regardless of the LLM’s prediction. These examples describe which responses humans prefer. For unlabeled comparisons, we use the LLM’s choices, which may agree or disagree with human preferences. Thus, “labeled” and “unlabeled” distinguish the availability of human preferences, while “positive” and “negative” indicate whether the choice used to represent a comparison agrees with the human preference. A comparison ordered according to the human choice is positive by construction.

In our experiments, the datasets contain human preferences for all comparisons, but only a subset is made available for constructing the positive set. For the comparisons being audited, human labels are withheld and used for evaluation. Comparisons where the LLM disagrees with the human are retained: when their human labels are available, we use the human preference to construct the positive example. We do not filter the labeled set by LLM–human agreement. Separately, the agreement rate on a held-out human-labeled subset is used to estimate the transport fraction mm.

Agreement and disagreement labels can also be obtained by comparing human and LLM choices at the same annotation cost. The motivation for our formulation is to use available human preferences as positive examples for auditing unverified judgments. We will revise the introduction to explain this setting without relying on selective annotation as the reason for missing negative labels, and clarify the experimental construction in Sections 3.1 and 5.1.

2.PUAudit is designed to audit and selectively correct an existing LLM judge. Directly using the reward head replaces the original judge with the reward model: the final preferences are determined by the RM’s scores, independently of the original judgments. PUAudit uses the RM as a replaceable frozen encoder to audit the original LLM judge against human-verified positive examples. The two methods address completely different tasks, and direct RM is used only as an unfair baseline.

This auditing procedure requires no additional model training or fine-tuning. Both the original judge and the encoder remain frozen, and the auditing stage takes only a few seconds on CPU in our experiments. This provides a low-cost way to audit existing judgments compared with approaches that require additional training. We will clarify these differences in purpose and computational requirements when presenting the baseline comparisons.

3.Unlike standard PU estimators which require selected completely at random assumption, we only requires supp(μP)⊆supp(μ+)\text{supp}(\mu_P) \subseteq \text{supp}(\mu_+) because our group denoising extracts the consensus direction within the positive cone, where Theorem A.3 proves that within the small threshold, the reference direction is ϵ\epsilon–aligned wit the true preference direction regardless of the non-uniform sampling density. In simulation (Table 1), both judge errors and human verifications concentrate near ambiguous boundary, while PUAudit still delivers an improvement on accuracy gain. Furthermore, we also evaluate the feature–dependent SAR across judges and benchmarks where verification probability varies with distance from the pool mean where PUAudit's accuracy under SAR differ from SCAR by at most 1%.

4.Flipping the sample with low score directly follows from Theorem 4.3 with the conservation constraints of POT. This theorem proves that the optimal plan is supported when the positive–negative cosine gap holds which implies that every true negative receives zero mass. In the meantime, the POT constraints enforce an upper capacity per unlabeled sample while transporting total mass. When mm equals the latent positive proportion, all the positive sample might saturate its capacity to fulfill the total mass. Therefore, the when m=αm=\alpha, it is necessary and sufficient for a judgement to be an error.

5.The positive set is constructed directly from human preferences. Each labeled comparison is ordered according to the human choice and is therefore positive, even when the original LLM prediction disagrees.

We estimate mm separately on a held-out set VV of human-labeled comparisons. Specifically, m^=∣V+∣/∣V∣\hat m=|V_+|/|V|, where V+V_+ contains comparisons on which the original LLM prediction agrees with the human label. Both the human preferences and the LLM predictions are available on VV, so the agreement rate can be computed directly. Being positive in the POT reference set does not imply that the original LLM prediction was correct: the reference pair follows the human choice, while estimating mm requires comparing the two choices. We will clarify these separate uses of human labels in Sections 3.1 and 5.1.

We additionally evaluated verification schemes based on ambiguity, judge–RM disagreement, and other features. These schemes model how existing human annotations may have been collected. We tested inverse-probability weighting to correct the estimate of mm, and to additionally reweight the verified positives. Both variants retain positive denoising.

The controlled experiment uses known inclusion probabilities, 14 configurations, and a 5% expected verification budget. Entries are mean accuracy changes in percentage points relative to unweighted PUAudit under the same selection scheme, averaged equally across configurations using paired valid runs.

Verification schemeCorrect mm onlyCorrect mm and reweight positives
Uniform0.000.00
Near the mean embedding+0.01-0.01
Small RM score gap, strength 10-0.15+0.09
Small RM score gap, strength 100-0.28+0.06
Large RM score gap-0.13-0.28
Judge–RM disagreement+0.55+3.48
Long responses-0.44-0.11
Far from the mean embedding+0.84+1.60

The RM score gap serves as a proxy for ambiguity or confidence; disagreement refers to different choices by the judge and RM. This experiment does not use human-rated absolute response quality to determine verification probabilities. We will include the selection rules, weighting procedure, and results in the revision.

Thank you for raising the concern for the motivation and other questions.

Reviewer_QmuZOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PUAUDIT, an LLM-as-a-Judge auditing method based on positive-unlabeled (PU) learning. The proposed approach leverages a small set of human-verified positive examples to identify and correct biased judges without retraining. The authors also provide a theoretical analysis and validate the proposed method using both simulated and real-world data.

Critical strengths
  1. The paper addresses an important problem, as identifying and correcting biases in LLM judges is critical when human annotations are limited.

  2. The proposed framework does not require model training and demonstrates improvements in some experiments.

Critical weaknesses
  1. I have concerns about the assumptions underlying the proposed method. First, the methodology assumes that correct and biased judge evaluations are separable in the embedding space, but it is unclear whether this assumption holds in practice. Additionally, a low alignment score does not necessarily indicate an incorrect evaluation, so flipping the judge's decision may introduce additional errors.

  2. The proposed method focuses on binary preference tasks. However, LLM evaluation can also involve multi-score evaluation tasks, for which the current distance-based approach may be difficult to extend.

  3. The writing and organization of the paper could be improved. In particular, some key methodological steps are described in the appendix rather than in the main body of the paper.

Questions for authors

See Weaknesses above.

5 comments

Human judgment considers several dimensions, including fluency, correctness, and appropriateness. In our geometric model, their combined contribution is represented by a quality vector: alignment with this vector supports the human preference, while the opposite direction supports the other response. We model biases that should not affect our judgment of response quality, such as verbosity bias, as noise approximately orthogonal to this vector. For example, we should not rate a response more highly just because it is longer. Positive group denoising aims to retain the common quality direction.

As for alignment score, we keep judgments that are sufficiently similar to the positives because they are expected to agree with human judgments, and flip those that are not. Theorem 4.3 provides a theoretical justification for this rule: under its conditions, transported mass is assigned only to positives, and if mm also equals the true positive proportion in the unlabeled set, every positive receives its full capacity. The normalized alignment score is therefore one for positives and zero for negatives, allowing us to distinguish correct judgments from incorrect ones. Since the task is binary, flipping an incorrect judgment means selecting the other option and thus corrects the decision.

We focus on binary preferences because they are widely used in important applications, including model comparison in Chatbot Arena, preference-based training such as DPO, and medical response evaluation, where clinicians compare alternative answers for factual accuracy and potential harm, as in Med-PaLM 2. Auditing these preferences is therefore relevant to both general model evaluation and applications that rely on expert judgment.

A natural extension to multi-score evaluation is to decompose it into binary judgments. For multiple dimensions, such as correctness and fluency, we could apply PUAudit to pairwise preferences within each dimension. For a 1–5 rating scale, we could use the standard decomposition from ordinal classification: whether the score exceeds 1, 2, 3, and 4. Human-verified agreements would provide positive examples for each binary subproblem, allowing us to apply the same auditing principle. The audited decisions could then be combined into a rating after enforcing consistency across thresholds; for example, “yes, yes, no, no” corresponds to a score of 3. This provides a concrete way to extend PUAudit by auditing binary judgments and combining their results. We will discuss this extension, while our current theory and experiments focus on strict binary preferences.

3.Thank you for the feedback. We will move the algorithm and key methodological steps into the main text, clarify the notation and how mm is estimated, and improve the explanation of the theoretical results and auditing procedure in the revision.

We additionally measured corrected errors and false flips on MT-Bench. We kept the judge, Skywork checkpoint, and data splits fixed, and increased the encoder's input limit from 1024 to 2048 tokens. The table reports means over ten random seeds at 5% verification.

SettingAccuracy (%)Corrected errorsFalse flips
Original judge77.87——
PUAudit, 1024 tokens75.60252.3307.2
PUAudit, 2048 tokens78.59288.5271.1

With the longer input limit, PUAudit corrects more errors and introduces fewer false flips. We will include these statistics in the revision to show how input length affects the audit.

Thank you for your detailed review and suggestion.

Thank you for the detailed responses and additional experiments. The clarifications address some of my concerns, particularly regarding the potential extension to multi-score evaluation and the presentation of the methodology.

However, I still have concerns about the assumptions underlying the method, especially whether correct and incorrect judge decisions are sufficiently separable in the embedding space in practice. The theoretical justification for flipping low-alignment decisions relies on conditions that may not always hold in real-world settings. While the additional MT-Bench results are encouraging, they do not fully resolve these concerns.

I appreciate the authors' efforts and have raised my rating to 2 (Weak Rejection).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.