acceptodds
Withdrawn from ICLR 2027

AURA: Adaptive Uncertainty-aware Refinement for LLM-as-a-Judge Auditing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) are widely used as judges for open-ended generation because human evaluation is costly and hard to scale, yet their preferences remain imperfect proxies for human judgment. Human verification can reveal which judge decisions to trust, but it is typically affordable for only a small fraction of them. We propose AURA, an adaptive, uncertainty-aware framework that jointly refines estimates of judge–human agreement and guides human verification for pairwise LLM-as-a-judge decisions. The framework updates a probability of agreement for each comparison using a learned human-consistency scorer and local and anchor-based evidence. Sparse transport propagates reliable signals to uncertain comparisons, while uncertainty guides the allocation of additional human verification. Our synthetic and real-data evaluations show how performance varies with judge quality and verification budget, and illustrate when adaptive refinement can help allocate limited human review.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM-as-a-judge, PU learning, LLM evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TobMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AURA, an adaptive framework for auditing LLM-as-a-Judge decisions through trust refinement, evidence propagation, and uncertainty-guided human verification. Experiments on synthetic and real-world datasets report improved agreement with human preferences under limited verification budgets.

Critical strengths
  1. The paper addresses a practically relevant problem: improving the reliability of LLM-as-a-Judge evaluations while reducing human annotation costs.
  2. The experiments cover multiple judge models and evaluation settings, with ablation studies examining the contributions of individual components.
Critical weaknesses
  1. Presentation and Clarity. The manuscript exhibits a strong AI-generated feel, with difficult-to-follow figures and rather mechanical theoretical analyses that lack clear intuition and motivation. The overall presentation appears insufficiently refined, making the core ideas and technical contributions unnecessarily difficult to understand.

  2. Insufficient Baseline Comparisons. The baselines mainly consist of conventional classifiers and label propagation methods. Stronger active learning and semi-supervised learning baselines under matched human verification budgets are needed to substantiate the claimed label efficiency.

  3. Concerns About Robustness. The ablation study reveals substantial performance degradation when the transport component is removed, with accuracy falling well below the original judge performance. This raises concerns about the stability of the overall framework and the strong dependency between its components.

Questions for authors
  1. What is the fundamental methodological novelty of AURA compared with existing PU learning and active learning frameworks?
  2. Why does removing the transport component cause such severe performance degradation? How robust is the framework to noisy or conflicting human annotations?
Reviewer_Qo1aOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents an approach to calibrate llm-as-a-judge approaches by aligning judges with human checker. Very similar work has been already introduced in the literature [1]. Missing baseline comparison.

[1] Agus Sudjianto, Aijun Zhang, Srinivas Neppalli, Tarun Joshi, Michal Malohlava: Human-Calibrated Automated Testing and Validation of Generative Language Models. CoRR abs/2411.16391 (2024)

Critical strengths
  • well-motivated problem
Critical weaknesses
  • missing baseline
  • missing RL-based baseline - that should be an easy baseline to do given the nature of the problem
  • evaluation protocol is unclear
  • i had difficulty interpreting the ablation study - please explain it more
  • Theory is only covering part of the approach - please clarify
Questions for authors

cf. questions above.

Reviewer_HfdROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces AURA, an adaptive framework for auditing pairwise LLM-as-a-judge decisions under a limited human-verification budget. AURA learns a human-consistency scorer from reward-model features, combines model, neighborhood, anchor, and transport evidence to estimate whether each LLM judgment agrees with humans, propagates verified evidence to uncertain comparisons, and adaptively selects additional examples for human review. Theoretical results characterize fixed-round stability and conditions under which transport reduces estimation error. Experiments on synthetic data, MT-Bench, and Chatbot Arena show improved judge–human agreement with fewer reported human verifications than supervised baselines, while ablations identify transport and adaptive verification as important components.

Critical strengths
  1. The paper is practically motivated and addresses a timely and meaningful problem: how to audit unreliable LLM-as-a-judge decisions while minimizing the cost of human verification.

  2. The empirical evaluation is extensive, covering synthetic settings, multiple question types, two real-world benchmarks, and five judge models. The reported results consistently improve judge–human agreement, while the ablation study provides evidence for the contributions of transport, representation learning, and adaptive verification.

Critical weaknesses
  1. The theoretical section is difficult to follow and the results are only weakly connected to the full algorithm and its empirical performance. For example, ℓi(t)\ell_i^{(t)} first appears around line 200 but is not defined until Appendix B.1, while Lenc(t)(Θ(t,s))L_{\mathrm{enc}}^{(t)}(\Theta^{(t,s)}) around line 269 is never explicitly introduced. More importantly, the stability results apply only within a fixed refinement round, and the transport-improvement condition depends on the unknown true labels. Consequently, the theory does not establish end-to-end convergence or explain the reported empirical gains, leaving a substantial gap between the theoretical analysis and the proposed method.

  2. The experimental evaluation does not yet provide sufficiently convincing evidence for AURA’s claimed label efficiency. AURA benefits from adaptive label selection, whereas the supervised and PU baselines use fixed verified sets, and no active or adaptive baseline is included.

Questions for authors
  1. Could the authors clarify Figure 3, including what the x- and y-axes represent, and how the distributions should be interpreted? In addition, the caption appears to reverse the two panels: the left panel is titled “Final Prediction” and the right panel “Verification / Waiting Pool,” whereas the caption describes them in the opposite order. Please correct this inconsistency and state the intended takeaway from the figure.

  2. What motivates the additive trust-logit formulation in Equation (2)? In particular, why is it appropriate to linearly combine the model log-odds, local-consistency score, anchor-support score, and previous transport inflow, given that these quantities have different meanings and scales? Please clarify whether this formulation has a probabilistic derivation or is primarily heuristic, how its weights are selected.

  3. Could the authors provide a sensitivity analysis for the loss coefficients in the encoder objective, particularly λsoft\lambda_{\mathrm{soft}}, λgeo\lambda_{\mathrm{geo}}, λanchor\lambda_{\mathrm{anchor}}? Appendix E.4 varies the pseudo-label weight but does not examine the geometry and anchor regularization weights.

  4. Since AURA is positioned as an adaptive PU-learning method, why are standard PU baselines omitted from the main comparison tables? On the other hand, how about other baseline related to active learning?

  5. I suggest polishing the Figure 2 (overview figure) with less words but more vivid procedure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.