acceptodds
Under review as a conference paper at ICLR 2027

BiMetaJudge: Bidirectional Metamorphic Auditing and Selective Risk Control for LLM Judges

Abstract

Large language models (LLMs) are increasingly used as judges. Repeated sampling estimates local decoding variability, and position swaps expose one presentation artifact, but neither establishes whether a judge tracks a known change in answer quality. We present BiMetaJudge, a black-box audit built around a graded bidirectional response curve: matched nuisance controls estimate spurious movement, while independently verified mild/severe degradations and partial/full repairs measure directional elasticity. A development-fitted risk model drives a staged policy whose score contains only purchased probes, and a disjoint split certifies the final accepted set. Stable-but-wrong cases are an evaluation slice rather than a claimed new phenomenon. We evaluate the complete pipeline on MetaJudge-MR, spanning four domains, five judge configurations, and 12,000 base comparisons. On the held-out test split, BiMetaJudge reaches AUROC 0.824 versus 0.808 for the strongest non-BiMeta baseline and stable-but-wrong recall 0.795 versus 0.280. Leave-one-domain and leave-one-judge gains remain +2.67 and +2.68 AUROC points over nuisance-only evidence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-Judge, metamorphic testing, evaluation reliability, selective risk control, stable errors, uncertainty estimation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_K7AcOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

BiMetaJudge audits black-box LLM judges for stable but wrong judgments by testing whether each verdict stays fixed under quality-preserving edits and moves as expected under verified degradations and repairs, then uses these responses for risk-based selective acceptance with a finite-sample certificate. On a self-built benchmark of four domains and five judges, it improves error detection over the strongest baselines, with AUROC 0.824 versus 0.808 and stable-error recall 0.795 versus 0.280.

Critical strengths

The paper targets a real blind spot of consistency-based confidence: judgments that stay stable under repeated sampling and position swaps yet are still wrong. It also proves that consistency alone cannot tell correct judgments from wrong ones.

Critical weaknesses
  1. The core ideas largely repeat prior work such as CheckList, FBI and Trust or Escalate, so the novelty is limited.
  2. All results come from a single self-built benchmark, so their credibility is hard to judge.
  3. The baselines are weak and the closest related methods are not compared.
  4. The writing is dense and poorly structured, and reads as obvious AI slop.
Questions for authors

See Weaknesses.

Reviewer_JxU1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents BiMetaJudge, which measures judge reliability by testing whether an LLM judge responds correctly to verified changes in answer quality. BiMetaJudge applies bidirectional transformations that deliberately degrade or repair candidate answers and measures the judge’s directional response. BiMetaJudge is particularly effective at detecting stable-but-wrong judgments that conventional consistency-based methods often miss. The authors further use adaptive auditing and risk certification to reduce evaluation cost and support reliable deployment.

Critical strengths
  1. This paper studies an important limitation of consistency-based judge reliability estimation: a judge can be highly consistent but systematically wrong.

  2. The bidirectional degradation-and-repair design provides a principled way to test whether a judge truly responds to answer quality, which is informative for judge reliability estimation.

  3. The proposed adaptive auditing strategy reduces evaluation cost while maintaining similar selective risk.

Critical weaknesses
  1. The proposed method relies heavily on the availability of reliable degradation and repair transformations, which may be difficult to construct for subjective or open-ended tasks.

  2. The proposed method requires external verification of transformations, raising questions about its scalability and practical cost.

  3. The performance gains are substantially larger on objectively verifiable domains such as math and code than on subjective preference tasks, limiting the generality of the approach.

  4. Some figures in this paper seem to be generated by AI, whose quality can be improved.

Questions for authors

Please refer to the weakness part.

Reviewer_ySGFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BiMetaJudge, which is a black box auditing framework designed to identify unreliable pairwise decisions made by LLM judges. BiMetaJudge also tests how a judge responds to different changes in candidate answers. Some changes affect presentation, but others make an answer worse or to improve it intentionally.

This paper further introduces MetaJudge-MR, a benchmark with four domains, 12000 comparisons and five LLM judges. Being evaluated on 36000 held-out judge decisions, BiMetaJudge achieves 0.824 on AUROC and 0.532 on AUPRC. BiMetaJudge also identifies 79.5% of stable-but-wrong decisions, while method using verbal confidence only achieve 28%.

Critical strengths

1). This paper addresses meaningful topic with a bidirectional auditing design. The paper discussed about an important distinction between judgment consistency and correctness. Multiple agreement does not necessarily mean that a judge responds in correct way when experiencing changes in answer quality. The method combines graded degradation and repair probes to gather instance-level evidence. Removing either component reduces stable-but-wrong recall from 79.5% to 10.6% or 6.6% (table 3).

Critical weaknesses

1). There is no enough information on how this benchmark is constructed. My major concern is that construction of MetaJudge-MR is not documented in details. For instance, in section 5.1, this paper only talked a brief descriptions of four domains. And in section 5.2, those evaluated judges are only identified by anonymized configurations. Although Section 4.1 describes the general validation process, but things such as source datasets, full probe-generation prompts, and detailed annotation procedures are not explicitly provided.

2.) The improvement in this manuscript is relatively small. For instance, BiMetaJudge only improves AUROC from 0.808 to 0.824 (table 1). Figure 3B also shows small or negative gains over nuisance-only evidence for some judge configurations. Improvement on preference and summarization task are also very small, and in some cases even negative (figure 3B). In general, AUROC gains are less consistent across settings.

Questions for authors

1). Can you clarify how the degradation or repair examples are generated and verified, including which candidate is modified?

2). Can authors provide confidence intervals for this comparison? And can author also report performance specifically on judgments that remain stable across repeated sampling and position swaps?

3). Can authors clarify the difference between the candidate regions fixed on development data and the routing thresholds selected using certification data? Can you also clarify this process and report the actual coverage and test error rate at the 8% risk target?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.