acceptodds
Under review as a conference paper at ICLR 2027

Certified Reuse or Recalibration of LLM Judges after Output Shift

Abstract

An LLM judge is often calibrated on one model release and then reused after the evaluated outputs change. Its reliability map may continue to transfer, but it can also become stale when visible response properties change their relationship with quality. Recalibration is not automatically safer: generator replacement need not make a repair useful, and a small audit can favor a noisy repair. We study this as a finite-sample reuse-or-recalibration decision. Controlled style conflict provides a positive case, prespecified generator transitions test whether reuse remains adequate, and expert-scored long-form updates provide an external case. On a specified target window, a disjoint audit deploys a repair only when it certifies lower risk; reuse otherwise means only that no registered repair is resolved as beneficial at the current radius. Observable source–repair disagreement determines the paired-loss range, yielding safe near-oracle selection up to certification radii and an exact continuous minimax allocation. A fixed-repair binary-Brier lower bound matches the resulting worst-case label-complexity rate. Simulations verify finite-sample control; grouped replays show that certification suppresses held-out harm from validation-selected repairs while retaining power under drift.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM evaluation, LLM-as-a-judge, judge calibration, evaluation reliability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VKPtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies when to keep or update an LLM judge’s calibration after model outputs change. It fits updates on one data split and checks them on another, using disagreement with the original calibration map to guide labeling. Experiments on controlled shifts, generator changes, and expert-scored reports show fewer harmful updates, but limited ability to detect smaller improvements.

Critical strengths
  • Separate fitting and checking, grouped data splits, and thresholds chosen in advance make the evaluation careful. The paper clearly distinguishes its statistical guarantee from performance on held-out data.

  • The experiments show that ordinary validation can select harmful updates, giving a useful motivation for the method. The JETTS results also show that changing generators does not always require recalibration.

Critical weaknesses
  • The benefit of the proposed allocation method is unclear.The minimax allocation performs almost the same as the simpler alternative on the main benchmark (Table 25). This leaves limited evidence that its added complexity provides practical value.
  • Practical usefulness remains limited at the tested budgets.The conservative gates never select an update on RealDR, and Qwen3.8 remains unresolved through 400 labels. In the main setting, checking uses labels for 200 of 232 groups, so substantial label savings are not demonstrated. The guarantee also covers the checked window and does not by itself ensure improvement on future traffic.
Questions for authors
  • What does the method add beyond existing fitting and safety-testing approaches? How does it compare with a valid variance-based bound under the same sampling setup and safety guarantee?

  • What label budgets would allow useful updates on RealDR and Qwen3.8, and what assumptions are needed to extend the guarantee to future traffic?

2 comments
AuthorsOpenReview ↗

Thank you for asking us to isolate design benefits and actionable budgets.

W1 / Q1. Allocation value and a valid variance-based comparison.

The canonical menu supports the simpler envelope: at budget 200, envelope/minimax switching is 11%/12%, with gains 0.0034/0.0040. For fixed states and capacities, the envelope is exactly minimax if one repair attains the largest movement in every state. A controlled 42-state heterogeneous-movement experiment requires up to 1.59 times as many envelope labels to match the minimax radius. This identifies a potential benefit regime, while the canonical data show little solver advantage (Tables 29/30).

For practical allocation value, seven-judge LLMBar fixes one repair, delta=0.05, and 25 fit plus 25 certification labels. Disagreement-range switches in 76.43%, versus 53.00% for SRS WoR betting, with gains 0.03165/0.02263 and zero observed window/test harm. Holding exact hypergeometric certification fixed, disagreement versus target-mass allocation switches in 53.71%/14.86%, favoring disagreement in all seven judges. This isolates allocation at 25/112 queried prompts (Table 4).

SRS means simple random sampling; WoR denotes sampling without replacement. For the variance-based comparison, identical disagreement queries give:

CertificateLLMBar switch
Range76.43%
Valid EB-Serfling43.00%
Exact hypergeometric53.71%

All target the same finite-window gain at level 0.05. Cellwise EB/exact pay simultaneous statewise control; the earlier range floor is removed (Table 36). With 25 fit labels fixed, RM-Bench style-contract switching reaches 82.2% at 50 disagreement-range certification labels and 82.4% at 100 SRS-betting labels: total budgets 75 versus 125. Betting is stronger in verdict-only and high-budget cases (Figure 3). We position the design within LTT and classical allocation theory.

W2 / Q2. RealDR and Qwen3.8 budgets.

Stronger certificates enable checked-window updates under both original menus. Qwen3.8's verdict-only six-repair menu at budget 400 (200 fit, 200 certify) gives:

RuleSwitchTest gainTest harm
SRS WoR betting51%0.003193%
Disagreement EB-Serfling67%0.004076%

Both have zero observed window non-improvement. On identical disagreement queries, replacing range with EB-Serfling raises switching from 0% to 67%. This is a certificate effect at 200/232 queries; the low-budget comparison above supplies separate label-saving evidence (Table 15).

RealDR keeps three repairs and 15 fit tasks. Betting is inactive through 12/15 certification tasks; at 14/15 it switches in 15.7%, gains 0.000161, and has 1.7% test harm. Census test harm is 24%. Selected partial-window replays pass deterministic worst-case completion, so action arises near census. The 15 fit plus 14 certification tasks cost 116 report labels (four per task). These 40 tasks identify a checked-window budget; future-task budgeting additionally needs independent-task evidence under population stability (Table 17).

W2 / Q2, continued. Future-traffic assumptions and measured power.

Population certification requires independent target prompts, a fixed judge/prompt, and a joint law stable from certification through deployment. Functions/support/fallback are frozen on history/fit before certification covariates. IID paired-loss tests then avoid state-mass estimation.

At 199 fit plus 201 certification labels, canonical style-contract IID betting switches in 100%/99.75% for one/six repairs, gains 0.06862/0.07043, and has zero observed test harm (Table 32). Qwen3.8's verdict-only six-repair population grid switches in 7%/80% at 400/597 actual labels, with gains 0.00054/0.00389 and test harm 0%/12%. Natural HumanEval at 20 fit plus 22 certification labels gives 9.2%/0.00287/0.8% (Table 21).

Chronological Arena replay retains eight judges and 793 unique prompts. At 50 fit plus 50 certification labels, next-block gains are 0.000042, 0.000226, and -0.0000043. The final-fold certification census still gives IID-betting future gain -0.000137 and 10.4% test harm (Table 38). Thus subsequent joint-law changes need a drift allowance or another audit. Test harm differs from the certified event; repeated seeds measure cached-data sensitivity.

AuthorsOpenReview ↗

Looking Forward to Your Feedback

Dear Reviewer VKPt, thank you again for your constructive feedback. We have provided detailed responses to your comments and would appreciate it if you could take a look at our rebuttal. Please let us know if any concerns remain; we would be happy to clarify further.

Reviewer_UozNOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether an LLM judge’s calibration map, fitted on one model release, can be reused after the evaluated outputs change, or should be recalibrated. Candidate recalibrations are fitted on one split of target labels. One is deployed only if a stratified Hoeffding lower bound on its Brier improvement is positive on a disjoint split. Audit labels go where the old and new maps disagree most. Experiments show that recalibration helps on RM-Bench’s constructed style conflict but not under generator changes, and that the certified gate avoids most of the held-out harm caused by naive validation.

Critical strengths

The question is practical, and the answer is sensible and honest. A model update doesn’t automatically make calibration stale, and recalibrating on a small audit without a significance check can hurt.

Critical weaknesses
  1. The method is a held-out significance test with a standard allocation. In effect, the procedure deploys the new calibrator only if a one-sided Hoeffding bound on a held-out split says it is better, which is single-candidate Learn-then-Test.
  • The loss-range observation is one line of algebra.
  • Allocating labels in proportion to mass × disagreement follows from Cauchy–Schwarz.
  • The menu version and the lower bound are standard.
  1. The certificate rarely acts, and its value isn’t shown.
  • At 50 labels the gate switches in only 1–8% of replays, because the radius (≈0.11) dwarfs the gains (≤0.02).
  • The main experiments use verdict-only states, so the style shift the paper is about never enters the map.
  • Harm is reported as a frequency, not a size, and no downstream effect is shown (e.g., on model rankings).
  1. The motivation is asserted rather than established, and the significance is not shown. The paper assumes a workflow in which practitioners maintain a verdict-to-probability calibration map for a judge and carry it across model releases, but cites no evidence that this is common practice. Judges are typically used through raw verdicts or per-evaluation corrections such as PPI.
Questions for authors
  1. How large are the losses that certification avoids, in Brier and in a downstream quantity such as model rankings?
  2. At what label budget does the gate become useful, and is it then meaningfully better than single-candidate LTT?
  3. Can you show the style shift on naturally updated outputs rather than on RM-Bench’s constructed split?
4 comments
AuthorsOpenReview ↗

Thank you for pressing us on practical value. We give the new results directly below. In deployment replays, gain is mean held-out Brier reduction, counting reuse as zero; switch/test harm are fractions of all replays.

W1, including the algebra, allocation, menu, and lower-bound concerns.

The contribution is a post-update map-maintenance decision with a shared movement-weighted audit. For binary Brier loss, paired-loss width is twice the absolute source-repair movement, yielding allocation proportional to state mass times movement. One shared sample supports simultaneous repair comparisons. The fixed-repair binary-Brier label-complexity rate scales as squared target-weighted movement over squared gain, times log(1/delta), under independent target labels. We position these results using LTT, Neyman allocation, and classical minimax/testing tools. The matched comparisons below quantify the design's operational increment; the canonical minimax/envelope gap is small.

W2a / Q2. Useful budgets and comparison with single-candidate LTT.

At 25 fit plus 25 certification prompt labels, seven-judge LLMBar gives the following, fixing one source-prior repair and delta=0.05:

RuleSwitchTest gainTest harm
Disagreement-range76.43%0.031650%
Single-candidate LTT-Hoeffding0.43%0.000190%
SRS WoR betting53.00%0.022630%

SRS means simple random sampling; WoR denotes sampling without replacement. Certification queries 25/112 prompts; all three have zero observed window non-improvement. Holding exact hypergeometric certification fixed, disagreement versus target-mass allocation switches in 53.71%/14.86%, favoring disagreement in all seven judges (Table 4). With 25 fit labels fixed, RM-Bench style-contract switching reaches 82.2% at 50 disagreement-range certification labels and 82.4% at 100 SRS-betting labels: tested total budgets 75 versus 125 (Figure 3). Betting is stronger for verdict-only small effects and higher LLMBar budgets. The original six-repair small-budget conservatism remains visible.

W2b. Verdict-only states and style shift.

P(Y|verdict) can change after an output update without style entering the map; this measures aggregate drift. On RM-Bench, our registered state ablation quantifies the richer-state alternative. At 400 total labels, verdict-only gives 23.0% switching and gain 0.0076; adding observable length/format gives 100% switching, gain 0.1142, and zero observed test harm. At 50 labels, the style contract resets 28.9 active cells on average for inadequate support and never switches. Richer contracts localize drift while fragmenting support; these are separate registered contracts (Tables 28/37).

W2c / Q1. Brier harm magnitude and downstream rankings.

On canonical RM-Bench at budget 50, naive validation harms 12.75% of replays. Conditional mean Brier increase is 0.00927, maximum 0.04086; minimax has zero observed test harm with 2.75% switching (Table 11). RealDR ranking error is 0.17811 under reuse versus 0.17950 under census selection; rankings worsen/improve in 5.67%/6.33% of replays (Table 18). Thus Brier-harm reduction has direct evidence, while ranking changes are relatively small and mixed.

W3. Workflow evidence and PPI.

The pipeline has documented implementations. SAJA (ACL Industry 2026) fits a human-aligned head over fixed-rubric judge features. CJE, Sec. 5.2/Fig. 6, reuses a base-policy calibration on target policies; its failing policy has mean residual -0.31. PRECISE (AAAI 2026) uses isotonic score calibration within PPI-based production search evaluation. Calibration and PPI can coexist. CJE already audits mean transport; our decision certifies whether a particular repair lowers target Brier risk using disjoint, movement-allocated labels.

Q3. Naturally updated outputs.

Unmodified Qwen2.5-to-Qwen3 outputs show presentation changes and HumanEval calibration staleness. On all 164 HumanEval/500 MATH500 prompts, mean length changes 132.6-to-94.4/218.9-to-255.0 words. Five-judge OOF gains are 0.02690 [0.00895, 0.04542]/0.00553 [-0.00025, 0.01117] (95% CIs). Among unchanged-verifier prompts, paired length changes are -39.3 [-47.2, -31.8] words on 102 code prompts and +30.4 [23.3, 37.5] on 433 math prompts; content can still change.

With prompt-disjoint source/fit/cert/test pools, HumanEval at 20 fit plus about 22 certification labels gives switch/gain/test harm of 5.0%/0.00142/0.2% for original disagreement-range, 28.6%/0.00670/3.4% for exact-cell bounds on the same allocation, and 9.2%/0.00287/0.8% for IID betting under stability. Naive test harm is 12.6% (Table 21). These are cached-data gain-harm tradeoffs; held-out harm differs from the certified event.

Thank you for your response and new experiment. The new comparisons are helpful. The results on naturally updated model outputs also make the motivation more convincing. I still have some concerns on the practical benefits of the proposed method, particularly its limited ability to select useful repairs in some settings. Therefore, I am updating the score to 2 (borderline reject). Though, I do not object if all other reviewers are willing to accept this.

Also, please update the manuscript correpsondingly.

AuthorsOpenReview ↗

Thank you for your time

Thank you for taking the time to read our rebuttal and for updating your assessment. We appreciate your acknowledgment that the new comparisons and naturally updated model outputs strengthen the paper. We also recognize your remaining concerns about the practical benefits and repair-selection capability, and will revise the paper to make these limitations clearer. Thank you again for your thoughtful feedback and constructive discussion.

Reviewer_2yZwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when an LLM judge's calibration should be reused or updated after output distribution shifts. It proposes a finite-sample certification framework with disagreement-based audit allocation and theoretical guarantees. Experiments across RM-Bench, JETTS, RealDR, and other benchmarks evaluate the safety and effectiveness of the proposed approach.

Critical strengths
  1. Interesting problem formulation. I like that the paper distinguishes generator changes from actual calibration shifts, and focuses on whether recalibration is worth deploying.

  2. Strong theoretical and empirical analysis. The finite-sample guarantees are well motivated, and I appreciate the experiments covering both significant calibration drift and cases where reuse remains sufficient.

Critical weaknesses
  1. Limited deployment guarantees. The guarantees primarily apply to a fixed certification window rather than future data. The RealDR results (Table 15) illustrate this gap, with 24% held-out test harm when using exact-window selection.

  2. Limited generalizability of the JETTS results. The practical-equivalence findings rely on a restricted subset of jointly pair-eligible prompts (Table 10). It's unclear whether the same conclusion holds for the full output distribution.

Questions for authors
  1. Can the authors demonstrate useful certification power on future data? Table 28 suggests that estimating target state masses substantially reduces the switching rate.

  2. Would the JETTS conclusions remain consistent when evaluating a more representative set of generator outputs?

2 comments
AuthorsOpenReview ↗

Thank you for asking for evidence beyond the checked window and the selected JETTS subset. We added stable-population tests, chronological replay, and two broader JETTS audits.

W1 / Q1. Future-data power, state-mass uncertainty, and RealDR harm.

We obtain useful power under a stable target law; chronological future-block results are mixed. The population certificate requires independent target prompts, a fixed judge/prompt, and a joint law stable through deployment. Functions, support, fallback, and loss bounds are frozen on history/fit before certification covariates. IID paired-loss testing then certifies population gain directly, without estimating state masses.

For four canonical RM-Bench judges, 100 replays each, we fix 199 fit and 201 certification labels and a contract using verdict, length, and formatting. Switching rates are:

CertificateOne repairSix repairs
IID betting100%99.75%
IID empirical Bernstein87%37.5%
Stratified range + mass uncertainty1.5%0%

IID betting's gains are 0.06862/0.07043, with zero observed test harm. Fits/pools are shared; IID rules share queries. Stratification additionally uses 199 independent state-only prompts. Thus mass uncertainty materially reduces power here, while direct IID testing avoids that cost. With 50 fit plus 50 certification labels, single-repair IID betting already switches in 64%, gaining 0.01793 (Tables 4/32). The artifact records the support-freezing implementation correction and query-seed change, preserving the earlier version.

On the natural HumanEval update, 20 fit plus 22 certification labels give IID-betting switch/gain/test harm of 9.2%/0.00287/0.8%, versus 0%/0/0% for IID empirical Bernstein on the same repair and queries. Labels are cached verifier outcomes (Table 21).

For chronological evidence, we retain eight judges and 793 unique Arena prompts ordered by battle timestamps, with successive ten-day fit/certification/future-test blocks. At 50 fit plus 50 certification labels, IID-betting next-block gains are 0.000042, 0.000226, and -0.0000043. In the final fold, even a certification census gives future gain -0.000137 and 10.4% test harm. This retrospective result demonstrates the need for a cert-to-deployment stability or drift assumption (Table 38).

RealDR's 24% test harm is from census selection; the original range gate always reuses. Keeping three repairs and 15 fit tasks, without-replacement betting at 14/15 certification tasks switches in 15.7%, gains 0.000161, and has 1.7% test harm, with zero observed window non-improvement. This costs 116 report labels: four per task across 15 fit plus 14 certification tasks. Harmful census replays have mean normalized squared-loss increase 0.000704. The available tasks therefore identify a near-census checked-window budget; a future-task budget also requires independent-task evidence and population stability (Table 17).

W2 / Q2. Representative JETTS outputs.

The pairwise finding survives broader eligibility; full-prompt correctness evaluation reveals staleness. We added 31,278 judgments for Qwen3-4B and Llama-3.1-8B across all four registered transitions. Applying pair eligibility independently in each environment expands target pools from 550-726 to 880-1,100 prompts. The eight OOF refit gains range from -0.00071 to 0.00048; all eight refitted-bootstrap 90% intervals lie within +/-0.005.

A separate audit selects one answer per generator/prompt without reading correctness labels, covering all 2,407 released prompts across four tasks. Under a pointwise correctness rubric, all eight OOF gains have positive 95% lower endpoints, ranging from 0.00213 to 0.05719; three remain equivalent at +/-0.01. For Llama-8B to Qwen-32B, gains are 0.02115 for Qwen3-4B and 0.05719 for Llama-3.1-8B. Thus the boundary extends to broader pairwise eligibility, while full-prompt correctness calibration can become stale. Population and rubric differences prevent attributing this contrast solely to selection or style (Table 39). The original 20 selected-scope intervals are unchanged; the shared-prompt denominator is corrected from 1,885 to 2,407 (Table 12).

Replay rates measure cached-data sensitivity. Held-out test harm is distinct from the non-improving-deployment event controlled by certification.

I appreciate the authors for their response, which has addressed most of my concerns. I will maintain my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.