acceptodds
Under review as a conference paper at ICLR 2027

VERA: Verdict-Conditioned Reliability for Adaptive LLM Judges

Abstract

Accurately estimating judgment reliability is a central challenge in adapting LLM judges to newly verified feedback while preserving previously learned behavior. However, existing approaches often rely on output-level confidence, which can be overconfident and poorly aligned with judgment correctness. We propose **VERA**, a *Verdict-conditioned Reliability Axis* that estimates reliability from hidden activations by distinguishing correct from incorrect judgments within each predicted-verdict group. Using **VERA** as a control signal, we develop a **VERA-guided periodic adaptation framework** that integrates reliability-ranked corrective updates, reliability-residual replay, and periodic refresh of the reliability directions. After VERA-guided adaptation on Chatbot Arena, 8B- and 14B-parameter judges outperform the strongest baseline on each of four held-out public benchmarks, with relative gains of up to **23.01%**. The framework also improves focal-class recall by up to **16.1%** relative to the strongest adaptive baselines on a separate proprietary temporal auditing task.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-judge, reliability estimation, model adaptation, internal representations, selective replay

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1rrJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the continuous adaptation problem of Large Language Models (LLMs) as judges (LLM-as-a-Judge) when periodically receiving new validation feedback. It explores how to accurately estimate judgment reliability while preserving prior capabilities during error correction. The authors point out that traditional output layer confidence (such as linguistic self-assessment or Softmax probability) is often overconfident and fails to faithfully reflect the actual correctness of the judgment. To address this, the paper proposes VERA (Verdict-Conditioned Reliability Axis), which extracts the hidden layer activations at the last cue position before generating a judgment and calculates the direction of the mean difference between correct and incorrect judgments within each "predicted judgment" group. Cosine similarity is then used to generate a reliability score in a single forward propagation.

Based on the VERA signal, the authors constructed a closed-loop periodic adaptation framework:

  • Reliability-ranked corrective update: The unlikely loss penalty for incorrect decisions is weighted using the empirical cumulative distribution rank of the reliability scores within the group before the update, combined with weighted cross-entropy and asymmetric KL divergence regularization.

  • Periodic reliability refresh: With the selected representation layer ℓ⋆\ell^\star fixed, each decision direction is periodically re-estimated and smoothed using exponential moving average (EMA) as the model is updated, while the logistic calibration mapping is refitted.

  • Reliability residual replay: High-priority historical samples are filtered and added to the cumulative replay buffer using the residual ∣rθt(x)−Vt(x)∣\vert{}r_{\theta_t}(x) - V_t(x)\vert{} between the actual correctness of the updated model and the calibrated reliability estimate.

Based on Qwen3-8B and Qwen3-14B, the authors conducted experiments on public benchmarks (initialized with JudgeLM, adapted through 5 batches of Chatbot Arena feedback streams, and evaluated on MT-Bench-HJ, Auto-J, JudgeBench, LLMBar, and RewardBench 2) and a dedicated time-series binary classification auditing task, verifying the improvement of the proposed method in cross-benchmark generalization and key class recall.

Critical strengths
  • Lightweight and efficient decision conditional representation design: Compared to global correctness orientation (which treats all categories the same) or logistic regression (LR) probes that require additional gradient training, VERA effectively removes the interference of category-specific representation structures by calculating the mean difference vector based on predicted decision groups (Definition 2). At frozen checkpoints, this training-free probe significantly outperforms linguistic confidence, softmax probability, global orientation, and LR probes (Tables 3 and 6).

  • Elegant "pre-update correction + post-update replay" closed-loop mechanism: The "intra-group relative ranking q(x)q(x)" of the original scores is used for non-likelihood-gated weighting before the update, while the calibrated "absolute probability residual ΔVERA,t(x)\Delta_{\text{VERA}, t}(x)" is used for post-update memory replay selection. This focuses on marginal samples where the model's understanding differs from the actual results (i.e., "high-confidence misclassification" and "low-confidence correct classification"), which is both intuitive and logically sound.

Critical weaknesses
  • There is a significant disconnect between the core motivation (based on the frozen model) and the actual starting point of adaptation (based on the IFT model)

    • The paper establishes VERA's superiority over Softmax probabilities in the introduction and Figure 1 using the substantial AUROC advantage of the frozen checkpoint (i.e., before IFT). However, in the actual periodic adaptation process, whether for public or proprietary tasks, the initial model θ0\theta_0 is always the checkpoint after instruction fine-tuning (IFT).

    • According to the data in Table 3, after IFT, the AUROC gap between VERA and Softmax narrowed dramatically (0.867 vs. 0.857 on proprietary 8B, and 0.869 vs. 0.866 on proprietary 14B). Even on MT-Bench-HJ (8B), the Softmax AUROC (0.732 and 0.736) in the IFT and DPO stages was actually higher than that of VERA (0.726 and 0.727).

    • Similarly, Table 6 shows that after JudgeLM IFT, the migration AUROC of the global direction (0.730) and LR probe (0.725) on MT-Bench-HJ is higher than that of the decision-conditional VERA (0.719); Table 8 also shows that when the JudgeLM-IFT checkpoint is fixed, the average AUROC of the source VERA on the five benchmarks (0.677) is lower than that of Softmax (0.689). Since periodic adaptation starts from the IFT checkpoint, and at this point VERA has no significant advantage over directly reading the Softmax probability or global direction, this weakens the argument for the necessity of extracting the decision-conditional hidden layer direction during the adaptation phase.

  • Lack of end-to-end adaptive ablation that completely replaces VERA with Softmax confidence

    • In the ablation experiments in Table 9, the “Softmax residual” only replaces VV in ∣r−V∣\vert{}r - V\vert{} with Softmax confidence during the replay phase. However, during the periodic update phase (algorithm 1, lines 4–10), it appears that VERA-based in-group ranking is still used to calculate the correction gate g~t(x)\tilde{g}_t(x). Given that the correctness discrimination ability of Softmax after IFT in Table 3 is extremely close to that of VERA, the paper lacks an end-to-end comparative baseline that completely replaces the signals in the correction gate g~t(x)\tilde{g}_t(x) and the replay residual Δt(x)\Delta_t(x) with Softmax probabilities.
Questions for authors
  • Regarding the gap between VERA and Softmax after IFT and end-to-end replacement experiments: Since periodic adaptation starts from the IFT checkpoint, and Table 3 shows that the AUROC of Softmax and VERA after IFT is almost the same (Softmax is even higher on MT-Bench-HJ 8B), what would the end-to-end performance be on proprietary tasks (Table 2) and public benchmarks (Table 1) if Softmax prediction probabilities (or within-group Softmax rank) were used to replace VERA scores throughout the entire periodic adaptation framework (including both the correction gating ordering of Equation 9 and the replay residuals of Equation 13)?

  • Regarding the root cause analysis of safety degradation: In Table 18, DPO only decreases by about 5–6% in safety compared to IFT, while VERA plummets by 24–31%. If the draw error weight in the public adaptation is restored from ρ(x)=3\rho(x)=3 to ρ(x)=1\rho(x)=1, or if safe/rejected samples are retained proportionally in the initial JudgeLM replay buffer, can the decrease in Safety crash and Auto-J order consistency be avoided without targeted post-event fixes (Appendix D.4)?

4 comments

Thank you for recognizing the lightweight verdict-conditioned construction and the complementary roles of pre-update correction and post-update replay as strengths of our work. Please find our responses to your concerns and questions below.

W1.1. How are the frozen reliability results connected to the post-IFT adaptation setting?

We thank the reviewer for asking about the connection between these stages. We would like to clarify that Figure 1 evaluates correctness discrimination before IFT, while the adaptation experiments evaluate how reliability guides further learning from an IFT-trained judge. The post-IFT use is assessed through correction, replay, and refresh experiments. In the proprietary 8B component study, Table 11 removes refresh while retaining the remaining procedure. Forward CRITICAL recall falls from 72.7% to 67.7% and F1 from 66.1% to 64.7%. This supports maintaining the reliability estimator as the judge changes during adaptation in that configuration.

W1.2. What does the narrower or reversed post-IFT AUROC gap in Table 3 imply?

We thank the reviewer for asking about the checkpoint-dependent AUROC results. We would like to clarify that Table 3 evaluates how effectively each reliability score distinguishes correct from incorrect judge verdicts at pre-IFT, IFT, and DPO checkpoints. It measures the score’s discrimination ability, separately from the judge’s decision accuracy. Before IFT, VERA exceeds softmax AUROC in all four settings, with gains of 0.219 and 0.203 on the proprietary 8B and 14B judges, and 0.074 and 0.034 on MT-Bench-HJ, respectively.

After IFT, model probabilities provide a substantially stronger reliability signal. On 8B MT-Bench-HJ, softmax AUROC increases from 0.636 to 0.732, while VERA increases from 0.710 to 0.726. This is consistent with task-specific adaptation making output probabilities more informative. VERA still exceeds softmax on the proprietary task after IFT (0.867 vs. 0.857 at 8B and 0.869 vs. 0.866 at 14B) and after DPO (0.866 vs. 0.852 at 8B and 0.872 vs. 0.870 at 14B). Across the Table 3 settings, VERA’s AUROC ranges from 0.710 to 0.872. These results support an informative VERA reliability signal at every evaluated training stage, even as post-training strengthens the probability baseline.

W1.3. What do the Global, LR-probe, and source-transfer comparisons imply about verdict conditioning?

We thank the reviewer for asking about the role of verdict conditioning during adaptation. We would like to clarify that its purpose is to extract correctness information separately for each decision the judge makes. A global difference of means mixes within-verdict correctness differences with differences in verdict composition and representation. During adaptation, changes in the frequency of predicted verdicts can therefore change the global direction even without changes in within-verdict correctness. VERA addresses this ambiguity by extracting a correct–incorrect hidden-state direction for each predicted verdict at the selected layer. These directions can then be refreshed from groupwise means as the judge changes, without gradient-based probe fitting.

Table 6 provides post-IFT evidence for this construction on the proprietary task, where VERA achieves 0.867 AUROC versus 0.848 for Global and 0.862 for LR. The higher public post-IFT AUROC of Global and LR in Table 6, and softmax’s higher source-transfer mean in Table 8, concern pooled correctness ranking. They do not determine the within-verdict correction weights or replay residuals used during adaptation.

W2&Q1. Does the ablation test replacing VERA throughout the adaptation pipeline?

We thank the reviewer for asking whether the ablation covers both uses of the signal. We conducted a public Qwen3-8B comparison that replaces VERA with Softmax in both pre-update correction ranks and initial and subsequent replay selection. All other experimental settings are matched. Table C1 reports this full-pipeline ablation together with Uniform and Global controls, as mean percentage scores. LLMBar averages its four adversarial subsets.

Results show that full Softmax substitution yields 66.23% MT-Bench-HJ accuracy and 63.93% LLMBar accuracy, compared with 66.72% and 64.71% for VERA. This comparison directly tests replacing VERA throughout the public adaptation pipeline while retaining the corrective objective. Due to resource constraints during the rebuttal period, we have not yet completed the corresponding full-pipeline Softmax substitution on the proprietary task. We will complete this experiment and include its earlier-anchor and forward evaluation results in the final version.

Table C1. Full-pipeline signal ablations.

Signal throughout adaptationMT-HJLLMBar
VERA66.7264.71
Softmax66.2363.93
Uniform + random replay66.6565.06
Global direction66.5264.77

Q2. Do the missed-tie weight and the composition of initial replay explain the Safety and order-consistency losses?

We thank the reviewer for asking whether missed-tie weighting or initial replay composition explains the Safety and order-consistency losses. We ran two single-factor Qwen3-8B controls from the same IFT checkpoint. One reduces the missed-tie weight from three to one. The other proportionally stratifies the initial replay pool using source safety/refusal proxy labels while fixing its capacity and leaving later appends unchanged. Table C3 reports Safety accuracy and Auto-J response-order consistency in percent. Safety is evaluated on 450 items and four response rotations. Reducing the missed-tie weight improves Auto-J consistency from 86.57% to 87.64%, while Safety changes from 52.22% to 51.22%. Proportional replay changes Safety by +0.28 points and consistency by +0.14 points. The interventions therefore affect consistency and Safety differently, with the larger Safety decline persisting (Table C3). The full signal controls reach 52.31% Safety with VERA, 52.28% with Softmax, and 53.97% with Uniform, from 82.22% at IFT. VERA is 1.67 points below Uniform, alongside a larger decline shared by all three procedures. These results indicate that missed-tie weight and the composition of initial replay cannot explain the performance losses.

Table C3. Safety and consistency prevention controls.

ConfigurationSafetyConsistency
Starting IFT82.2288.58
missed-tie weight 352.2286.57
missed-tie weight 151.2287.64
proportional initial replay52.5086.71

We sincerely thank the reviewer again for the thoughtful comments and constructive suggestions. We hope these clarifications resolve your concerns.

Thank you again for your careful review. As the discussion period draws to a close, we would appreciate your feedback on whether our rebuttal has resolved your concerns. We remain happy to clarify any outstanding points.

Thank you for providing the extra ablation results. While I appreciate the additional analysis, the performance gains of VERA over Softmax still seem modest, leaving my primary concern largely unresolved.

Reviewer_T8aZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes VERA, a method uses the hidden states to estimate the reliability of the judge model, specfically, it calculates the cosine similarity between the hidden states of the prediction and the direction from the mean hidden states of incorrect predictions to the correct predictions. During training, it assigns training samples with lower prediction reliability with higher weights. Additionally, VERA score can be used to select replay training sample during periodic adaption. The experiments show that on four reward model benchmarks, the proposed method achieves higher score compared to the standard SFT and DPO training methods, and VERA can achieve higher recall rate compared to the baseline methods.

Critical strengths

VERA score can be a strong indicator of the judge model's confidence. The method is post-hoc, which doesn't require additional training.

Critical weaknesses

The models in the experiments are limited to Qwen3-8B and Qwen3-14B, and the gain Qwen3-8B experiment (Table 1) is also small. Experiments with a different LLM famility can be helpful to demonstrate the generalizabilty of the paper.

In table 2, while VERA achieives higher recall compared to the baselines, VERA doesn't improve the F1 scores, which means the precision of the model is damanged. While higher recall maybe considered better in this specific proprietary task, it does not support VERA is generally a better method compared to other baseslines.

The robust of VERA to different distributions remains unclear, the method requires calculating the mean hidden states of correct and incorrect prediction on a given training dataset, and it may not be able to transfer to out-of-distritbution exmples.

In table 3, VERA score doesn't provide better confidence estimation compared to prediction probability method for post-trained (SFT/DPO) models, this weakens the paper's major contribution.

Lack of the mathematical justification of why VERA can help trainig.

Questions for authors

Could the authors also report precision alongside recall and F1 in table 2?

Could the authors show that how does the AUROC looks like using mean hidden states calculated from datasets with different distributions? Additionally, how many data examples are needed for VERA score? Will VERA score be reliable when the labelled data is small? If you reduce the number of the training data, how does it compare to prediction probability?

Could the authors explain why VERA loses its advantage in confidence estimation on post-trained methods?

4 comments

Thank you for recognizing the usefulness of VERA as a confidence indicator and the practicality of estimating reliability post hoc without training an additional probe. Please find our responses to your concerns and questions below.

W1. Do the gains extend beyond Qwen, and how should the smaller 8B improvements be interpreted?

We thank the reviewer for asking about generalization beyond Qwen and the size of the 8B gains. We added Mistral-7B-Instruct-v0.3 experiments with the same direction construction, correction, replay, and refresh procedure. All methods start from one fixed JudgeLM-IFT checkpoint and undergo five feedback stages. Table B1 reports mean percentage scores ± sample standard deviation across repeated adaptation runs, with LLMBar averaged over its four adversarial subsets. Each adapted checkpoint is evaluated on all four main public benchmarks, as in the original Qwen experiments after Arena adaptation.

On Mistral, VERA exceeds DPO by 1.76 points on MT-Bench-HJ, 4.91 on Auto-J, 2.95 on JudgeBench, and 6.06 on LLMBar. For Qwen8B, the 0.16-point JudgeBench difference accompanies improvements from 64.57% to 66.69% on MT-Bench-HJ and 59.63% to 64.95% on LLMBar. At 14B, LLMBar improves from 57.05% to 70.18%. These results extend the evidence to a second model family, with the Mistral improvements measured relative to DPO across the four main benchmarks.

Table B1. Mistral-7B-Instruct-v0.3 adaptation.

MethodMT-HJAuto-JJudgeBenchLLMBarSafety
Starting JudgeLM-IFT60.7656.9744.0046.6466.89
Arena DPO60.52 ± 0.0957.88 ± 0.2543.62 ± 0.1648.45 ± 0.3544.67 ± 0.54
VERA62.28 ± 0.2662.79 ± 0.6946.57 ± 0.8654.51 ± 0.9131.43 ± 2.76

W2&Q1. Does improved CRITICAL recall also benefit F1, and what is the corresponding precision?

We thank the reviewer for asking for precision alongside recall and F1. We would like to clarify that our auditing task places greater emphasis on recall because reducing missed CRITICAL cases is a primary objective. This priority is reflected in the higher corrective weight assigned to such errors. Table B2 summarizes precision alongside the reported recall and F1 for the CRITICAL class on the earlier-anchor and forward sets. All values are percentages.

At 8B, VERA reaches 72.7% forward recall and 66.1% F1, exceeding the strongest adaptive-baseline values of 62.6% recall for DPO and 64.0% F1 for IFT. At 14B, it reaches 75.8% recall versus 69.7% for DPO, the strongest adaptive recall baseline. Against RAG, the highest-F1 baseline, it reduces the fraction of missed CRITICAL cases from 31.3% to 24.2%, approximately 22.7% in relative terms. The corresponding precision is approximately 63.5% versus 73.0%, and F1 is 69.1% versus 70.8%. Thus, the 8B benefit includes improved recall and F1. At 14B, the gain directly addresses our priority of reducing missed CRITICAL cases, with the accompanying precision and F1 trade-off reported explicitly.

Table B2. CRITICAL precision, recall, and F1.

Method8B earlier anchor8B forward14B earlier anchor14B forward
Frozen initial72.9 / 85.8 / 78.853.2 / 65.7 / 58.875.3 / 82.6 / 78.847.3 / 70.7 / 56.7
RAG76.2 / 82.0 / 79.061.6 / 61.6 / 61.677.1 / 77.9 / 77.573.0 / 68.7 / 70.8
IFT83.0 / 72.5 / 77.472.0 / 57.6 / 64.081.6 / 76.6 / 79.072.9 / 65.7 / 69.1
DPO74.0 / 84.1 / 78.753.9 / 62.6 / 57.976.1 / 81.5 / 78.753.9 / 69.7 / 60.8
VERA73.2 / 86.0 / 79.160.6 / 72.7 / 66.173.7 / 85.3 / 79.163.5 / 75.8 / 69.1

W3. Can VERA’s reliability directions transfer to out-of-distribution judgments?

We thank the reviewer for asking about robustness to distribution shifts. We would like to clarify that VERA can score judgments from a new distribution using directions constructed entirely on a source dataset. In Table 8, we construct directions and select the hidden layer on JudgeLM, then transfer the fixed estimator to held-out judging tasks without using their labels for construction or layer selection.

The transferred estimator achieves correctness AUROCs of 0.732 on MT-Bench-HJ, 0.711 on Auto-J, 0.616 on JudgeBench, and 0.565 on LLMBar. These results demonstrate that correctness-ranking information transfers beyond the construction distribution.

Q2. How do the construction distribution and labeled-data budget affect AUROC and the comparison with model probabilities?

We thank the reviewer for asking about construction data and small-label reliability. We varied the source-label budget from 32 to the full source set for Frozen and IFT Qwen3-8B, fixing the checkpoint, layer, and target predictions within each study. Table B3 reports JudgeLM-source VERA AUROC by label budget, with a separate Softmax baseline row for each checkpoint. Subsampled VERA scores are shown as mean ± standard deviation. Source holdouts are excluded from direction fitting, although JudgeLM data were available during IFT. The first fully covered tested budgets are 256 labels for Frozen and 512 for IFT. With 512 labels, Frozen VERA exceeds Softmax on MT-HJ (0.699 vs. 0.636), Auto-J, and JudgeBench, while LLMBar remains lower. After IFT, MT-HJ VERA AUROC increases from 0.7174 at 512 labels to 0.7346 at full budget, compared with Softmax’s 0.7325.

Table B3. JudgeLM-source reliability across label budgets.

StageMethodSource labelsMT-HJAuto-JLLMBarLLMBar
FrozenSoftmax—0.6360.6580.5300.530
FrozenVERA2560.655 ± 0.0230.657 ± 0.0340.507 ± 0.0170.507 ± 0.017
FrozenVERA5120.699 ± 0.0140.704 ± 0.0120.504 ± 0.0130.504 ± 0.013
FrozenVERA1,0240.719 ± 0.0200.720 ± 0.0080.505 ± 0.0090.505 ± 0.009
FrozenVERA2,0480.709 ± 0.0210.708 ± 0.0290.502 ± 0.0060.502 ± 0.006
FrozenVERA4,0960.715 ± 0.0130.720 ± 0.0080.506 ± 0.0070.506 ± 0.007
FrozenVERA8,1920.726 ± 0.0140.726 ± 0.0070.511 ± 0.0040.511 ± 0.004
FrozenVERA79,718 (full)0.7260.7290.5110.511
IFTSoftmax—0.7330.7220.5800.580
IFTVERA5120.717 ± 0.0240.698 ± 0.0340.568 ± 0.0080.568 ± 0.008
IFTVERA1,0240.720 ± 0.0110.700 ± 0.0180.569 ± 0.0060.569 ± 0.006
IFTVERA2,0480.723 ± 0.0230.702 ± 0.0330.568 ± 0.0070.568 ± 0.007
IFTVERA4,0960.727 ± 0.0130.707 ± 0.0180.567 ± 0.0060.567 ± 0.006
IFTVERA8,1920.732 ± 0.0050.714 ± 0.0070.568 ± 0.0020.568 ± 0.002
IFTVERA79,718 (full)0.7350.7160.5680.568

Table B4 repeats the study with MT-Bench construction data and includes the Frozen source comparison. Softmax is listed in a separate baseline row for each checkpoint. Its MT-Bench column evaluates the source holdout, while Auto-J, JudgeBench, and LLMBar use the same external predictions as Table B3. Complete coverage first occurs at 64 labels for both stages. Source choice affects discrimination: at 512 labels, Frozen Auto-J AUROC is 0.7044 with JudgeLM construction versus 0.5799 with MT-Bench construction. After IFT, MT-Bench construction instead improves Auto-J from 0.7023 to 0.7129 at 2,048 labels without updating the judge. Together, the results establish useful low-label settings and show how construction data can refine a fixed judge’s reliability estimate.

Table B4. MT-Bench-source reliability across label budgets.

StageMethodSource labelsMT-Bench holdoutAuto-JJudgeBenchLLMBar
FrozenSoftmax—0.6230.6580.5770.530
FrozenVERA640.586 ± 0.0630.561 ± 0.0640.513 ± 0.0230.521 ± 0.024
FrozenVERA1280.621 ± 0.0450.561 ± 0.0320.523 ± 0.0290.502 ± 0.012
FrozenVERA2560.633 ± 0.0450.571 ± 0.0600.535 ± 0.0280.494 ± 0.018
FrozenVERA5120.636 ± 0.0140.580 ± 0.0240.529 ± 0.0340.490 ± 0.016
FrozenVERA1,0240.679 ± 0.0390.620 ± 0.0390.541 ± 0.0330.494 ± 0.003
FrozenVERA2,0480.713 ± 0.0090.658 ± 0.0130.555 ± 0.0100.497 ± 0.006
FrozenVERA2,684 (full)0.7150.6610.5530.497
IFTSoftmax—0.7320.7220.6760.580
IFTVERA640.694 ± 0.0190.676 ± 0.0240.634 ± 0.0250.545 ± 0.007
IFTVERA1280.702 ± 0.0250.683 ± 0.0320.638 ± 0.0140.545 ± 0.004
IFTVERA2560.717 ± 0.0150.702 ± 0.0180.643 ± 0.0060.543 ± 0.004
IFTVERA5120.719 ± 0.0170.703 ± 0.0150.645 ± 0.0080.545 ± 0.003
IFTVERA1,0240.729 ± 0.0040.712 ± 0.0030.650 ± 0.0050.547 ± 0.003
IFTVERA2,0480.731 ± 0.0010.713 ± 0.0010.650 ± 0.0030.548 ± 0.001
IFTVERA2,684 (full)0.7310.7120.6500.548

W4&Q3. Why does the AUROC advantage narrow after instruction tuning, and does that invalidate its use during adaptation?

We thank the reviewer for asking why the confidence-estimation gap narrows after post-training. We would like to clarify that Table 3 measures how effectively each reliability score distinguishes correct from incorrect judge verdicts at pre-IFT, IFT, and DPO checkpoints. It does not measure the judge’s decision accuracy. Before IFT, VERA exceeds softmax in all four settings, with AUROC gains of 0.219 and 0.203 on the proprietary 8B and 14B judges, and 0.074 and 0.034 on MT-Bench-HJ, respectively.

After IFT, probability-based reliability becomes substantially stronger: on 8B MT-Bench-HJ, softmax AUROC increases from 0.636 to 0.732, while VERA increases from 0.710 to 0.726. This is consistent with task-specific adaptation making output probabilities more informative. VERA still exceeds softmax on the proprietary task after IFT (0.867 vs. 0.857 at 8B and 0.869 vs. 0.866 at 14B) and after DPO. Across the Table 3 settings, VERA’s AUROC ranges from 0.710 to 0.872, supporting an informative reliability signal at every evaluated training stage. Table B5 reports supplementary post-IFT AUROCs as VERA / softmax, using a fixed JudgeLM construction source and hidden layer per model and identical target predictions for the two scores.

Table B5. Post-IFT correctness AUROC.

Post-IFT judgeMT-HJAuto-JJudgeBenchLLMBar
Qwen3-8B0.735 / 0.7330.716 / 0.7220.674 / 0.6760.568 / 0.580
Qwen3-14B0.736 / 0.7350.703 / 0.7020.660 / 0.6720.567 / 0.572
Mistral-7B0.721 / 0.7230.698 / 0.7110.626 / 0.6260.508 / 0.529

W5. What mathematical reasoning supports the direction construction and reliability-weighted training?

We thank the reviewer for asking for the mathematical basis of VERA and its training rule. We would like to clarify that the direction has a direct mean-separation interpretation. Within verdict cc, define δc=E[H∣correct,c]−E[H∣incorrect,c]\delta_c = \mathbb{E}[H \mid \mathrm{correct}, c] - \mathbb{E}[H \mid \mathrm{incorrect}, c]. For a unit vector vv, Cauchy–Schwarz gives v⊤δc≤∥δc∥2v^\top \delta_c \leq \lVert \delta_c \rVert_2, attained by v=δc/∥δc∥2v = \delta_c / \lVert \delta_c \rVert_2 for nonzero δc\delta_c. Conditioning estimates this contrast within a decision category and removes the between-verdict mixture term in a global difference. This identity concerns raw projection means. It extends exactly to cosine mean separation when hidden norms are constant within the group. The directions are fitted directly from groupwise means.

For correction, the mean-one gate is proportional to 1+4(1−ri)21 + 4(1 - r_i)^2, with fixed pre-update rank rir_i. At equal error-type weight and probability gradient, a rank-zero error receives five times the gate multiplier of a rank-one error. At a fixed checkpoint and row distribution, define ai=∇Ltarget⊤uia_i = \nabla L_{\mathrm{target}}^\top u_i for ungated corrective gradient uiu_i. The first-order target-loss difference from uniform correction is −ηCov⁡(gi,ai)-\eta \operatorname{Cov}(g_i, a_i), making positive covariance the local condition for improvement. We checked the analytical gradient against autograd in 12 cases. Tables 9 and 11 and the matched public substitutions evaluate the subsequent effects of replay, refresh, and signal choice over repeated updates.


Thank you for the thoughtful questions and for helping us make the practical and mathematical basis of the method more explicit. We hope these clarifications resolve your concerns.

Thank you again for your constructive comments. As the discussion period draws to a close, we would be grateful for your feedback on whether our responses have addressed your concerns. We remain happy to provide further clarification.

Reviewer_oeJfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VERA, a hidden-state reliability estimator for LLM judges that constructs separate correctness directions for each predicted verdict. These scores guide corrective training, replay selection, and periodic reliability refresh as new verified judgments become available. The authors evaluate Qwen3-8B and Qwen3-14B on four public judging benchmarks and a proprietary temporal auditing task. The complete framework improves mean performance over the evaluated IFT and DPO baselines on the four principal public benchmarks, with particularly substantial gains on LLMBar. Additional experiments examine reliability estimation, adaptation components, and Safety degradation and repair.

Critical strengths

I think the basic idea is quite reasonable: judgments with different predicted verdicts may have different representation structures, so estimating their reliability separately seems well motivated. I also like the simplicity of the difference-of-means construction and the attempt to use reliability to improve subsequent learning. This makes the work potentially useful beyond estimating whether an individual judgment is correct.

The empirical improvements are interesting, especially the 14B LLMBar result, which increases from 57.05% under the strongest evaluated baseline to 70.18%. Evaluating the same adapted checkpoint across several benchmarks gives these results more weight. I also appreciate that the authors include meaningful ablations and discuss negative results, including the Safety and response-order consistency losses.

Critical weaknesses

My main concern is that I had a hard time distinguishing the contribution of VERA itself from that of the overall training procedure. From my understanding, several things change together, including corrective unlikelihood training, error weighting, KL regularization, and replay. Tables 9 and 11 help, but removing rank-UL also removes the corrective term, and the replay-selection comparison is limited to the proprietary 8B setting. Could the authors help clarify whether the public DPO baseline receives the same Arena examples, response-order augmentation, and feedback schedule, and how ties are handled? More importantly, are otherwise matched public comparisons available using uniform or softmax-based corrective weights and simpler replay selection, or global rather than verdict-conditioned directions? I am asking for clarification of the existing protocol and, if available, additional evidence isolating the mechanism. This seems particularly relevant because VERA does not consistently improve reliability discrimination after IFT, for example, 0.726 versus 0.732 AUROC for softmax on 8B MT-Bench-HJ. I may be missing an important distinction here, and I would be happy if the authors could help me understand it.

My other concern is how to interpret the retention results. Safety accuracy drops from 81.43% to 50.36% at 8B and from 81.09% to 57.29% at 14B, which seems substantial given the motivation of preserving previously learned behavior. The repair experiment is encouraging, but my understanding is that it uses additional labeled target examples and produces different checkpoints. Could the authors clarify how they view this trade-off in the overall assessment of the method? Also, are Auto-J, JudgeBench, and LLMBar results available for the repaired checkpoints in Table 13? Those results would help me understand whether Safety recovery preserves the broader improvements. I appreciate that the authors explicitly report this issue, but I am currently unsure how broadly the retention benefits should be interpreted.

Questions for authors

Please respond to the questions and concerns raised in the main review above.

3 comments

Thank you for recognizing the motivation and simplicity of the verdict-conditioned estimator, as well as the value of testing the same adapted checkpoint across benchmarks. Please find our responses to your concerns and questions below.

Q1.1. Do DPO and VERA receive matched data, response orders, feedback schedules, and tie handling?

We thank the reviewer for asking about the comparison protocol. We would like to clarify that the public DPO baseline and VERA share the same Arena examples, response-order augmentation, and feedback schedule. For tie handling, we follow prior DPO work on Chatbot Arena data and exclude tie-labeled examples during the DPO stage.[1]

Q1.2. What do the Uniform and Softmax controls show about reliability-guided correction and replay?

We thank the reviewer for asking how to isolate VERA’s contribution to correction and replay. We conducted an experiment comparing VERA with Uniform and Softmax controls on Qwen3-8B while retaining the same IFT checkpoint, corrective objective, and training budgets. Uniform uses a constant confidence gate and uniform sampling from the replay pool. Softmax supplies probability-based ranks and residuals for correction and replay. Given the computational resource constraints for this rebuttal, we report the new public comparison only on MT-Bench-HJ in Table A1. Scores are mean percentages.

Results indicate that VERA improves MT-Bench-HJ accuracy over Softmax by 0.49 percentage points, while Uniform remains competitive. This comparison isolates the joint choice of the correction signal and replay selector while retaining the corrective objective. Table 9 separately isolates replay selection by fixing the corrective updates, replay budget, and other training settings on the proprietary 8B task: VERA-residual replay achieves 72.7% forward CRITICAL recall and 66.1% F1, compared with 69.7% recall and 63.0% F1 for both uniform sampling and Softmax-residual selection.

Table A1. Uniform and Softmax controls on MT-HJ.

MetricVERASoftmaxUniform
MT-HJ66.7266.2366.65

Q1.3. What does the matched Global-direction control show about verdict conditioning?

We thank the reviewer for asking whether verdict conditioning contributes beyond a global direction. We conducted an experiment comparing VERA with a Global control on Qwen3-8B. Global replaces the verdict-conditioned directions with one shared correctness direction throughout correction and replay, while retaining the within-verdict rank operator, IFT checkpoint, corrective objective, and training budgets. Given the computational resource constraints for this rebuttal, we report this comparison only on MT-Bench-HJ. The results are shown in Table A2.

Results indicate that VERA outperforms Global in mean MT-Bench-HJ accuracy, achieving 66.72% versus 66.52%, an absolute improvement of 0.20 percentage points. Verdict conditioning constructs a separate correctness contrast within each predicted-verdict group, providing category-specific reliability directions for correction and replay.

Table A2. VERA and Global controls on MT-HJ.

MetricVERAGlobal
MT-HJ66.7266.52

Q1.4. Why use VERA after IFT when softmax has similar or higher correctness AUROC?

We thank the reviewer for asking about VERA’s post-IFT reliability. We would like to clarify that Table 3 evaluates each reliability score’s AUROC for distinguishing correct from incorrect judge verdicts at Frozen (pre-IFT), IFT, and DPO checkpoints. It evaluates score discrimination, separately from the judge’s decision accuracy. Before IFT, VERA exceeds softmax AUROC in all four settings, with gains of 0.219 and 0.203 on the proprietary 8B and 14B judges, and 0.074 and 0.034 on MT-Bench-HJ, respectively.

After IFT, model probabilities provide a substantially stronger reliability signal: on 8B MT-Bench-HJ, softmax improves from 0.636 to 0.732, while VERA improves from 0.710 to 0.726. This is consistent with task-specific adaptation making output probabilities more informative. VERA still exceeds softmax on the proprietary task after IFT (0.867 vs. 0.857 at 8B and 0.869 vs. 0.866 at 14B) and DPO (0.866 vs. 0.852 at 8B and 0.872 vs. 0.870 at 14B). Across the Table 3 settings, VERA’s AUROC ranges from 0.710 to 0.872, supporting informative correctness ranking at every evaluated training stage.

Q2.1. How much does targeted repair improve Safety under matched supervision budgets?

We thank the reviewer for asking how additional supervision supports Safety recovery. We compared VERA-selected and uniformly selected repair under matched budgets at both model scales. These repaired checkpoints come from a separate archived experiment and are distinct from the exact historical Table 13 weights. The results are shown in Table A3 as percentage scores.

Results indicate that VERA-selected repair substantially improves Safety accuracy, from 45.56% to 84.58% at 8B and from 52.36% to 78.89% at 14B. Under the same repair budget, it exceeds uniform selection by 9.86 and 11.81 percentage points, respectively. The repaired 8B judge exceeds its initial IFT Safety accuracy of 80.42%, while the repaired 14B judge approaches its initial 80.28%. These results demonstrate effective targeted recovery.

Table A3. Safety recovery with matched repair budgets.

ScaleInitial IFTBefore repairRandom repairVERA repair
8B80.4245.5674.7284.58
14B80.2852.3667.0878.89

Q2.2. Does Safety repair preserve performance on the other judging benchmarks?

We thank the reviewer for asking whether Safety recovery preserves the broader judging improvements. We evaluated the same unrepaired, uniformly repaired, and VERA-repaired checkpoints from Q2.1 on the complete public evaluation sets, alongside their initial IFT references. Table A4 reports percentage scores for MT-Bench-HJ, Auto-J, JudgeBench two-order accuracy, and LLMBar adversarial macro-accuracy.

Results indicate that Safety recovery maintains JudgeBench accuracy within 0.57 percentage points of the unrepaired judge at both scales and improves LLMBar from 67.66% to 72.21% at 8B and from 70.22% to 75.28% at 14B. Both benchmarks retain their gains over the initial IFT references. MT-Bench-HJ also remains above its initial IFT reference at both scales, although it decreases by 1.65 and 2.56 points from the unrepaired checkpoint. Auto-J decreases by 6.75 and 4.81 points. The results therefore support substantial Safety recovery with preserved JudgeBench and LLMBar gains, alongside task-dependent costs on MT-Bench-HJ and Auto-J.

Table A4. Cross-benchmark performance after Safety repair.

ScaleMetricInitial IFTBefore repairRandom repairVERA repair
8BMT-HJ64.3466.5965.0464.93
8BAuto-J58.4158.7656.1852.01
8BJudgeBench56.2959.1458.8659.14
8BLLMBar55.5867.6673.5172.21
14BMT-HJ64.0867.3965.5964.83
14BAuto-J60.7062.3658.0557.54
14BJudgeBench58.8663.4364.8662.86
14BLLMBar56.3170.2275.9175.28

We sincerely thank the reviewer again for the thoughtful comments and constructive suggestions. We hope these clarifications resolve your concerns.

References

[1] Ivison, H., Wang, Y., Liu, J., Wu, Z., Pyatkin, V., Lambert, N., Smith, N. A., Choi, Y., & Hajishirzi, H. (2024). Unpacking DPO and PPO: Disentangling best practices for learning from preference feedback. Advances in Neural Information Processing Systems, 37, 36602–36633.

Thank you again for your thoughtful feedback. As the discussion period draws to a close, we would appreciate your feedback on whether our rebuttal has addressed your concerns. We remain happy to answer further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Authors' remark for reviewers and ACs

We thank reviewers oeJf, T8aZ, and 1rrJ for their constructive feedback and for recognizing the motivation, simplicity, and practical value of VERA. We also sincerely thank the Area Chair for their time and consideration.

We summarize the main experimental additions and clarifications below:

  1. Reliability before and during adaptation. VERA exceeds Softmax in all four pre-IFT settings and remains informative after IFT, with higher correctness AUROC on the proprietary task at both model scales. Its verdict-specific directions are constructed from hidden-state means and can be refreshed without training an additional probe.
  2. Evidence for reliability-guided learning. We added matched public signal controls and clarified the separate contributions of replay and refresh. With correction and replay budgets fixed, VERA replay improves forward CRITICAL recall from 69.7% to 72.7% and F1 from 63.0% to 66.1% over uniform and Softmax selection, while removing refresh reduces recall to 67.7%.
  3. Generalization beyond Qwen. We added Mistral-7B-Instruct-v0.3 experiments starting from a common IFT checkpoint. VERA exceeds DPO across all four main public benchmarks, including gains of 4.91 percentage points on Auto-J and 6.06 on LLMBar.
  4. Useful reliability with limited construction labels. We added construction-source and label-budget studies for Frozen and IFT Qwen3-8B. With 512 JudgeLM labels, Frozen VERA achieves AUROCs of 0.699 on MT-Bench-HJ and 0.704 on Auto-J, compared with Softmax’s 0.636 and 0.658, demonstrating useful reliability from limited labels in these settings.
  5. Targeted Safety recovery. With additional verified feedback and matched repair budgets, VERA-selected repair raises Safety accuracy from 45.56% to 84.58% at 8B and from 52.36% to 78.89% at 14B. At both scales, this substantial Safety recovery is accompanied by JudgeBench accuracy remaining within 0.57 percentage points of its pre-repair level and further gains on LLMBar.

These findings strengthen VERA’s central contribution: a lightweight, verdict-specific reliability mechanism that connects correctness estimation to corrective learning, selective replay, and periodic refresh. We sincerely thank the reviewers and the Area Chair again for helping us strengthen the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.