- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the continuous adaptation problem of Large Language Models (LLMs) as judges (LLM-as-a-Judge) when periodically receiving new validation feedback. It explores how to accurately estimate judgment reliability while preserving prior capabilities during error correction. The authors point out that traditional output layer confidence (such as linguistic self-assessment or Softmax probability) is often overconfident and fails to faithfully reflect the actual correctness of the judgment. To address this, the paper proposes VERA (Verdict-Conditioned Reliability Axis), which extracts the hidden layer activations at the last cue position before generating a judgment and calculates the direction of the mean difference between correct and incorrect judgments within each "predicted judgment" group. Cosine similarity is then used to generate a reliability score in a single forward propagation.
Based on the VERA signal, the authors constructed a closed-loop periodic adaptation framework:
-
Reliability-ranked corrective update: The unlikely loss penalty for incorrect decisions is weighted using the empirical cumulative distribution rank of the reliability scores within the group before the update, combined with weighted cross-entropy and asymmetric KL divergence regularization.
-
Periodic reliability refresh: With the selected representation layer fixed, each decision direction is periodically re-estimated and smoothed using exponential moving average (EMA) as the model is updated, while the logistic calibration mapping is refitted.
-
Reliability residual replay: High-priority historical samples are filtered and added to the cumulative replay buffer using the residual between the actual correctness of the updated model and the calibrated reliability estimate.
Based on Qwen3-8B and Qwen3-14B, the authors conducted experiments on public benchmarks (initialized with JudgeLM, adapted through 5 batches of Chatbot Arena feedback streams, and evaluated on MT-Bench-HJ, Auto-J, JudgeBench, LLMBar, and RewardBench 2) and a dedicated time-series binary classification auditing task, verifying the improvement of the proposed method in cross-benchmark generalization and key class recall.
Critical strengths
-
Lightweight and efficient decision conditional representation design: Compared to global correctness orientation (which treats all categories the same) or logistic regression (LR) probes that require additional gradient training, VERA effectively removes the interference of category-specific representation structures by calculating the mean difference vector based on predicted decision groups (Definition 2). At frozen checkpoints, this training-free probe significantly outperforms linguistic confidence, softmax probability, global orientation, and LR probes (Tables 3 and 6).
-
Elegant "pre-update correction + post-update replay" closed-loop mechanism: The "intra-group relative ranking " of the original scores is used for non-likelihood-gated weighting before the update, while the calibrated "absolute probability residual " is used for post-update memory replay selection. This focuses on marginal samples where the model's understanding differs from the actual results (i.e., "high-confidence misclassification" and "low-confidence correct classification"), which is both intuitive and logically sound.
Critical weaknesses
-
There is a significant disconnect between the core motivation (based on the frozen model) and the actual starting point of adaptation (based on the IFT model)
-
The paper establishes VERA's superiority over Softmax probabilities in the introduction and Figure 1 using the substantial AUROC advantage of the frozen checkpoint (i.e., before IFT). However, in the actual periodic adaptation process, whether for public or proprietary tasks, the initial model is always the checkpoint after instruction fine-tuning (IFT).
-
According to the data in Table 3, after IFT, the AUROC gap between VERA and Softmax narrowed dramatically (0.867 vs. 0.857 on proprietary 8B, and 0.869 vs. 0.866 on proprietary 14B). Even on MT-Bench-HJ (8B), the Softmax AUROC (0.732 and 0.736) in the IFT and DPO stages was actually higher than that of VERA (0.726 and 0.727).
-
Similarly, Table 6 shows that after JudgeLM IFT, the migration AUROC of the global direction (0.730) and LR probe (0.725) on MT-Bench-HJ is higher than that of the decision-conditional VERA (0.719); Table 8 also shows that when the JudgeLM-IFT checkpoint is fixed, the average AUROC of the source VERA on the five benchmarks (0.677) is lower than that of Softmax (0.689). Since periodic adaptation starts from the IFT checkpoint, and at this point VERA has no significant advantage over directly reading the Softmax probability or global direction, this weakens the argument for the necessity of extracting the decision-conditional hidden layer direction during the adaptation phase.
-
-
Lack of end-to-end adaptive ablation that completely replaces VERA with Softmax confidence
- In the ablation experiments in Table 9, the “Softmax residual” only replaces in with Softmax confidence during the replay phase. However, during the periodic update phase (algorithm 1, lines 4–10), it appears that VERA-based in-group ranking is still used to calculate the correction gate . Given that the correctness discrimination ability of Softmax after IFT in Table 3 is extremely close to that of VERA, the paper lacks an end-to-end comparative baseline that completely replaces the signals in the correction gate and the replay residual with Softmax probabilities.
Questions for authors
-
Regarding the gap between VERA and Softmax after IFT and end-to-end replacement experiments: Since periodic adaptation starts from the IFT checkpoint, and Table 3 shows that the AUROC of Softmax and VERA after IFT is almost the same (Softmax is even higher on MT-Bench-HJ 8B), what would the end-to-end performance be on proprietary tasks (Table 2) and public benchmarks (Table 1) if Softmax prediction probabilities (or within-group Softmax rank) were used to replace VERA scores throughout the entire periodic adaptation framework (including both the correction gating ordering of Equation 9 and the replay residuals of Equation 13)?
-
Regarding the root cause analysis of safety degradation: In Table 18, DPO only decreases by about 5–6% in safety compared to IFT, while VERA plummets by 24–31%. If the draw error weight in the public adaptation is restored from to , or if safe/rejected samples are retained proportionally in the initial JudgeLM replay buffer, can the decrease in Safety crash and Auto-J order consistency be avoided without targeted post-event fixes (Appendix D.4)?
4 comments
Q2. Do the missed-tie weight and the composition of initial replay explain the Safety and order-consistency losses?
We thank the reviewer for asking whether missed-tie weighting or initial replay composition explains the Safety and order-consistency losses. We ran two single-factor Qwen3-8B controls from the same IFT checkpoint. One reduces the missed-tie weight from three to one. The other proportionally stratifies the initial replay pool using source safety/refusal proxy labels while fixing its capacity and leaving later appends unchanged. Table C3 reports Safety accuracy and Auto-J response-order consistency in percent. Safety is evaluated on 450 items and four response rotations. Reducing the missed-tie weight improves Auto-J consistency from 86.57% to 87.64%, while Safety changes from 52.22% to 51.22%. Proportional replay changes Safety by +0.28 points and consistency by +0.14 points. The interventions therefore affect consistency and Safety differently, with the larger Safety decline persisting (Table C3). The full signal controls reach 52.31% Safety with VERA, 52.28% with Softmax, and 53.97% with Uniform, from 82.22% at IFT. VERA is 1.67 points below Uniform, alongside a larger decline shared by all three procedures. These results indicate that missed-tie weight and the composition of initial replay cannot explain the performance losses.
Table C3. Safety and consistency prevention controls.
| Configuration | Safety | Consistency |
|---|---|---|
| Starting IFT | 82.22 | 88.58 |
| missed-tie weight 3 | 52.22 | 86.57 |
| missed-tie weight 1 | 51.22 | 87.64 |
| proportional initial replay | 52.50 | 86.71 |
We sincerely thank the reviewer again for the thoughtful comments and constructive suggestions. We hope these clarifications resolve your concerns.
Thank you again for your careful review. As the discussion period draws to a close, we would appreciate your feedback on whether our rebuttal has resolved your concerns. We remain happy to clarify any outstanding points.
Thank you for providing the extra ablation results. While I appreciate the additional analysis, the performance gains of VERA over Softmax still seem modest, leaving my primary concern largely unresolved.
Thank you for recognizing the lightweight verdict-conditioned construction and the complementary roles of pre-update correction and post-update replay as strengths of our work. Please find our responses to your concerns and questions below.
W1.1. How are the frozen reliability results connected to the post-IFT adaptation setting?
We thank the reviewer for asking about the connection between these stages. We would like to clarify that Figure 1 evaluates correctness discrimination before IFT, while the adaptation experiments evaluate how reliability guides further learning from an IFT-trained judge. The post-IFT use is assessed through correction, replay, and refresh experiments. In the proprietary 8B component study, Table 11 removes refresh while retaining the remaining procedure. Forward CRITICAL recall falls from 72.7% to 67.7% and F1 from 66.1% to 64.7%. This supports maintaining the reliability estimator as the judge changes during adaptation in that configuration.
W1.2. What does the narrower or reversed post-IFT AUROC gap in Table 3 imply?
We thank the reviewer for asking about the checkpoint-dependent AUROC results. We would like to clarify that Table 3 evaluates how effectively each reliability score distinguishes correct from incorrect judge verdicts at pre-IFT, IFT, and DPO checkpoints. It measures the score’s discrimination ability, separately from the judge’s decision accuracy. Before IFT, VERA exceeds softmax AUROC in all four settings, with gains of 0.219 and 0.203 on the proprietary 8B and 14B judges, and 0.074 and 0.034 on MT-Bench-HJ, respectively.
After IFT, model probabilities provide a substantially stronger reliability signal. On 8B MT-Bench-HJ, softmax AUROC increases from 0.636 to 0.732, while VERA increases from 0.710 to 0.726. This is consistent with task-specific adaptation making output probabilities more informative. VERA still exceeds softmax on the proprietary task after IFT (0.867 vs. 0.857 at 8B and 0.869 vs. 0.866 at 14B) and after DPO (0.866 vs. 0.852 at 8B and 0.872 vs. 0.870 at 14B). Across the Table 3 settings, VERA’s AUROC ranges from 0.710 to 0.872. These results support an informative VERA reliability signal at every evaluated training stage, even as post-training strengthens the probability baseline.
W1.3. What do the Global, LR-probe, and source-transfer comparisons imply about verdict conditioning?
We thank the reviewer for asking about the role of verdict conditioning during adaptation. We would like to clarify that its purpose is to extract correctness information separately for each decision the judge makes. A global difference of means mixes within-verdict correctness differences with differences in verdict composition and representation. During adaptation, changes in the frequency of predicted verdicts can therefore change the global direction even without changes in within-verdict correctness. VERA addresses this ambiguity by extracting a correct–incorrect hidden-state direction for each predicted verdict at the selected layer. These directions can then be refreshed from groupwise means as the judge changes, without gradient-based probe fitting.
Table 6 provides post-IFT evidence for this construction on the proprietary task, where VERA achieves 0.867 AUROC versus 0.848 for Global and 0.862 for LR. The higher public post-IFT AUROC of Global and LR in Table 6, and softmax’s higher source-transfer mean in Table 8, concern pooled correctness ranking. They do not determine the within-verdict correction weights or replay residuals used during adaptation.
W2&Q1. Does the ablation test replacing VERA throughout the adaptation pipeline?
We thank the reviewer for asking whether the ablation covers both uses of the signal. We conducted a public Qwen3-8B comparison that replaces VERA with Softmax in both pre-update correction ranks and initial and subsequent replay selection. All other experimental settings are matched. Table C1 reports this full-pipeline ablation together with Uniform and Global controls, as mean percentage scores. LLMBar averages its four adversarial subsets.
Results show that full Softmax substitution yields 66.23% MT-Bench-HJ accuracy and 63.93% LLMBar accuracy, compared with 66.72% and 64.71% for VERA. This comparison directly tests replacing VERA throughout the public adaptation pipeline while retaining the corrective objective. Due to resource constraints during the rebuttal period, we have not yet completed the corresponding full-pipeline Softmax substitution on the proprietary task. We will complete this experiment and include its earlier-anchor and forward evaluation results in the final version.
Table C1. Full-pipeline signal ablations.