- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies when to keep or update an LLM judge’s calibration after model outputs change. It fits updates on one data split and checks them on another, using disagreement with the original calibration map to guide labeling. Experiments on controlled shifts, generator changes, and expert-scored reports show fewer harmful updates, but limited ability to detect smaller improvements.
Critical strengths
-
Separate fitting and checking, grouped data splits, and thresholds chosen in advance make the evaluation careful. The paper clearly distinguishes its statistical guarantee from performance on held-out data.
-
The experiments show that ordinary validation can select harmful updates, giving a useful motivation for the method. The JETTS results also show that changing generators does not always require recalibration.
Critical weaknesses
- The benefit of the proposed allocation method is unclear.The minimax allocation performs almost the same as the simpler alternative on the main benchmark (Table 25). This leaves limited evidence that its added complexity provides practical value.
- Practical usefulness remains limited at the tested budgets.The conservative gates never select an update on RealDR, and Qwen3.8 remains unresolved through 400 labels. In the main setting, checking uses labels for 200 of 232 groups, so substantial label savings are not demonstrated. The guarantee also covers the checked window and does not by itself ensure improvement on future traffic.
Questions for authors
-
What does the method add beyond existing fitting and safety-testing approaches? How does it compare with a valid variance-based bound under the same sampling setup and safety guarantee?
-
What label budgets would allow useful updates on RealDR and Qwen3.8, and what assumptions are needed to extend the guarantee to future traffic?
2 comments
Looking Forward to Your Feedback
Dear Reviewer VKPt, thank you again for your constructive feedback. We have provided detailed responses to your comments and would appreciate it if you could take a look at our rebuttal. Please let us know if any concerns remain; we would be happy to clarify further.
Thank you for asking us to isolate design benefits and actionable budgets.
W1 / Q1. Allocation value and a valid variance-based comparison.
The canonical menu supports the simpler envelope: at budget 200, envelope/minimax switching is 11%/12%, with gains 0.0034/0.0040. For fixed states and capacities, the envelope is exactly minimax if one repair attains the largest movement in every state. A controlled 42-state heterogeneous-movement experiment requires up to 1.59 times as many envelope labels to match the minimax radius. This identifies a potential benefit regime, while the canonical data show little solver advantage (Tables 29/30).
For practical allocation value, seven-judge LLMBar fixes one repair, delta=0.05, and 25 fit plus 25 certification labels. Disagreement-range switches in 76.43%, versus 53.00% for SRS WoR betting, with gains 0.03165/0.02263 and zero observed window/test harm. Holding exact hypergeometric certification fixed, disagreement versus target-mass allocation switches in 53.71%/14.86%, favoring disagreement in all seven judges. This isolates allocation at 25/112 queried prompts (Table 4).
SRS means simple random sampling; WoR denotes sampling without replacement. For the variance-based comparison, identical disagreement queries give:
All target the same finite-window gain at level 0.05. Cellwise EB/exact pay simultaneous statewise control; the earlier range floor is removed (Table 36). With 25 fit labels fixed, RM-Bench style-contract switching reaches 82.2% at 50 disagreement-range certification labels and 82.4% at 100 SRS-betting labels: total budgets 75 versus 125. Betting is stronger in verdict-only and high-budget cases (Figure 3). We position the design within LTT and classical allocation theory.
W2 / Q2. RealDR and Qwen3.8 budgets.
Stronger certificates enable checked-window updates under both original menus. Qwen3.8's verdict-only six-repair menu at budget 400 (200 fit, 200 certify) gives:
Both have zero observed window non-improvement. On identical disagreement queries, replacing range with EB-Serfling raises switching from 0% to 67%. This is a certificate effect at 200/232 queries; the low-budget comparison above supplies separate label-saving evidence (Table 15).
RealDR keeps three repairs and 15 fit tasks. Betting is inactive through 12/15 certification tasks; at 14/15 it switches in 15.7%, gains 0.000161, and has 1.7% test harm. Census test harm is 24%. Selected partial-window replays pass deterministic worst-case completion, so action arises near census. The 15 fit plus 14 certification tasks cost 116 report labels (four per task). These 40 tasks identify a checked-window budget; future-task budgeting additionally needs independent-task evidence under population stability (Table 17).
W2 / Q2, continued. Future-traffic assumptions and measured power.
Population certification requires independent target prompts, a fixed judge/prompt, and a joint law stable from certification through deployment. Functions/support/fallback are frozen on history/fit before certification covariates. IID paired-loss tests then avoid state-mass estimation.
At 199 fit plus 201 certification labels, canonical style-contract IID betting switches in 100%/99.75% for one/six repairs, gains 0.06862/0.07043, and has zero observed test harm (Table 32). Qwen3.8's verdict-only six-repair population grid switches in 7%/80% at 400/597 actual labels, with gains 0.00054/0.00389 and test harm 0%/12%. Natural HumanEval at 20 fit plus 22 certification labels gives 9.2%/0.00287/0.8% (Table 21).
Chronological Arena replay retains eight judges and 793 unique prompts. At 50 fit plus 50 certification labels, next-block gains are 0.000042, 0.000226, and -0.0000043. The final-fold certification census still gives IID-betting future gain -0.000137 and 10.4% test harm (Table 38). Thus subsequent joint-law changes need a drift allowance or another audit. Test harm differs from the certified event; repeated seeds measure cached-data sensitivity.