- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates whether evaluation criteria that agree on existing responses are equally sensitive to changes induced by LLM training. It argues that high agreement does not guarantee similar coverage of training-induced policy changes. The main contributions are: (1) introducing the agreement–coverage gap and a geometric method for computing criterion overlap before training, (2) providing controlled experimental evidence that removing a checker-aligned component from the training signal reduces the checker's detection of policy changes while largely preserving the driving criterion's signal, and (3) demonstrating that pre-training geometric overlap predicts the relative responses of different evaluation criteria to training-induced changes. For validation, the authors train Qwen2.5 models using DPO and compare original and modified training signals at matched KL divergence.
Critical strengths
- The paper highlights that agreement between evaluation criteria on existing responses does not necessarily imply similar sensitivity to training-induced changes. Its geometric formulation provides a way to estimate this sensitivity before training, offering a potentially useful perspective on evaluation reliability.
- The authors compare DPO training runs at matched KL divergence and demonstrate reduced checker detection across all five criterion pairs while largely preserving the driving criterion's signal. Additional placebo controls, model-scale experiments, and external reward signals strengthen the evidence that the observed effect is associated with the targeted geometric intervention rather than simply changing the training direction.
Critical weaknesses
-
The paper measures evaluator coverage through agreement between linear criterion readouts and training-induced changes in response probabilities. However, it is unclear whether this metric captures a meaningful failure of evaluation. The treatment deliberately removes the component of the training signal aligned with criterion B, so reduced agreement with B may simply reflect fewer changes in the direction preferred by B, rather than B failing to detect changes it should recognize. To establish practical evaluator blindness, the experiments would need independent evidence of meaningful changes in B-related quality that evaluator B fails to detect. The independent generated-response evaluations do not establish such a selective failure: both A and B improve relative to the base model, and their changes are not significantly different. Thus, while the experiments support the proposed geometric relationship, its implications for practical evaluation reliability remain unclear.
-
Although geometric overlap correlates with relative criterion responses within the controlled experiments, it does not reliably predict detection-loss magnitudes across different criterion pairs. Furthermore, the main experiments are restricted to linear readouts and DPO, with relatively few independent training runs. These limitations raise questions about the broader applicability and practical usefulness of the proposed approach.
-
The paper is unnecessarily difficult to read and follow. The distinction between the quantities actually measured and their broader implications is also not always clearly communicated. The paper would benefit from substantial improvements in clarity, organization, and language fluency, as the current presentation makes it difficult to identify the precise contributions and assess their significance.
Questions for authors
-
The paper defines coverage through agreement between criterion rankings and training-induced probability changes. However, reduced agreement with criterion B may simply reflect fewer changes in the direction preferred by B, rather than a failure of B to detect meaningful changes. While the authors acknowledge that their generated-response experiments do not establish selective checker blindness, could they clarify why reduced coverage, as defined in this work, should be considered a concern for practical evaluation reliability? What distinguishes low coverage from a legitimate difference between the optimized and monitored criteria?
-
The reported confidence intervals for and are question-bootstrapped and conditional on fixed checkpoints, so they do not account for run-to-run variation. This seems important given that the two seeds of pair 2 differ beyond their question-level intervals (§4.5), while pairs 1 and 5 have only one matched seed each. Appendix V also reports substantial KL variation between same-seed reruns, although this does not directly measure variation in detection loss. Can the authors quantify the uncertainty in or per-pair across independent training runs, potentially through additional seeds or a sensitivity analysis? In particular, does the evidence support a reliably positive when training-run variability is considered?
4 comments
Response to Reviewer 8cfm (2/2)
W2: Magnitude prediction, training runs, scorers, readout class and optimiser.
"Although geometric overlap correlates with relative criterion responses within the controlled experiments, it does not reliably predict detection-loss magnitudes across different criterion pairs. Furthermore, the main experiments are restricted to linear readouts and DPO, with relatively few independent training runs."
Overlap empirically predicts the ordering of criterion responses within a comparison (ρ=0.81 pooled; ρ=0.71 under ordinary reward-model DPO), but not calibrated magnitudes: pair 3 was predicted at +0.0247 and measured at +0.0029, and pair 4 changes sign between early training and matched KL (Table 4). The Abstract and §5 now state both the ordering and this limit.
Training runs: nine seeds and 37 cells (E1, previous response).
Scoring and policy: the loss holds when B's ranking is replaced by GPT-4, a DeepSeek-V4-Pro judge or an ArmoRM head (E3; under these scorers A declines similarly, E10), at 0.5B (four pairs; App. H) and 7B (one pair; App. L), and on Qwen2.5-1.5B-Instruct:
| ID | Question | What was done | Result | Revised paper |
|---|---|---|---|---|
| E8 | Does it hold for an instruction-tuned policy? | Full pipeline rebuilt on Qwen2.5-1.5B-Instruct; seeds 2–4 | 9/9 cells positive, +0.079 [0.075,0.083]; directional > placebo in 9/9 | §4.5; App. AI |
Readout class and optimiser: the pre-specified nonlinear probes failed their eligibility gate, and IPO/SimPO produced no admissible matched comparison (App. J); these settings are outside the matched evidence.
W3: Presentation.
"The paper is unnecessarily difficult to read and follow. The distinction between the quantities actually measured and their broader implications is also not always clearly communicated."
To make the paper easier to follow, we rewrote the Abstract and Introduction around a concrete use case, added a worked example and an equation showing where the measured difference comes from, checked every reference against its published version, and marked all revisions in blue. The table lists each change with its location:
| Change | Reading difficulty it addresses | Location |
|---|---|---|
| Worked example on a real FLASK response pair | How a readout's ranking, the policy's margin change and the agreement statistic relate | §3, p. 4 |
| Eq. 4 with the discordant-pair breakdown | Which response pairs produce ΔB−ΔA (the 27% ranked differently) | §3; App. AG |
| Table 3 | Collects seed and external-score replication of the fixed-response measurement, apart from generated-text results (App. AH) | p. 7 |
| Rewritten opening and explicit measurement boundaries | Concrete proxy-monitor scenario; separates static grading, fixed-response coverage and generated-output quality | Abstract; §1; §5 |
| One wording for overlap | "Computable before training; predicts relative responsiveness, not magnitudes" | Abstract; §5 |
| Overview table of Apps. AB–AJ | Which question each new analysis answers | App. AA |
Summary. Together, the new runs, scorers and policy reinforce the core claim that static agreement does not ensure the same training response, and the revision states each measurement at the level at which it was made. If these additions address your main concerns, we would be grateful if you would consider reflecting this in your assessment.
I appreciate the new experiments, particularly the additional seeds and independent evaluator analyses. These address my concerns about the robustness of the measured agreement–coverage gap. However, I remain uncertain about the practical significance.
Although the generated-response experiments demonstrate that the intervention affects actual model behavior, they do not establish a practical monitoring failure. Both A and B exhibit measurable quality changes, and the evaluators detect these changes. Thus, it remains unclear whether the reduced coverage measured on fixed responses translates into a situation where a proxy evaluator fails to identify a meaningful change in model quality. Furthermore, Appendix X reports that the tested checker-selection strategies do not outperform random selection. Thus, while the geometric relationship is empirically supported, it remains unclear whether the proposed metric identifies meaningful evaluation failures or improves evaluator selection in practice.
Thank you for your thoughtful follow-up, and for confirming that the additional seeds and independent evaluator analyses address your concerns about the robustness of the measured agreement–coverage gap.
Practical significance: reduced coverage measured on fixed responses (§4.2–4.3, Tables 1–3). Static agreement alone does not ensure that a proxy evaluator will reliably track a given training update. In practice, a proxy evaluator B is often trusted to monitor A-driven training because it agrees with A on existing responses. Tables 1–2 show where reduced coverage measured on fixed responses matters for evaluation reliability: logical efficiency (A) and conciseness (B) agree on 65% of fixed pairs, yet at matched KL, A tracks the control and treated updates equally (0.645 vs 0.644) while B falls from 0.539 to 0.500. Relying on B alone makes the treated update look much weaker on a shift that A tracks equally well. Independent evaluators confirm this checker loss (Table 3), and ordinary reward-model DPO already moves criteria unevenly in the order of their pre-training overlap (ρ=0.71, §4.2).
Generated-response experiments and checker-selection strategies (Apps. AH and X).
- Generated-response experiments (App. AH): Both A and B exhibit measurable quality changes and the evaluators detect these changes in aggregate. Testing whether a proxy evaluator fails to identify a meaningful change in model quality on generations is a distinct question that requires predefined quality-drop and alert thresholds.
- Checker-selection strategies (App. X): At k=1, ranking by relative sensitivity alone placed conciseness first in all five pairs despite near-chance detection, so it did not outperform random selection. Rather than relying on a single static ratio, improving evaluator selection in practice requires checking baseline responsiveness alongside pre-training overlap in a short pilot update on held-out pairs (§5).
Thank you again for helping us sharpen these distinctions. If this clarifies the practical significance and scope of our findings, we would be grateful if you would consider reflecting this in your assessment.
Response to Reviewer 8cfm (1/2)
Thank you for the careful assessment and for recognising the agreement–coverage distinction and the matched-KL design.
W1(Q1): Why reduced coverage matters for evaluation reliability.
Low coverage is not, by itself, an evaluator failure. If B is used only to judge its own attribute, a weaker response to A-driven optimisation may be entirely legitimate. The reliability concern is a validation mismatch: when B is chosen to monitor A-driven policy changes because its rankings agree with A on existing answers, that static agreement does not establish the required monitoring responsiveness. The distinction was already observable in ordinary reward-derived DPO, without any checker-targeted removal: eleven criteria respond unevenly, broadly ordered by overlap (ρ=0.71, §4.2). The matched-KL experiment confirms this separation across five pairs with 65–80% static agreement (Table 2).
The actionable implication is what to validate before relying on a proxy checker (§5): supplement static agreement with overlap against the intended direction, which ranks criterion responses before training (ρ=0.71, §4.2), and an A/B pilot-run response comparison; use independent generated-output assessment when needed. This is a diagnostic recommendation (App. X compares candidate selection rules). External DeepSeek scores replicate the positive B loss (E3, 5/5 pairs); because external judges rank A and B alike on most pairs, their A loss is similar (E10), so the separation of B from A is measured with the readouts.
W1 (cont.): Independent evidence about B-related quality in generated answers.
E7 adds that the removal changes actual answers and that an independent judge and the checkers register the change. It does not establish selective blindness: A and B both change, and both are seen. Coverage on fixed responses and detection rate on generated quality are different measurements; we report both and do not equate them.
W2(Q2): Uncertainty across independent training runs.
Yes. Over nine training seeds, all 37 matched cells are positive, and the loss stays positive when questions and seeds are resampled jointly.
Pair 2: all seven matched seeds are positive (0.029 to 0.056; mean +0.0420 [0.0325,0.0520]); the gap between the two original seeds (0.046 vs 0.031) is within this seed-to-seed spread. Per pair (cells, ΔB): 1 (6) +0.0709; 2 (7) +0.0420; 3 (9) +0.0745; 4 (8) +0.0809; 5 (7) +0.0341, every crossed interval above zero.
A cell is one pair at one seed whose two arms meet the pre-specified KL tolerance and extension rule; 6 of 45 combinations had none and are left empty, not replaced (adding the two seed-0 cells without stored per-response log-probabilities at their reported values gives +0.0645). The interval is conditional on this rule. A stricter rule agrees (E11): with KL on 256 held-out contexts and both arms within ±0.025 of the target and 0.02 of each other, ΔB−ΔA is +0.0141 [0.0085,0.0193], +0.0288 [0.0217,0.0358] and +0.0363 [0.0316,0.0410] at KL 0.10, 0.20 and 0.45. The same-seed KL variation you note (App. H) is why arms are matched on measured KL; E11 tightens that match, and the nine seeds measure run-to-run variation in the loss directly.