acceptodds
Under review as a conference paper at ICLR 2027

Agreement Is Not Coverage: What Evaluation Criteria Register of a Training Change

Abstract

Criteria that monitor optimisation are usually chosen because they agree with human judgements, or with one another, on existing responses. Such agreement does not show whether a criterion registers the changes that training makes. We call this difference the agreement–coverage gap and study it with linear criterion readouts on a shared representation, where the part of a training direction that a readout can register is its overlap with that readout and can be computed before training. Training Qwen2.5-1.5B with DPO along an unmodified reward-model direction already moves eleven FLASK criteria unevenly, in the order of their overlap (Spearman ρ = 0.71). In a controlled test, we remove from a driver criterion's direction only the component shared with a checker criterion and compare two DPO arms at matched KL. Across five pairs whose readouts rank 65%–80% of response pairs alike, the checker's detection of the policy change falls in every pair (aggregate +0.063, 95% CI [0.060, 0.067]) while the driver keeps 79%–99% of its above-chance signal. An equal-length placebo removal has a smaller effect in all three pairs where it could be matched, and pre-training overlap orders the responses of the other ten criteria (ρ = 0.81). The pattern holds at 0.5B and 7B and with a mis-specified checker readout. On generated text, an independent judge sees both criteria move, so our claims concern criterion readouts.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM evaluation, evaluation criteria, directional coverage, linear readouts, preference optimization, direct preference optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8cfmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether evaluation criteria that agree on existing responses are equally sensitive to changes induced by LLM training. It argues that high agreement does not guarantee similar coverage of training-induced policy changes. The main contributions are: (1) introducing the agreement–coverage gap and a geometric method for computing criterion overlap before training, (2) providing controlled experimental evidence that removing a checker-aligned component from the training signal reduces the checker's detection of policy changes while largely preserving the driving criterion's signal, and (3) demonstrating that pre-training geometric overlap predicts the relative responses of different evaluation criteria to training-induced changes. For validation, the authors train Qwen2.5 models using DPO and compare original and modified training signals at matched KL divergence.

Critical strengths
  1. The paper highlights that agreement between evaluation criteria on existing responses does not necessarily imply similar sensitivity to training-induced changes. Its geometric formulation provides a way to estimate this sensitivity before training, offering a potentially useful perspective on evaluation reliability.
  2. The authors compare DPO training runs at matched KL divergence and demonstrate reduced checker detection across all five criterion pairs while largely preserving the driving criterion's signal. Additional placebo controls, model-scale experiments, and external reward signals strengthen the evidence that the observed effect is associated with the targeted geometric intervention rather than simply changing the training direction.
Critical weaknesses
  1. The paper measures evaluator coverage through agreement between linear criterion readouts and training-induced changes in response probabilities. However, it is unclear whether this metric captures a meaningful failure of evaluation. The treatment deliberately removes the component of the training signal aligned with criterion B, so reduced agreement with B may simply reflect fewer changes in the direction preferred by B, rather than B failing to detect changes it should recognize. To establish practical evaluator blindness, the experiments would need independent evidence of meaningful changes in B-related quality that evaluator B fails to detect. The independent generated-response evaluations do not establish such a selective failure: both A and B improve relative to the base model, and their changes are not significantly different. Thus, while the experiments support the proposed geometric relationship, its implications for practical evaluation reliability remain unclear.

  2. Although geometric overlap correlates with relative criterion responses within the controlled experiments, it does not reliably predict detection-loss magnitudes across different criterion pairs. Furthermore, the main experiments are restricted to linear readouts and DPO, with relatively few independent training runs. These limitations raise questions about the broader applicability and practical usefulness of the proposed approach.

  3. The paper is unnecessarily difficult to read and follow. The distinction between the quantities actually measured and their broader implications is also not always clearly communicated. The paper would benefit from substantial improvements in clarity, organization, and language fluency, as the current presentation makes it difficult to identify the precise contributions and assess their significance.

Questions for authors
  1. The paper defines coverage through agreement between criterion rankings and training-induced probability changes. However, reduced agreement with criterion B may simply reflect fewer changes in the direction preferred by B, rather than a failure of B to detect meaningful changes. While the authors acknowledge that their generated-response experiments do not establish selective checker blindness, could they clarify why reduced coverage, as defined in this work, should be considered a concern for practical evaluation reliability? What distinguishes low coverage from a legitimate difference between the optimized and monitored criteria?

  2. The reported confidence intervals for TT and ΔB\Delta_B are question-bootstrapped and conditional on fixed checkpoints, so they do not account for run-to-run variation. This seems important given that the two seeds of pair 2 differ beyond their question-level intervals (§4.5), while pairs 1 and 5 have only one matched seed each. Appendix V also reports substantial KL variation between same-seed reruns, although this does not directly measure variation in detection loss. Can the authors quantify the uncertainty in TT or per-pair ΔB\Delta_B across independent training runs, potentially through additional seeds or a sensitivity analysis? In particular, does the evidence support a reliably positive TT when training-run variability is considered?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8cfm (1/2)

Thank you for the careful assessment and for recognising the agreement–coverage distinction and the matched-KL design.

W1(Q1): Why reduced coverage matters for evaluation reliability.

"[...] reduced agreement with criterion B may simply reflect fewer changes in the direction preferred by B, rather than a failure of B to detect meaningful changes." "[...] could they clarify why reduced coverage, as defined in this work, should be considered a concern for practical evaluation reliability? What distinguishes low coverage from a legitimate difference between the optimized and monitored criteria?"

Low coverage is not, by itself, an evaluator failure. If B is used only to judge its own attribute, a weaker response to A-driven optimisation may be entirely legitimate. The reliability concern is a validation mismatch: when B is chosen to monitor A-driven policy changes because its rankings agree with A on existing answers, that static agreement does not establish the required monitoring responsiveness. The distinction was already observable in ordinary reward-derived DPO, without any checker-targeted removal: eleven criteria respond unevenly, broadly ordered by overlap (ρ=0.71, §4.2). The matched-KL experiment confirms this separation across five pairs with 65–80% static agreement (Table 2).

The actionable implication is what to validate before relying on a proxy checker (§5): supplement static agreement with overlap against the intended direction, which ranks criterion responses before training (ρ=0.71, §4.2), and an A/B pilot-run response comparison; use independent generated-output assessment when needed. This is a diagnostic recommendation (App. X compares candidate selection rules). External DeepSeek scores replicate the positive B loss (E3, 5/5 pairs); because external judges rank A and B alike on most pairs, their A loss is similar (E10), so the separation of B from A is measured with the readouts.

W1 (cont.): Independent evidence about B-related quality in generated answers.

"To establish practical evaluator blindness, the experiments would need independent evidence of meaningful changes in B-related quality that evaluator B fails to detect."

IDQuestionWhat was doneResultRevised paper
E7Does the removal change generated answers, and do checkers see it?200 EVAL questions; control, treated, placebo (31 cells); DeepSeek judge scores A and B; B readout rescoresTreated vs control: B −0.27, A −0.32 (placebo B −0.06); readout matches judged B changes in 72% of 2,956 treated vs 61% of 2,462 placebo eventsApp. AH

E7 adds that the removal changes actual answers and that an independent judge and the checkers register the change. It does not establish selective blindness: A and B both change, and both are seen. Coverage on fixed responses and detection rate on generated quality are different measurements; we report both and do not equate them.

W2(Q2): Uncertainty across independent training runs.

"Can the authors quantify the uncertainty in [T] or per-pair [ΔB] across independent training runs [...]? In particular, does the evidence support a reliably positive [T] when training-run variability is considered?"

Yes. Over nine training seeds, all 37 matched cells are positive, and the loss stays positive when questions and seeds are resampled jointly.

IDQuestionWhat was doneResultRevised paper
E1Is T positive across training runs?Seeds 2–8 added under the frozen protocol; KL-only checkpoint selection saved before any agreement statistic; crossed question × seed bootstrapΔB=+0.0638 [0.0587,0.0690]; ΔA=+0.0254 [0.0212,0.0297]; ΔB−ΔA=+0.0384 [0.0341,0.0431]; 37/37 cells positiveTable 3; App. AB

Pair 2: all seven matched seeds are positive (0.029 to 0.056; mean +0.0420 [0.0325,0.0520]); the gap between the two original seeds (0.046 vs 0.031) is within this seed-to-seed spread. Per pair (cells, ΔB): 1 (6) +0.0709; 2 (7) +0.0420; 3 (9) +0.0745; 4 (8) +0.0809; 5 (7) +0.0341, every crossed interval above zero.

A cell is one pair at one seed whose two arms meet the pre-specified KL tolerance and extension rule; 6 of 45 combinations had none and are left empty, not replaced (adding the two seed-0 cells without stored per-response log-probabilities at their reported values gives +0.0645). The interval is conditional on this rule. A stricter rule agrees (E11): with KL on 256 held-out contexts and both arms within ±0.025 of the target and 0.02 of each other, ΔB−ΔA is +0.0141 [0.0085,0.0193], +0.0288 [0.0217,0.0358] and +0.0363 [0.0316,0.0410] at KL 0.10, 0.20 and 0.45. The same-seed KL variation you note (App. H) is why arms are matched on measured KL; E11 tightens that match, and the nine seeds measure run-to-run variation in the loss directly.

AuthorsOpenReview ↗

Response to Reviewer 8cfm (2/2)

W2: Magnitude prediction, training runs, scorers, readout class and optimiser.

"Although geometric overlap correlates with relative criterion responses within the controlled experiments, it does not reliably predict detection-loss magnitudes across different criterion pairs. Furthermore, the main experiments are restricted to linear readouts and DPO, with relatively few independent training runs."

Overlap empirically predicts the ordering of criterion responses within a comparison (ρ=0.81 pooled; ρ=0.71 under ordinary reward-model DPO), but not calibrated magnitudes: pair 3 was predicted at +0.0247 and measured at +0.0029, and pair 4 changes sign between early training and matched KL (Table 4). The Abstract and §5 now state both the ordering and this limit.

Training runs: nine seeds and 37 cells (E1, previous response).

Scoring and policy: the loss holds when B's ranking is replaced by GPT-4, a DeepSeek-V4-Pro judge or an ArmoRM head (E3; under these scorers A declines similarly, E10), at 0.5B (four pairs; App. H) and 7B (one pair; App. L), and on Qwen2.5-1.5B-Instruct:

IDQuestionWhat was doneResultRevised paper
E8Does it hold for an instruction-tuned policy?Full pipeline rebuilt on Qwen2.5-1.5B-Instruct; seeds 2–49/9 cells positive, +0.079 [0.075,0.083]; directional > placebo in 9/9§4.5; App. AI

Readout class and optimiser: the pre-specified nonlinear probes failed their eligibility gate, and IPO/SimPO produced no admissible matched comparison (App. J); these settings are outside the matched evidence.

W3: Presentation.

"The paper is unnecessarily difficult to read and follow. The distinction between the quantities actually measured and their broader implications is also not always clearly communicated."

To make the paper easier to follow, we rewrote the Abstract and Introduction around a concrete use case, added a worked example and an equation showing where the measured difference comes from, checked every reference against its published version, and marked all revisions in blue. The table lists each change with its location:

ChangeReading difficulty it addressesLocation
Worked example on a real FLASK response pairHow a readout's ranking, the policy's margin change and the agreement statistic relate§3, p. 4
Eq. 4 with the discordant-pair breakdownWhich response pairs produce ΔB−ΔA (the 27% ranked differently)§3; App. AG
Table 3Collects seed and external-score replication of the fixed-response measurement, apart from generated-text results (App. AH)p. 7
Rewritten opening and explicit measurement boundariesConcrete proxy-monitor scenario; separates static grading, fixed-response coverage and generated-output qualityAbstract; §1; §5
One wording for overlap"Computable before training; predicts relative responsiveness, not magnitudes"Abstract; §5
Overview table of Apps. AB–AJWhich question each new analysis answersApp. AA

Summary. Together, the new runs, scorers and policy reinforce the core claim that static agreement does not ensure the same training response, and the revision states each measurement at the level at which it was made. If these additions address your main concerns, we would be grateful if you would consider reflecting this in your assessment.

I appreciate the new experiments, particularly the additional seeds and independent evaluator analyses. These address my concerns about the robustness of the measured agreement–coverage gap. However, I remain uncertain about the practical significance.

Although the generated-response experiments demonstrate that the intervention affects actual model behavior, they do not establish a practical monitoring failure. Both A and B exhibit measurable quality changes, and the evaluators detect these changes. Thus, it remains unclear whether the reduced coverage measured on fixed responses translates into a situation where a proxy evaluator fails to identify a meaningful change in model quality. Furthermore, Appendix X reports that the tested checker-selection strategies do not outperform random selection. Thus, while the geometric relationship is empirically supported, it remains unclear whether the proposed metric identifies meaningful evaluation failures or improves evaluator selection in practice.

Thank you for your thoughtful follow-up, and for confirming that the additional seeds and independent evaluator analyses address your concerns about the robustness of the measured agreement–coverage gap.

Practical significance: reduced coverage measured on fixed responses (§4.2–4.3, Tables 1–3). Static agreement alone does not ensure that a proxy evaluator will reliably track a given training update. In practice, a proxy evaluator B is often trusted to monitor A-driven training because it agrees with A on existing responses. Tables 1–2 show where reduced coverage measured on fixed responses matters for evaluation reliability: logical efficiency (A) and conciseness (B) agree on 65% of fixed pairs, yet at matched KL, A tracks the control and treated updates equally (0.645 vs 0.644) while B falls from 0.539 to 0.500. Relying on B alone makes the treated update look much weaker on a shift that A tracks equally well. Independent evaluators confirm this checker loss (Table 3), and ordinary reward-model DPO already moves criteria unevenly in the order of their pre-training overlap (ρ=0.71, §4.2).

Generated-response experiments and checker-selection strategies (Apps. AH and X).

  • Generated-response experiments (App. AH): Both A and B exhibit measurable quality changes and the evaluators detect these changes in aggregate. Testing whether a proxy evaluator fails to identify a meaningful change in model quality on generations is a distinct question that requires predefined quality-drop and alert thresholds.
  • Checker-selection strategies (App. X): At k=1, ranking by relative sensitivity alone placed conciseness first in all five pairs despite near-chance detection, so it did not outperform random selection. Rather than relying on a single static ratio, improving evaluator selection in practice requires checking baseline responsiveness alongside pre-training overlap in a short pilot update on held-out pairs (§5).

Thank you again for helping us sharpen these distinctions. If this clarifies the practical significance and scope of our findings, we would be grateful if you would consider reflecting this in your assessment.

Reviewer_kwUPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper separates two properties of an evaluation criterion: whether it ranks responses the way another criterion does (agreement), and whether it registers the change that training makes (coverage). For linear readouts on a frozen representation, what a readout can register is the overlap between the training direction and that readout, which can be computed on the base model before training. The authors construct a direction that preserves a driver criterion while being orthogonal to a checker, train two DPO arms at matched KL, and show across five pre-registered FLASK pairs that agree on 65-80% of response pairs that checker detection falls in all five (aggregate +0.063, 95% CI [0.060, 0.067]) while the driver keeps 79-99% of its signal.

Critical strengths

The experimental discipline is the best I have seen in a submission of this kind: partitions frozen and pinned by hash, predictions hashed before measurement, refuted pre-registrations reported rather than buried, and an equal-length placebo that the authors openly admit is not inert. The two-arm matched-KL design is the right instrument, and the held-out HelpSteer2 result on real generations (ArmoRM ratio 0.951 for the control arm vs 0.156 for the constructed arm) is genuine external evidence. Props to the authors.

Critical weaknesses

The core finding is close to entailed by the construction: if preferences are labeled by a direction from which the checker's component has been removed, it is not surprising that the checker stops seeing the change. What is empirical is that this survives DPO and the scoring map, which is worth measuring, but the paper never separates the part that follows from the algebra from the part that could have come out otherwise, so the reader cannot tell how surprised to be.

The "coverage, computable before training" framing is also stronger than the evidence. Table 3 gets the sign right but the magnitude off in several pairs (pair 3 predicted +0.0247, measured +0.0029 [-0.0026, +0.0082]), and the paper itself says overlap orders criteria within a comparison rather than predicting effect size; the abstract should carry that limit. The safety-relevant version of the claim, a deployed monitor missing a real behavioral change, is not shown: the generation-level gap appears in Qwen2.5-1.5B but reverses in SmolLM2-1.7B (ratio 1.31), and everything runs on small base models. The notation load is heavy; a worked example for one response pair would help a lot.

Questions for authors
  1. Which share of the five-of-five result is entailed by the construction? A null arm or an analytic bound would settle this.
  2. What separates the Qwen and SmolLM2 generation-level results?
  3. Does the effect hold with an LLM judge or a deployed reward model as the checker, which is what practitioners actually use?
2 comments
AuthorsOpenReview ↗

Response to Reviewer kwUP (1/2)

Thank you for the careful reading and for recognising the frozen protocol, the matched-KL design and the reporting of refuted predictions.

W1(Q1): What the construction entails, and what training adds.

"The core finding is close to entailed by the construction [...] the paper never separates the part that follows from the algebra from the part that could have come out otherwise." "Which share of the five-of-five result is entailed by the construction? A null arm or an analytic bound would settle this."

The construction guarantees orthogonality to B's projected coefficient direction. On the fixed candidate data, it also produces DPO labels that agree less with B's readout; the amount is measured rather than algebraically predetermined. The experiments measure what training does with that: the size of the checker loss after DPO at matched KL, how much of A is retained, whether the loss exceeds a removal of the same size in another direction, and whether it holds across runs.

IDQuestionWhat was doneResultRevised paper
E4How large is the effect already in the labels?Agreement of each arm's DPO labels with B's readoutLabel-level loss +0.067 to +0.175; trained policies show 42–56% of it, same order except the close pairs 3 and 4App. AE
E2Null arm: does the direction matter?Equal-length placebo (same length and angle, other direction), over seedsDirectional − placebo > 0 in 29/29 cells, all intervals above zero; pair means +0.034 to +0.073App. AC
E1Does it hold across runs?9 training seeds37/37 cells positive; ΔB−ΔA=+0.0384 [0.0341,0.0431]App. AB

E4 separates what is present before training from what is realised after it. E2 tests a different question: whether the removal's direction, rather than its size, causes the loss. The placebo has its own effect, so we report the paired difference. The labels also lose agreement with A (+0.020 to +0.101), so A's retention is part of the empirical result. We did not derive an analytic bound; τ is descriptive.

New App. AJ answers this at the label level: the label-level loss is exactly f·(2p−1), with f the share of flipped labels and p the share of flips on which B sides with the control. In pair 4 the placebo flips as many labels as the treatment (23% vs 24%) but nearly at random with respect to B (p=0.41–0.69 vs 0.85–0.90 across pairs), so the loss is carried by which labels flip, not how many. The initial gradient's infinitesimal effect gives the checker-loss sign in only 4/5 pairs, so the 5/5 result after DPO is not fixed by the first update.

W2: What "computable before training" supports.

"The 'coverage, computable before training' framing is also stronger than the evidence [...] the abstract should carry that limit."

The overlap and its change under the construction are computable before training, and they empirically predict the ordering of criterion responses within a comparison (ρ=0.81 pooled; ρ=0.71 in §4.2). They do not calibrate magnitudes: pair 3 was predicted at +0.0247 and measured at +0.0029 [−0.0026,+0.0082], and pair 4 changes sign between early training and matched KL (Table 4). Over nine seeds, overlap and ΔB have Spearman 0.9 across the five pairs, which is descriptive; five pairs cannot establish magnitude prediction. The Abstract, §1 and §5 now use this wording.

W2: Model scale and instruction tuning.

"[...] and everything runs on small base models."

IDQuestionWhat was doneResultRevised paper
E8Does it hold for an instruction-tuned policy?Readouts, directions and placebos rebuilt on Qwen2.5-1.5B-Instruct; seeds 2–49/9 cells positive, +0.079 [0.075,0.083]; at least the base model's loss in each pair; directional > placebo 9/9§4.5; App. AI

Existing checks: 0.5B, four pairs, T=+0.0633 [0.0589,0.0676]; 7B, one pair, ΔB=+0.0319 [0.0254,0.0379] (Apps. H and L).

W2: Notation and a worked example.

"The notation load is heavy; a worked example for one response pair would help a lot."

E6 adds one (§3, p. 4): for a FLASK question on speed and velocity, completeness prefers one answer and insightfulness the other; the control policy moves its margin towards B's choice, the treated policy towards A's. Eq. 4 shows that only such discordantly ranked pairs (27%) contribute to ΔB−ΔA; App. AG verifies it in all 37 cells.

AuthorsOpenReview ↗

Response to Reviewer kwUP (2/2)

Q2: The Qwen/SmolLM2 generation-level difference.

"What separates the Qwen and SmolLM2 generation-level results?"

At similar training extent the two models are close (0.79 vs 0.86); most of the original gap comes from unmatched training extent and reference checkpoints. The original contrast (0.156 vs 1.306) compared Qwen at KL ≈ 0.67 with SmolLM2 at KL ≈ 0.32, against different reference checkpoints (Aonly@1200 vs Aonly@3000); scorer and length control were the same. Our audit (E9; App. U, Table 28) separates these factors as far as the archived runs allow:

Model / comparisonKL of the two armsRaw ratio [95% CI]
Qwen, original vs Aonly@12000.678 / 0.6560.22 [0.03,0.39]
Qwen, separate KL series near 0.330.330 / 0.3860.79 [0.71,0.87]
SmolLM2 vs Aonly@12000.317 / 0.3310.86 [0.58,1.29]
SmolLM2 vs Aonly@30000.317 / 0.3151.31 [0.95,1.97]

Ratio: the external scorer's excess win rate above 0.5 divided by the driver's; the published 0.156 and 1.306 are the length-controlled values of the first and last rows. In a separate Qwen run, the ratio falls as training proceeds (0.91, 0.95, 0.79, 0.58 near KL 0.05, 0.20, 0.35, 0.45); near KL 0.33, Qwen (0.79) and SmolLM2 (0.86), each against an Aonly arm at similar KL, are close. The reference checkpoint also matters (SmolLM2: 0.86 vs 1.31 at the same KL), and two smaller models give ratios near zero, so generation-level visibility depends jointly on the model, the training extent and the reference; the archived runs do not isolate each factor's causal share. The readout-level ordering reproduces in both families (App. U). The original 0.951 reference (KL 0.012) is not KL-matched.

Q3 (and W2): LLM judges and a deployed reward model as checkers.

"Does the effect hold with an LLM judge or a deployed reward model as the checker, which is what practitioners actually use?"

Yes for the checker loss: with an LLM judge or a deployed reward model as the checker, it holds in every pair; separating it from the driver's loss needs criterion-specific scores, which holistic judges do not provide. We replaced B's ranking of the same candidate responses with external scores:

IDQuestionWhat was doneResultRevised paper
E3Does the B loss hold under judges and a reward model?B's ranking of fixed responses replaced by external scores (28 cells)GPT-4 +0.0415 [0.0338,0.0492]; DeepSeek-V4-Pro +0.0521 [0.0426,0.0612]; ArmoRM +0.0273 [0.0232,0.0313]; 5/5 pairs eachTable 3; App. AD

GPT-4 supplied the readouts' fitting labels; DeepSeek shares neither labels nor provider with them; ArmoRM's heads were mapped to criteria on FIT2 only. We also checked what happens when the same external judge scores A as well. Re-scoring the EVAL responses on all ten criteria with DeepSeek-V4-Pro and comparing A and B on the same response pairs gives ΔB=+0.0294 and ΔA=+0.0283, ΔB−ΔA=+0.0010 [−0.0003,0.0023] (E10, 37 cells; App. AD.1); GPT-4 and ArmoRM agree. The judge ranks A and B alike on 93–98% of response pairs (readouts: 65–80%), and by Eq. 4 only differently ranked pairs can separate them. External scores therefore confirm the B loss; the readouts, which rank A and B less alike, measure its separation from A.

The deployment version you raise in W2, a monitor missing a real behavioural change, is a different measurement. E7 (App. AH) shows that the removal changes generated answers, lowering judged B by 0.27 and A by 0.32 against control, and that the checkers register these changes. The new evidence therefore establishes the fixed-response B loss under practical scorers, not a missed behavioural change; the revised Table 3 and §5 keep the two apart.

Summary. These results support the core claim with instruments practitioners use, and the revision states each result at the level it was measured. If these additions address your main concerns, we would be grateful if you would consider reflecting this in your assessment.

Reviewer_R7dXOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

From what I understand, the paper asks whether two evaluation scores that agree on existing answers will also respond to the same changes during training. Working with simple scoring rules built from a language model’s internal features, the authors modify the training signal so that it loses its connection to one score while keeping its connection to another. Across five such pairs, training keeps improving the preserved score while becoming less consistent with the targeted one. The main contribution is a controlled empriical demonstration that agreement on existing answers does not guarantee similar sensitivity to training.

Critical strengths

The paper makes a useful distinction between grading existing answers and tracking what training changes. It shows that a score can do well at the first and still miss the second. This is worth have in mind when deciding which evaluation scores are trushworthy during training.

The experiments makes a lot of sense. The authors check if the separation holds up in training, compare runs with similar amounts of change, and use equally large changes in random directions (as a control). These comparisons make the effect generally convincing to me.

Critical weaknesses
  • The practical significance is unclear (to me). The main result is about how simple internal scoring rules relate to changes in the probabilities the model assigns to existing answers.

And it does not show that an evaluator misses a change in the quality it is meant to measure. When the generated answers are assessed independently both move in similar ways. This means a gap between the effect shown and what it means for evaluating actual model outputs.

  • The geometric explanation is missing a step. The theory establishes a property of the scoring rules, but it does not show how that property determines the model's response to training. The experiments support this relationship in the setting they test. As a result, it is hard to know how much to trust the explanation.
Questions for authors
  • Under what assumptions does the relationship between the scoring rules predict how they respond to training? Is this connection meant as a mathematical result, or as an empirical finding?
2 comments
AuthorsOpenReview ↗

Response to Reviewer R7dX (1/2)

Thank you for the careful review, and for recognising both the distinction between grading existing answers and tracking training and the matched-KL and placebo comparisons.

W1: Practical significance of a coverage gap on fixed responses.

"The practical significance is unclear (to me). The main result is about how simple internal scoring rules relate to changes in the probabilities the model assigns to existing answers."

The concern is which evaluator to trust as a proxy monitor, not whether two criteria must assess the same attribute. When a preferred criterion A is costly or difficult to assess repeatedly, a more accessible evaluator B may be used to monitor A-driven training because it agrees closely with A on existing responses (revised Introduction, p. 1). Such agreement validates how B ranks those answers, not whether it responds to the training changes it is asked to track. This distinction already appears without targeted removal: ordinary reward-derived DPO produces unequal responses among eleven criteria, approximately ordered by their pre-training overlap (ρ=0.71, §4.2). The five-pair, matched-KL intervention then isolates and measures the directional separation: despite 65–80% static agreement, B loses more fixed-response detection while A retains most of its signal (Table 2).

This changes a concrete validation decision (§5): before treating B as a proxy monitor of A, examine its overlap with the intended direction and compare A/B detection on a pilot policy update using fixed response pairs; retain independent generated-output evaluation where needed. The overlap step is available before any training and empirically predicts relative criterion responsiveness (ρ=0.71, §4.2; ρ=0.81 pooled across the controlled comparisons), so evaluators likely to respond weakly can be flagged in advance; the pilot run then measures the size. Low coverage alone is not a failure of B at grading B. It is a warning against inferring proxy monitoring suitability from static agreement. We propose this additional audit, not an experimentally validated optimal checker-selection rule: Appendix X does not find a selection rule that clearly beats random choice.

To test whether the fixed-response B contrast survives outside the fitted readouts, we replaced B's ranking with external scores:

IDQuestionWhat was doneResultRevised paper
E3Is the loss specific to the readouts?B's ranking of the same fixed responses replaced by external scores (28 cells)GPT-4 +0.0415 [0.0338,0.0492]; DeepSeek-V4-Pro +0.0521 [0.0426,0.0612]; ArmoRM +0.0273 [0.0232,0.0313]; 5/5 pairs eachTable 3; App. AD

Under these scorers A also declines by a similar amount: re-scoring every response on all ten criteria, DeepSeek-V4-Pro gives ΔB=+0.029 and ΔA=+0.028 on the same response pairs (E10), because it ranks A and B alike on 93–98% of pairs. The external scorers thus confirm the B loss; its separation from A is measured with the readouts.

W1 (cont.): Does an evaluator miss a change in generated answers?

"And it does not show that an evaluator misses a change in the quality it is meant to measure. When the generated answers are assessed independently both move in similar ways."

We agree, and the revision says so. E7 adds what the removal does to generated answers:

IDQuestionWhat was doneResultRevised paper
E7Does the removal change generated answers, and is the change seen?200 EVAL questions answered by control, treated and placebo arms (31 cells); DeepSeek judge scores A and B; B readout rescoresTreated vs control: B −0.27, A −0.32 (placebo B −0.06); readout matches judged B changes in 72% of treated vs 61% of placebo eventsApp. AH

These results connect the intervention to judged generation changes, but not to selective missed B-quality degradation: both A and B fall and the checkers register the shifts. Fixed-response directional coverage and generation-level quality detection are different endpoints (revised Abstract, §1 and §5). Our practical claim is correspondingly specific: agreement alone does not validate B for monitoring A-driven policy changes.

AuthorsOpenReview ↗

Response to Reviewer R7dX (2/2)

W2(Q1): How the geometry connects to training, and whether the link is mathematical or empirical.

"The geometric explanation is missing a step. The theory establishes a property of the scoring rules, but it does not show how that property determines the model's response to training." "Is this connection meant as a mathematical result, or as an empirical finding?"

Both, at different steps. From geometry to preference labels and from labels to the first update, the link is exact; through multi-step training it is measured. We traced the chain in the actual training code (DPO β=0.1, LoRA, AdamW) for all five pairs, comparing control (C), treated (T) and the equal-length placebo (P) on identical candidate pairs (new App. AJ).

Step 1. Geometry → labels (exact). Write a=s·b+r with r⊥b. The control labels a pair by sign(s·bᵀΔh+rᵀΔh), the treated arm by sign(rᵀΔh). They differ only where the b-term outweighs and opposes the r-term, so on every flipped pair the control label follows B's coefficient-space ranking (s>0 in all five pairs). The label-level checker loss is then exactly f·(2p−1): f is the share of flipped pairs, p the share of flips on which B's readout sides with the control.

Step 2. Labels → first update (exact at a common θ). Since σ(u)+σ(−u)=1, ∇L_C(θ)−∇L_T(θ)=−(β/N)·Σ_flipped s_i^C·∇M_i(θ): the arms' gradients differ only through flipped pairs. At initialisation the LoRA A-gradient is zero and the first AdamW step is −lr·sign of the B-gradient. Numerical checks: relative errors 1.4×10⁻⁵, 2.6×10⁻⁶, 1.1×10⁻⁷.

Step 3. Through training (empirical). After the first step the arms' parameters, optimiser states and resampled pairs diverge, so nothing above is guaranteed. Measured on seeds 2–8:

  • Each arm's accumulated LoRA update stays more aligned with its own initial gradient than with the other arm's in all 45 pair × arm × checkpoint combinations (step 100, step 3000, matched KL).
  • The initial gradient's infinitesimal first-order effect on EVAL margins gives the checker loss the right sign in 4/5 pairs. Pair 3 fails, and in pair 4 the placebo exceeds the treatment at this level.
  • Per response pair, the initial gradient does not reliably predict final margin changes (r from −0.46 to +0.66 across pairs).
  • Given the trained update, its linearisation recovers 64% of the aggregate checker loss, 4/5 signs (E5, App. AF).
PairLabel-level loss, T (f, p)Label-level loss, P (f, p)Initial-gradient level, TTrained, T (nine seeds)
1+0.119 (0.15, 0.90)+0.027 (0.12, 0.62)+0.134+0.071
2+0.098 (0.13, 0.89)−0.017 (0.10, 0.41)+0.013+0.042
3+0.175 (0.24, 0.86)+0.018 (0.13, 0.57)−0.014+0.075
4+0.167 (0.24, 0.85)+0.086 (0.23, 0.69)+0.017+0.081
5+0.067 (0.10, 0.85)+0.003 (0.08, 0.52)+0.024+0.034

The table also answers why the placebo, which removes the same length c, does less: it flips a similar share of labels (pair 4: 24% vs 23%) but nearly at random with respect to B (p=0.41–0.69 vs 0.85–0.90). The treated loss reproduces App. AE.

What was predicted before training. The original submission predicted the step-100 checker contrast from base-model gradients accumulated over a finite, KL-sized step (a different instrument from the infinitesimal one in Step 3), hash-frozen before measurement (Table 4): signs agree in 5/5 point estimates, including the negative pair 4, which reverses by matched KL.

Assumptions. Steps 1–2 assume frozen linear readouts on a shared representation, the projected subspace with c<1, the actual DPO loss, identical candidate pairs and a common starting and reference policy. Nothing beyond the first update is assumed; it is measured.

Summary. Exact: Steps 1–2 and Eq. 4. Conditional: first-order sign preservation once an update is given (App. Y.2). Empirical: everything through multi-step training, including overlap's prediction of relative responsiveness (ρ=0.71 under ordinary reward-model DPO; 0.81 pooled), not of magnitudes. If these additions address your main concerns, we would be grateful if you would consider reflecting this in your assessment.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers for their careful reading. All three recognised the distinction between agreement on existing responses and response to training, and the matched-KL comparisons with placebo controls (R7dX, kwUP, 8cfm); kwUP also highlighted the frozen, hash-pinned protocol and the reporting of refuted predictions.

Shared concerns. The reviews raise five shared concerns: (i) the practical value of a coverage gap measured on fixed responses (all three); (ii) which part of the result follows from the construction, and how the geometry connects to training (R7dX, kwUP); (iii) reproducibility across independent training runs (8cfm); (iv) external evaluators and model scope (kwUP, 8cfm); and (v) readability (kwUP, 8cfm). Our core claim is unchanged: agreement on existing responses does not guarantee that a criterion registers the change that training makes. The new experiments test this claim from further directions, and the revision states more precisely what each measurement shows and where it applies.

New experiments and analyses (cited as E1–E11 in the individual responses)

IDQuestionWhat was doneResultRevised paper
E1Does the loss survive run-to-run variation?7 further training seeds under the frozen protocol; checkpoints chosen on KL alone; questions and seeds resampled jointly37/37 matched cells positive; ΔB=+0.0638 [0.0587,0.0690]Table 3; App. AB
E2Is it the direction of the removal?Equal-length placebo (same length and angle, other direction) repeated over seedsDirectional > placebo in 29/29 cellsApp. AC
E3Is it specific to the fitted readouts?B's ranking of the same fixed responses replaced by GPT-4 scores, a DeepSeek-V4-Pro judge, an ArmoRM headB loss positive in 5/5 pairs under each; A declines similarly (E10)Table 3; App. AD
E4How much does the algebra imply?Checker loss already present in the DPO labels, compared with the trained lossTrained policies show 42–56% of itApps. AE, AJ
E5Does the first-order account hold after training?Trained LoRA increments linearised at matched checkpointsRight sign in 4/5 pairs; 64% of aggregate sizeApp. AF
E6Which response pairs carry the effect?Real FLASK response pair; Eq. 4 checked in all 37 cellsOnly pairs the readouts rank differently contribute§3, p. 4; App. AG
E7Does the removal change generated answers?200 EVAL questions, control/treated/placebo; DeepSeek judge scores A and B; checkers rescoreTreated lower on B (−0.27) and A (−0.32); checkers register the changesApp. AH
E8Does instruction tuning close the gap?Full pipeline rebuilt on Qwen2.5-1.5B-Instruct9/9 cells positive; directional > placebo 9/9§4.5; App. AI
E9Why do Qwen and SmolLM2 differ in generation?Audit of App. U: KL of each arm, reference checkpoint, scoring conventionTraining extent and reference checkpoint are confounded with model family; causal contributions remain unidentifiedApp. U, Table 28
E10Do external judges also separate checker from driver?Every EVAL response rescored by DeepSeek-V4-Pro on all ten criteria; driver loss ΔA and checker loss ΔB compared on the same response pairs (37 cells)ΔB=+0.0294 [0.0229,0.0359]; ΔA similar, ΔB−ΔA=+0.0010 [−0.0003,0.0023]; the judge ranks A and B alike on 93–98% of pairsApp. AD.1
E11Does the result depend on the KL tolerance?KL recomputed on 256 held-out contexts; checkpoints re-selected so both arms lie within ±0.025 of targets 0.10–0.45 and within 0.02 of each otherΔB−ΔA=+0.014, +0.029, +0.036 at KL 0.10, 0.20, 0.45; all intervals above zeroApp. AB.1

Summary. Together, the new evidence strengthens the controlled agreement–coverage finding: the fixed-response checker loss holds across nine training seeds, stricter KL matching, three external B-scoring instruments, and an instruction-tuned policy. On generated text, the removal changes both A and B and the checkers register both changes, so the generation-level evidence concerns measurable behavioural change rather than selectively missed degradation. The practical implication has two parts: static agreement alone does not validate a proxy monitor for a training direction, and overlap, computable before training, empirically predicts which criteria respond more weakly to that direction (ρ=0.71 under ordinary reward-model DPO; ρ=0.81 pooled), though not the magnitude of detection after DPO. We thank the reviewers; the revision clarifies these distinctions.

Dear Area Chair,

Thank you for handling our submission. To help minimize your workload, we provide a brief summary of the discussion outcomes and current status of our manuscript.

Reviewer feedback and rebuttal outcomes. Reviewers R7dX and kwUP recognized the controlled empirical contribution and matched-KL design, and we addressed their questions on theoretical framing, geometric mechanisms, and scope in our general response (E1–E11), individual replies, and revised manuscript. Following the rebuttal, Reviewer 8cfm explicitly confirmed that the additional seeds and independent evaluators addressed their concerns about the robustness of the agreement–coverage gap. The expanded nine-seed experiments show positive checker loss (ΔB) in all 37 admissible matched cells, with an aggregate ΔB−ΔA = +0.0384 [0.0341, 0.0431].

Final discussion point: scope and practical significance. Reviewer 8cfm asked whether reduced fixed-response coverage translates into missed quality-change alerts on generated outputs or improves checker selection over random choice. To avoid conflating distinct evaluation outcomes, we noted that static agreement alone does not guarantee comparable responsiveness to a training update, even when criteria agree on existing responses. We illustrated this with a controlled checker-orthogonal intervention in pair 2: despite 65% static agreement, the driver's detection remains essentially unchanged (0.645 vs. 0.644), while the checker's drops from 0.539 to 0.500 at matched KL. Relying on B alone would therefore make the treated update appear substantially weaker, even though A registers both updates equally well. Independent evaluators also confirm the checker loss, and ordinary reward-model DPO exhibits uneven criterion responsiveness (ρ=0.71).

We further clarified that the generated-response experiments show quality changes detected by both criteria in aggregate; establishing missed quality-change alerts would require predefined quality-drop and alert thresholds. Appendix X also shows that relative sensitivity alone did not outperform random checker selection. Accordingly, §5 recommends combining pre-training overlap, baseline responsiveness, and short matched-control pilot checks rather than relying on static agreement or a single sensitivity measure alone.

All revisions are highlighted in blue in the updated manuscript, with an overview of the new analyses in Appendix AA.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.