- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how to determine the direction of performance changes brought about by language model updates in the absence of probe labels. The authors point out that symmetric metrics such as answer disagreement rate and JS divergence can only measure the magnitude of change, but cannot distinguish between improvement and degradation. Therefore, the paper proposes the Directional Disagreement Estimator (DDE), which combines the answer disagreement rate between the old and new models with the net preference of a blind judge model on the disagreement cases. Experiments were conducted on 22 model transitions in GPQA. The results show that DDE has a moderate correlation with accuracy changes on the held-out set, but cannot accurately estimate the magnitude of change, and its effectiveness depends on the judge model itself having discriminative ability above the chance level.
Critical strengths
1.The paper clearly distinguishes between the “magnitude” and “direction” of model changes and demonstrates that symmetric metrics cannot independently determine signed performance changes. This problem is of practical significance, and the theoretical argument is concise and clear.
2.The experimental reports are relatively transparent. The authors not only demonstrate the effectiveness of DDE but also report negative results such as the chance-level judge causing DDE failure and DDE’s absolute error being inferior to the zero-change baseline. Therefore, the paper’s overall description of the method’s capabilities is quite restrained.
Critical weaknesses
1.Figure 1 shows only three operating points on the right: two from the same GPQA experiment and the third a post-hoc exploratory result. These operating points demonstrate that the experimental phenomena are consistent with the theoretical trend, but are insufficient to verify a stable and generalizable relationship between judge discriminative accuracy and DDE attenuation. Therefore, the evidence for the mechanism by which judge quality determines the effectiveness of DDE remains weak.
2.The gain of DDE relative to directly using judge preference (J) is small, with the correlation coefficient only increasing from 0.588 to 0.600, and DDE exhibits large errors on several substantial performance changes. Given that the experiments are primarily limited to 22 related model-update transitions on GPQA, it remains unclear whether DDE provides sufficiently stable practical value compared with directly using judge preference.
Questions for authors
1.There are only three operating points on the right side of Figure 1. Do the authors consider these results sufficient to support a general attenuation relationship, or can they only be regarded as preliminary evidence consistent with the theoretical predictions?
2.The results of DDE are very close to those obtained by directly using judge preference (J). Can the authors further explain the practical gain from the answer disagreement rate, and whether this conclusion is stable with respect to the choice of model-update transitions?
2 comments
Thank you for your detailed response and additional experiments. I will maintain my original rating.
Response to Reviewer cCV5
Thank you for a careful review. Your two questions are the ones the revision was built to answer.
Q1. Are three operating points enough to support the attenuation relationship? No, and we agree they were only preliminary evidence. We pre-registered a direct test before any new API call:
H1 is confirmed: Spearman 0.77, one-sided p = 0.036, with the pre-registered rule that drops judges with >5% unparseable verdicts. With all nine judges it is 0.82 (p = 0.004), and q against DDE's ranking quality is 0.93 (Figure 1, Table 2). The sweep covers all three regimes the theory predicts:
A second pre-registered study on MMLU-Pro (16 updates among 14 non-OpenAI models) adds 4 more references, which fall on the same curve. Counting judges and models used as solvers, there are now 26 references.
Two qualifications are stated in the paper. The ordering by q is robust, but the magnitude 2q−1 is not exact: strong judges overshoot it. Section 5.5 and Appendix M break the slope down exactly into votes on fix, break and both-wrong items. The excess comes from judges being more accurate on items that move with the update's overall direction, and from both-wrong votes favouring the stronger version. So we now present 2q−1 as a first-order guide, not a calibration law.
Q2. Practical gain of the disagreement rate over raw judge preference J; stability. We agree, and say so plainly in the revision (Section 3.1): d_parsed only rescales J, so it cannot change the ranking much.
The gain is on scale, which matters for absolute error. With gpt-5-mini, raw J has MAE 0.279 on GPQA and 0.171 on MMLU-Pro, against 0.107 and 0.042 for DDE; on MMLU-Pro DDE also beats the zero-change predictor (0.121). The stable conclusion is that the judge supplies the direction and d puts it in accuracy units.
"Large errors on several substantial changes." These were judge failures, not estimator failures. GPT-5 at minimal reasoning split its votes evenly on the three largest GPQA updates (M11: 21 vs 21; Δp = −0.38). On the same items, gpt-5-mini gives DDE −0.34, +0.33 and +0.16 for M11, S01 and M06 (true −0.38, +0.29, +0.27). Figure 2 shows every update for both benchmarks.
Scope. In Study C the judges are much stronger than the evaluated models, which makes that setting easier. We state this in the Limitations and do not claim the MMLU-Pro numbers transfer to frontier-on-frontier monitoring.
We would be grateful if you would consider whether the new evidence addresses your concerns.