acceptodds
Under review as a conference paper at ICLR 2027

Did the Model Get Better or Just Different? Directional Evaluation of LLM Updates from Unlabeled Probes

Abstract

Model updates can change behavior without revealing whether task reliability improved or degraded. We show that this distinction is structural: symmetric behavioral distances can quantify change magnitude but cannot consistently encode the sign of a reliability change. We introduce the Directional Disagreement Estimator (DDE), a parameter-free estimator that combines the fraction of unlabeled probes on which two model versions disagree with a blind judge’s net preference over those disagreements. Under a simple judge-error model, DDE is attenuated toward zero as judge discriminative accuracy approaches chance. In a pre-specified confirmatory evaluation on GPQA across 22 model-update transitions, our primary endpoint—the correlation of DDE with held-out signed accuracy change—is Spearman ρ=0.600 (95% CI [0.22, 0.87], p=0.0031), and DDE attains AUROC 0.85 for detecting regressions exceeding five accuracy points. A secondary sign-accuracy comparison gives 0.68 against the 0.50 constant-sign baseline on a balanced set. DDE does not outperform a zero-change predictor on absolute error, so we position it as a directional statistic rather than a calibrated estimate of magnitude. Symmetric answer disagreement instead predicts change magnitude but not direction. A judge measured above chance yields useful DDE signal; one measured at chance yields none. Labeled canaries remain stronger, positioning DDE as a screening method that needs no probe labels rather than a replacement for labeled evaluation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, LLM evaluation, model updates, reliability evaluation, model regression, LLM-as-a-judge, unlabeled evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cCV5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how to determine the direction of performance changes brought about by language model updates in the absence of probe labels. The authors point out that symmetric metrics such as answer disagreement rate and JS divergence can only measure the magnitude of change, but cannot distinguish between improvement and degradation. Therefore, the paper proposes the Directional Disagreement Estimator (DDE), which combines the answer disagreement rate between the old and new models with the net preference of a blind judge model on the disagreement cases. Experiments were conducted on 22 model transitions in GPQA. The results show that DDE has a moderate correlation with accuracy changes on the held-out set, but cannot accurately estimate the magnitude of change, and its effectiveness depends on the judge model itself having discriminative ability above the chance level.

Critical strengths

1.The paper clearly distinguishes between the “magnitude” and “direction” of model changes and demonstrates that symmetric metrics cannot independently determine signed performance changes. This problem is of practical significance, and the theoretical argument is concise and clear.

2.The experimental reports are relatively transparent. The authors not only demonstrate the effectiveness of DDE but also report negative results such as the chance-level judge causing DDE failure and DDE’s absolute error being inferior to the zero-change baseline. Therefore, the paper’s overall description of the method’s capabilities is quite restrained.

Critical weaknesses

1.Figure 1 shows only three operating points on the right: two from the same GPQA experiment and the third a post-hoc exploratory result. These operating points demonstrate that the experimental phenomena are consistent with the theoretical trend, but are insufficient to verify a stable and generalizable relationship between judge discriminative accuracy and DDE attenuation. Therefore, the evidence for the mechanism by which judge quality determines the effectiveness of DDE remains weak.

2.The gain of DDE relative to directly using judge preference (J) is small, with the correlation coefficient only increasing from 0.588 to 0.600, and DDE exhibits large errors on several substantial performance changes. Given that the experiments are primarily limited to 22 related model-update transitions on GPQA, it remains unclear whether DDE provides sufficiently stable practical value compared with directly using judge preference.

Questions for authors

1.There are only three operating points on the right side of Figure 1. Do the authors consider these results sufficient to support a general attenuation relationship, or can they only be regarded as preliminary evidence consistent with the theoretical predictions?

2.The results of DDE are very close to those obtained by directly using judge preference (J). Can the authors further explain the practical gain from the answer disagreement rate, and whether this conclusion is stable with respect to the choice of model-update transitions?

2 comments
AuthorsOpenReview ↗

Response to Reviewer cCV5

Thank you for a careful review. Your two questions are the ones the revision was built to answer.

Q1. Are three operating points enough to support the attenuation relationship? No, and we agree they were only preliminary evidence. We pre-registered a direct test before any new API call:

  • Ten further judges from seven families vote on the same 22 GPQA updates.
  • Each judge's accuracy q is measured on items where exactly one version is right. Labels are used only for this; DDE never sees them.
  • The hypothesis H1 was "DDE slope increases with q".

H1 is confirmed: Spearman 0.77, one-sided p = 0.036, with the pre-registered rule that drops judges with >5% unparseable verdicts. With all nine judges it is 0.82 (p = 0.004), and q against DDE's ranking quality is 0.93 (Figure 1, Table 2). The sweep covers all three regimes the theory predicts:

  • judges at chance (q 0.52–0.55) give ρ ≈ 0;
  • the one judge clearly below chance (Llama-3.1 8B, q = 0.457) gives an inverted ranking, ρ = −0.51, CI [−0.76, −0.12];
  • gpt-5-mini at q = 0.843 reaches ρ = 0.83.

A second pre-registered study on MMLU-Pro (16 updates among 14 non-OpenAI models) adds 4 more references, which fall on the same curve. Counting judges and models used as solvers, there are now 26 references.

Two qualifications are stated in the paper. The ordering by q is robust, but the magnitude 2q−1 is not exact: strong judges overshoot it. Section 5.5 and Appendix M break the slope down exactly into votes on fix, break and both-wrong items. The excess comes from judges being more accurate on items that move with the update's overall direction, and from both-wrong votes favouring the stronger version. So we now present 2q−1 as a first-order guide, not a calibration law.

Q2. Practical gain of the disagreement rate over raw judge preference J; stability. We agree, and say so plainly in the revision (Section 3.1): d_parsed only rescales J, so it cannot change the ranking much.

  • Across 12 judges, ρ_DDE ≥ ρ_J for 10, but the differences are small.
  • On MMLU-Pro, J ranks marginally better (0.934 vs 0.928).
  • A transition bootstrap of the ρ_DDE − ρ_J gap for GPT-5 gives 0.012, CI [−0.05, 0.10].

The gain is on scale, which matters for absolute error. With gpt-5-mini, raw J has MAE 0.279 on GPQA and 0.171 on MMLU-Pro, against 0.107 and 0.042 for DDE; on MMLU-Pro DDE also beats the zero-change predictor (0.121). The stable conclusion is that the judge supplies the direction and d puts it in accuracy units.

"Large errors on several substantial changes." These were judge failures, not estimator failures. GPT-5 at minimal reasoning split its votes evenly on the three largest GPQA updates (M11: 21 vs 21; Δp = −0.38). On the same items, gpt-5-mini gives DDE −0.34, +0.33 and +0.16 for M11, S01 and M06 (true −0.38, +0.29, +0.27). Figure 2 shows every update for both benchmarks.

Scope. In Study C the judges are much stronger than the evaluated models, which makes that setting easier. We state this in the Limitations and do not claim the MMLU-Pro numbers transfer to frontier-on-frontier monitoring.

We would be grateful if you would consider whether the new evidence addresses your concerns.

Thank you for your detailed response and additional experiments. I will maintain my original rating.

Reviewer_q3f5OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether paired, unlabeled responses from an old and a new LLM version can reveal the sign of the accuracy change. It first notes that a statistic symmetric in the two versions cannot encode an antisymmetric target (Proposition 1). It then proposes the Directional Disagreement Estimator, DDE = (W_new − W_old)/N. DDE is a blind pairwise judge's net preference on the probes where the two versions' answers differ, scaled by the disagreement rate. Under a symmetric judge-error model, E[DDE] = (2q − 1)Δ\Deltap. The evaluation uses GPQA and 14 OpenAI endpoints forming 22 transitions. DDE correlates with held-out Δ\Deltap at Spearman ρ\rho = 0.600, has a sign accuracy of 0.68, and reaches an AUROC of 0.85 for regressions larger than 5 points. However, DDE fails to beat a trivial zero-change predictor on MAE/RMSE.

Critical strengths
  1. The question is well posed and practically motivated. The paper is unusually candid about its negative results: DDE does not beat the zero-change predictor on MAE, labeled canaries are stronger, and one of the two judges fails.

  2. The statistics are careful as the split is fixed before evaluation, the transition is the unit of analysis, and intervals are paired (McNemar-style). Robustness is checked with leave-one-endpoint-out and vertex-bootstrap sensitivity analyses and with reverse-pair diagnostics.

  3. The two-judge contrast, together with the decomposition by hidden label (Table 7), clearly shows the failure condition. The informative judge reverses its vote with correctness; the chance-level judge does not.

Critical weaknesses
  1. Proposition 1 follows directly from symmetry. It rules out a procedure that, to my knowledge, is not used in practice, namely inferring direction from a distance alone. DDE itself is pairwise LLM-as-judge, restricted to disagreeing items and rescaled. The paper's own results show that the rescaling adds little:
  • Ranking is essentially unchanged relative to the raw judge preference J (ρ\rho = 0.600 vs 0.588, Table 1).
  • Calibration does not beat Δp^=0\widehat{\Delta p} = 0 (MAE 0.1115 vs 0.0990).
  1. The evaluation uses a single multiple-choice dataset (GPQA), 64 probes, and 22 non-independent transitions across 14 OpenAI configurations judged by an OpenAI model (gpt-5), raising same-family bias concerns. Sign accuracy (15/22=68.2%15/22 = 68.2\%, p=0.067p = 0.067) is not statistically significant at α=0.05\alpha = 0.05. In Table 4, two of the three reverse diagnostics (D02_REV_M07, D03_REV_P01) are incomplete (10 missing judge calls due to exhausted API credits, despite total spend being 3.93 ), and P01 fails to flip sign under reversal (+0.109+0.109 forward, +0.044+0.044 reverse).
  2. Because a judge can perform at chance (gpt-4o), deploying DDE requires a labeled in-domain calibration set to verify q>0.5q > 0.5. However, Section 5.7 shows that labeling just 16–32 canary items directly achieves ρ=0.604–0.847\rho = 0.604–0.847 and AUROC=0.98\text{AUROC} = 0.98 without any judge calls, and DDE's exact-match gating does not directly apply to open-ended generation where labels are genuinely unavailable.
  3. There are a few presentation issues in the paper. (a. The main paper contains a single figure. Several central results appear only as tables or inline numbers: the direction/magnitude dissociation (Table 1), DDE versus the canary ladder (§5.7), and the judge decomposition (Table 7). A per-signal plot of ρ(S, Δ\Deltap) against ρ(S, |Δ\Deltap|), or a labeled-budget curve with DDE shown as a reference band, would make the argument much easier to follow. b. The Limitations section is longer than the Conclusion and largely repeats caveats from §§5.4–5.7. This space would be better used for a conclusion that states concretely when a practitioner should prefer DDE over labeled canaries.
Questions for authors
  1. Why did the gpt-5 judge yield J≈0J \approx 0 (or the wrong sign) on the three largest transitions (M11, S01, M06) in Table 3?
  2. Does DDE maintain ρ≈0.60\rho \approx 0.60 when evaluated across non-OpenAI model updates or when using a judge from a different model family (e.g., Claude or Qwen)?
  3. Why did P01 and its reversal D03_REV_P01 both produce positive DDE values (+0.109+0.109 and +0.044+0.044) in Table 6?
1 comment
AuthorsOpenReview ↗

Response to Reviewer q3f5

Thank you for the most detailed review. I addressed each point and have changed the paper accordingly.

Novelty; Proposition 1 is trivial; DDE ≈ J. We agree that Proposition 1 is elementary. It is now a short design constraint ("a change monitor needs an asymmetric signal"), not a contribution in its own right.

We also agree that d_parsed only rescales J. Ranking is nearly identical (0.600 vs 0.588 in Study A). The rescaling matters for MAE: with gpt-5-mini, raw J has MAE 0.279 / 0.171 on GPQA / MMLU-Pro, against 0.107 / 0.042 for DDE.

The revised contribution is the noisy-reference account, backed by two new pre-registered studies. A judge and a model that solves the probe itself are the same estimator with different voters, and their usefulness is ordered by their accuracy q on discriminable items. Across 26 references on two benchmarks, q predicts ranking quality (0.89 on GPQA, with the pre-registered test H1 confirmed at p = 0.036).

Single dataset, OpenAI-only, same-family bias. Study C uses MMLU-Pro, with 14 Llama/Qwen/Gemma/Mistral/Phi versions and 16 updates fixed before any answer was seen. With no OpenAI model among the evaluated versions:

  • gpt-5-mini DDE ρ = 0.93, CI [0.74, 0.99];
  • AUROC 1.0 for the four large regressions;
  • MAE 0.042 against 0.121 for the zero-change predictor.

The result is stronger without family overlap, which argues against self-preference driving it. Study B also adds non-OpenAI judges. They are weak (Claude Haiku 5.5 and DeepSeek-V3.2 at q ≈ 0.66–0.68), and the pre-registered strong non-OpenAI reasoning judges (DeepSeek-R1, Qwen3 thinking) were dropped for budget, as logged. The strongest judges in our data are therefore OpenAI reasoning models; we state this.

Sign accuracy not significant. Correct for GPT-5 at minimal reasoning (15/22, p = 0.067). In the paper this remains a descriptive secondary metric. With gpt-5-mini, sign accuracy is 16/22 on GPQA (p = 0.026) and 14/16 on MMLU-Pro (p = 0.002).

Q1. Why J ≈ 0 for GPT-5 on M11, S01, M06? The judge failed on those items; the estimator did not. GPT-5 at minimal reasoning split its votes evenly (M11: 21 new vs 21 old, against Δp = −0.38). On M11 it was at chance on the items the old model got right. On S01 and M06, its votes on both-wrong items leaned to the old model and cancelled the signal. A stronger judge on the same cached responses gets all three right: gpt-5-mini gives −0.34, +0.33, +0.16 (true −0.38, +0.29, +0.27), and gpt-oss-120b gives −0.50, +0.33, +0.39.

Q2. Non-OpenAI updates or a different judge family? See Study C above, and Table 2 and Appendix J for all ten Study B judges.

Q3. P01 and its reversal both positive. Two facts explain it.

  • P01 hardly changes accuracy: held-out Δp = +0.018, CI [−0.011, 0.047]. On the 64 probes, however, it is +0.109.
  • The reverse run is incomplete: 12 of 17 disagreements were judged before credit ran out, and they split 7 vs 5. Perfect antisymmetry would predict about −0.41.

We read this as a style or position preference of the minimal-reasoning judge on a prompt update, which changes wording more than content. It is the kind of non-antisymmetry Section 3.3 now lists. We have not re-run the five missing calls. The original endpoint was called directly and is not available to us now, so re-running through another route would not be the same judge. The diagnostics stay labelled as partial.

Labels are needed to check the judge; 16–32 canaries do well. We agree, and the paper now says so directly. Our recommended protocol uses the labels twice: to confirm q > 0.5, and to combine with DDE (rank average, no fitting). With gpt-5-mini:

  • 4 labeled probes alone give ρ 0.52 (GPQA) and 0.42 (MMLU-Pro); with DDE they give 0.77 and 0.79;
  • at 32 labels, 0.83 → 0.86 and 0.80 → 0.89;
  • on MMLU-Pro, DDE alone (0.93) exceeds 32 labels.

A label-free screen, swap consistency, correlates 0.82 with q across judges. It is post hoc and only a screen.

Open-ended generation. Agreed; this is outside our evidence and is stated in the Limitations.

Presentation. The main text now has three figures:

  • every reference on one plot (q against ρ);
  • labels against labels plus DDE;
  • one point per update for both benchmarks.

Limitations is shorter, and the conclusion states when to use DDE.

Reviewer_decUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates whether the direction of LLM reliability changes can be estimated from unlabeled probes when models are updated. The authors combine the response disagreement rate between model versions with preference judgments from an LLM judge to estimate whether an update improves or degrades performance. The contributions of this paper include a theoretical analysis demonstrating that symmetric behavioral distances cannot consistently capture the direction of reliability changes, a parameter free directional estimator, and an analysis of how judge accuracy affects estimation reliability. The method is primarily evaluated on the GPQA benchmark with additional exploratory experiments on MMLU-Pro.

Critical strengths
  • The paper addresses a practical problem of determining whether LLM updates improve or degrade model reliability without requiring labeled evaluation data.

  • The authors provide a theoretical analysis demonstrating the limitations of symmetric behavioral distance measures and explaining how judge accuracy affects directional estimation.

  • The proposed method is a straightforward and parameter free estimator that combines response disagreement with LLM-based preference judgments without requiring model training or ground truth labels for the probes.

Critical weaknesses
  • The primary evaluation is conducted only on the GPQA benchmark and MMLU-Pro is only used only for exploratory analysis.

  • The proposed DDE method achieves only a marginal 0.012 improvement in Spearman correlation over baselines.

  • The proposed DDE method fails to outperform a trivial zero-change predictor in absolute estimation error.

  • The effectiveness of the proposed DDE relies heavily on the accuracy of the LLM judge and verifying judge reliability may require labeled data.

Questions for authors
  • Could the authors further clarify the necessity of the disagreement based component and demonstrate its contribution through additional ablation studies?

  • Could the authors conduct additional experiments on other public datasets to demonstrate the generalizability of the proposed approach across different tasks and domains?

  • How can the authors ensure reliable directional estimation when judge quality is unknown, particularly when obtaining labeled data for judge validation contradicts the motivation of unlabeled evaluation?

  • How does the performance of the proposed DDE change with different numbers of unlabeled probes?

2 comments
AuthorsOpenReview ↗

Response to Reviewer decU

Thank you for the review and the concrete questions. All four are answered with new results in the revision.

"Only a marginal 0.012 improvement over baselines." To clarify, 0.012 is the gain over raw judge preference J, not over the baselines. Against the symmetric baselines the difference is large: the agreement distance has ρ = −0.06 with signed change and DDE has 0.60.

We agree DDE ranks almost exactly like J. That is expected, because d_parsed only rescales J (Section 3.1).

Q1. Necessity of the disagreement component (ablation). The disagreement rate does not drive ranking; it supplies the scale needed for absolute error. With gpt-5-mini:

  • raw J has MAE 0.279 on GPQA and 0.171 on MMLU-Pro;
  • DDE has 0.107 and 0.042;
  • the zero-change predictor has 0.099 and 0.121.

So DDE beats the zero-change predictor on MMLU-Pro, and J does not on either benchmark. The disagreement set also limits judging to items whose answers differ, which is what makes the vote sum equal to fixes minus breaks in expectation (Equation 5).

Q2. Other datasets. We ran a pre-registered replication on MMLU-Pro (Study C), excluding every item of our earlier exploratory study. It used 14 open models from five families (Llama, Qwen, Gemma, Mistral, Phi) and 16 updates fixed before any answer was seen.

  • gpt-5-mini DDE ρ = 0.93, CI [0.74, 0.99], p = 1e-7; gpt-oss-120b 0.90.
  • All four large regressions are ranked lowest (AUROC 1.0).
  • The sign is correct for 14/16 updates.

A caveat we state: here the judges are much stronger than the evaluated models, which makes the task easier than GPQA.

Q3. Unknown judge quality; labels contradict the motivation. This is the right concern, and we no longer claim a fully label-free method. Three results make the trade-off concrete.

  • Judge quality is predictable. Across ten judges plus the original two, accuracy q orders DDE quality (pre-registered test p = 0.036; q against ρ 0.93), so q tells you in advance whether DDE will work.
  • A few labels go further with DDE than alone. Four labeled probes give ρ 0.52 / 0.42 on GPQA / MMLU-Pro; the same four labels rank-averaged with DDE give 0.77 / 0.79. The labels both check q and improve the ranking.
  • A label-free screen exists but is not sufficient. Swap consistency correlates 0.82 with q across judges; it is post hoc and we present it only as a screen.

Q4. Number of unlabeled probes. New in the revision (Section 5.6). Mean ρ of gpt-5-mini DDE on random probe subsets of 8 / 16 / 32 / 48 / 64 probes:

  • GPQA: 0.60 / 0.71 / 0.79 / 0.81 / 0.83
  • MMLU-Pro: 0.60 / 0.73 / 0.85 / 0.90 / 0.93

The curve is still rising at 64, and probes cost only model calls, so a larger probe set is the cheapest improvement. With the weaker GPT-5 (minimal) judge the curve flattens at about 0.60 from 48 probes. There the judge, not the probe count, is the bottleneck.

We hope the new studies address your concerns about generality and judge validation.

Thanks for the authors' response. I will maintain my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision: two new pre-registered studies

I thank all three reviewers. The reviews agreed on four problems: one benchmark and one provider; too little evidence that judge quality drives the method (three points in Figure 1); DDE adding little over raw judge preference and losing to the zero-change predictor on MAE; and the need for labels to validate the judge. We ran two new studies and rewrote the paper around them. Each was pre-registered before any of its API calls, with a dated deviation log. The protocols, code and outputs are in the updated supplementary material.

Study B: judge sweep (same 22 GPQA updates, cached responses). We ran ten further judges from seven families: Llama 3B/8B/70B, Mistral-Small, Gemma-3, Qwen3-235B, DeepSeek-V3.2, Claude Haiku 5.5, gpt-oss-120b and gpt-5-mini. Each model also answered the probes itself as a "solver" baseline.

  • The pre-registered test H1 (DDE slope increases with judge accuracy q) is confirmed: Spearman 0.77, one-sided p = 0.036, under the pre-registered exclusion rule. Over all nine pre-registered judges it is 0.82 (p = 0.004).
  • Judge accuracy q against DDE's ranking quality gives 0.93.
  • All three regimes of the theory appear:
    • five open-weight non-reasoning judges are at chance and give no signal;
    • Llama-3.1 8B is below chance (q = 0.457) and its DDE is inverted (ρ = −0.51);
    • gpt-5-mini (q = 0.843) reaches ρ = 0.83, CI [0.60, 0.93], matching 32 labeled probes.

Study C: replication on a new benchmark and non-OpenAI updates. MMLU-Pro (excluding every item of our earlier exploratory study), with 14 Llama/Qwen/Gemma/Mistral/Phi versions and 16 updates fixed before any answer was seen.

  • The pre-registered test H7 is confirmed: gpt-5-mini DDE ρ = 0.93, CI [0.74, 0.99], p = 1e-7; gpt-oss-120b 0.90.
  • AUROC is 1.0 for the four large regressions, and sign accuracy is 14/16 (p = 0.002).
  • DDE now beats the zero-change predictor on MAE: 0.042 vs 0.121.

What we report against ourselves:

  • With bare option letters (63% of GPQA pairs), the same model is a better reference as a solver than as a judge (11/11 models). With worked answers (Study C), the gap closes.
  • Strong judges lean toward the overall stronger model. They are more accurate on items that move with the update's direction (0.88 vs 0.71 on GPQA), which inflates DDE beyond 2q−1. We state this as a limitation.
  • In Study C the judges are much stronger than the evaluated models.
  • Every post-hoc analysis is labelled as such.

Changes to the paper:

  • Proposition 1 is shortened and presented as a design constraint.
  • The main text has three new figures:
    • every judge and solver on one plot;
    • slope against theory, and labels combined with DDE;
    • one point per update on both benchmarks.
  • Limitations are condensed. The conclusion gives a concrete protocol: label a few probes, use them to check the judge, and rank updates with labels plus DDE. Four labels plus DDE give ρ ≈ 0.78, against 0.42–0.52 for the labels alone.

The main text is 9 pages.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.