acceptodds
Under review as a conference paper at ICLR 2027

When the Evaluator Changes: Measuring False Progress in Self-Improving Language Models

Abstract

Self-improving language models rely on evaluators both to guide learning and to measure progress. When evaluators evolve alongside the models they assess, rising scores may reflect improved behavior, changes in evaluation criteria, or adaptation to evaluator weaknesses. We study false progress: apparent improvement that does not persist when model versions are compared under a diagnostic evaluation held fixed across training. We introduce a retrospective measurement framework based on a cross-version evaluation matrix, in which outputs from successive model versions are evaluated under multiple evaluator versions. This separates changes in model behavior from changes induced by evaluator revision, while a progress decomposition quantifies how much apparent improvement survives under a common reference. To connect measurement failures to training history, we build on **Loom**, a replayable trajectory dataflow that records which supervision signals were consumed by which policy updates. This lets us distinguish three properties that are often conflated: whether measured progress remains valid under evaluator revision, whether a model state descends from disputed supervision, and whether that state remains recoverable after the evaluator is repaired. In controlled reinforcement-learning post-training experiments, we observe cases where reported scores increase while performance under a fixed diagnostic reference deteriorates. We further find that descent from affected supervision does not by itself imply irreversible behavioral failure: some intermediate states remain recoverable even after subsequent training substantially degrades reference performance. These results motivate treating progress measurement, training ancestry, and recovery as distinct questions. More broadly, they suggest that self-improving systems with evolving evaluators should be assessed through versioned, retrospective evaluation rather than contemporaneous scores alone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
self-improving language models, reinforcement learning, evolving evaluators, false progress, reward hacking, evaluation reliability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UkMGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates false progress in self-improving language models caused by changing evaluators. The authors propose a retrospective auditing framework that uses cross-version evaluation and Loom-based training lineage to assess progress validity, supervision ancestry, and recoverability. Experiments using Qwen2.5-Math-1.5B with GRPO demonstrate that reported performance can increase while accuracy under a fixed reference evaluator deteriorates significantly. Additional recovery experiments show that checkpoints affected by faulty supervision may still retain performance and recover after evaluator correction, while fully collapsed checkpoints may fail to recover within the tested training budget.

Critical strengths
  1. The research problem is meaningful and interesting. As self-improving systems become increasingly autonomous and intelligent, designing reliable evaluation mechanisms may be even more important than optimization itself. Moreover, evaluation should evolve dynamically alongside the model's capabilities, making this research direction particularly relevant.
  2. The paper investigates three aspects: progress validity, supervision ancestry, and recoverability, which form a clear and logically organized framework for analyzing evaluator changes.
Critical weaknesses
  1. The experimental design mainly introduces artificially biased evaluators, such as those favoring shorter answers. However, in realistic self-improving systems, evaluator changes may aim to assess different capabilities or adapt to increasingly capable models, rather than introduce biased evaluation criteria. Therefore, the current experiments may not fully reflect realistic evaluator evolution scenarios.
  2. The experiments focus on a single model, method and task, without exploring more complex scenarios involving automatically evolving LLM-as-a-Judge evaluators. This limits the generalizability of the findings.
Questions for authors
  1. Could you investigate multiple reasonable evaluator changes within a single training trajectory, potentially at a higher frequency, to examine whether different model capabilities are preserved or degraded over time?
  2. How does your work differ from existing research on reward hacking, and what are your main novel contributions beyond prior approaches?
  3. Could you extend the experiments to other optimization methods, such as DPO, PPO, and SFT, to examine whether the findings generalize beyond GRPO?
  4. In the relatively simple linear training setting considered in this paper, directly rolling back to an earlier checkpoint seems to be a straightforward alternative. Under what scenarios would Loom provide clear advantages over checkpoint rollback? Could you provide specific examples?
2 comments
AuthorsOpenReview ↗

Response to UkMG: Evaluator Revisions, Replication, and Lineage

Thank you for the specific questions. We agree that the main GRPO experiment is a deliberately misspecified stress test, not an autonomous co-evolving judge. We have investigated the concerns as follows.

Q1. More realistic evaluator revisions. We ran a small matched, evaluator-filtered LoRA-SFT continuation with Qwen2.5-Math-1.5B-Instruct (one seed, 48 updates, 64 fixed held-out questions). Both arms share initialization, the first 16 updates, the optimizer state at the fork, and a candidate stream. The control retains exact checking; the revised arm uses exact checking (updates 1–16), normalized checking (17–32), and a symbolically augmented/text-repaired checker (33–48). All versions attempt to assess mathematical correctness rather than replace it with brevity.

On the same held-out questions, the shared C0 scores 64.06% exact / 71.88% normalized / 73.44% repaired. At C48, both the fixed and revised arms score 56.25% / 64.06% / 65.63%. Under the contemporaneous evaluator schedule, C0→C48 shows +1.56 pp apparent gain (95% paired interval [−14.06, 17.19]), versus −7.81 pp under the fixed normalized reference ([−21.88, 4.69]), yielding +9.38 pp excess reported gain ([3.13, 17.19]). This demonstrates reference-sensitive measurement, not a significant gain or decline in either individual series. The final revised-minus-control accuracy is 0.00 pp [−4.69, 4.69], so we do not attribute degradation to evaluator revision or claim equivalence of the arms. The revised/control arms consumed 171/153 selected answers; we recorded consumption IDs, but this is not a full Loom recovery experiment. We corrected a preliminary 128-token decoding truncation by uniformly re-evaluating all states at 1,024 tokens. This is one short SFT experiment, not a high-frequency or multi-capability co-evolution study.

Q2. Novelty beyond reward hacking. We agree that proxy overoptimization is not new. Our contribution is an audit of historical progress claims: paired rescoring of identical stored outputs across evaluator versions; explicit accounting of apparent versus reference-conditioned gain; and records of which disputed signals were actually consumed by which updates. These support distinct conclusions. Historical reward curves alone cannot reconstruct paired cross-version scores; rescoring alone does not identify consumed signals; and lineage alone does not establish causal damage or recoverability. We do not claim that the accounting identity itself is a new optimization algorithm.

Q3. Generalization beyond GRPO. The SFT continuation demonstrates the feasibility of versioned evaluation and consumption logging with a different update mechanism; the original results remain the principal GRPO evidence. We additionally checked two archived clean-Base GRPO seeds: their C60 reference accuracies are 67.58%/67.97%, both falling to 0% at C160 while proxy pass rates reach 100%. We have no controlled PPO/DPO or co-trained LLM-judge experiments and do not claim such validation.

Q4. Advantage over checkpoint rollback. We agree that timestamp rollback and Loom both select C60 in the synchronous global-switch incidents; Loom has no demonstrated advantage there. The key difference is deployment time versus actual consumption. In a synthetic dependency diagnostic run through the released library, a disputed signal produced at step 1 is first consumed at update 3: consumption lineage finds C2 as the latest unaffected state, while deployment-time rollback selects C0. If the signal is never consumed, C4 remains unaffected. This validates the record-level dependency rule, not an empirical training benefit. The current library over-approximates branched traces, so we do not claim validated branched-training support.

Finally, repaired training from affected intermediate checkpoints remains viable: after 40 updates, C80 minus C60 recovery differs by +4.30 pp [−0.39, 8.98] in one additional seed and −2.73 pp [−7.81, 2.34] in the other. Thus disputed ancestry does not imply irreversible damage, nor do these runs establish a consistently better restart strategy.

We hope these additional results and explicit limits address the key soundness concerns. We would welcome further discussion of which remaining concern is most consequential to your assessment.

AuthorsOpenReview ↗

Reminder to reply

Dear reviewer,

We kiindly ask you to consider our comments and would be happy to address any further discussion.

Thank you!

Reviewer_hmj1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on false progress in self-improving models, where changing evaluators make performance appear to rise even as true accuracy collapses. The authors introduce a retrospective audit that rescores past outputs against a fixed benchmark and traces training lineage with Loom. This separates genuine progress from bad training signals, demonstrating that models exposed to flawed feedback can still recover once the evaluator is repaired.

Critical strengths
  • Formulates an exact math decomposition and cross-evaluation matrix that cleanly separates genuine model gains from evaluator shifts.
  • Rigorously decouples reference measurement validity, training ancestry, and post-repair recovery using Loom's trajectory dataflow.
  • Proves that a model's test scores can look like they are improving even as real performance completely fails, but training on those bad signals doesn't mean the model is ruined beyond recovery.
Critical weaknesses
  • Evaluations are relatively narrow, relying on a single model family, one math task, and an artificially injected rule flaw rather than more real-world co-evolving evaluators.
  • Tracking training history only shows which steps received bad feedback, not whether that feedback actually broke the model or how much damage it caused.
Questions for authors

How easily does the framework extend to coding tasks where unit test suites frequently change or patch leaky edge cases?

1 comment
AuthorsOpenReview ↗

Response to hmj1: Coding Test-Suite Evolution and Multi-Seed Recovery Evidence

Thank you for the encouraging assessment and the important question about coding tasks. We agree that evolving unit-test suites provide a particularly natural application of our framework, and that disputed supervision ancestry alone does not establish causal damage.

Q. How does the framework extend to coding tasks with changing unit tests?

The core retrospective audit extends directly, without requiring a coding-specific measurement framework. We distinguish three operations:

(1) Measurement under evolving test suites. For each generated program, we preserve its source code, task ID, content hash, and original test verdict. When the test suite is revised (e.g., to cover a previously untested edge case), we re-execute the identical stored program under both versions. This produces the cross-version evaluation matrix without regenerating outputs.

For example, a program accepted by an incomplete suite may fail after additional edge-case tests are introduced. The original PASS and revised FAIL constitute a retrospective disagreement for the same program, rather than a difference caused by new sampling. If the updated tests preserve the intended task specification, the disagreement can identify an earlier overly permissive verification signal.

(2) Consumption-level ancestry. If the original PASS verdict was used as training supervision, Loom links the corresponding signal ID to the updates that actually consumed it and their descendant checkpoints. This distinguishes a test-suite revision from its influence on training history: a disputed verdict that was never consumed does not create disputed ancestry. Evaluator version timestamps alone cannot establish this distinction when signals are buffered, filtered, or replayed.

(3) Behavioral damage and recovery. Neither a flipped test verdict nor recorded ancestry proves that the disputed signal caused model degradation. Establishing a causal training effect requires a matched intervention, while assessing recoverability requires further training with repaired supervision and evaluation on fixed held-out tasks.

Our completed coding work is limited to exploratory rescoring of stored programs from released models. We have not performed a coding-agent training-lineage-and-recovery experiment, and therefore do not claim full empirical generalization to coding post-training. Furthermore, stronger tests remain imperfect references; changes to the underlying task specification must be distinguished from repairs to test coverage.

Additional evidence on ancestry versus recovery. We also verified two additional archived clean-Base GRPO runs on the same 256 held-out MATH questions. Reference accuracy declines from 67.58%/67.97% at C60 to 0% at C160, while proxy pass rates reach 100%.

After 40 repaired-reference updates, the affected C80 checkpoints reach 71.09%/69.14%, compared with 66.80%/71.88% from unaffected C60. The paired C80–C60 differences are +4.30 pp [−0.39, 8.98] and −2.73 pp [−7.81, 2.34]. Recovery from C160 remains at 0% within the tested budget.

These observations reinforce the distinction between recorded ancestry, observed performance, and recoverability. They do not establish causal damage from individual signals, statistical equivalence of restart choices, or irreversible failure.

We hope this clarifies both the operational extension to versioned coding tests and the scope of the current empirical evidence. We appreciate the opportunity to make these distinctions more explicit.

Reviewer_MQSnOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper distinguishes false progress (either made by changes in evaluation criteria or models being adaptive to evaluator weakness) from progress made by improving behaviours. Such discrimination is achieved by a cross-version evaluation matrix, called Loom, a replayable trajectory dataflow that can match supervision signals to policy updates.

Critical strengths
  • This paper identifies a neglected dimension in measuring self-improving progress: the evaluator itself is evolving/changing in a self-improving system. False progress has been reported alongside genuine improvements because the evaluator is assumed to be fixed, but it evolves with self-improving systems. Experiment results recorded in Figure 2 suggest that the pass rate of a trained policy may suddenly jump when its fixed reference goes through performance collapse.
  • It proposes a cross-version evaluation matrix, where versioned, retrospective evaluation is reported alongside the previous contemporaneous scores. In staged experiments reported in Figure 3, it's clearly demonstrated that the pass rate would usually suddenly jump whenever the reference performance collapses.
Critical weaknesses
  • The experiments conducted are narrower than their motivation has suggested. The authors intend to distinguish evaluator-change-based false progress from true progress, but in this paper's empirical experiments, the two contradictory criteria (correctness and length-check) are only artificially constructed. Yes, length optimisation is false progress, but in reality the problem is that experimenters do not always have good judgment of what is true progress as opposed to what is false progress. When evaluations are contradictory, we still do not get to discriminate true progress from false progress. In essence, what this paper establishes is a reward-misspecification problem, which seems to fall short of the authors' expectations of measuring false progress.
  • The writing is unnecessarily confusing from time to time because of jargon or typos (078 "make the audit’s estimand"). The write-up can also enjoy benefits from better signposting (e.g., when writing contributions, or identifying four questions evaluated in experiments). Each of those issues is non-critical, but improvements made on those lines would give readers an easier time engaging.
Questions for authors

Have you ever tested on tasks where you do not know in advance what is "true progress" as opposed to "false progress"? This seems to be the core motivation of the paper. However, in the experiments conducted, we clearly knew in advance that correctness is true progress, whereas brevity is not. In contrast, in reality, with RSI, humans'd be in uncharted waters where we do not have good judgment of what is true progress and what is false one. A reward misspecification and over-optimisation problem is different from a self-evolving evaluation in RSI.

1 comment
AuthorsOpenReview ↗

Response to MQSn: Reference Uncertainty, Additional Evidence, and Scope

Thank you for the thoughtful review and for identifying the important distinction between a known-objective stress test and open-ended self-improvement. We agree that this distinction deserves greater emphasis.

Q1. Have we tested settings where true progress is unknown?

We have not tested an open-ended setting in which the intended objective itself is unknown. Our additional experiments instead examine uncertainty in how a specified objective is evaluated. The framework does not determine which of two conflicting objectives should be preferred; it asks which historical progress claims remain supported under an explicitly declared diagnostic reference.

This distinction is important. When reasonable evaluators disagree, the cross-version matrix makes the disagreement observable, but does not automatically resolve it. Our framework provides reference-conditioned evidence about historical progress, rather than an unconditional determination of true progress.

Contribution beyond reward misspecification.

We agree that reward misspecification and proxy overoptimization are established phenomena. Our intended contribution beyond demonstrating reward misspecification is the retrospective audit protocol: (i) paired rescoring of identical historical outputs under evaluator versions, (ii) explicit accounting of apparent versus reference-conditioned gains, and (iii) linking disputed supervision signals to the updates that actually consumed them.

These operations distinguish measurement validity, supervision ancestry, and recoverability. Neither cross-version rescoring nor lineage alone establishes causal damage, and recovery requires a separate training experiment.

To clarify a terminology issue in our presentation: the cross-version matrix is the measurement component; Loom is the separate trajectory and consumption-lineage infrastructure. The audit combines these components rather than treating the matrix itself as Loom.

Q2. Additional evidence under alternative correctness references.

To examine reference uncertainty, we conducted additional retrospective evaluations of 8,960 stored outputs from 35 states, using four correctness-checking variants: exact, normalized, symbolically augmented, and text-repaired.

The large reference-performance collapses persist under all four variants, despite differences in their initial accuracy estimates. However, smaller changes depend on the reference: for one additional seed, C60→C80 changes by +1.17 percentage points under normalized checking but 0.00 pp under exact checking. We do not interpret these point estimates as a statistically established difference.

Importantly, this analysis also revealed an error in our original normalized checker: it accepted "9 p.m." as equivalent to "9 a.m.". Correcting this error removes nine previously accepted output occurrences associated with one question. The symbolic extension additionally accepts 61 output occurrences involving eight distinct answer–target pairs across seven questions.

These references are not independent ground-truth judges: they share extraction procedures, and the symbolic extension inherits normalized acceptances. We therefore do not claim that combining them determines absolute correctness. Rather, these results illustrate what the audit can establish: whether a progress conclusion is stable across the tested reference implementations, and where it remains evaluator-dependent.

We agree that these experiments do not resolve normative disagreements about progress in open-ended RSI. Our length-based incident is a controlled reward-misspecification stress test, not an empirical demonstration of autonomous evaluator co-evolution.

Writing and presentation.

We appreciate the suggestions regarding clarity. We will replace "make the audit's estimand" with "make explicit what the audit measures," distinguish the cross-version measurement framework from Loom's lineage implementation in the contributions, and improve signposting of the four experimental questions. We will also clarify that recorded ancestry does not itself imply causal harm or irrecoverability.

We hope these additional results and clarifications help establish both the contribution of retrospective progress auditing and its intended scope. We thank the reviewer again for the constructive feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response: Additional Evidence and Clarification of Scope

We thank all three reviewers for their careful feedback. Detailed responses to MQSn, hmj1, and UkMG are posted separately. Here we summarize the contribution, new evidence, and remaining scope.

Core contribution. When evaluators change, reward curves alone cannot establish whether historical gains survive under a common reference, which updates consumed disputed feedback, or whether affected checkpoints recover. Our retrospective audit protocol addresses these questions separately: (i) paired rescoring of identical saved outputs measures reference-conditioned progress; (ii) Loom links supervision to consuming updates; and (iii) repaired-training runs test recovery. The cross-version matrix provides the measurement layer; Loom provides lineage. This goes beyond demonstrating reward misspecification, without claiming to discover an unknown true objective. Ancestry alone proves neither causal harm nor irreversibility.

Additional Evidence

1. Replication and recovery (hmj1, UkMG). We verified two additional archived clean-Base GRPO runs on 256 fixed held-out MATH questions. Reference accuracy falls from 67.58%/67.97% at C60 to 0% at C160, while proxy pass rates reach 100%. After 40 repaired updates, affected C80 checkpoints reach 71.09%/69.14%, versus 66.80%/71.88% from unaffected C60; C160 remains at 0% within this budget. This replicates the proxy/reference divergence and shows disputed ancestry need not prevent recovery, without establishing causal damage, irreversibility, or a better restart choice.

2. Reference robustness and uncertainty (MQSn, UkMG). We rescored 8,960 stored outputs from 35 states under four correctness checkers. Large collapses persist, whereas smaller changes depend on the reference (seed-2 C60→C80: +1.17 pp normalized versus 0.00 pp exact). We also found an error in the original normalized checker: it accepted "9 p.m." for "9 a.m." in nine outputs from one question. This was corrected in the follow-up analysis. These checkers share components and are not independent gold standards. The protocol tests robustness under specified references, not absolute correctness when the objective itself is unknown.

3. Reasonable checker revisions beyond GRPO (UkMG). We newly ran a matched 48-update evaluator-filtered LoRA-SFT continuation (one seed, two mathematical-correctness checker revisions, fixed-checker control, 64 held-out questions). Applying the evaluator active at each checkpoint to its held-out outputs yields +1.56 pp apparent change (95% paired-question interval [−14.06, 17.19]), versus −7.81 pp under a fixed normalized reference ([−21.88, 4.69]): +9.38 pp excess reported gain ([3.13, 17.19]). Final revised-minus-control accuracy is 0.00 pp [−4.69, 4.69]. These intervals do not capture across-seed uncertainty. This demonstrates evaluator-dependent reporting and consumption logging with SFT, not a causal effect of evaluator revision or equivalence of the arms. All saved states were reevaluated with a uniform 1,024-token limit after an initial 128-token evaluation caused truncation.

4. Consumption lineage versus rollback (UkMG). In synchronous global-switch GRPO runs, timestamp rollback and Loom both select C60; we claim no advantage there. In a separate synthetic linear diagnostic using the released library, a disputed signal produced at step 1 is first consumed at update 3. Lineage identifies C2 as the latest reference-unaffected checkpoint, versus C0 under deployment-time rollback. This validates record-level dependency handling, not a real-training benefit; branched traces remain over-approximated.

Scope and Implications

The original length-biased incident is a controlled stress test, not autonomous judge co-evolution. Unknown-objective RSI, multi-capability evaluation, PPO/DPO, and full coding-training lineage/recovery remain unvalidated; coding evidence is exploratory rescoring. In a manuscript revision, we will clarify these limits, correct checker descriptions, distinguish the cross-version matrix from Loom, and improve signposting.

Together, the replications, reference checks, and limited SFT extension strengthen the evidence for a reproducible retrospective audit of historical progress, consumed supervision, and separately tested recovery. We hope this evidence and its stated limits help the reviewers and Area Chair assess the paper's contribution. We would be grateful if the reviewers could reconsider their assessments in light of these results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.