- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates false progress in self-improving language models caused by changing evaluators. The authors propose a retrospective auditing framework that uses cross-version evaluation and Loom-based training lineage to assess progress validity, supervision ancestry, and recoverability. Experiments using Qwen2.5-Math-1.5B with GRPO demonstrate that reported performance can increase while accuracy under a fixed reference evaluator deteriorates significantly. Additional recovery experiments show that checkpoints affected by faulty supervision may still retain performance and recover after evaluator correction, while fully collapsed checkpoints may fail to recover within the tested training budget.
Critical strengths
- The research problem is meaningful and interesting. As self-improving systems become increasingly autonomous and intelligent, designing reliable evaluation mechanisms may be even more important than optimization itself. Moreover, evaluation should evolve dynamically alongside the model's capabilities, making this research direction particularly relevant.
- The paper investigates three aspects: progress validity, supervision ancestry, and recoverability, which form a clear and logically organized framework for analyzing evaluator changes.
Critical weaknesses
- The experimental design mainly introduces artificially biased evaluators, such as those favoring shorter answers. However, in realistic self-improving systems, evaluator changes may aim to assess different capabilities or adapt to increasingly capable models, rather than introduce biased evaluation criteria. Therefore, the current experiments may not fully reflect realistic evaluator evolution scenarios.
- The experiments focus on a single model, method and task, without exploring more complex scenarios involving automatically evolving LLM-as-a-Judge evaluators. This limits the generalizability of the findings.
Questions for authors
- Could you investigate multiple reasonable evaluator changes within a single training trajectory, potentially at a higher frequency, to examine whether different model capabilities are preserved or degraded over time?
- How does your work differ from existing research on reward hacking, and what are your main novel contributions beyond prior approaches?
- Could you extend the experiments to other optimization methods, such as DPO, PPO, and SFT, to examine whether the findings generalize beyond GRPO?
- In the relatively simple linear training setting considered in this paper, directly rolling back to an earlier checkpoint seems to be a straightforward alternative. Under what scenarios would Loom provide clear advantages over checkpoint rollback? Could you provide specific examples?
2 comments
Reminder to reply
Dear reviewer,
We kiindly ask you to consider our comments and would be happy to address any further discussion.
Thank you!
Response to UkMG: Evaluator Revisions, Replication, and Lineage
Thank you for the specific questions. We agree that the main GRPO experiment is a deliberately misspecified stress test, not an autonomous co-evolving judge. We have investigated the concerns as follows.
Q1. More realistic evaluator revisions. We ran a small matched, evaluator-filtered LoRA-SFT continuation with Qwen2.5-Math-1.5B-Instruct (one seed, 48 updates, 64 fixed held-out questions). Both arms share initialization, the first 16 updates, the optimizer state at the fork, and a candidate stream. The control retains exact checking; the revised arm uses exact checking (updates 1–16), normalized checking (17–32), and a symbolically augmented/text-repaired checker (33–48). All versions attempt to assess mathematical correctness rather than replace it with brevity.
On the same held-out questions, the shared C0 scores 64.06% exact / 71.88% normalized / 73.44% repaired. At C48, both the fixed and revised arms score 56.25% / 64.06% / 65.63%. Under the contemporaneous evaluator schedule, C0→C48 shows +1.56 pp apparent gain (95% paired interval [−14.06, 17.19]), versus −7.81 pp under the fixed normalized reference ([−21.88, 4.69]), yielding +9.38 pp excess reported gain ([3.13, 17.19]). This demonstrates reference-sensitive measurement, not a significant gain or decline in either individual series. The final revised-minus-control accuracy is 0.00 pp [−4.69, 4.69], so we do not attribute degradation to evaluator revision or claim equivalence of the arms. The revised/control arms consumed 171/153 selected answers; we recorded consumption IDs, but this is not a full Loom recovery experiment. We corrected a preliminary 128-token decoding truncation by uniformly re-evaluating all states at 1,024 tokens. This is one short SFT experiment, not a high-frequency or multi-capability co-evolution study.
Q2. Novelty beyond reward hacking. We agree that proxy overoptimization is not new. Our contribution is an audit of historical progress claims: paired rescoring of identical stored outputs across evaluator versions; explicit accounting of apparent versus reference-conditioned gain; and records of which disputed signals were actually consumed by which updates. These support distinct conclusions. Historical reward curves alone cannot reconstruct paired cross-version scores; rescoring alone does not identify consumed signals; and lineage alone does not establish causal damage or recoverability. We do not claim that the accounting identity itself is a new optimization algorithm.
Q3. Generalization beyond GRPO. The SFT continuation demonstrates the feasibility of versioned evaluation and consumption logging with a different update mechanism; the original results remain the principal GRPO evidence. We additionally checked two archived clean-Base GRPO seeds: their C60 reference accuracies are 67.58%/67.97%, both falling to 0% at C160 while proxy pass rates reach 100%. We have no controlled PPO/DPO or co-trained LLM-judge experiments and do not claim such validation.
Q4. Advantage over checkpoint rollback. We agree that timestamp rollback and Loom both select C60 in the synchronous global-switch incidents; Loom has no demonstrated advantage there. The key difference is deployment time versus actual consumption. In a synthetic dependency diagnostic run through the released library, a disputed signal produced at step 1 is first consumed at update 3: consumption lineage finds C2 as the latest unaffected state, while deployment-time rollback selects C0. If the signal is never consumed, C4 remains unaffected. This validates the record-level dependency rule, not an empirical training benefit. The current library over-approximates branched traces, so we do not claim validated branched-training support.
Finally, repaired training from affected intermediate checkpoints remains viable: after 40 updates, C80 minus C60 recovery differs by +4.30 pp [−0.39, 8.98] in one additional seed and −2.73 pp [−7.81, 2.34] in the other. Thus disputed ancestry does not imply irreversible damage, nor do these runs establish a consistently better restart strategy.
We hope these additional results and explicit limits address the key soundness concerns. We would welcome further discussion of which remaining concern is most consequential to your assessment.