- Rating
- 3(+1)
- Confidence
- 3(+1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper investigates rise-and-collapse during code RL with verifiable rewards. Its main contribution is distinguishing within-campaign performance loss from cross-campaign retention, showing that better final performance does not necessarily imply more stable training. In its 7B experiments, GRPO improves final performance while leaving local collapse largely unchanged. A live stopping-and-rollback controller increases held-out pass@1 from 11.8% to 25.6% and reduces the mean online peak-to-end gap from 17.6 to 6.2 percentage points. The paper also proposes reporting trajectory stability alongside final accuracy.
Critical strengths
- Separating local collapse from cross-campaign retention helps explain why methods with similar final scores can exhibit different training behavior.
- Checkpoint continuation experiments demonstrate downstream consequences of checkpoint selection, while the leave-one-out baseline helps qualify the reinforce grpo comparison.
Critical weaknesses
- The experiments primarily study chained, verifier-guided RL fine-tuning. They do not establish a failure mechanism specific to broader recursive self-improvement systems.
- Most conditions use five seeds, some use three, and the cross-family check is a single pilot. Evidence is concentrated on Qwen models and custom code-task splits.
Questions for authors
- Could the authors clarify the concrete problem the paper aims to address, and why early stopping or best-checkpoint selection is insufficient for that objective?
- How do the continuations in Table 2 differ from the original run’s next 20 steps?
3 comments
Response - part 2 (continuing part 1)
We will revise both the Table 2 caption and the corresponding Section 4 description.
Revised Table 2 caption:
"Separate 20-update continuation experiments initialized from checkpoints at updates 50, 100, and 150 of the original 200-update diagnostic trajectory. The trajectory-region labels refer to the original online CodeGrader measurements, not to held-out starting performance. Each continuation is a newly sampled optimization branch, not the next 20 updates of the original trajectory."
The original trajectory and each continuation branch must also be distinguished by their stochastic execution paths. A continuation may use different rollout samples and training-example ordering from the original run after the branching point, even when initialized from the same saved checkpoint state.
The submitted Table 2 reports one continuation per checkpoint. It is therefore an illustrative diagnostic rather than a replicated statistical comparison.
4. Repeated continuations and optimizer-state control
We repeated the 20-update continuation protocol five times from each saved checkpoint. Each repetition restored the same checkpoint model weights and the same saved Adam optimizer state, including moment estimates and optimizer update count. We preserved the optimizer hyperparameters and learning-rate schedule.
The five continuation seeds varied rollout sampling and training-example ordering. The initial model and optimizer state were held fixed within each checkpoint group.
Each endpoint was evaluated on the same fixed 30-problem diagnostic set, with ten sampled completions per problem. Pass@1 was computed as the mean per-problem fraction of correct completions, corresponding to 300 completion evaluations per endpoint.
We define continuation improvement as the change in held-out pass@1 from the checkpoint's starting score to its continuation endpoint.
| Checkpoint | Original start | Original endpoint | Repeated endpoints, five seeds | Mean change from start |
|---|---|---|---|---|
| Update 50 | 0.63 | 0.81 | 0.74, 0.76, 0.78, 0.80, 0.82 | +0.15 |
| Update 100 | 0.31 | 0.77 | 0.60, 0.72, 0.75, 0.78, 0.80 | +0.42 |
| Update 150 | 0.41 | 0.00 | 0.04, 0.06, 0.08, 0.12, 0.15 | −0.32 |
The repeated endpoint means and sample standard deviations were 0.780 ± 0.032, 0.730 ± 0.079, and 0.090 ± 0.045 for updates 50, 100, and 150, respectively.
All five update-50 and all five update-100 continuations improved relative to their starting checkpoints. None of the five update-150 continuations improved relative to its starting checkpoint.
The corresponding mean changes from the starting held-out scores were +0.15, +0.42, and −0.32.
Thus, in these repeated continuations, update-100 improved in 5/5 stochastic branches, whereas update-150 improved in 0/5 branches.
5. Interpreting the original update-150 endpoint of 0.00
The submitted update-150 continuation ended at 0.00 held-out pass@1, whereas the five repeated endpoints ranged from 0.04 to 0.15.
We do not treat the original zero score as representative of the mean continuation outcome.
The repeated results indicate that the original 0.00 endpoint was a more extreme deterioration event than the five new continuation endpoints, while the direction of deterioration remained consistent across all five repetitions.
A zero score could reflect severe loss of task-solving ability, a discrete output-format failure, truncation, or another generation/evaluation failure mode.
The endpoint score alone cannot distinguish these possibilities.
We therefore separate the observed performance regression from its underlying mechanism. Diagnosing the original zero endpoint requires examining the saved generations, parsing and compilation failures, output lengths, and evaluation logs.
We do not claim that the current continuation experiment uniquely identifies the mechanism of collapse.
6. Contribution and limitations
The paper investigates three related quantities:
- Within-campaign online performance.
- Held-out quality of checkpoints retained for subsequent training.
- Optimization outcomes when training resumes from different saved checkpoint states.
The independent checkpoint evaluation establishes held-out deterioration in the evaluated trajectories. The deployed baseline comparison shows improved retained-checkpoint quality under the tested trajectory-dependent selection policy. The repeated continuations show that update-100 improved in all five tested branches, whereas update-150 deteriorated in all five.
These findings are specific to the evaluated checkpoints, model, training configuration, and verifier-guided code-RL setting.
The three checkpoint groups originate from one diagnostic trajectory, so the continuation experiment does not establish a universal causal relationship between checkpoint age and intrinsic trainability.
Thank you for providing a detailed and sufficient reply. I will raise my score to 3.
Response- Part 1
We sincerely thank the reviewer for the thoughtful assessment. We agree that the paper should more precisely define its scope, distinguish checkpoint carryover from conventional early stopping, and clarify the continuation protocol and interpretation of Table 2.
1. Scope and contribution
We study checkpoint carryover and performance regression in chained verifier-guided reinforcement learning (RLVR), where a selected checkpoint initializes the next training campaign.
We recognize that the term "self-improvement" in the submitted title may suggest autonomous self-modification or self-generated training data, neither of which is required by our experimental setting.
We therefore propose revising the title to:
"Checkpoint Carryover and Performance Regression in Chained Verifier-Guided RL"
Our contribution concerns the reliability of policy states retained and reused across successive training campaigns, rather than the discovery of a unique RL collapse mechanism or a new stopping algorithm.
The submitted 7B experiments show that GRPO improves held-out end-of-chain pass@1 from 11.8% to 20.7%, while its descriptive online peak-to-end gap remains comparable to REINFORCE (0.165 versus 0.176).
Additional evaluation of 90 saved peak/endpoint checkpoint pairs on an independent 300-problem test set showed mean held-out deterioration of 15.2 percentage points for REINFORCE and 14.6 points for GRPO.
2. Comparison with conventional checkpoint selection
We agree that the submitted live-versus-naive comparison does not establish superiority over shorter fixed schedules or conventional validation-based checkpoint selection.
We therefore conducted a separate deployed comparison using ten fresh paired training seeds, disjoint from the five submitted seeds, with a fixed independent 300-problem LiveCodeBench-derived test set.
Live control improved final pass@1 by 5.7 percentage points over fixed-budget training, 4.2 points over periodic validation, and 3.4 points over validation-based best-checkpoint selection.
Compute-budget definition.
The five submitted live-controller runs used 108, 116, 121, 127, and 133 cumulative updates (mean 121.0, sample SD 9.67). Their mean per-campaign budget was therefore 12.1 updates, rounded to 12 to define the fixed-update baseline. This was not an optimized fixed budget.
The ten fresh live-controller runs used 108, 114, 119, 123, 127, 130, 134, 138, 121, and 132 updates (mean 124.6, sample SD 9.38).
The fixed baseline used 120 updates over ten campaigns. Thus, the comparison is near-matched in training updates rather than exactly compute-matched.
Live monitoring reused the CodeGrader rewards already produced during training, without additional monitoring executions. The validation-based baselines required dedicated checkpoint-selection evaluations.
We do not interpret the reduced online peak-to-end gap under live stopping as independent evidence of improved stability, because stopping mechanically affects that statistic.
Instead, the retained-checkpoint comparison measures the held-out quality of policies actually passed to subsequent campaigns.
3. Clarification of Table 2 and the original continuation protocol
Table 2 reports separate 20-update continuation experiments initialized from checkpoints saved at updates 50, 100, and 150 of the original 200-update diagnostic trajectory.
These are new continuation branches, not contiguous segments of the original trajectory.
We agree that the original trajectory-region labels were ambiguous.
The labels "near peak," "degrading," and "collapsed" describe positions in the original online CodeGrader trajectory, not held-out starting-performance rankings.
In particular, the update-150 checkpoint starts at 0.41 held-out pass@1, above the update-100 checkpoint's 0.31, despite its worse position in the original online trajectory.