acceptodds
Under review as a conference paper at ICLR 2027

Self-Improvement Can Self-Regress: The Rise-and-Collapse Failure Mode of LLM Self-Training

Abstract

Self-improvement can self-regress. In verifiable-reward code RL, continued optimization can first improve a language model and then erase those gains. We study this instability with Qwen-2.5-3B/7B on custom competitive-programming splits drawn from HumanEval, MBPP, and APPS, using binary CodeGrader reward and chained training campaigns. In a 200-step Qwen-2.5-7B REINFORCE diagnostic run, the online success signal rises from 25% to 81% within roughly 50 updates and later falls to nearly zero; checkpoint-continuation tests show that checkpoints from different parts of this trajectory lead to sharply different downstream outcomes. We therefore ask an operational question: When does regression become actionable, and where should the control loop intervene? We separate two temporal quantities: a within-campaign cliff, measured by the loss from a campaign's online peak to its endpoint, and cross-campaign endpoint persistence, measured from the sequence of checkpoints passed forward through the chain. The distinction exposes a striking asymmetry. On the matched 7B held-out splits, the frozen base reaches 10.4% [8.1, 12.8], vanilla REINFORCE 11.8% [5.2, 18.3], and GRPO 20.7% [15.7, 25.1] over five seeds, yet the mean within-campaign peak-to-end gap remains nearly unchanged under REINFORCE and GRPO (17.6 vs. 16.5 points). A matched REINFORCE+leave-one-out-baseline (RLOO-style) control reaches 13.0% [8.5, 17.5] while retaining a 17.0-point mean peak-to-end gap. GRPO raises the floor, but does not remove the cliff. The RLOO control further shows that this endpoint gain is not reproduced by adding a simple baseline alone. Control effectiveness is also regime-dependent. A simple campaign-end controller raises 3B end-of-chain pass@1 from 4.9% to 9.5% across five matched seeds (paired bootstrap difference [+0.4,+8.9] points), but provides no clear gain at 7B. A trajectory-informed peak-rollback/budget schedule reaches 22.2% [14.1,28.0] at 7B (). Turning the causal-prefix three-decline rule into a live controller raises 7B held-out end-of-chain pass@1 to 25.6% [20.3,30.4] over five seeds and reduces the mean peak-to-end gap from 17.6 to 6.2 points. The central lesson is that final performance is not a sufficient statistic for training stability: update-rule changes can raise the floor while same-campaign control directly suppresses the cliff. We advocate reporting local peak retention, failure geometry, endpoint persistence, and detection lead time alongside final pass@1 when evaluating self-improving RL systems.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-improving, failure mode

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tdHSOpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates rise-and-collapse during code RL with verifiable rewards. Its main contribution is distinguishing within-campaign performance loss from cross-campaign retention, showing that better final performance does not necessarily imply more stable training. In its 7B experiments, GRPO improves final performance while leaving local collapse largely unchanged. A live stopping-and-rollback controller increases held-out pass@1 from 11.8% to 25.6% and reduces the mean online peak-to-end gap from 17.6 to 6.2 percentage points. The paper also proposes reporting trajectory stability alongside final accuracy.

Critical strengths
  1. Separating local collapse from cross-campaign retention helps explain why methods with similar final scores can exhibit different training behavior.
  2. Checkpoint continuation experiments demonstrate downstream consequences of checkpoint selection, while the leave-one-out baseline helps qualify the reinforce grpo comparison.
Critical weaknesses
  1. The experiments primarily study chained, verifier-guided RL fine-tuning. They do not establish a failure mechanism specific to broader recursive self-improvement systems.
  2. Most conditions use five seeds, some use three, and the cross-family check is a single pilot. Evidence is concentrated on Qwen models and custom code-task splits.
Questions for authors
  1. Could the authors clarify the concrete problem the paper aims to address, and why early stopping or best-checkpoint selection is insufficient for that objective?
  2. How do the continuations in Table 2 differ from the original run’s next 20 steps?
3 comments
AuthorsOpenReview ↗

Response- Part 1

We sincerely thank the reviewer for the thoughtful assessment. We agree that the paper should more precisely define its scope, distinguish checkpoint carryover from conventional early stopping, and clarify the continuation protocol and interpretation of Table 2.

1. Scope and contribution

We study checkpoint carryover and performance regression in chained verifier-guided reinforcement learning (RLVR), where a selected checkpoint initializes the next training campaign.

We recognize that the term "self-improvement" in the submitted title may suggest autonomous self-modification or self-generated training data, neither of which is required by our experimental setting.

We therefore propose revising the title to:

"Checkpoint Carryover and Performance Regression in Chained Verifier-Guided RL"

Our contribution concerns the reliability of policy states retained and reused across successive training campaigns, rather than the discovery of a unique RL collapse mechanism or a new stopping algorithm.

The submitted 7B experiments show that GRPO improves held-out end-of-chain pass@1 from 11.8% to 20.7%, while its descriptive online peak-to-end gap remains comparable to REINFORCE (0.165 versus 0.176).

Additional evaluation of 90 saved peak/endpoint checkpoint pairs on an independent 300-problem test set showed mean held-out deterioration of 15.2 percentage points for REINFORCE and 14.6 points for GRPO.

2. Comparison with conventional checkpoint selection

We agree that the submitted live-versus-naive comparison does not establish superiority over shorter fixed schedules or conventional validation-based checkpoint selection.

We therefore conducted a separate deployed comparison using ten fresh paired training seeds, disjoint from the five submitted seeds, with a fixed independent 300-problem LiveCodeBench-derived test set.

Deployed methodFinal pass@1Mean retained pass@1
Fixed-budget19.1%15.72%
Periodic validation20.6%16.41%
Validation best-checkpoint21.4%16.92%
Live causal-prefix24.8%18.53%

Live control improved final pass@1 by 5.7 percentage points over fixed-budget training, 4.2 points over periodic validation, and 3.4 points over validation-based best-checkpoint selection.

Compute-budget definition.

The five submitted live-controller runs used 108, 116, 121, 127, and 133 cumulative updates (mean 121.0, sample SD 9.67). Their mean per-campaign budget was therefore 12.1 updates, rounded to 12 to define the fixed-update baseline. This was not an optimized fixed budget.

The ten fresh live-controller runs used 108, 114, 119, 123, 127, 130, 134, 138, 121, and 132 updates (mean 124.6, sample SD 9.38).

The fixed baseline used 120 updates over ten campaigns. Thus, the comparison is near-matched in training updates rather than exactly compute-matched.

Live monitoring reused the CodeGrader rewards already produced during training, without additional monitoring executions. The validation-based baselines required dedicated checkpoint-selection evaluations.

We do not interpret the reduced online peak-to-end gap under live stopping as independent evidence of improved stability, because stopping mechanically affects that statistic.

Instead, the retained-checkpoint comparison measures the held-out quality of policies actually passed to subsequent campaigns.

3. Clarification of Table 2 and the original continuation protocol

Table 2 reports separate 20-update continuation experiments initialized from checkpoints saved at updates 50, 100, and 150 of the original 200-update diagnostic trajectory.

These are new continuation branches, not contiguous segments of the original trajectory.

Original checkpointHeld-out startOriginal continuation endpoint
Update 500.630.81
Update 1000.310.77
Update 1500.410.00

We agree that the original trajectory-region labels were ambiguous.

The labels "near peak," "degrading," and "collapsed" describe positions in the original online CodeGrader trajectory, not held-out starting-performance rankings.

In particular, the update-150 checkpoint starts at 0.41 held-out pass@1, above the update-100 checkpoint's 0.31, despite its worse position in the original online trajectory.

AuthorsOpenReview ↗

Response - part 2 (continuing part 1)

We will revise both the Table 2 caption and the corresponding Section 4 description.

Revised Table 2 caption:

"Separate 20-update continuation experiments initialized from checkpoints at updates 50, 100, and 150 of the original 200-update diagnostic trajectory. The trajectory-region labels refer to the original online CodeGrader measurements, not to held-out starting performance. Each continuation is a newly sampled optimization branch, not the next 20 updates of the original trajectory."

The original trajectory and each continuation branch must also be distinguished by their stochastic execution paths. A continuation may use different rollout samples and training-example ordering from the original run after the branching point, even when initialized from the same saved checkpoint state.

The submitted Table 2 reports one continuation per checkpoint. It is therefore an illustrative diagnostic rather than a replicated statistical comparison.

4. Repeated continuations and optimizer-state control

We repeated the 20-update continuation protocol five times from each saved checkpoint. Each repetition restored the same checkpoint model weights and the same saved Adam optimizer state, including moment estimates and optimizer update count. We preserved the optimizer hyperparameters and learning-rate schedule.

The five continuation seeds varied rollout sampling and training-example ordering. The initial model and optimizer state were held fixed within each checkpoint group.

Each endpoint was evaluated on the same fixed 30-problem diagnostic set, with ten sampled completions per problem. Pass@1 was computed as the mean per-problem fraction of correct completions, corresponding to 300 completion evaluations per endpoint.

We define continuation improvement as the change in held-out pass@1 from the checkpoint's starting score to its continuation endpoint.

CheckpointOriginal startOriginal endpointRepeated endpoints, five seedsMean change from start
Update 500.630.810.74, 0.76, 0.78, 0.80, 0.82+0.15
Update 1000.310.770.60, 0.72, 0.75, 0.78, 0.80+0.42
Update 1500.410.000.04, 0.06, 0.08, 0.12, 0.15−0.32

The repeated endpoint means and sample standard deviations were 0.780 ± 0.032, 0.730 ± 0.079, and 0.090 ± 0.045 for updates 50, 100, and 150, respectively.

All five update-50 and all five update-100 continuations improved relative to their starting checkpoints. None of the five update-150 continuations improved relative to its starting checkpoint.

The corresponding mean changes from the starting held-out scores were +0.15, +0.42, and −0.32.

Thus, in these repeated continuations, update-100 improved in 5/5 stochastic branches, whereas update-150 improved in 0/5 branches.

5. Interpreting the original update-150 endpoint of 0.00

The submitted update-150 continuation ended at 0.00 held-out pass@1, whereas the five repeated endpoints ranged from 0.04 to 0.15.

We do not treat the original zero score as representative of the mean continuation outcome.

The repeated results indicate that the original 0.00 endpoint was a more extreme deterioration event than the five new continuation endpoints, while the direction of deterioration remained consistent across all five repetitions.

A zero score could reflect severe loss of task-solving ability, a discrete output-format failure, truncation, or another generation/evaluation failure mode.

The endpoint score alone cannot distinguish these possibilities.

We therefore separate the observed performance regression from its underlying mechanism. Diagnosing the original zero endpoint requires examining the saved generations, parsing and compilation failures, output lengths, and evaluation logs.

We do not claim that the current continuation experiment uniquely identifies the mechanism of collapse.

6. Contribution and limitations

The paper investigates three related quantities:

  1. Within-campaign online performance.
  2. Held-out quality of checkpoints retained for subsequent training.
  3. Optimization outcomes when training resumes from different saved checkpoint states.

The independent checkpoint evaluation establishes held-out deterioration in the evaluated trajectories. The deployed baseline comparison shows improved retained-checkpoint quality under the tested trajectory-dependent selection policy. The repeated continuations show that update-100 improved in all five tested branches, whereas update-150 deteriorated in all five.

These findings are specific to the evaluated checkpoints, model, training configuration, and verifier-guided code-RL setting.

The three checkpoint groups originate from one diagnostic trajectory, so the continuation experiment does not establish a universal causal relationship between checkpoint age and intrinsic trainability.

Thank you for providing a detailed and sufficient reply. I will raise my score to 3.

Reviewer_MxQDOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the rise-and-collapse behaviour observed during successive rounds of RLVR training. The authors analyse training trajectories and checkpoints on code generation tasks, distinguish within-campaign performance deterioration from cross-campaign performance retention, and compare different RL algorithms and training control strategies. They further propose online monitoring, early stopping, and checkpoint rollback to mitigate performance deterioration. The main contribution is an empirical analysis of training dynamics in specific RLVR settings and an exploration of corresponding training control methods.

Critical strengths
  • The authors conduct a relatively systematic set of experiments on performance deterioration during RLVR training, providing empirical results on performance changes across training stages and comparisons between different control strategies.
  • The authors explore using online training signals to determine when to stop training and roll back to a previous checkpoint. Some experiments show that this approach can improve final model performance, suggesting potential practical value.
Critical weaknesses
  • The distinction between the proposed phenomenon and existing problems is unclear. The paper frames the observed rise-and-collapse behaviour as a self-improvement failure mode, but the experiments primarily concern conventional RLVR training. Chaining multiple training campaigns does not, by itself, establish a distinct self-improvement process. The observed deterioration is also consistent with conventional RL optimisation instability, overfitting, and checkpoint sensitivity. The paper does not sufficiently distinguish the proposed failure mode from these familiar phenomena.
  • The underlying mechanism of performance deterioration is insufficiently analysed. The paper primarily characterises training trajectories and reports correlations with signals such as entropy contraction and increasing gradient norms. However, it does not adequately explain the causes of the collapse or distinguish among plausible explanations, including overfitting, unstable gradient estimates, policy collapse, and reward-related effects. The proposed two-timescale analysis therefore remains largely descriptive.
  • Important experimental factors and alternative explanations are insufficiently investigated. The paper does not systematically examine the effects of rollout batch size or group size, leaving open whether the observed instability is sensitive to sampling variance or policy gradient estimation. It also lacks sufficient analysis of reward signals, potential training data overlap, and the relationship between optimising the training reward and generalisation to independent test problems. Furthermore, the experiments focus primarily on two Qwen2.5 model sizes and specific code generation tasks, limiting the generality of the conclusions.
  • The methodological novelty is overstated. The proposed online causal-prefix controller is essentially an early-stopping heuristic that triggers when the online reward declines for three consecutive steps, followed by rollback to the best checkpoint within the current campaign. The term causal-prefix merely indicates that decisions rely on the trajectory observed up to the current step; it does not introduce a new causal inference method or a substantially novel control algorithm. The other proposed strategies also rely primarily on checkpoint selection and training budget adjustment. The paper does not sufficiently demonstrate that these methods outperform conventional early stopping and best-checkpoint selection under matched computational and evaluation budgets. The terminology and conceptual framing therefore overstate the novelty of the proposed approach.
Questions for authors
  • In what sense does the observed behaviour constitute a distinct self-improvement failure mode, rather than conventional RLVR training instability or overfitting?
  • Would larger rollout batch sizes or group sizes change the extent of performance deterioration? Can the authors rule out sampling variance and unstable gradient estimation as contributing factors?
  • What clear advantages do the proposed control strategies offer over conventional early stopping and best-checkpoint selection under the same computational budget?
5 comments
AuthorsOpenReview ↗

Response - part 1

We agree that the paper should distinguish its contribution from established RL instability, evaluate conventional alternatives under comparable computational budgets, and clarify the roles of reward signals, sampling, and overfitting.

1. Research scope, terminology, and novelty

We study checkpoint carryover in chained verifier-guided reinforcement learning (RLVR): the checkpoint selected after one training campaign initializes the next campaign.

We agree that this setting does not, by itself, constitute autonomous self-modification or self-generated training data. The term "self-improvement" in our submitted title was intended to describe repeated policy improvement through chained training, but we recognize that it can suggest a broader autonomous-learning setting than we evaluate.

We therefore propose revising the title:

"Checkpoint Carryover and Performance Regression in Chained Verifier-Guided RL"

which more precisely describes the investigated phenomenon without implying autonomous self-modification.

Our contribution is not the discovery of a new RL collapse mechanism. Rather, it is the study of how within-campaign optimization behavior affects the quality of checkpoints retained and reused across successive training campaigns.

The distinction is operationally important: a checkpoint-selection decision affects not only the current campaign's retained policy but also the initialization of subsequent campaigns.

2. Evidence beyond known online RL collapse

We agree that policy collapse, entropy contraction, and over-optimization are established concerns.

The submitted 7B experiments show that GRPO improves end-of-chain held-out pass@1 relative to REINFORCE, while the descriptive online peak-to-end gaps remain comparable.

Submitted 7B resultsREINFORCEGRPORLOO-style
Held-out end-of-chain pass@111.8%20.7%13.0%
Mean online peak-to-end gap0.1760.1650.170

The online gaps characterize training trajectories; they are not themselves measurements of held-out forgetting.

We additionally evaluated 90 saved peak/endpoint checkpoint pairs on an independent 300-problem LiveCodeBench-derived test set, disjoint from the original HumanEval/MBPP/APPS-derived training and development splits after duplicate filtering.

Independent checkpoint evaluationREINFORCEGRPO
Recoverable checkpoint pairs4446
Mean held-out deterioration15.2 pp14.6 pp
Positive deterioration38/4440/46

These results support the persistence of checkpoint-level deterioration beyond the original online CodeGrader measurements, including under GRPO, which improves end-of-chain performance.

The checkpoint pairs are correlated within training seeds and are not treated as independent training replicates.

3. Mechanisms, reward signals, sampling, and overfitting

We agree that the current evidence does not isolate a unique mechanism.

The submitted KL stress test shows that the rise-and-collapse pattern persists under the tested fixed and increasing reference-policy KL schedules. The diagnostic measurements show approximately 18% entropy contraction and a 1.9× increase in gradient norm relative to the peak region, while response length and compile-failure rate change comparatively little.

We also acknowledge that the observed behavior may depend on the reward signal and sampling configuration.

FactorEvidence or limitation
Reward signalCodeGrader-based verifier feedback; reward-model or verifier misspecification is not independently isolated
SamplingSubmitted protocol uses 16 completions per prompt
Group/batch sizeNo systematic group-size or batch-size ablation establishing invariance
KL regularizationTested fixed and increasing KL schedules do not eliminate the observed pattern
OverfittingTraining-time and independent held-out deterioration are both relevant, but do not isolate every form of overfitting

Regarding overfitting, the simplest explanation would be that the policy increasingly fits the training examples while losing generalization.

However, the observed online CodeGrader performance itself declines after its peak. To the extent that this metric evaluates the training-time task distribution, this behavior is not the typical signature of ordinary train–test overfitting, which would predict improving or sustained training performance alongside declining held-out performance.

Moreover, the independent checkpoint evaluation shows that held-out quality also deteriorates on previously unseen coding problems.

Together, these observations argue against the simplest explanation of monotonically improving training reward accompanied only by declining generalization.

They do not rule out distributional overfitting, verifier exploitation, reward misspecification, or interactions with group size and sampling variance. We will make these limitations explicit.

AuthorsOpenReview ↗

Response - part 2

4. Comparison with fixed-budget and validation-based alternatives

We agree that the benefit of causal-prefix control must be distinguished from simply reducing the number of training updates. We therefore compare it against a fixed-step baseline and conventional validation-based checkpoint selection, explicitly accounting for training and evaluation executions.

Fixed-budget selection. The fixed budget was determined before the new comparison using the five submitted live-controller runs.

Those runs used 108, 116, 121, 127, and 133 cumulative gradient updates over ten campaigns (mean 121.0, sample SD 9.67). This corresponds to 12.1 updates per campaign on average. We rounded this value to 12 updates per campaign to define a simple fixed-budget baseline.

The 12-step budget was not selected by searching for the best fixed schedule, nor was it fitted to the results of the new evaluation.

We then conducted a separate deployed comparison using ten fresh, paired training seeds, disjoint from the submitted five seeds. The new live-controller runs used 108, 114, 119, 123, 127, 130, 134, 138, 121, and 132 cumulative updates, respectively (mean 124.6, sample SD 9.38).

Computational accounting.

Each gradient update used one training prompt with 16 sampled completions, each receiving one CodeGrader evaluation. Live causal-prefix monitoring read the already computed training rewards and required no additional CodeGrader calls.

The periodic-validation and validation best-checkpoint baselines used a separate 30-problem selection set, with one completion per problem and one validation round every two updates.

Compute per seedFixedLivePeriodicValidation best
Mean gradient updates120.0124.6120.0120.0
Mean training rollouts1,9201,993.61,9201,920
Training CodeGrader calls1,9201,993.61,9201,920
Extra monitoring calls0000
Validation rounds006060
Validation CodeGrader calls001,8001,800
Total CodeGrader calls1,9201,993.63,7203,720

Thus, live control used 3.83% more training CodeGrader executions than fixed-budget training, while requiring substantially fewer total CodeGrader executions than the two validation-based baselines.

We characterize the live-versus-fixed comparison as near-matched computational budget, not exact compute matching. The comparison with validation-based methods separately accounts for dedicated selection-set evaluations. These counts exclude final test evaluations, which are not used for checkpoint selection.

Deployed performance.

All methods were evaluated on the same independent 300-problem LiveCodeBench-derived test set, which was not used for controller decisions or hyperparameter selection.

MethodIndependent final pass@1Mean retained pass@1
Fixed-budget19.1%15.72%
Periodic validation20.6%16.41%
Validation best-checkpoint21.4%16.92%
Live causal-prefix24.8%18.53%

Live control improved final pass@1 by 5.7 percentage points over fixed-budget training, 4.2 points over periodic validation, and 3.4 points over validation best-checkpoint selection.

These results indicate that trajectory-dependent checkpoint selection improves retained policy quality beyond the tested shorter fixed schedule, with a small additional training-execution cost relative to fixed-budget training and substantially lower evaluation overhead than the validation-based alternatives.

We do not claim that this establishes superiority over every fixed-budget schedule or every possible checkpoint-selection policy. The comparison specifically tests a fixed budget derived from the submitted live-controller training budget and conventional validation-based alternatives.

AuthorsOpenReview ↗

Response - part 3

5. Checkpoint carryover across campaigns

We agree that early stopping mechanically reduces the observed online peak-to-end gap. We therefore do not use the smaller online gap of the stopped trajectory as independent evidence of controller effectiveness.

Instead, we evaluate the held-out quality of the checkpoints actually carried forward into subsequent campaigns.

For each of the ten new paired seeds, we evaluated the retained checkpoint after every campaign on the same independent test set, without using these measurements for training decisions.

MethodMean retained pass@1 across 10 campaignsFinal pass@1
Fixed-budget15.72%19.1%
Periodic validation16.41%20.6%
Validation best-checkpoint16.92%21.4%
Live causal-prefix18.53%24.8%

The live controller maintained higher mean independently evaluated retained-checkpoint quality than the tested fixed-budget and validation-based alternatives.

These results support the value of trajectory-dependent checkpoint selection for preserving the quality of policies actually reused in the evaluated chained-training process, rather than merely truncating online trajectories.

We distinguish this from the stronger claim that checkpoint selection changes intrinsic trainability. Establishing such a mechanism would require controlled continuation experiments from matched-quality checkpoints.

6. Revised contribution and scope

We position the paper as an investigation of performance regression and checkpoint carryover in chained verifier-guided RL.

The contributions are:

  1. Distinguishing transient within-campaign performance from the quality of checkpoints retained across successive training campaigns.
  2. Evaluating whether checkpoint deterioration persists under independent held-out testing, including for optimizers that improve end-of-chain performance.
  3. Comparing trajectory-dependent checkpoint selection against fixed-budget and validation-based alternatives under explicitly reported computational budgets.

We do not claim a novel RL optimizer, a unique collapse mechanism, or universal generalization across reward signals, group sizes, model families, and task domains.

We thank the reviewer for motivating a more precise framing and a stronger separation between established RL instability and its consequences for checkpoint carryover.

Reviewer_dE6LOpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies rise-and-collapse trajectories in verifiable-reward code RL. The authors separate the within-campaign peak-to-endpoint drop from endpoint persistence across chained training campaigns. They compare REINFORCE, GRPO, a leave-one-out baseline, and several checkpoint controllers on custom splits drawn from HumanEval/MBPP/APPS with Qwen2.5 3B/7B. At 7B, GRPO improves final pass@1 but leaves the mean peak-to-end gap close to REINFORCE's; a live three-decline stopping rule improves final performance and shrinks the online signal's peak-to-end gap.

Critical strengths

1.The paper separates "high final performance" from "not collapsing within a training stage," and provides operationalizable peak-to-end gap, cliffness, and detection lead time. Table 5 and Table 6's 7B results (GRPO 20.7% vs. REINFORCE 11.8%, peak-to-end gap 0.165 vs. 0.176) support this distinction and serve as a direct reminder to RLVR work that reports only endpoints. 2.The control experiments go beyond post-hoc best-checkpoint selection: they include a leave-one-out baseline, a KL stress test, and distinguish post-hoc selection from live control based on trajectory-visible information. The live controller achieves 25.6% [20.3, 30.4] final pass@1 over five 7B seeds and reduces the online peak-to-end gap to 0.062 (Table 5, Section 5.2).

Critical weaknesses

1.The core extrapolation remains limited by evaluation scale. Each campaign uses only 50 training problems and 30 same-source held-out problems; the ten-campaign chain covers only two model scales. Fluctuations across 30 problems can substantially affect peaks, three-decline triggering, and cliffness. Treating the 47/50 campaign traces as descriptive statistics is appropriate, but they are not independent samples. A larger, genuinely independent, out-of-distribution fixed test set is still needed to establish the generality of "collapse" (Section 3, Tables 4–6). 2.The selection procedure for the three-decline threshold and budget rules, the matched computational/training budget across controllers, and the use of a separate tuning set need to be clearer. TB has only 3 seeds, and one of the three GRPO+TB seeds collapses to zero. Final performance and peak-to-end gap use different signals (held-out pass@1 vs. online CodeGrader), so one cannot claim from their co-movement alone that the same kind of held-out collapse was predicted (Tables 5–6, Sections 5.2–5.3).

Questions for authors

1.On what development data were the three-decline rule, the 0.02 threshold, and the 8–20 step budget determined? Please freeze the rules completely and report step-level trigger times, false alarm rates, and paired seed differences on a larger fixed held-out set. 2.Does the live controller still improve over early stopping, periodic validation, and best-checkpoint baselines matched for equal training steps/equal code executions? Do the peak-to-end gap and lead time persist when using a different source of code problems or a stronger model?

5 comments
AuthorsOpenReview ↗

Response- PART 1

We sincerely thank the reviewer for the constructive feedback. We agree that independent evaluation, parameter provenance, sensitivity analysis, and comparisons against conventional checkpoint-selection methods are essential for assessing the reliability of our conclusions.

1. Independent evaluation of checkpoint deterioration

The submitted experiments use 50 training and 30 held-out problems per campaign from HumanEval/MBPP/APPS-derived slices. We acknowledge both the limited evaluation size and the statistical dependence among campaigns within each training seed.

We additionally evaluated 90 saved peak/endpoint checkpoint pairs from the original complete training trajectories on an independent set of 300 LiveCodeBench-derived coding problems. The set was constructed separately from the original training and development splits, with exact-match and normalized near-duplicate filtering. It was never used for checkpoint selection, controller decisions, or parameter tuning.

MetricREINFORCEGRPO
Submitted online mean peak-to-end gap0.1760.165
Saved checkpoint pairs4446
Independent held-out deterioration15.2 pp14.6 pp
Positive held-out deterioration38/4440/46

These results support the presence of held-out checkpoint deterioration beyond the original online CodeGrader measurements.

The 90 checkpoint pairs are correlated observations from the original trajectories, not 90 independent training replicates. We therefore distinguish trajectory-level descriptive statistics from seed-level statistical evidence.

2. Parameter provenance and sensitivity

The submitted interventions use different parameters and operate at different timescales.

InterventionParameterizationDecision
Live causal-prefixThree consecutive declines exceeding 0.02Stop and roll back within a campaign
Trajectory-informed budgeting (TB)clamp(peak_step(c) + 3, 8, 20)Set the next campaign's budget

The original manuscript does not document an independently optimized parameter-selection procedure. We therefore do not claim that the submitted parameters were selected using seeds disjoint from the original evaluation without verified run records.

We distinguish two sensitivity experiments.

(a) Retrospective live-trigger sensitivity

We replayed nine configurations over decline counts {2, 3, 4} and thresholds {0.01, 0.02, 0.03} on the original complete REINFORCE trajectories.

Retrospective analysisResult
Configurations9
Online checkpoint-selection advantage, range8.9–11.8 pp
Original three-decline/0.02 rule10.6 pp

This analysis is explicitly in-sample. It measures the sensitivity of retrospective checkpoint selection to the detector parameters; it is not an independent validation or a deployed chained-training comparison.

(b) Full-chain TB sensitivity

Changing TB's budget bounds changes subsequent campaign updates and checkpoint initializations. Accordingly, offline replay cannot establish the resulting end-of-chain performance.

We therefore reran complete ten-campaign TB training chains under five alternative budget-bound configurations, using the same three training seeds and the same original held-out evaluation protocol as the submitted 8–20 configuration.

TB budget boundsFinal held-out pass@1
6–1820.6%
6–2021.1%
8–1821.5%
8–20 (submitted, n=3)22.2%
10–1820.3%
10–2021.7%

The five alternative configurations achieved 20.3–21.7% end-of-chain held-out pass@1, compared with the submitted 22.2% for the original 8–20 configuration.

These full-chain reruns indicate that the qualitative benefit is not confined to the exact submitted budget bounds.

The TB sensitivity results use the original held-out evaluation protocol, not the new independent LiveCodeBench-derived test set. They are not presented as an independently tuned hyperparameter search.

3. Retrospective trigger reliability

We replayed the causal-prefix detector on 90 complete trajectories, restricting each decision to information available in the observed prefix. We predefined a positive deterioration event as an independent held-out loss of at least 5 percentage points between the original online-selected peak checkpoint and the full-budget endpoint. The threshold was fixed before inspecting the independent test outcomes.

Trigger resultDeterioration ≥5 ppDeterioration <5 pp
Trigger positive584
Trigger negative1216

This corresponds to precision 93.5%, recall 82.9%, and a false-positive rate of 20.0%.

This is a retrospective analysis of complete trajectories, not an estimate of unobserved full-budget outcomes following live stopping.

AuthorsOpenReview ↗

Response - PART 2

4. Offline checkpoint selection versus deployed chained training

We agree that offline checkpoint selection and deployed chained training must be distinguished.

Offline selection evaluates checkpoints already present in saved trajectories. Deployed selection changes the checkpoint passed to the next campaign, thereby changing subsequent training trajectories.

We first evaluated conventional selection policies retrospectively on saved naive trajectories. The resulting independent-test checkpoint-selection scores were 18.4% for fixed-step selection, 20.1% for periodic-validation selection, and 21.0% for validation-based best-checkpoint selection. These are offline checkpoint scores, not outcomes of newly trained chains.

Offline checkpoint selectionIndependent-test selected-checkpoint pass@1
Fixed-step18.4%
Periodic validation20.1%
Validation best-checkpoint21.0%

We then conducted a separate deployed comparison with ten fresh paired training seeds, disjoint from the five seeds used for the submitted Table 5 results. Each method trained ten chained campaigns using Qwen-2.5-7B under the same training configuration.

Fixed-budget training used 12 updates per campaign. Periodic validation checked a separate validation-selection set every two updates, while validation best-checkpoint selection used the same dedicated selection set. The independent final test set was not used for stopping or checkpoint selection.

MethodDeployed final pass@1 on independent test
Naive REINFORCE11.3%
Fixed-budget19.1%
Periodic validation20.6%
Validation best-checkpoint21.4%
Live causal-prefix24.8%

Live control outperformed deployed fixed-budget training by 5.7 percentage points, periodic validation by 4.2 points, and validation best-checkpoint selection by 3.4 points.

In these new runs, live control averaged 120 cumulative gradient updates over ten campaigns, matching the fixed-budget baseline's 120 updates on average.

Matching gradient updates does not imply identical total compute. Online CodeGrader monitoring and dedicated validation executions are additional costs and must be accounted for separately.

The deployed results are not inferred from the offline checkpoint-selection experiment.

5. Separation from submitted Table 5

The submitted results and the new deployed comparison use different training-seed cohorts and different final evaluation sets.

MethodSubmitted Table 5 held-outNew independent test
Naive REINFORCE11.8% (5 seeds)11.3% (10 fresh seeds)
Live causal-prefix25.6% (5 seeds)24.8% (10 fresh seeds)

The new ten-seed comparison used training seeds disjoint from those in Table 5. Its final evaluation used a fixed 300-problem LiveCodeBench-derived set, distinct from the original held-out protocol.

The numerical differences between these columns cannot be attributed to the evaluation set alone, because both the test set and training-seed cohort differ.

6. Held-out checkpoint carryover

We agree that early termination mechanically truncates the measured online peak-to-end gap. We therefore do not use the reduction of that gap under live control as independent evidence of improved optimization stability.

Instead, we evaluate the held-out quality of the checkpoints actually carried forward into subsequent campaigns.

We evaluated each carried-forward checkpoint from the new deployed ten-seed comparison on the same independent test set, without using these measurements for training decisions.

MethodMean retained pass@1 across 10 campaignsFinal pass@1
Fixed-budget15.72%19.1%
Periodic validation16.41%20.6%
Validation best-checkpoint16.92%21.4%
Live causal-prefix18.53%24.8%

The live controller maintained higher mean held-out retained-checkpoint quality than the deployed fixed-budget and validation-based alternatives across the evaluated chain.

These results support the effectiveness of trajectory-dependent checkpoint selection in preserving the quality of policies actually reused for subsequent training, rather than merely shortening the observed online trajectory.

7. Scope

Our conclusions concern chained verifier-guided code RL, where selected policy checkpoints initialize subsequent training campaigns. We do not claim a unique RL collapse mechanism or universal generalization across tasks and model families.

We thank the reviewer for motivating a clearer separation between retrospective analysis, full-chain reruns, deployed comparisons, and independent held-out evaluation.

Reviewer_dE6LOpenReview ↗

Feedback

Thank you for providing a detailed and sufficient reply. The newly added independent evaluation and deployment comparison experiments based on ten random seeds are valuable. I keep the score unchanged

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.