- Rating
- 3(+1)
- Confidence
- 2(+1)
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies selection-development lock-in (SDL), a failure mode in on-policy RL where a behavior branch with low initial competence loses probability mass to a safer fallback before getting enough samples to raise above 's reward. In a 3-outcome tabular GRPO model ( with rewards ), the authors show that below maturity , expected selection drift is negative while conditional competence drift is positive (Theorem 1), scaling as and for small . Integrating yields an exponential support valley and reversal time despite effective exposure (Theorem 2). Proposition 2 extends this to general trajectories with and , paired with MNIST MLP experiments.
Critical strengths
- The 3-outcome tabular GRPO analysis (Prop. 1, Theorems 1-2) is clean and exact. Showing that success-free mixed groups ( and , with non-zero reward variance) still boost conditional competence () via within-group standardization while inducing a phase slope is a nice insight.
- Separating effective exposure from wall-clock sampling time (Eqs. 9-10) clarifies exploration starvation.
Critical weaknesses
- The neural runs in Sec. 5 violate the monotone competence growth required by the theory. Both Thms. 1-2 and Prop. 2 (and Prop. 6) require competence to grow () along the pre-reversal segment up to a reversal level . Yet in Table 4 (p. 24) and l. 1354 (p. 26), the MLP loses competence before , and at (100k steps vs. 20k for , l. 1269) overall at reversal is below initialization ( pp; pp from to at ). Also, Eq. (66) gives , but is never tracked to test if context routing drives reversal.
- In Eqs. (24, 32), () relies on a 1D sigmoid () without KL/entropy penalties. Also, Eqs. (88-89) fit only 5 points () with unequal horizons (20k vs. 100k) and give different slopes ( vs. ).
Questions for authors
- Since regresses before ( pp at , Table 4), violating in Prop. 2, did you track (Eq. 66) to see if routing drives reversal?
- Why was only run 100k steps (l. 1269), and how do Eqs. (88-89) change at 20k/100k?
Above all, my major concern is the non-monotone neural dynamics, I'm open to discuss.
Rating 2
3 comments
The authors have partly addressed my concerns, so I have raised my score to 3.
Response to Reviewer YAzn
Thank you for taking the time to review our response and reconsider your assessment.
We are glad that our clarifications and additional evidence addressed part of your concerns. Your comments have been very helpful in sharpening the scope of our claims. In particular, you prompted us to make the distinction more explicit between the exact theoretical results and the neural experiments. We have incorporated these clarifications into the revised manuscript to improve the clarity and precision of our presentation.
Thank you again for your thoughtful review and constructive engagement. Your feedback has helped us present both the contribution and its scope more precisely.
Response to Reviewer YAzn
We sincerely thank you for sparing the significant time and effort reading our paper and giving many insightful comments. The major concerns are summarized below and addressed point by point, to help understand our work better.
1. Non-monotone neural dynamics, routing, and the scope of Prop. 2
In the shared-neural experiments, aggregate fixed-context competence can be non-monotone due to shared parameters and other non-canonical effects. We do not claim that these neural trajectories strictly satisfy all assumptions of Prop. 2. Routing and shared parameters may also affect the detailed reversal dynamics, but not eliminate the underlying access bottleneck. Rather, the neural experiments are intended to test whether the SDL signatures predicted by the canonical construction: support valleys, access-dependent delay, and lost developmental opportunity, persist beyond the exactly solvable setting.
Importantly, a temporary regression in q does not weaken the starvation interpretation. If the learner is ultimately to reach the same competence level, any decrease in q must later be compensated by additional competence gain. More precisely, Prop. 2 characterizes the access cost of monotone development phases, while competence regression in the neural system introduces additional development that must subsequently be recovered.
2. Training horizons and regression fits
Regarding the training horizons, in fact, we extended all conditions with q0=0.10–0.18 to 100K updates to resolve censoring. For q0=0.18, the number of resolved seeds increased from 7/10 at 20K to 10/10 at 100K, whereas all runs with q0=0.10–0.16 remained horizon-censored even at 100K. This led us to report only q0=0.18 as a 100K condition in the paper because it was the only one whose censoring status changed. We agree that this was insufficiently clear. Thank you for pointing it out.
But we would also like to emphasize that Eqs. 88–89 do not fit five points with q0=0.10–0.18. The regressions use the seven resolved conditions q0=0.18,0.20,…,0.30, while q0=0.10–0.16 are excluded entirely because they remain censored. All conditions with q0=0.20–0.30 were already resolved before 20K updates. Therefore, for the purpose of censoring and the reported regression, the data are effectively equivalent to using a common 100K horizon across all conditions: extending the horizon changes the status of q0=0.18, but neither introduces any q0=0.10–0.16 point into the fit nor changes the already-resolved q0=0.20–0.30 observations.
3. Additional RLVR evidence
For completeness, we add an RLVR experiment below as complementary evidence beyond the controlled MNIST setting. We use Qwen2.5-0.5B-Instruct on GSM8K. The model chooses between SOLVE and SKIP: SKIP receives reward 0.5, while SOLVE receives 1 only for a correct answer and 0 otherwise. Across three seeds and 300 rollout batches, we measure conditional competence q as forced-SOLVE accuracy and access p as the model's probability of choosing SOLVE.
Thus, Natural rapidly loses almost all SOLVE access, after which conditional competence improves only marginally. We forked Rescue from Natural checkpoint at batch 100 and restored behavioral access by sampling SOLVE/SKIP with probability 0.5 each. The result shows that access starvation is a major cause of slow development.
As a complementary learnability control, Forced access from initialization reaches 67.27% conditional competence at batch 300.
Together, the Natural trajectories show that conditional development stalls as access collapses, while the same-checkpoint interventions show that development resumes once access is restored. This further supports SDL in this RLVR environment.