acceptodds
Under review as a conference paper at ICLR 2027

Selection–Development Lock-in: Access Starvation in On-Policy Learning

Abstract

On-policy learning changes not only the current policy but also the distribution of data available for future learning, a feedback particularly relevant to modern generative-model fine-tuning. A behavioral region may initially perform poorly yet improve with continued exploration, while policy optimization can reduce its future sampling opportunities to negligible levels before its competence develops sufficiently. We call this selection–development lock-in (SDL). In a canonical environment, we formalize and analyze SDL and show that conditional development requires only polynomial effective learning exposure, yet acquiring that exposure through on-policy sampling takes exponential time as initial competence decreases. We extend the analysis to general on-policy dynamics through two quantities: the support lost per unit of conditional improvement and the dependence of development on sampling access. Under explicit regularity conditions, sufficiently steep support loss combined with access-dependent development produces an exponential barrier. We then conduct controlled shared-parameter neural experiments that reproduce the predicted support valley and delayed development beyond the exactly solvable model. These results identify how locally reward-improving selection can make attainable future improvement prohibitively costly within a training budget.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, reinforcement learning theory, Group Relative Policy Optimization, on-policy learning, policy optimization, exploration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YAznOpenReview ↗
Rating
3(+1)
Confidence
2(+1)
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies selection-development lock-in (SDL), a failure mode in on-policy RL where a behavior branch XX with low initial competence q0=P(GX∣X)q_0 = P(G_X \mid X) loses probability mass p=P(X)p = P(X) to a safer fallback YY before getting enough samples to raise qq above YY's reward. In a 3-outcome tabular GRPO model (X+,X−,YX_+, X_-, Y with rewards 1,0,1/21, 0, 1/2), the authors show that below maturity qm=1/2q_m=1/2, expected selection drift is negative while conditional competence drift is positive (Theorem 1), scaling as −gl∼SG(p)-g_l \sim S_G(p) and dq/dt∼ηdDG(p)q2dq/dt \sim \eta_d D_G(p) q^2 for small qq. Integrating dL/dq∼−(ρ/2)q−2dL/dq \sim -(\rho/2)q^{-2} yields an exponential support valley −log⁡pmin⁡∼ρ/(2q0)-\log p_{\min} \sim \rho/(2q_0) and reversal time log⁡Trev∼ρ/(2q0)\log T_{\text{rev}} \sim \rho/(2q_0) despite O(q0−1)O(q_0^{-1}) effective exposure (Theorem 2). Proposition 2 extends this to general trajectories with −dL/dq∼κq−δ-dL/dq \sim \kappa q^{-\delta} and dq/dt=pγddq/dt = p^\gamma d, paired with MNIST MLP experiments.

Critical strengths
  • The 3-outcome tabular GRPO analysis (Prop. 1, Theorems 1-2) is clean and exact. Showing that success-free mixed groups (X−X_- and YY, with non-zero reward variance) still boost conditional competence (gh>0g_h > 0) via within-group standardization while inducing a −(ρ/2)q−2-(\rho/2)q^{-2} phase slope is a nice insight.
  • Separating effective exposure τ(t)=Θ(q0−1)\tau(t) = \Theta(q_0^{-1}) from wall-clock sampling time Trev=exp⁡(Θ(q0−1))T_{\text{rev}} = \exp(\Theta(q_0^{-1})) (Eqs. 9-10) clarifies exploration starvation.
Critical weaknesses
  1. The neural runs in Sec. 5 violate the monotone competence growth required by the theory. Both Thms. 1-2 and Prop. 2 (and Prop. 6) require competence to grow (dq/dt>0,dmin⁡>0dq/dt > 0, d_{\min}>0) along the pre-reversal segment up to a reversal level qIq_I. Yet in Table 4 (p. 24) and l. 1354 (p. 26), the MLP loses competence before Tpmin⁡T_{p_{\min}}, and at q0=18%q_0=18\% (100k steps vs. 20k for 10–16%10\text{--}16\%, l. 1269) overall qq at reversal is below initialization (Δq=−0.48\Delta q = -0.48 pp; +1.87+1.87 pp from 0.20200.2020 to 0.22070.2207 at 20%20\%). Also, Eq. (66) gives qvisit=q+Covν(po,qo)/pˉq_{\text{visit}} = q + \text{Cov}_\nu(p_o,q_o)/\bar{p}, but Covν(po,qo)/pˉ\text{Cov}_\nu(p_o,q_o)/\bar{p} is never tracked to test if context routing drives reversal.
  2. In Eqs. (24, 32), dq/dt∼q2dq/dt \sim q^2 (δ=2\delta=2) relies on a 1D sigmoid (∂hlog⁡(1−q)=−q\partial_h\log(1-q)=-q) without KL/entropy penalties. Also, Eqs. (88-89) fit only 5 points (q0∈{10–18}%q_0 \in \{10\text{--}18\}\%) with unequal horizons (20k vs. 100k) and give different 1/q01/q_0 slopes (2.2742.274 vs. 3.3113.311).
Questions for authors
  1. Since qq regresses before Tpmin⁡T_{p_{\min}} (Δq=−0.48\Delta q = -0.48 pp at 18%18\%, Table 4), violating dq/dt>0dq/dt > 0 in Prop. 2, did you track Covν(po,qo)/pˉ\text{Cov}_\nu(p_o,q_o)/\bar{p} (Eq. 66) to see if routing drives reversal?
  2. Why was only 18%18\% run 100k steps (l. 1269), and how do Eqs. (88-89) change at 20k/100k?

Above all, my major concern is the non-monotone neural dynamics, I'm open to discuss.

Rating 2

3 comments
AuthorsOpenReview ↗

Response to Reviewer YAzn

We sincerely thank you for sparing the significant time and effort reading our paper and giving many insightful comments. The major concerns are summarized below and addressed point by point, to help understand our work better.


1. Non-monotone neural dynamics, routing, and the scope of Prop. 2

In the shared-neural experiments, aggregate fixed-context competence can be non-monotone due to shared parameters and other non-canonical effects. We do not claim that these neural trajectories strictly satisfy all assumptions of Prop. 2. Routing and shared parameters may also affect the detailed reversal dynamics, but not eliminate the underlying access bottleneck. Rather, the neural experiments are intended to test whether the SDL signatures predicted by the canonical construction: support valleys, access-dependent delay, and lost developmental opportunity, persist beyond the exactly solvable setting.

Importantly, a temporary regression in qq does not weaken the starvation interpretation. If the learner is ultimately to reach the same competence level, any decrease in qq must later be compensated by additional competence gain. More precisely, Prop. 2 characterizes the access cost of monotone development phases, while competence regression in the neural system introduces additional development that must subsequently be recovered.

2. Training horizons and regression fits

Regarding the training horizons, in fact, we extended all conditions with q0=0.10q_0=0.10–0.180.18 to 100K updates to resolve censoring. For q0=0.18q_0=0.18, the number of resolved seeds increased from 7/10 at 20K to 10/10 at 100K, whereas all runs with q0=0.10q_0=0.10–0.160.16 remained horizon-censored even at 100K. This led us to report only q0=0.18q_0=0.18 as a 100K condition in the paper because it was the only one whose censoring status changed. We agree that this was insufficiently clear. Thank you for pointing it out.

But we would also like to emphasize that Eqs. 88–89 do not fit five points with q0=0.10q_0=0.10–0.180.18. The regressions use the seven resolved conditions q0=0.18,0.20,…,0.30q_0=0.18,0.20,\ldots,0.30, while q0=0.10q_0=0.10–0.160.16 are excluded entirely because they remain censored. All conditions with q0=0.20q_0=0.20–0.300.30 were already resolved before 20K updates. Therefore, for the purpose of censoring and the reported regression, the data are effectively equivalent to using a common 100K horizon across all conditions: extending the horizon changes the status of q0=0.18q_0=0.18, but neither introduces any q0=0.10q_0=0.10–0.160.16 point into the fit nor changes the already-resolved q0=0.20q_0=0.20–0.300.30 observations.

3. Additional RLVR evidence

For completeness, we add an RLVR experiment below as complementary evidence beyond the controlled MNIST setting. We use Qwen2.5-0.5B-Instruct on GSM8K. The model chooses between SOLVE and SKIP: SKIP receives reward 0.5, while SOLVE receives 1 only for a correct answer and 0 otherwise. Across three seeds and 300 rollout batches, we measure conditional competence qq as forced-SOLVE accuracy and access pp as the model's probability of choosing SOLVE.

Natural050100150200250300
q32.56%37.97%38.82%38.93%38.89%39.39%41.02%
p79.92%0.0382%0.0148%0.0104%0.0129%0.0214%1.05%

Thus, Natural rapidly loses almost all SOLVE access, after which conditional competence improves only marginally. We forked Rescue from Natural checkpoint at batch 100 and restored behavioral access by sampling SOLVE/SKIP with probability 0.5 each. The result shows that access starvation is a major cause of slow development.

Rescue100150200250300
q38.82%55.99%58.92%60.16%60.33%
p0.0148%77.31%78.28%75.81%80.27%

As a complementary learnability control, Forced access from initialization reaches 67.27% conditional competence at batch 300.

Together, the Natural trajectories show that conditional development stalls as access collapses, while the same-checkpoint interventions show that development resumes once access is restored. This further supports SDL in this RLVR environment.

The authors have partly addressed my concerns, so I have raised my score to 3.

AuthorsOpenReview ↗

Response to Reviewer YAzn

Thank you for taking the time to review our response and reconsider your assessment.

We are glad that our clarifications and additional evidence addressed part of your concerns. Your comments have been very helpful in sharpening the scope of our claims. In particular, you prompted us to make the distinction more explicit between the exact theoretical results and the neural experiments. We have incorporated these clarifications into the revised manuscript to improve the clarity and precision of our presentation.

Thank you again for your thoughtful review and constructive engagement. Your feedback has helped us present both the contribution and its scope more precisely.

Reviewer_deFNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses selection-development lock-in (SDL). The paper formalizes and analyzes SDL. It separates the probability of entering a behavioral region from the conditional competence within that region. The authors show that the effective learning exposure required for conditional development can be only polynomial, while the natural on-policy sampling time can become exponential. The experiment on MNIST classification identifies how locally reward-improving selection can make attainable future improvement prohibitively costly within a training budget.

Critical strengths
  • The analysis is interesting: the paper clarifies that insufficient exploration can be harmful and characterizes when support loss and access-dependent development produce an exponential barrier.
  • The separation between access probability and conditional competence is clean and important. It can distinguish losing behavioral support from losing the opportunity.
Critical weaknesses
  • I am afraid that the experiment is constructed too purposefully to support the proposed mechanism. The fallback action receives a fixed reward, while the digit region is initialized with conditional accuracy below that reward, so it is apparent that on-policy optimization may suppress digit actions. This shows the predicted dynamics, but provides limited evidence for the strength of SDL.
  • The central intuition is related to well-known exploration, coverage, and mode-collapse phenomena. The main novelty appears to be the formalization of the mechanism through separate selection and development dynamics.
Questions for authors
  • Do you have any other experimental results or evidence of SDL emerging naturally in more natural RL settings? Or do you claim that the MNIST result generalizes well?
2 comments
AuthorsOpenReview ↗

Response to Reviewer deFN

We sincerely thank you for sparing the significant time and effort reading our paper and giving many insightful comments. The major concerns are summarized below and addressed point by point, to help understand our work better.


1. Controlled construction and the relevance of the fallback setting

We agree that, because the fallback reward is higher than the initial conditional accuracy of the digit region, the direction of the initial selection pressure is expected by construction. This is intentional because this situation is likely to occur as well in real scenarios. For example, in code RL, a model may already execute a familiar heuristic reliably, while a different algorithmic strategy is initially less successful because the model has not yet learned to implement it consistently but could become more reliable after sufficient training. The familiar strategy can have higher verified reward at first, which can correspond to the high fallback reward in our neural experiment. This may also occur in scenarios involving RLHF, text-to-image generation models, etc.

How common this situation is remains uncertain and requires further research and this is precisely what we discuss in limitations and future work, but the phenomenon itself is meaningful enough. We choose such a scenario to facilitate better and more convenient construction. Moreover, the experiments also show that it is difficult to achieve sufficient performance under natural on-policy sampling which is not so obvious.

2. Distinction from exploration, coverage, and mode collapse

Recent RLVR mode-collapse literature studies probability concentration away from behaviors that are already useful or verifier-correct. SDL instead concerns a region that can be genuinely worse under the current policy, yet whose conditional competence could improve through further learning enough to become the better alternative. Selection against such a region can be locally reward-improving, not an erroneous estimate of its current value. This is precisely the theoretical tension highlighted by SDL: reducing the probability of a currently worse region is a correct update with respect to the current reward, yet the same update also reduces that region's future learning access and can therefore prevent it from developing into the better attainable solution.

Exploration and coverage problems concern how a learner acquires sufficiently informative experience. Our analysis characterizes when their interaction produces a support valley and an exponential sampling barrier. Therefore, SDL provides a more specific formalization of how locally reward-improving selection can endogenously remove the learning opportunities required for a currently weaker but improvable behavior to develop.

3. Additional experimental results

We add an RLVR experiment using Qwen2.5-0.5B-Instruct on GSM8K. The model chooses between SOLVE and SKIP: SKIP receives reward 0.5, while SOLVE receives 1 only for a correct answer and 0 otherwise. Across three seeds and 300 rollout batches, we measure conditional competence qq as forced-SOLVE accuracy and access pp as the model's probability of choosing SOLVE.

Natural050100150200250300
q32.56%37.97%38.82%38.93%38.89%39.39%41.02%
p79.92%0.0382%0.0148%0.0104%0.0129%0.0214%1.05%

Thus, Natural rapidly loses almost all SOLVE access, after which conditional competence improves only marginally. We forked Rescue from Natural checkpoint at batch 100 and restored behavioral access by sampling SOLVE/SKIP with probability 0.5 each. The result shows that access starvation is a major cause of slow development.

Rescue100150200250300
q38.82%55.99%58.92%60.16%60.33%
p0.0148%77.31%78.28%75.81%80.27%

As a complementary learnability control, Forced access from initialization reaches 67.27% conditional competence at batch 300.

Together, the Natural trajectories show that conditional development stalls as access collapses, while the same-checkpoint interventions show that development resumes once access is restored. This further supports SDL in this RLVR environment.

Thank you again for your time and constructive review. As the discussion period is coming to an end, we would like to check whether our responses have addressed your concerns. We look forward to receiving your feedback at your earliest convenience.

Reviewer_NcodOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents an analysis of "selection-dependent lock-in" in RL, where an on-policy algorithm performs probabilistic action selection, which in turn impacts observations. SDL describes a potential failure mode in regions of the state-action-space with initially moderate returns, which could become huge if more learning were to happen there . Since returns are not excessive, this region is sampled less often due to learning, and at some point the huge rewards are no longer attainable due to insufficient sampling. The paper formulates condition under which this can or cannot happen, and provides limited experimental support both for the effect and the conditions for its occurrence.

Critical strengths

The paper treats a relevant problem in RL, and proposes an in-depth mathematical analysis under certain assumptions.

Critical weaknesses

It is unclear what would be the application of this analysis. Either in practice: for LLM training, or RL in robotics, ... Or as a contribution to advance theoretical understanding: in this case, I would expect a much clearer link to common RL algorithms (e.g., PPO), and an argument as to why the assumptions made here hold. The experimental support is constructed for a clear toy problem: are there no standard RL benchmarks (gymnasium or similar) that could be used here? In any case, anyone would agree that SDL exists. The question is: does it occur in a lot of relevant scenarios, and what can this paper's results achieve for fixing the problem?

Questions for authors
  • What is the impact of these results on RL theory, or RL applications? Can you give concrete examples to motivate this research?
  • What is the rationale behind the conducted experimental work? Why not use one of the standard RL benchmarks as well?
  • How generally applicable are the assumptions being made in sec. 3.3?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Ncod

We sincerely thank you for sparing the significant time and effort reading our paper and giving many insightful comments. The major concerns are summarized below and addressed point by point, to help understand our work better.


1. Impact and concrete examples

The contribution is not merely the observation that insufficient exploration or support loss can occur. It is by separating behavioral access from conditional competence that we expose a distinct failure mode of on-policy learning that is obscured when support and competence are treated jointly. Existing support-loss analyses largely ask whether useful or diverse behaviors are preserved. Our decomposition asks a different question: whether a behavior that is currently inferior but developmentally improvable can lose the very access required to become superior.

More importantly, our analysis separates the intrinsic learning requirement of such development from the access cost of obtaining effective learning exposure under the evolving on-policy distribution. We show that these two complexities can be fundamentally different: conditional development may require only polynomial effective exposure, while acquiring that exposure through natural on-policy sampling can require exponential time. Therefore, we show that the barrier is a data-access complexity barrier, not an intrinsic learning-complexity barrier.

This failure mode can also occur in real scenarios. For example, in code RL, an alternative algorithmic strategy may initially achieve lower reward because the model has not yet learned to implement it consistently but could become more reliable after sufficient training. On-policy optimization may suppress its access before that development occurs. This may also occur in scenarios involving RLHF, text-to-image generation models, etc. Moreover, we add an RLVR experiment and you can read below.

Regarding the questions raised under Weaknesses, our goal is to identify and characterize this mechanism, not to establish its prevalence across applications or propose a new mitigation algorithm. We view both as natural follow-up questions enabled by the present characterization.

2. Experimental rationale, RL benchmarks and additional RLVR evidence

The controlled experiments were chosen for mechanism identification rather than benchmark performance. Testing SDL requires separately measuring access pp and conditional competence qq, varying their initialization, and constructing counterfactual controls that determine whether a suppressed region would improve if access were maintained. In a generic benchmark these quantities are difficult to isolate, and ordinary task failure would not distinguish lack of learnability from access starvation.

More importantly, we add an RLVR experiment using Qwen2.5-0.5B-Instruct on GSM8K. The model chooses between SOLVE and SKIP: SKIP receives reward 0.5, while SOLVE receives 1 only for a correct answer and 0 otherwise. Across three seeds and 300 rollout batches, we measure conditional competence qq as forced-SOLVE accuracy and access pp as the model's probability of choosing SOLVE.

Natural050100150200250300
q32.56%37.97%38.82%38.93%38.89%39.39%41.02%
p79.92%0.0382%0.0148%0.0104%0.0129%0.0214%1.05%

Thus, Natural rapidly loses almost all SOLVE access, after which conditional competence improves only marginally. We forked Rescue from Natural checkpoint at batch 100 and restored behavioral access by sampling SOLVE/SKIP with probability 0.5 each. The result supports that access starvation is a major cause of slow development.

Rescue100150200250300
q38.82%55.99%58.92%60.16%60.33%
p0.0148%77.31%78.28%75.81%80.27%

As a complementary learnability control, Forced access from initialization reaches 67.27% conditional competence at batch 300.

Together, the Natural trajectories show that conditional development stalls as access collapses, while the same-checkpoint interventions show that development resumes once access is restored. This further supports SDL in this RLVR environment.

3. General applicability of the assumptions in Sec. 3.3

The assumptions specific to Sec. 3.3 are chosen to make the selection–development feedback exactly solvable. The general claim is instead made in Sec. 4. We do not intend this canonical construction itself to be a general model of RL. Its purpose is instead to isolate the mechanism and identify the two quantities that control the barrier.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.