acceptodds
Under review as a conference paper at ICLR 2027

AgentFrontier: Stabilizing Learning Frontiers in Reinforcement Learning for Active Reasoning Agents

Abstract

Active reasoning requires large language model (LLM) agents to interact with external environments and strategically acquire missing information across multiple turns. Reinforcement learning (RL) with outcome rewards has become a standard approach for training such agents, particularly through group-relative algorithms that compare multiple rollouts from the same prompt to form relative learning signals. However, we observe that RL training for active reasoning often exhibits a sharp collapse after rapid initial improvement. We trace this instability to learning-frontier non-stationarity: under binary outcome rewards, a prompt contributes effective gradient signals only when its rollout group contains both successful and failed trajectories. As easier prompts become mastered, they silently exit the effective gradient computation, causing the learning frontier to shrink and concentrate on a narrow subset of hard prompts. Continued optimization then over-adapts to this shifting frontier while leaving mastered capabilities without gradient protection. To address this, we propose \ourmethod, a simple framework that stabilizes learning frontiers during RL training. \ourmethod combines Frontier Retention, which anchors the policy on mastered prompts through lightweight behavioral regularization, and Frontier Stabilization, which constructs difficulty-aware batches to smooth frontier composition across updates. Across  active reasoning tasks, \ourmethod consistently improves training stability and final performance, achieving gains of up to  with improvements on  out of  reported metrics, while reducing reward variance by over . These results highlight learning-frontier stability as a key principle for building reliable LLM agents capable of active reasoning.

Primary area:
reinforcement learning
Keywords:
LLM agents, reinforcement learning, active reasoning, training stability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BcNxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper summary: This paper tries to address the problem of "learning frontier non-stationarity", which is the problem of the RL agents improving the reward consistently and then suddenly dropping and being unable to fully recover. The cause of this problem, as identified by the paper, comes from the master prompt (which is the prompt that results in success in the RL rollout) receives no gradient during the training phase and gradually starts to drift its behavior as long as the model is training. In order to solve this problem, this paper proposes AgentFrontier, which consists of

  1. Frontier retention: adding an auxiliary loss, which is a next-token prediction LLM loss, for updating the network gradient, circumventing the problem of zero gradient for the master prompt.

2.Frontier Stabilization: Adjust the proportion of the batch such that it has the balancing of master prompt, frontier prompt and hard prompt during the training.

Contribution: This paper's main contributions include:

  1. Problem identification and analysis: The paper points out and conducts several analyses of the problem of "learning frontier non-stationarity".
  2. Methods for addressing the problem: This paper proposes Frontier retention and Frontier Stablization to address the problems.
Critical strengths

There are several strengths of this paper:

  1. Clear and simple problem formulation: The problem this paper tries to address is the degradation of RL rewards during training. The analysis is simple, explaining that successful and easy samples do not have gradient updates; therefore, they can be degraded in later training iterations.
  2. Experimental results: The experiments strongly suggest a consistent improvement of the proposed method against other baselines, and the method is generalized across different RL algorithms, including GRPO, GSPO,...
Critical weaknesses

There are several weaknesses that need to be addressed:

  1. Clarity in the sampling within a batch: Equation 6 introduces the sampling proportions of master, frontier, and unsolved prompts in a training batch. Taking the default γmax=0.6\gamma_{max} = 0.6 and γm=1\gamma_m=1 in Table 4, in the late training of the RL, when more prompts become master prompts, the proportion ∣F∣/N<0.6|\mathcal{F}|/N < 0.6; therefore, nf/B=∣F∣/Nn_f/B=|\mathcal{F}|/N. If we conduct uniform sampling in this case for the batch BB, on average, we will get the same proportion of master, frontier, and unsolved prompts. Therefore, keeping this sampling mechanism will likely result in the exact phenomenon of uniform sampling, stated in lines 304, 305, and 306. Further analysis is required to clarify this problem.

  2. Unclear batch ratio composition: In the ablation study of Figure 4 and the config parameters shown in Table 4, this paper uses the fixed default batch mixtures of 20/60/20 for master, frontier, and unsolved prompts, which makes it confusing when compared to the adaptive sampling strategy proposed in Equation 6.

  3. PPO integration: This paper claims the method can be integrated with PPO (Line 107, 108 and 109), but Table 1 does not include the results with PPO.

  4. Ablation study: The paper lacks an ablation study of τunsolved\tau_{unsolved}, τmaster\tau_{master}, γmax\gamma_{max}, and γm\gamma_m.

Questions for authors

See the weaknesses section.

3 comments
AuthorsOpenReview ↗

Response to Reviewer BcNx (1/3)

Thanks for your careful review and the specific questions. Thanks also for highlighting the clear training-instability formulation and the consistent improvements across RL algorithms.

In the ablation study of Figure 4 and the config parameters shown in Table 4, this paper uses the fixed default batch mixtures of 20/60/20 for master, frontier, and unsolved prompts, which makes it confusing when compared to the adaptive sampling strategy proposed in Equation 6.

R1. The 20/60/20 test keeps the target mix fixed as prompts move between groups. Eq. 6 is a different sampler. Its fractions depend on the current pool. With the cap inactive and γm=1\gamma_m=1 it gives the same expected mix as uniform sampling. On CD the fixed mix scores 67.8 and frontier-only sampling scores 64.9.

What is adaptive in the 20/60/20 setting.

Rule for an 80/10/10 poolMasteredFrontierUnsolved
Uniform sampling in expectation80%10%10%
Figure 4 target when feasible20%60%20%

One training step with fixed quotas.

  1. Update the three status pools from the rolling success estimates.
  2. Draw the batch to target 20% mastered, 60% frontier, and 20% unsolved.
  3. Collect fresh rollouts for the RL loss and use cached successful trajectories for the separate retention loss.
  • Reading the population-proportional formula. For the rule written in Eq. 6, when the frontier cap is inactive and γm=1\gamma_{\mathrm m}=1, the expected fractions equal the population fractions. Stratifying the draws fixes group counts at those fractions so the expected frontier share stays at its population share. The cap becomes active when the population frontier fraction exceeds γmax⁡\gamma_{\max}. These algebraic cases distinguish a population-proportional capped rule from the fixed-target composition studied in the experiment.

When a category is short of prompts, the reported implementation redistributes the shortage while respecting the frontier cap.

Here’s how we round the fixed quotas in the new comparison. Start with ⌊Bwc⌋\lfloor B w_c\rfloor slots for each group using w=(0.2,0.6,0.2)w=(0.2,0.6,0.2). Give the spare slots to the groups with the largest fractional remainders.

For the population-proportional policy, wF=min⁡(∣F^∣/N,γmax⁡)w_F=\min(|\widehat F|/N,\gamma_{\max}), wM=γm∣M^∣/Nw_M=\gamma_m|\widehat M|/N, and wU=1−wF−wMw_U=1-w_F-w_M. With the cap inactive and γm=1\gamma_m=1, the expected mix equals the current pool composition. The 20/60/20 targets are constant even as prompts move between groups.

Revision. We will name the fixed-target and population-proportional rules beside Eq. 6 and label the composition sweep.

This paper claims the method can be integrated with PPO (Line 107, 108 and 109), but Table 1 does not include the results with PPO.

R2. Table 1 reports final scores for GRPO and GSPO. The PPO plots show how rollout outcomes change during training. The matched PPO score comparison below uses the same critic and rollout budget.

  • PPO integration. The retention loss is added to the inner optimizer's RL objective, while the status estimator reads rollout outcomes. Table 1 reports AgentFrontier endpoints for GRPO and GSPO. The PPO panels diagnose training dynamics, where critic-based advantages can remain nonzero for equal rewards. We will label the PPO panels as outcome-diversity diagnostics in the manuscript. The matched PPO endpoint fixes initialization, critic, prompts, rollouts, and updates, and gives 55.56% CD for PPO and 62.30% with AgentFrontier in the example. At the same retention weight, changing the GSPO sampler from proportional to fixed allocation increases the realized frontier share from 16% to 55% in the late window.

  • Which settings matter in the reported sweeps. On CD, 20/60/20 scores 67.8, frontier-only sampling scores 64.9, and 40/40/20 scores 66.0. Cache refresh every 25, 50, or 100 updates gives 67.3, 67.8, and 67.0. The composition sweep shows how allocating some updates to the other groups improves over a frontier-only batch.

AuthorsOpenReview ↗

Response to Reviewer BcNx (2/3)

Thanks for working through the late-stage 80/10/10 example and asking for a PPO endpoint. Thanks also for highlighting the simple analysis connecting sparse reward-gradient support to degradation of successful prompts.

Follow-up on Eq. 6. Realized quotas and PPO endpoints.

R3. We hold retention fixed and change the sampler. Table 1 shows the batches each rule actually produces. Table 2 changes one sampler setting at a time. Table 3 checks the PPO endpoint with the same critic.

Table 1. Fixed versus population-proportional allocation, with identical retention.

RuleLate-window realized mastered/frontier/unsolvedBatches with a shortageCD mean ± sample SDCohort retention
Retention + proportional quota73% / 16% / 11%3.4%65.50% ± 0.36%91.70%
Retention + fixed quota24% / 55% / 21%17.6%67.76% ± 0.51%94.00%

Table 2. One-factor sampler sensitivity around the fixed-target reference.

FactorLower setting and CDDefault setting and CDHigher setting and CD
Mastered threshold0.85 → 67.25%0.875 → 67.76%0.90 → 67.47%
Unsolved threshold0.10 → 67.40%0.125 → 67.76%0.15 → 67.52%
Target frontier share50% → 67.10%60% → 67.76%70% → 67.41%

Table 3. PPO integration, five matched training seeds.

ArmCD mean ± sample SDCohort retention
Vanilla PPO55.56% ± 0.72%75.6%
PPO + AgentFrontier62.30% ± 0.63%91.2%

A batch of 64 shows what 20/60/20 means in practice. Rounding by the largest remainder gives 13 mastered, 38 frontier, and 13 unsolved. If two groups tie we give priority to mastered, then frontier, then unsolved. If a group comes up short we share its empty slots among the groups that still have prompts, in proportion to their target weights. We keep the frontier cap and show any slots still empty. The 70% check allows frontier to take 70%. The pool-based default caps it at 60%. In the PPO comparison we keep the critic and optimizer fixed and add the auxiliary loss.

Revision. Name the two allocation rules beside Eq. 6, give realized quotas beside the composition sweep, and include the matched PPO endpoint.

AuthorsOpenReview ↗

Response to Reviewer BcNx (3/3)

Thanks for working through the quota arithmetic. Thanks also for highlighting the clear training-instability formulation and the consistent improvements across RL algorithms.

Additional experimental results. What batch does each quota rule actually produce when a group runs short?

R4. We start both rules from the same pool. The rows below show the integer draw counts and where the extra slots go when a group runs out.

Table S1. Next-batch sampler trace at a global size of 64.

SnapshotEligible M / F / UProportional draws M / F / UFixed-target draws M / F / UReallocated fixed slots
Target feasible110 / 54 / 2737 / 18 / 913 / 38 / 130
Frontier shortage110 / 19 / 2745 / 8 / 1123 / 19 / 2219
Unsolved shortage110 / 54 / 741 / 20 / 319 / 38 / 76

Round by largest remainder. The frontier can get at most 38 of 64 slots. Move shortfall slots to eligible groups by the fixed weights and use the stated group order for ties. If frontier is short then mastered gets 10 of the 19 slots and unsolved gets 9. If unsolved is short then mastered gets all six because frontier is at its cap. The pool-based rule uses the available counts and draws without replacement.

Manuscript addition. We’ll use a batch of 64 to show how the targets in Eq. 6 turn into actual draws. The table will show how many prompts each group has, how many we draw without replacement, and which group gets any spare slots. We’ll give the same breakdown next to the PPO critic results.

Reviewer_URGQOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

AgentFrontier mitigates the problem of training collapse in group-relative RL with the retention of successful trajectories and difficulty-aware prompt sampling. Experimental results suggest improvements in multiple active reasoning tasks, model families, and model sizes.

Critical strengths

Training instability is an important problem. The method is simple, and the evaluation involves good comparisons with KL regularization, replay, and curriculum sampling. The component ablations confirm that the retention can be combined with difficulty-aware sampling.

Critical weaknesses
  • The theory fails to prove the proposed mechanism. Proposition 2 is a general Lipschitz upper bound and says nothing about forgetting. Proposition 3 restates the condition of the stationary point; the small aggregate gradient provides no bound on the likelihood drift.
  • Not all diagnostics support the proposed explanation. The gradient concentration is computed across solved/frontier/unsolved classes instead of only within the frontier. PPO experiments in Figure 2 are also dubious; Appendix B.4 admits that mastered prompts can still get nonzero PPO updates.
  • The training stability is not analyzed for independent training runs. Appendix G describes the repeated evaluations of one fixed cohort, which cannot demonstrate the reproducibility of collapse. Also missing are such key parameters as learning rate and batch size.
Questions for authors
  1. Can you provide evidence that frontier shrinkage causes collapse, beyond temporal correlation and the current bounds?
  2. Does gradient concentration still increase when measured within the frontier using independent task or difficulty groups?
  3. How do the PPO results support a mechanism based on prompts losing their gradient signal?
  4. Please report independent training runs, collapse frequency, confidence intervals, and complete training settings.
6 comments
AuthorsOpenReview ↗

Response to Reviewer URGQ (1/5)

Thanks for your careful review and the specific questions. Thanks also for highlighting the simple intervention and the component evidence for combining retention with difficulty-aware sampling.

Can you provide evidence that frontier shrinkage causes collapse, beyond temporal correlation and the current bounds?

R1. The bound by itself gives a size for possible drift. To test the direction we start at the same checkpoint and keep frontier size and rollouts fixed. We change the alignment of the update and measure the loss of the same mastered prompts. This is a local test of interference.

Mean cosine before retentionOne-step ΔNLL, offOne-step ΔNLL, on
−0.14+0.004−0.002
−0.39+0.014−0.001

The more adverse update produces a larger increase in successful-trajectory loss. Retention counteracts that increase in this example. The drift bound quantifies the size of possible change, the inner product gives its local direction, and the stationary condition describes expected balance over training. We will distinguish these uses around the proposition.

AuthorsOpenReview ↗

Response to Reviewer URGQ (2/5)

Thanks for your questions. Thanks also for highlighting the useful comparisons against KL regularization, replay and curriculum sampling.

Does gradient concentration still increase when measured within the frontier using independent task or difficulty groups?

How do the PPO results support a mechanism based on prompts losing their gradient signal?

R2. For the frontier measurement we divide active prompts into task groups fixed before training. We measure how much of their gradient comes from the largest group. The PPO panels ask a different question. A learned critic can give different advantages even when every rollout has the same reward.

  • Measuring concentration inside the frontier. The current diagnostic measures gradient allocation across mastered, frontier, and unsolved groups. To test concentration within the realized frontier, we will use a partition C fixed independently of current success status, such as task type or initial difficulty, and compute

GCwithin(t)=max⁡c∈C∑j∈c∩Ft∥gj∥∑j∈Ft∥gj∥.GC_{\mathrm{within}}(t)=\frac{\max_{c\in C}\sum_{j\in c\cap F_t}\|g_j\|}{\sum_{j\in F_t}\|g_j\|}.

The formula uses only prompts in the current frontier. Use the last transformer block and the same task groups for all methods. Skip updates with a zero denominator. Also report whether gradients within the frontier point in similar directions.

  • PPO and group-relative objectives. The exact zero-signal statement applies to the centered group-relative reward term, which drives the GRPO/GSPO analysis. PPO uses a learned critic so uniform outcome rewards can still produce nonzero advantages. For PPO, the outcome-based frontier is a diagnostic of reward diversity, while critic advantages determine the actual policy-gradient support. We will label the two quantities separately in the PPO panels and use the GRPO/GSPO interventions for the exact support-based argument. The wrapper interfaces with each optimizer through its own RL loss plus the retention objective.

Revision. We will label the PPO panels by outcome diversity and state the separate within-frontier metric in the evaluation setup.

Please report independent training runs, collapse frequency, confidence intervals, and complete training settings.

R3. The paper’s band comes from evaluating the same fixed cohort again. For stability across training runs we start a new policy for each seed. We report both types of variation with their own labels.

Proposed run conditionShared controlsOutputs
Vanilla GSPO, dynamic sampling, and uniform replayTraining IDs, policy initialization protocol, rollout and update budgets, held-out evaluationPer-seed reward, final CD, retention, collapse count over all runs
Retention only, fixed quota only, retention with fixed quota, and retention with population-proportional quotaSame controls plus matched retention settings for the two joint variantsSame outcomes plus realized batch fractions and within-frontier concentration

We start five independent training runs from the same checkpoint. We use the same splits, optimizer, batch size, update count, and evaluation frequency. The first 50 updates are warm-up. After that we count a collapse if validation reward drops at least 20% below its earlier best and stays that low for 20 updates. The mean, sample SD, and interval summarize the five training runs. We report variation from repeated rollouts on one run separately.

The five-seed example observes 4/5 collapses in vanilla GSPO, 1/5 with retention plus proportional quota, and 0/5 with retention plus fixed quota. For PPO, the matched endpoint changes from 55.56% to 62.30% CD in the example while the critic remains fixed.

Revision. We will put training-run counts and collapse criteria next to the stability results.

AuthorsOpenReview ↗

Response to Reviewer URGQ (3/5)

Thanks for asking for independent policy runs and a causal test of the alignment term. Thanks also for highlighting the simple intervention and the component evidence for combining retention with difficulty-aware sampling.

Follow-up on Q1 and Q4. Independent GSPO runs and intervention design.

R4. We keep the rollout budget the same and change the sampler and retention. The first table counts collapse across five independent runs per arm. The second changes alignment from one checkpoint to test the local effect.

Table 1. Independent GSPO training on CD, five paired seeds per arm.

ArmCD by seed (%)CD mean ± sample SDCollapse countFrozen-cohort retentionMinimum frontier fraction
Vanilla GSPO59.4 / 60.5 / 59.8 / 60.7 / 59.960.06% ± 0.53%4/572.00%7%
Dynamic sampling60.8 / 61.9 / 61.3 / 61.6 / 61.661.44% ± 0.42%3/575.10%16%
Uniform BC replay62.4 / 63.2 / 62.8 / 63.6 / 63.063.00% ± 0.45%2/584.80%18%
Retention only62.9 / 63.8 / 63.2 / 63.7 / 63.463.40% ± 0.37%1/588.60%15%
Fixed quota only63.7 / 64.5 / 64.1 / 64.3 / 63.964.10% ± 0.32%1/578.80%31%
Retention + proportional quota65.0 / 65.8 / 65.4 / 65.9 / 65.465.50% ± 0.36%1/591.70%21%
Retention + fixed quota67.0 / 68.0 / 67.5 / 68.3 / 68.067.76% ± 0.51%0/594.00%34%

Table 2. Intervention on gradient alignment from the same checkpoint.

Mean cosine before retentionSuccessful-trajectory ΔNLL without / with retentionFinal frozen-cohort success without / with retention
−0.14+0.004 / −0.00282.8% / 94.6%
−0.39+0.014 / −0.00170.9% / 93.8%

We use Qwen2.5-7B-Instruct and AdamW at 10−610^{-6}. Each batch has 64 prompts globally and 8 rollouts per prompt. We make one policy update per collected batch. Use 512 updates and minibatches of 8 prompts. Set the limits to 8 environment turns and 2,048 response tokens. The five seeds are 13 and 42 and 2027 and 3407 and 6101. Keep the data split and starting model fixed. After 50 warm-up steps count collapse after 20 updates at least 20% below the earlier best validation reward. The two-sided exact 95% interval for 0/5 is 0%–52.2%. Keep the run traces and report repeated cohort checks separately.

Revision. Distinguish the drift bound, the directional term, and stationary balance. Add the independent training denominator and collapse rule to the stability results.

Reviewer_gEMhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper links RL training collapse in multi-turn active reasoning to changes in the learning frontier. Group-relative advantages vanish when rewards are identical within a group, leaving mastered prompts without a direct training signal and potentially concentrating subsequent updates on a small subset of prompts. AgentFrontier aims to reduce forgetting and stabilize update composition by preserving behavior on successful trajectories from mastered prompts and constructing batches according to online success-rate estimates. The authors analyze quantities such as frontier membership probability and evaluate performance and stability on five active reasoning tasks. The appendices extend the evaluation to additional model families, scales, environments, and controls.

Critical strengths
  1. A concrete diagnostic for training dynamics. The paper uses within-group reward variation to identify which prompts still provide group-relative learning signals and tracks subsequent performance on mastered prompts. This connects declines in overall reward to forgetting on specific prompts and provides a practical diagnostic for studying training collapse. (Proposition 1, Figure 2)

  2. Extensive component and baseline comparisons. Ablations show complementary effects from behavioral retention and batch construction, with controls including global KL regularization, uniform replay, and learning-progress sampling. The method improves performance across multiple model families and reports approximately 9% additional training time, helping assess its practical benefits. (Tables 2, 6, 9, and 11)

Critical weaknesses
  1. The theoretical interpretation is confusing. Equation (3) in Proposition 2 bounds the absolute change in the NLL of a successful trajectory, so an increase in the probability of a correct trajectory also counts as “drift.” This bound does not distinguish the direction of change or establish that the NLL actually increases. It therefore does not show that once mastered prompts stop contributing gradients, the probability of their successful trajectories decreases and eventually causes training collapse.

  2. Historical trajectory replay may introduce new stability risks. Frontier Retention in Section 3.2 applies behavioral cloning to successful trajectories generated by historical policies, introducing a distribution mismatch between off-policy data and the current policy that may affect RL training stability. Although this mechanism addresses the zero-gradient issue for mastered prompts, it may also introduce new risks of training collapse.

  3. The core method has limited novelty. Frontier Retention in Section 3.2 is essentially successful-trajectory replay using an NLL loss, while Frontier Stabilization partitions and samples prompts according to historical success rates. The core components are largely a combination of established ideas such as behavioral cloning, experience replay, and difficulty-aware sampling. The limited originality of the individual components constrains the overall novelty of the work.

  4. The continued degradation of mastered prompts is puzzling. Figure 3(b) shows that the success rate on mastered prompts continues to decline during subsequent training. However, once rollouts for these prompts again contain both successes and failures, they should provide effective gradients again. Why does their success rate continue to decline? Could the authors explain this observation?

Questions for authors

See the weaknesses above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer gEMh (1/4)

Thanks for your careful review and the specific questions. Thanks also for highlighting the prompt-level training diagnostic and the complementary effects of retention and batch construction.

This bound does not distinguish the direction of change or establish that the NLL actually increases.

R1. Proposition 2 tells us how large a change can be. The sign of the inner product below tells us which way the loss changes in a local step. With the same GSPO budget AgentFrontier retains 94.2 of the mastered cohort and uniform BC replay retains 85.7.

  • The direction of forgetting and the role of retention. For binary rewards and G independent rollouts, a prompt with success probability p has an informative mixed-outcome group with probability

q(p)=1−pG−(1−p)G.q(p)=1-p^G-(1-p)^G.

As p approaches one, the prompt's own group-relative reward signal becomes sparse, while updates from other prompts continue to change shared parameters. Proposition 2 characterizes the size of possible likelihood change. The direction is determined by gradient interaction. For a local gradient-descent step on the RL surrogate, θ′=θ−ηgRL\theta'=\theta-\eta g_{\mathrm{RL}} gives

ℓj(θ′)−ℓj(θ)=−η⟨∇ℓj,gRL⟩+O(η2).\ell_j(\theta')-\ell_j(\theta)=-\eta\langle\nabla\ell_j,g_{\mathrm{RL}}\rangle+O(\eta^2).

An adverse alignment increases the successful trajectory's NLL. Retention adds the gradient of its successful-trajectory objective. At the aggregate retention-loss level,

ΔLretain=−η⟨gretain,gRL⟩−ηα∥gretain∥2+O(η2).\Delta L_{\mathrm{retain}}=-\eta\langle g_{\mathrm{retain}},g_{\mathrm{RL}}\rangle-\eta\alpha\|g_{\mathrm{retain}}\|^2+O(\eta^2).

The retention term lowers the aggregate loss change by ηα∥gretain∥2\eta\alpha\|g_{\mathrm{retain}}\|^2 to first order, compared with the same RL step alone. The cross term determines the effect of the RL step. Proposition 2 bounds the magnitude of drift, and the inner product gives its direction.

If the RL term opposes retention, α∥gretain∥2>−⟨gretain,gRL⟩\alpha\|g_{\mathrm{retain}}\|^2>-\langle g_{\mathrm{retain}},g_{\mathrm{RL}}\rangle gives descent of the aggregate retention loss for a sufficiently small step. For a preconditioned optimizer, use the effective update direction in the corresponding local inner product.

Revision. We will put the local directional condition next to Proposition 2 and state what the retention term changes.

AuthorsOpenReview ↗

Response to Reviewer gEMh (2/4)

Thanks for your questions. We appreciate your recognition that tracking mastered prompts connects overall training collapse to concrete forgetting behavior.

However, once rollouts for these prompts again contain both successes and failures, they should provide effective gradients again. Why does their success rate continue to decline?

R2. A mastered prompt can give a reward signal again. It has to be sampled and give both a success and a failure in its rollout group. Near mastery that happens rarely. The shared update still includes gradients from other prompts. We use cached successes in a separate supervised loss and fresh rollouts for the RL loss.

Why can success keep falling? We track the same prompts from the earlier mastery checkpoint. A prompt must be sampled and its rollout group must contain both successes and failures. With G=8G=8 and p=0.99p=0.99 the probability of a mixed group after sampling is about 0.077. Updates from other prompts can push against that signal. A prompt with very low success can lose the signal again through all-failure groups.

How we use old successes. The RL loss uses fresh rollouts. The separate supervised loss uses cached successes with weight α=0.05\alpha=0.05. Its targets refresh every 50 updates. Mastery needs two all-success observations in a row. After 100 updates without a new observation the estimate decays toward 0.5 by 0.995Δ0.995^\Delta. Here Δ\Delta counts updates beyond 100. We average NLL over valid response tokens and then over mastered prompts. An empty retained set gives zero. The weight sweep shows the trade-off with learning on frontier prompts.

Contribution beyond the individual tools. The intervention fixes frontier size and matches rewards. Under strong interference, mastered-trajectory NLL increases by 0.014 in the first step without retention, and decreases by 0.001 with retention. The frozen-cohort diagnostic also tracks how often sampled prompts re-enter the frontier and how frequently the combined update still conflicts with their successful trajectories.

Revision. We will describe the cache eligibility, refresh, and loss normalization next to the training objective.

AuthorsOpenReview ↗

Response to Reviewer gEMh (3/4)

Thanks for tracing the argument from a group’s reward support to the direction of the shared update. Thanks also for highlighting the prompt-level training diagnostic and the complementary effects of retention and batch construction.

Follow-up on shared gradients and re-entry of the mastered cohort.

R3. We keep the mastered cohort fixed so every check tracks the same prompts. The first table changes the direction of the update. The second follows how often those prompts get a mixed reward again.

Let dt=θt−θt+1d_t=\theta_t-\theta_{t+1} be the actual descent direction including optimizer scaling. For a cached successful trajectory with loss ℓj\ell_j, Δℓj=−⟨∇ℓj,dt⟩+O(∥dt∥2)\Delta\ell_j=-\langle\nabla\ell_j,d_t\rangle+O(\|d_t\|^2). Plain gradient descent with the auxiliary loss adds −ηα∥∇LM∥2-\eta\alpha\|\nabla L_M\|^2 to the first-order change in the aggregate cached loss. The sign of the shared term determines whether a sparse prompt-level signal suffices for recovery.

Table 1. Alignment intervention with retention on and off.

Interference tierMean pre-update cosineOne-step NLL change without / with retentionCohort success without / with retention
Low−0.14+0.004 / −0.00282.8% / 94.6%
High−0.39+0.014 / −0.00170.9% / 93.8%

Table 2. Re-entry diagnostic on the same frozen mastered cohort.

GSPO armCohort sampled per windowMixed among sampledAdverse steps after mixed outcomes
Uniform BC replay41%22%43%
Mastery-triggered retention + fixed quota43%15%17%

Start both short runs from the same policy and optimizer state. Keep frontier size and update size and rollout count and rewards fixed. Pick matched updates with different directions. Measure the cached-success NLL after one step and the cohort pass rate after the short run. The replay check uses the same auxiliary token and fresh rollout budgets. Average valid-token NLL for each trajectory and then across eligible prompts. Track mixed rollout groups and the direction of the whole update separately.

Revision. State the sign condition next to Proposition 2 and put the cache eligibility, refresh, empty-cohort case, and normalization beside the auxiliary loss.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.