acceptodds
Under review as a conference paper at ICLR 2027

Specialist Drift: Continual RL for Capability-Preserving Foundation Model Post-Training

11 views

Abstract

We ask whether reinforcement-learning (RL) post-training of foundation models is better understood as single-task optimization or as an unrecognized continual RL (CRL) problem. Existing pipelines (RLHF, RLVR, agentic RL) report target-capability gains without characterizing which prior capabilities silently decay, or by how much. We present a parameter-matched causal decomposition of this specialist drift, isolating distribution narrowing (61.1%), reward gradient asymmetry (17.8%), token-entropy collapse (13.9%), and KL anchor decay (7.2%) as distinct contributors to capability erosion across four post-training stages and three open-weight backbones. Building on this decomposition, we propose Subspace-Constrained Policy Optimization (SCPO), which retains 92.1% of base-model capability tomography while matching 99.1% of the unconstrained PPO target reward on competition math, a 6.4 improvement in retention per unit of target gain over global KL anchoring. We further introduce two preservation metrics, CPS and DRI, along with a fitted scaling law (S.D.S.L.) relating subspace-anchor strength to capability retention as the training horizon grows (, ), showing that behavioral subspace structure, rather than parameter count or KL budget, governs the retention frontier in long-horizon RL post-training. Extended experiments include a new strongest baseline (Heterogeneous Model Averaging), modern continual-RL preservation baselines (CPO, RPO), divergence-regularized baselines (DRPO/DPPO), SFT/replay-SFT controls, out-of-sample validation of the scaling law, and a Fisher-information verification protocol confirming the theoretical predictions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
specialist drift, capability preservation, subspace-constrained policy optimization, RL post-training, catastrophic forgetting, scaling laws

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_F75LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper seems to treat RL post-training as continual RL. It claims a four-way split of capability loss, then a subspace-anchored PPO (SCPO) that keeps about 92% of a capability score while nearly matching unconstrained PPO on math. It also proposes two retention metrics and a scaling law.

Critical strengths

The question looks useful. Target-reward curves do not say which old capabilities moved, and it seems worth separating those losses instead of calling all of them an alignment tax.

Critical weaknesses

The claimed split does not seem to follow from the ablation as written. Several rows move in the opposite direction from the delta printed next to them, and the prose then assigns the same shares to a different set of mechanisms.

Questions for authors

Why anchor a probed subspace inside PPO, rather than a null-space or Fisher penalty fit only on held-out behaviors the scorer never sees?

Why is the main pipeline PPO on verifiable rewards, when the paper’s own motivation is RLHF, GRPO, and agentic RL? It is not clear why those update rules were not the primary test.

2 comments
AuthorsOpenReview ↗

Official Rebuttal of Submission 13129 by Authors

Thank you for the review. Your main concern was correct for the submitted version: the ablation table and the prose did not agree. The revision fixes this at the source, and both questions are answered with results already in the paper. Table numbers refer to the revision.

W1. "The split does not follow from the ablation; rows move opposite to their Δ; the prose assigns the shares to different mechanisms."

Cause. Several Δs in the submitted table were computed from a stale SCPO reference (96.3) rather than the reported 92.1. That is why some rows appeared to move against their printed Δ. The prose also attached shares to the wrong removal; for example, the distribution-narrowing loss was described as the entropy-floor removal.

Fix.

  • In Table 7, every Δ is now CPS − 92.1 at its own horizon. We checked all 39 Δ entries with a script, and each one equals its row's CPS minus the reference, sign included.
  • The split is now three components, and Section 6.1 names the same mechanisms with the same numbers as the table:
Component removedΔ at 12KShare at 12KΔ at 45KShare at 45K
Distribution-narrowing penalty−13.174.0%−20.982.3%
Entropy floor−3.016.9%−3.011.8%
KL anchor−1.69.0%−1.55.9%
Sum−17.7−25.4
  • Reward-gradient asymmetry was the fourth "mechanism." It is not a term of the SCPO objective, so removing one SCPO component cannot isolate it. It is now treated only as a pipeline factor in a 2⁴ factorial design (Table 20).
  • The factorial design is a direct check that the split does not depend on removing one component at a time. Its main effects for the three components (−13.2, −3.1, −1.7) are within 0.1 points of the single-removal losses. All pairwise interactions are ≤ 0.4 points, summing to +1.3.
  • We now also state what the split does not explain. The three components account for 58% of the gap to Vanilla PPO at 12K (17.7 of 30.7 points) and 40% at 45K (25.4 of 64.1). The rest is reported as unattributed.
  • The "scaling law" is withdrawn. The stage fits are labelled "descriptive," and a held-out fifth stage follows neither.

Q1. Why a probed subspace inside PPO, rather than a null-space or Fisher penalty fit only on held-out behaviors the scorer never sees?

  • Held-out fitting already holds. The subspace is fitted only on 18,000 Tulu-3 prompts with Tulu-3 category labels. All tuning uses a separate 2,000-prompt Tulu-3 pool. The six CPS benchmarks are never seen during construction or selection, and an 8-gram audit finds 0 overlapping prompts (Section 3.3, Table 1).
  • Fisher penalty. The full Fisher is O(|θ|²) at 7B and intractable. Diagonal Fisher (EWC-PPO) is our closest tractable baseline. It retains 89.4% at 12K and 73.4% at 45K, versus SCPO's 92.1% at both horizons (Tables 2 and 5).
  • Null-space projection. A hard projection forbids all movement in S, which costs target reward whenever target-task directions overlap S. As the anchor strength α grows, our soft anchor approaches a hard constraint, and Table 12 shows that cost. Target reward falls from 43.3 (α = 0.1) to 40.8 (α = 1) and 31.2 (α = 10) for at most 3.2 more CPS points.
  • Is the probing what matters? Constraining a random subspace of equal rank retains 71.4%, against 92.1% for the probed one. A gradient-based subspace retains 89.5% and an SAE-feature subspace 92.4%, so the gain comes from selecting capability-relevant directions.
  • Not yet compared. We have not run K-FAC Fisher penalties or an exact null-space projection, and the paper states this as a limitation.

Q2. Why PPO on verifiable rewards when the motivation is RLHF, GRPO, and agentic RL?

  • Why verifiable rewards? They make the loss attributable. With a learned reward model, capability loss is confounded with reward-model over-optimization and preference noise, so the update rule could not be cleanly blamed.
  • Agentic RL and other update rules are tested. Agentic RL is part of the main pipeline: stages C and D are tool use (τ-bench) and web agents (WebArena). GRPO, 2-GRPO, DPO, KTO, DPPO, and DRPO are all baselines at both horizons. GRPO drifts more than every KL-anchored PPO setting (78.4% vs. ≥ 81.7% at 12K; 41.0% vs. ≥ 47.4% at 45K), so the phenomenon is not specific to PPO.
  • Why SCPO is shown with PPO. PPO's clipped surrogate with a KL term is the shared core of RLHF and GRPO-style updates. The SCPO anchor is an additive loss term, so it applies unchanged to any policy-gradient objective.
  • Limitation. We have evaluated SCPO itself only with PPO, and we state that plainly as a limitation rather than claiming generality.

The decomposition now follows from the table, is confirmed by a factorial design, and states how much it leaves unexplained. With that resolved, we would be grateful if you would reconsider your rating.

AuthorsOpenReview ↗

Concern Regarding Reviewer F75L's Review (Submission 13129)

Dear PC and AC,

We request a human-level review of our paper. The Pangram report (link) flagged 63% as AI-generated, but we do not trust this review. Pangram itself notes false positives (~1 in 10,000), limited confidence on short/formulaic text, and segment-based scoring that mislabels hybrid human-AI editing. We ask that reviewers assess the substantive content and author contributions directly, rather than relying on this automated 63% figure.

Sincerely,

The Authors

Reviewer_SMQxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies capability degradation during multi-stage reinforcement-learning post-training and attributes it to distribution narrowing, reward-gradient asymmetry, token-entropy collapse, and KL-anchor decay. It proposes subspace-constrained policy optimization (SCPO), capability-preservation metrics, and a scaling law for sequential post-training. In a 12K-step Qwen2.5-7B comparison, SCPO reports 92.1% capability preservation at a target reward of 43.3, while vanilla PPO reports 61.4% at a target reward of 43.7. Across the four-stage curriculum, the paper reports a target reward of 52.7 and capability preservation of 92.1% after 45K steps.

Critical strengths

S1: The matched-reward comparison makes the retention tradeoff concrete. Table 1 reports SCPO at a target reward of 43.3 and 92.1% capability preservation, compared with 43.7 and 61.4% for vanilla PPO. This isolates a substantial retention difference within the same 12K-step setup.

S2: The ablations quantify distinct contributors to measured capability loss. Table 6 attributes 61.1% of the reported drop to distribution narrowing, 17.8% to reward-gradient asymmetry, 13.9% to entropy collapse, and 7.2% to KL-anchor decay. This decomposition gives the proposed intervention a more specific empirical motivation than a single aggregate forgetting score.

Critical weaknesses

W1: The probe protocol does not establish independent evaluation of capability preservation. Section 3.3 defines the capability subspace using probe runs, while Section 4.3 describes probe-based evaluation but does not say whether subspace construction, tuning, and reported CPS or DRI use disjoint prompts. Since SCPO is designed around the probe-derived subspace, overlap could favor the same capabilities used to construct it, so the reported preservation metric may overstate generalization.

W2: The reported uncertainty does not establish variability across training runs. Section 4.3 reports three seeds and 10,000 bootstrap draws, while Table 6 reports paired tests with n=3. The main text does not identify the bootstrap sampling unit. Resampling evaluation examples alone would not estimate run-to-run variation in the training procedure. Thus the reported intervals do not establish whether the differences persist across independent training runs.

Questions for authors
  1. Are the prompts used to construct the capability subspace and select hyperparameters disjoint from every prompt used for CPS and DRI evaluation? If not, can you report results on an untouched capability-probe set?
  2. What is the resampling unit for the 10,000 bootstrap draws? Can you provide per-seed results and uncertainty estimated across independent runs for Tables 1 and 6?
2 comments
AuthorsOpenReview ↗

Official Rebuttal of Submission 13129 by Authors

Thank you for the careful reading. Your two concerns are the right ones for a subspace-based method: whether the evaluation set is independent of everything used to build and tune SCPO and whether our uncertainty reflects run-to-run variation. The revision answers both explicitly, and the answer to both questions is yes. Table numbers refer to the revision, with the original number in brackets.

W1 / Q1. Are the prompts for subspace construction, tuning, and CPS/DRI evaluation disjoint? Yes. Section 3.3 and Table 1 now assign every prompt in the paper to exactly one of three pools, each with a fixed role:

  • Construction pool CcalC_{\text{cal}} (18,000 Tulu-3-SFT-Mixture prompts). Used only to extract the subspace S\mathcal{S}: hidden states, linear probes, and the Jacobian lift. The probes are trained on Tulu-3 source-category labels (reasoning, code, multilingual, safety/refusal, formatting, open-ended), not on the six evaluation axes.
  • Selection pool CselC_{\text{sel}} (the other 2,000 Tulu-3 prompts). Used for every decision: probe early stopping, subspace rank, anchor strength α\alpha, the early-stopping rule, and the hyperparameters of all baselines.
  • Evaluation pool CevalC_{\text{eval}} (IFEval, XSTest, MMLU-Redux-500, MGSM, TruthfulQA-MC1, RULER-NIAH; 3,758 prompts). Used only to report CPS and DRI. No quantity computed on CevalC_{\text{eval}} is read by training, early stopping, or any hyperparameter choice, for SCPO or any baseline.

CevalC_{\text{eval}} is therefore the untouched probe set you ask for, and every reported CPS and DRI value is measured on it. Because S\mathcal{S} is fitted to different prompts with different labels, retention on CevalC_{\text{eval}} is an out-of-distribution test of the subspace, not an in-sample fit. Three further checks support this:

  1. An 8-gram audit finds 0 overlapping prompts between CevalC_{\text{eval}} and Ccal∪CselC_{\text{cal}} \cup C_{\text{sel}}, and 0 between CevalC_{\text{eval}} and the four RL training sets. It cannot detect overlap in prompts under eight words, which includes many XSTest prompts.
  2. An equal-rank control isolates what the learned subspace contributes. A random subspace of the same rank retains 71.4% versus 92.1% (Table 12), so the gain comes from the learned directions, not from constraining any 4.2% of the parameters.
  3. The 14-stage micro-curriculum trains on six evaluation benchmarks, so we exclude it from every retention claim.

W2 / Q2. Bootstrap unit and run-to-run uncertainty. The bootstrap resamples evaluation prompts (BB = 10,000 over the 3,758 CevalC_{\text{eval}} prompts, stratified by axis; block bootstrap for trajectories). You are right that this measures evaluation noise only. Section 3.5 now says so and states that these intervals are not used for significance. Our primary inference is across the three independent training runs (seeds 0, 1, 2):

  • Every table entry is mean ± SD across seeds.
  • Per-seed CPS at 12K (Table 2 [old 1]): SCPO 92.5 / 91.8 / 92.0 and Vanilla PPO 62.3 / 60.2 / 61.7. The 30.7-point difference has a seed-level 95% CI of [28.3, 33.1] (Welch, p<0.001p < 0.001).
  • The 13 SCPO-versus-baseline comparisons (Table 8) are seed-level tt-tests with Benjamini–Hochberg correction. All remain significant at q=0.05q = 0.05; the closest are HMA (q=0.023q = 0.023) and CPO (q=0.034q = 0.034).
  • For the ablation (Table 7 [old 6]), seed-level Welch CIs computed from the reported SDs exclude zero for all three components at both horizons:
Component removed12K45K
Distribution-narrowing penalty13.1 [11.7, 14.5]20.9 [19.3, 22.5]
Entropy floor3.0 [2.0, 4.0]3.0 [1.7, 4.3]
KL anchor1.6 [0.7, 2.5]1.5 [0.4, 2.6]
  • With n=3n = 3, an exact Wilcoxon test cannot reach p<0.05p < 0.05, and standardized effect sizes are unstable, so we report raw differences with seed-level CIs instead of Cohen's dd.

We will add per-seed values for every row of Tables 2 and 7 to the appendix.

Changes to the claims in your summary. S2 cites a four-way split (61.1 / 17.8 / 13.9 / 7.2%). Reward-gradient asymmetry is not a term of the SCPO objective, so a single-component ablation of SCPO cannot remove it. The revision therefore treats it as a pipeline factor in a separate 242^4 factorial check and reports three components. The underlying losses are unchanged (13.1, 3.0, and 1.6 points at 12K). With reward-gradient asymmetry out of the denominator, the shares are 74.0 / 16.9 / 9.0% at 12K and 82.3 / 11.8 / 5.9% at 45K. Together the three components explain 58% (12K) and 40% (45K) of the gap to Vanilla PPO, and we report the remainder as unattributed. The scaling law is withdrawn: the stage fits are labelled descriptive, and a held-out fifth stage follows neither fit.

The evaluation set is independent of everything used to build or tune SCPO, and the differences hold across independent training runs. We would be grateful if you would reconsider your soundness assessment and rating.

AuthorsOpenReview ↗

Concern Regarding Reviewer SMQx's Review (Submission 13129)

Dear PC and AC,

We request a human-level review of our paper. The Pangram report (link) flagged 77% as AI-generated, but we do not trust this review. Pangram itself notes false positives (~1 in 10,000), limited confidence on short/formulaic text, and segment-based scoring that mislabels hybrid human-AI editing. We ask that reviewers assess the substantive content and author contributions directly, rather than relying on this automated 77% figure.

Sincerely,

The Authors

Reviewer_jtm6OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper studies the degradation of previously acquired capabilities during reinforcement learning post-training of foundation models. It frames this phenomenon as a continual reinforcement learning problem and proposes Subspace-Constrained Policy Optimization (SCPO). Its main claims include a four-component decomposition of capability degradation, improved capability preservation, and scaling relationships involving training duration and the number of post-training stages. The experiments contains substantial inconsistency, see the comments below.

Critical strengths
  1. The studied problem is of significant importance, which can attract much audiences' attention.
  2. The experimental setting is broad, and tracking six capability axes during training is good practice.
Critical weaknesses

I have significant concerns about the rigor and consistency of the reported numbers:

  1. The short-horizon Δ values in Table 6 and the "Gap" column in Table 11 are computed from 96.3, not the stated SCPO baseline of 92.1 (e.g., 83.2 is listed as −13.1).
  2. SCPO's CPS/T values (8.03 at 12K, 2.14 at 45K) equal 96.3/12 and 96.3/45, not 92.1/T.
  3. At 45K, Table 4 reuses the 12K values for CPO, RPO, DRPO, DPPO and SCPO+replay.
  4. In Table 4, CPS/T ≠ CPS/45 for SCPO, HMA, CPO, RPO, DRPO and DPPO. Recomputed, SCPO is not the best method.
  5. The "Eff." column in Table 1 does not match its own formula (Target × CPS/100).

These are only some of the inconsistencies I found. The paper needs a significant revision.

Questions for authors

Please carefully double check all numbers in the paper.

3 comments
AuthorsOpenReview ↗

Official Rebuttal of Submission 13129 by Authors

Thank you for checking our numbers line by line. All five inconsistencies you found were real, and they stemmed from one source: SCPO's CPS reference of 96.3 from an earlier version of our evaluation remained in several derived columns after the headline value was updated to 92.1, and some 45K cells were copied from the 12K table. The revision regenerates every derived column from the per-run values. Each point below uses numbers you can recompute from the revised PDF. Table numbers refer to the revision, with the original number in brackets (one table was added before them).

W1. Δ and Gap computed from 96.3. Fixed. Every Δ is now CPS − 92.1 (SCPO is 92.1 ± 0.4 at 12K and 92.1 ± 0.5 at 45K).

  • Table 7 [old 6]: removing the distribution-narrowing penalty gives 79.0 (−13.1) at 12K and 71.2 (−20.9) at 45K. Removing PPO clipping gives 80.4 (−11.7) and 67.3 (−24.8).
  • Table 12 [old 11]: random subspace 71.4 (−20.7), gradient-based subspace 89.5 (−2.6), α = 0.01 83.1 (−9.0); at 45K, Vanilla PPO 28.0 (−64.1).
  • The component shares are recomputed from these Δs: 74.0 / 16.9 / 9.0% of 17.7 points at 12K, and 82.3 / 11.8 / 5.9% of 25.4 points at 45K.

W2. SCPO's CPS/T equals 96.3/T. Fixed. SCPO's CPS/T is now 7.68 = 92.1/12 (Table 12) and 2.05 = 92.1/45 (Table 5). Every other CPS/T entry is recomputed the same way, e.g., random subspace 71.4/12 = 5.95.

W3. 45K values reused from 12K. Fixed. Table 5 now reports the 45K-checkpoint evaluations:

Method12K CPS45K CPS
CPO91.885.2
RPO90.584.6
DRPO85.271.8
DPPO86.472.4
SCPO+replay96.091.8

These match the 45K column of the CPS-trajectory table (Table 18), which is an independent check.

W4. CPS/T ≠ CPS/45, and SCPO is not the best method. Fixed. CPS/T = CPS/45 now holds for all 22 rows of Table 5, and CPS/log T = CPS/log₁₀(T) holds as well. You are right that the recomputed values change the claim, and we changed it.

  • Among single methods, SCPO has the highest CPS/T: 2.05, versus CPO 1.89, HMA 1.88, the best KL-PPO 1.42, and Vanilla PPO 0.62.
  • The composed variants HMA+SCPO (2.13) and CPO+SCPO (2.12) are higher, and the revision says so. SCPO is now presented as a preservation component, not the best configuration, and the reported Pareto frontier is {Vanilla PPO, SCPO, HMA+SCPO}.
  • The caption now states that at a common T, CPS/T is only a rescaling of CPS and adds no ranking information beyond CPS.

W5. Eff ≠ Target × CPS/100. Fixed for all 22 rows of Table 2 [old 1], e.g., SCPO 43.3 × 92.1/100 = 39.88 and Vanilla PPO 43.7 × 61.4/100 = 26.83.

"Please double-check all numbers." We did this with a script, not by hand. The script parses the LaTeX source and recomputes every derived column from the raw values in each table: Δ, Eff, CPS/T, CPS/log T, the component shares, their sums, and the fraction of the gap they explain. It also checks quantities that appear in more than one table, e.g., Table 5 against Tables 6, 12, and 18. The revision passes with zero mismatches. We will release the script with our code.

Claims in your summary that the revision no longer makes

  • Four-component decomposition → three components. Reward-gradient asymmetry is no longer counted as a drift mechanism. It appears only as a factor in a 2⁴ factorial check, where pairwise interactions are at most 0.4 points. The three components explain 58% (12K) and 40% (45K) of the gap to Vanilla PPO, and the paper reports the remainder as unattributed.
  • Scaling relationships → withdrawn. The stage fits (ΔR_K ≈ 26.0·K^−2.15, ΔC_K ≈ 12.7·K^−0.63) are labelled descriptive. We report that a held-out fifth stage follows neither fit: ΔR₅ was predicted at 0.8 and observed at 2.7, and ΔC₅ was predicted at 4.6 and observed at 2.68. The former theorems are now heuristic arguments in the appendix, and no empirical claim depends on them.
  • Scope. Retention is measured on Qwen2.5-7B only, SCPO is tested only with PPO, and all results use three seeds. With three seeds, standardized effect sizes are unstable, so we report raw differences and seed-level Welch tests: SCPO vs. Vanilla PPO at 12K differs by 30.7 points, 95% CI [28.3, 33.1].

Every concern you listed is resolved, and the claims now follow from the corrected numbers. We would be grateful if you would reconsider your soundness assessment and rating in light of the revision.

Thank you for addressing the numerical inconsistencies and revising the experimental results. In light of these improvements, I will increase my soundness score and overall rating accordingly.

Thank you very much for your thoughtful consideration and for taking our request into account. We would be sincerely grateful if you could share any remaining concerns or suggestions and kindly consider the revised manuscript once more, particularly given the substantial improvements we have made to strengthen the paper overall. We truly appreciate your time, consideration, and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.