acceptodds
Under review as a conference paper at ICLR 2027

Preserving Pretrained Capabilities during RLHF via Cumulative Fisher Regularization

Abstract

Reinforcement learning from human feedback can improve preference satisfaction while degrading capabilities acquired before alignment. We introduce a Fisher-based update correction for proximal policy optimization (PPO) to address this capability-preservation problem. Our method estimates the initial policy’s output Fisher on capability-relevant prompts and corrects optimizer-proposed updates through a quadratic penalty on cumulative parameter displacement. Unlike a stepwise penalty, this formulation accounts for the interaction between each new update and the policy’s existing displacement from initialization. We derive a closed-form correction and a low-rank implementation that avoids constructing the full Fisher matrix. The correction selectively attenuates displacement along directions with high estimated output sensitivity, without requiring benchmark evaluation at every training step. We establish an evaluation protocol combining paired update diagnostics with PPO training trajectories, comparing against standard PPO, uniform update scaling, and activation-based correction. Mathematical task accuracy and frozen reward-model scores jointly assess capability retention and alignment gains, while held-out behavioral divergence tests the fidelity of the sensitivity estimate. This work provides a concrete optimization framework for preserving pre-existing capabilities during preference optimization, with cumulative behavioral sensitivity as its organizing principle.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning from Human Feedback, Capability Preservation, Fisher Information, Proximal Policy Optimization, Catastrophic Forgetting, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ro2sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies capability degradation during reinforcement learning from human feedback (RLHF), arguing that conventional reference-policy KL regularization on alignment prompts may insufficiently protect behaviors associated with specific pretrained capabilities. The authors propose Cumulative Fisher Protection (CFP), which constrains cumulative parameter displacement along directions estimated to be sensitive to selected capabilities.

CFP constructs an output-Fisher approximation from calibration prompts and response log-probability gradients evaluated at the initial policy. Given the cumulative displacement proposed by a base optimizer, it applies a Fisher-weighted proximal correction:

zt∗=(I+λtF^0)−1(Δt+dt0).z_t^*=(I+\lambda_t\hat F_0)^{-1}(\Delta_t+d_t^0).

The paper derives a closed-form solution and a low-rank implementation using the Woodbury identity. In its main experiments, the authors use diagonal Fisher approximations, implementing a loss-level Fisher penalty for PPO and a post-optimizer proximal correction for DPO.

Experiments are conducted on Qwen2.5-7B-Instruct, using PKU-SafeRLHF for alignment and evaluating mathematical reasoning, coding, general knowledge, and safety. The results show substantial reductions in Fisher-weighted parameter displacement. For example, mathematics-anchored PPO reduces the corresponding quadratic drift by approximately 67% at step 64. The authors also report partial recovery of OlympiadBench performance under PPO, recovery of MATH accuracy under DPO, and substantial preservation of safety improvements.

The empirical evidence, however, is less conclusive than these point estimates suggest, particularly regarding statistically reliable capability retention and the connection between the implemented calibration estimates and the theoretically defined Fisher metric.

Critical strengths

The paper identifies a meaningful limitation of conventional alignment regularization that protecting behavior on alignment prompts is not necessarily equivalent to protecting capabilities on their relevant input distributions.

The proposed correction provides an explicit mechanism for controlling accumulated displacement relative to initialization, rather than only modifying individual update directions. Its closed-form solution is straightforward to analyze, and the formulation offers a useful connection between capability-sensitive geometry and optimizer-level regularization.

Critical weaknesses
  1. The experimental calibration procedure does not support the claimed output-Fisher interpretation

A central premise of CFP is that its empirical matrix approximates

F0=Ex∼Pcap, y∼πθ0[s0(x,y)s0(x,y)⊤].F_0=\mathbb E_{x\sim P_{\mathrm{cap}},\,y\sim\pi_{\theta_0}} \left[s_0(x,y)s_0(x,y)^\top\right].

However, Table 3 reveals that the actual calibration responses are selected differently:

  • Mathematics: Official OlympiadBench reference solutions, rather than responses sampled from the initial policy.
  • Code: Initial-policy solutions selected according to whether they pass official tests.
  • Safety: PPO checkpoint responses selected according to whether they are judged refusals.

These procedures induce response distributions different from πθ0\pi_{\theta_0}. As a result, their score second moments generally do not equal the initial-policy Fisher information and cannot automatically be interpreted as the Hessian of the reference-policy KL.

Importantly, the mathematical proximal correction remains well defined for these matrices. The problem is the claimed connection between the empirical metric and the theoretically motivated capability-preservation geometry.

The mismatch between the defined and implemented quantities makes it difficult to determine whether the reported effects arise from the claimed Fisher geometry or from supervised response selection and more general parameter regularization.

  1. The experiments do not establish statistically reliable capability preservation or an advantage over existing regularization methods

The primary empirical claim is that CFP protects capabilities that would otherwise degrade during alignment. However, the evaluated setting exhibits relatively limited capability regression, and the evidence for improved retention is weak.

Addationally, the paper does not include the critical matched comparisons needed to isolate Fisher geometry from simpler regularization effects. The authors themselves identify these experiments as planned work in Appendix F.4.

Without significant baseline forgetting, reliable improvements across independent training runs, and appropriate matched controls, the results cannot convincingly attribute capability retention to cumulative output-Fisher protection. The evidence currently supports suppression of the selected quadratic displacement metric more strongly than it supports the broader behavioral claim.

Questions for authors
  1. Section 2 defines the Fisher using responses sampled from the initial policy, but Appendix C.2 describes reference solutions, correctness-filtered solutions, and later-policy refusals.

Can the authors clarify how these empirically constructed matrices should be interpreted as approximations to the initial-policy output Fisher? In particular, can the authors provide a comparison using genuinely initial-policy-sampled responses, together with measurements of held-out reference-policy KL on the protected capability distribution?

  1. Can the authors demonstrate statistically reliable capability retention in a setting where the unregularized alignment baseline exhibits a significant capability decline?

Ideally, this should use disjoint calibration and evaluation data, multiple independent training seeds, and comparisons at matched training checkpoints and alignment performance. It would also be useful to distinguish genuine task-correctness improvements from output-format changes.

  1. Can the authors provide controlled comparisons against simpler alternatives, particularly a standard diagonal Fisher penalty, cumulative L2 anchoring or uniform displacement scaling, and a stepwise Fisher correction, with coefficients selected under comparable alignment performance?

Additionally, can the authors clarify whether the main PPO results depend on the exact proximal correction or on the explicitly regularized Adam objective?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Ro2s (1/3)

We thank Reviewer Ro2s for the rigorous review. We have added all the requested experimental controls in the revised paper.

W1/Q1. Initial-policy Fisher versus selected responses. We agree with this distinction and address it in two ways: (1) explicitly clarifying the theoretical interpretation of selected-response matrices, and (2) adding true πθ0\pi_{\theta_0}-sampled anchors paired with held-out KL evaluations. First, the 7B anchors in the submission are positive semidefinite empirical score second moments evaluated on selected responses rather than unbiased estimates of the initial-policy Fisher. Proposition 1 holds for any positive semidefinite matrix, so they define valid capability-aware metrics, though only πθ0\pi_{\theta_0}-sampled anchors correspond to the exact Hessian of the θ0\theta_0-KL (§2.3, Appendix B.2, C.2). Second, our new 1B SFT anchors use one unfiltered πθ0\pi_{\theta_0} sample on each of 128 training prompts disjoint from all evaluation splits. In five-seed math SFT, CFP with this πθ0\pi_{\theta_0}-sampled anchor reduces held-out forward KL from θ0\theta_0 (on 330 disjoint prompts; Appendix D.1) from 0.066 to 0.005 while keeping SORRY-Bench at 33.6 and HarmBench at 15.0. Anchors constructed from Qwen2.5-7B-Instruct responses provide equivalent protection (SORRY-Bench 33.6 vs. 33.1, HarmBench 14.2 for both; three seeds), and across 306 fine-tuned models both quadratic forms rank held-out KL with a Spearman correlation of 0.81, compared to 0.73 for Euclidean parameter distance (Appendix D.6).

Crucially, this protection is not generic weight-space regularization: baselines that constrain parameter movement much more tightly still suffer larger safety regressions. In 1B math SFT at matched GSM8K, uniform displacement scaling (F=IF=I) and L2-SP both remain much closer to θ0\theta_0 in Euclidean norm than CFP, yet both leave substantial safety regressions; L2-SP also incurs a 2.4× higher held-out KL (0.0120.012 vs. 0.0050.005).

1B math SFT, 5 seeds (Tables 2 and 4)Displacement ∥Δθ∥\lVert\Delta\theta\rVertSORRY ↓HarmBench ↓
CFP1.4133.615.0
Uniform scaling, F=IF=I0.4646.128.5
L2-SP0.0435.918.6

The same holds at 7B scale. After 200 steps of math GRPO on Qwen2.5-7B-Instruct, applying a single post-hoc CFP projection to shrink the displacement norm to 25% reduces SORRY-Bench below θ0\theta_0 (33.9 vs. 34.3), whereas uniform shrinkage to the exact same norm does not.

7B math GRPO (Table 1; Appendix E.13, Table 30)Remaining displacementSORRY ↓HarmBench ↓MATH ↑
Step 200 (θ0\theta_0: 34.3 / 3.5 / 75.3)100%36.85.075.8
+ one CFP retraction25%33.9 (p=0.019p=0.019)3.075.3
+ uniform retraction25%35.74.575.1

p-value against step 200; in Table 30, the uniform retraction at 7B is not significantly below step 200.

AuthorsOpenReview ↗

Response to Reviewer Ro2s (2/3)

W2/Q2. Reliable retention under a significant decline. We added three experimental settings in which unprotected training causes a clear, statistically significant capability decline, using disjoint calibration and evaluation splits, comparisons at identical training steps, and format-agnostic answer scoring. In math SFT, the result holds consistently across all five seeds: every unprotected seed raises SORRY-Bench to 69.3–78.0, whereas every CFP seed stays at 33.2–33.9. In the reverse direction, refusal SFT and safety RLHF significantly degrade GSM8K (from 46.4 to 40.9, p=5×10−5p=5\times10^{-5}, and to 43.5, p=0.014p=0.014), and CFP recovers performance to 44.4 (p=0.009p=0.009) and 46.3 (p=0.018p=0.018). GSM8K is evaluated with a format-agnostic parser (Qwen2.5-Math), and under lenient matching (counting responses as correct when they derive the right answer before repeating a prompt number at the end), the refusal-SFT drop widens to 6.4 points while CFP's recovery remains highly significant (p=0.001p=0.001). With four anchors at λ=10\lambda=10, math SFT retains its full GSM8K gain (49.6 vs. 49.6) while restoring IFEval from 32.7 to 49.4 and eliminating 73% of the SORRY-Bench increase (69.3→43.969.3 \to 43.9). Finally, at matched GSM8K in safety RLHF, standard regularizers sacrifice substantially more of the alignment objective: a larger KL penalty (β=0.2\beta=0.2) and L2-SP reach rewards of only 20.5 and 20.1 (vs. 29.3 for CFP) with significantly higher harmful compliance (SORRY-Bench 20.5 and 15.2 vs. 11.4, p<10−3p<10^{-3}).

Llama-3.2-1B, disjoint splitsθ0\theta_0Unprotected+ CFPStatistical evidence
Math SFT → safety (5 seeds; Table 2)SORRY 35.0, HarmBench 15.074.5, 62.033.6, 15.0all 5 seeds: CFP 33.2–33.9 vs. SFT 69.3–78.0
Refusal SFT → GSM8K ↑ (App. Table 13)46.440.9 (p=5×10−5p=5\times10^{-5})44.4recovery p=0.009p=0.009
Safety RLHF → GSM8K ↑, step 64 (App. Table 12)46.443.5 (p=0.014p=0.014)46.3recovery p=0.018p=0.018
Math SFT, four anchors, λ=10\lambda=10 (App. Table 17)GSM8K 45.3, IFEval 50.1, SORRY 34.349.6, 32.7, 69.349.6, 49.4, 43.9full GSM8K gain retained
AuthorsOpenReview ↗

Response to Reviewer Ro2s (3/3)

Q3. Controlled comparisons. We evaluated all four controlled ablations in 1B math SFT at matched GSM8K (~47.2); removing any single component of CFP degrades safety retention (Table 4).

1B math SFT (Table 4)SeedsGSM8K ↑SORRY ↓HarmBench ↓WildJailbreak ↓∥Δθ∥\lVert\Delta\theta\rVert
CFP, λ=100\lambda=100547.233.615.026.71.41
Uniform scaling (F=IF=I)547.246.128.535.10.46
L2-SP547.435.918.628.70.04
Stepwise Fisher correction346.733.616.828.51.56
Fisher penalty in the loss, λ=2×102\lambda=2\times10^2547.039.320.332.50.78
Fisher penalty in the loss, λ=2×104\lambda=2\times10^4546.534.215.126.50.18
  • The Fisher metric. Uniform scaling differs from CFP only by setting F=IF=I; at matched GSM8K, it leaves SORRY-Bench 12.5 points higher and HarmBench 13.5 points higher, and L2-SP similarly leaves a HarmBench regression (+3.6), even though both baselines move the weights much less than CFP.
  • The cumulative form. The stepwise Fisher correction matches CFP on SORRY-Bench after one epoch, but underperforms on HarmBench and WildJailbreak. Over five epochs, its held-out KL triples (0.006→0.0180.006 \to 0.018) and its HarmBench score rises from 16.8 to 20.0, whereas CFP's held-out KL decreases (0.0055→0.00240.0055 \to 0.0024; Figure 3b, Appendix D.5). This directly reflects Proposition 2: under CFP, drift along each eigendirection vjv_j of F^\hat F satisfies ∣vj⊤ΔT∣≤max⁡k∣vj⊤dk0∣/(λμj)|v_j^\top\Delta_T|\le\max_k|v_j^\top d_k^0|/(\lambda\mu_j) for any horizon TT, whereas stepwise corrections have no horizon-independent bound.
  • The post-optimizer proximal step. Adding the Fisher penalty to the loss matches CFP's safety retention only when its coefficient is increased 100× (λ=2×104\lambda=2\times10^4), because Adam preconditions and clips the penalty gradient together with the task gradient; at that strength, it over-constrains task-relevant directions (∥Δθ∥=0.18\lVert\Delta\theta\rVert = 0.18 vs. 1.411.41, reducing GSM8K to 46.5 vs. 47.2).

Regarding the 7B PPO results: the submission's 7B PPO runs applied the Fisher penalty in the loss prior to Adam (as documented in Appendix C.1, whereas the Table 1 caption of the submission inadvertently mislabelled them as post-optimizer). In the revision, we run both formulations side by side in the exact same 7B PPO setup using identical anchors (Appendix E.4, Table 20). Unprotected PPO exhibits a statistically significant 0.30-point drop on MMLU (p=0.003p=0.003); exact post-optimizer CFP eliminates this drop across all three anchors (+0.31, +0.28, and +0.21 points over PPO, p≤0.033p\le0.033; Table 21), whereas the loss-level penalty does so only with F^adv\hat F_{\mathrm{adv}} (+0.25, p=0.006p=0.006; Appendix E.8, Table 26).

Revision: §3.1–3.2, §4.5 (Table 4, Figure 3); Appendix B.5, B.8, D.5, E.4, E.5, and E.8.

We hope these new controls fully address your concerns, and we are happy to clarify any further questions.

Dear Reviewer Ro2s,

We apologize for writing once more, as we understand this is a busy period. With the discussion period closing soon, we wanted to send a brief reminder of our three-part response above.

We hope it addresses everything you asked and suggested: (1) true unfiltered πθ0\pi_{\theta_0}-sampled anchors paired with held-out forward-KL evaluations and an explicit distinction from target-conditioned empirical Fishers (§2.3, Table 8, Appendix D.6); (2) statistically reliable, format-agnostic capability retention across three new regimes with severe bidirectional forgetting, disjoint splits, and five seeds (§4.2–4.3, Tables 1–3); and (3) all four matched controls—uniform displacement scaling (F=I), L2-SP, five-epoch stepwise Fisher correction, and loss-level Fisher penalty—alongside side-by-side proximal vs. loss-level runs in the 7B PPO setup (§4.5, Table 4, Figure 3, Tables 20–21, 26).

If the response resolves your questions, we would be grateful if you could confirm this or update your assessment. If anything remains unclear, we would be glad to clarify it in the time remaining.

Thank you again for your time and for a thorough review that substantially improved the paper.

Reviewer_azH3OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Cumulative Fisher Protection (CFP) to preserve model capabilities during RLHF. It uses capability-specific examples to estimate Fisher sensitivity and limit cumulative parameter drift from the initial model. PPO and DPO experiments report reduced drift in sensitive directions and improvements on selected capability benchmarks.

Critical strengths
  1. The paper explains why regularization on alignment prompts may not preserve abilities such as math or coding. CFP builds its protection term from task examples, giving a practical way to choose which abilities to protect during alignment.
  2. The derivation is easy to follow, particularly the treatment of changes accumulated over earlier updates. It gives an exact solution to the local quadratic problem and makes the role of the correction clear.
  3. Appendix helps assess the experimental claims. It reports significance tests and checks how much of the GSM8K gain comes from answer formatting. It also reports OlympiadBench results after removing the calibration examples.
Critical weaknesses
  1. The PPO baseline shows little capability loss, making CFP’s benefit difficult to assess.

    Most gains are small and not statistically significant. The clearest gains, on GSM8K, mainly come from answer formatting. These results provide limited evidence that CFP prevents a loss of task-solving ability.

  2. CFP applies a correction after the optimizer step, but the main PPO experiments add a Fisher penalty to the loss before Adam.

    Appendix B.8 establishes first-order equivalence for gradient descent, but not for Adam. The authors test the correction with DPO and on one PPO weight matrix. However, it is still unclear whether the correction works better than the loss-level penalty in the main PPO setup.

  3. The theory assumes responses sampled from the initial model. The experiments instead use reference solutions for math, code that passes tests, and refusals from the PPO-trained model.

    The Fisher is therefore estimated from a different response distribution.

Questions for authors
  1. What evidence supports capability retention beyond answer formatting?

    The PPO baseline shows little capability loss, and the GSM8K gains mainly come from formatting. The paper’s main claim is that CFP preserves capabilities during alignment. We would like to see whether this holds in a setting where PPO causes a clear drop in task-solving accuracy.

  2. Does the post-optimizer correction improve on the loss-level Fisher penalty?

    We would like to see a comparison of the two updates using the same PPO setup, protected parameters, and calibration data.

  3. The derivation assumes responses sampled from the initial policy. Does the connection between the estimated Fisher and the initial policy’s forward KL still hold with the response selection in Table 3?

5 comments
AuthorsOpenReview ↗

Response to Reviewer azH3 (1/2)

We thank Reviewer azH3 for the positive assessment and for clearly structuring the three core questions.

W1/Q1. Retention beyond answer formatting. In two new settings where safety training causes a statistically significant GSM8K drop under a format-agnostic parser, CFP reliably recovers mathematical performance. Specifically, we evaluate GSM8K using extracted final numbers (via the Qwen2.5-Math parser) rather than the strict #### regex that caused the formatting sensitivity in the submission (Appendix D.3 of the submission). Under format-agnostic scoring, refusal SFT lowers GSM8K from 46.4 to 40.9 and CFP recovers it to 44.4 (p=0.009p=0.009); safety RLHF lowers GSM8K to 43.5 and CFP restores it to 46.3 (p=0.018p=0.018) while retaining 79% of the SORRY-Bench safety improvement. To verify that this drop and recovery are not last-number extraction artifacts (e.g., when a model derives the correct answer but repeats a number from the prompt in its closing sentence), we also tested lenient answer-anywhere matching: the refusal-SFT drop widens to 6.4 points (p=4.5×10−6p=4.5\times10^{-6}), and CFP's recovery remains highly significant (p=0.001p=0.001; Appendix D.3).

Llama-3.2-1Bθ0\theta_0Unprotected+ CFPStatistical evidence
Refusal SFT → GSM8K ↑ (Table 1, App. Table 13)46.440.9 (p=5×10−5p=5\times10^{-5})44.4recovery p=0.009p=0.009
Safety RLHF → GSM8K ↑, step 64 (App. Table 12)46.443.5 (p=0.014p=0.014)46.3recovery p=0.018p=0.018
Reverse direction: math SFT → SORRY ↓ (5 seeds; Table 2)35.074.533.6all 5 seeds: CFP 33.2–33.9 vs. SFT 69.3–78.0

Revision: §4.2–4.3 (Figure 2, Table 1); Appendix D.2–D.4 (Tables 12–13, 15).

W2/Q2. Post-optimizer correction versus the loss-level penalty. We directly compared exact post-optimizer CFP against the loss-level Fisher penalty in the 7B PPO setup using identical anchors and protected parameters (Appendix E.4, Table 20), as well as in 1B math SFT (Table 4). In 7B PPO, the only statistically significant capability degradation is a 0.30-point drop on MMLU (p=0.003p=0.003; Appendix E.1); exact post-optimizer CFP eliminates this drop across all three anchors, whereas the loss-level penalty succeeds only with F^adv\hat F_{\mathrm{adv}} (Tables 21 and 26). As you noted, when the Fisher penalty is added to the loss, its gradient is clipped and preconditioned by Adam's v^\sqrt{\hat v} alongside the task gradient, making its effective strength dependent on optimizer state; in contrast, the post-optimizer proximal factor (1+λf^i)−1(1+\lambda\hat f_i)^{-1} acts directly in the Fisher metric. We observe the same behavior in 1B math SFT, where the loss-level penalty requires a 100× larger coefficient to match CFP's safety protection. We also note that the Table 1 caption of the submission inadvertently mislabelled the 7B PPO runs as proximal (whereas Appendix C.1 documented the loss-level formulation); the revision reports both formulations side by side.

7B safety PPO, MMLU (Appendix Tables 20, 21, 26)Exact Post-Optimizer CFPLoss-Level Fisher Penalty
Anchor F^math\hat F_{\mathrm{math}}74.26 (+0.31 vs. PPO, p=9×10−4p=9\times10^{-4})74.07 (+0.12, p=0.20p=0.20)
Anchor F^adv\hat F_{\mathrm{adv}}74.23 (+0.28, p=0.003p=0.003)74.20 (+0.25, p=0.006p=0.006)
Anchor F^code\hat F_{\mathrm{code}}74.16 (+0.21, p=0.033p=0.033)74.03 (+0.08, p=0.47p=0.47)

Initial model θ0\theta_0: 74.25; unprotected PPO: 73.95 (Table 20). Paired differences against PPO with exact McNemar p-values: Table 21 for CFP, Table 26 for the loss-level penalty. The loss-level F^math\hat F_{\mathrm{math}} run is evaluated at step 118.

1B math SFT, 5 seeds (Table 4)CoefficientGSM8K ↑SORRY ↓HarmBench ↓
Post-optimizer CFPλ=100\lambda=10047.233.615.0
Loss-level Fisher penaltyλ=2×102\lambda=2\times10^247.039.320.3
Loss-level Fisher penaltyλ=2×104\lambda=2\times10^446.534.215.1

Revision: §3.1, §4.5 (Table 4); Appendix B.8, C.1, E.1, E.4, E.5, and E.8.

AuthorsOpenReview ↗

Response to Reviewer azH3 (2/2)

W3/Q3. Selected responses and the Fisher–KL connection. Strictly speaking, the exact Fisher–KL Hessian identity requires responses sampled from πθ0\pi_{\theta_0} (Appendix B.2); when computed on reference or selected responses, F^\hat F is a positive semidefinite empirical score second moment rather than the exact Hessian of the θ0\theta_0-KL. We now state this distinction explicitly (§2.3, Appendix B.2, Table 8) and use unfiltered πθ0\pi_{\theta_0}-sampled anchors for all new 1B SFT experiments. Because Proposition 1 requires only a positive semidefinite matrix, the proximal projection properties hold under either construction. With one unfiltered πθ0\pi_{\theta_0} sample per prompt, CFP reduces the held-out forward KL from θ0\theta_0 (evaluated on 330 disjoint prompts) from 0.066 to 0.005 (Table 2). Empirically, both anchor constructions behave nearly identically: across 306 fine-tuned models, the quadratic forms Δθ⊤F^Δθ\Delta\theta^\top\hat F\Delta\theta built from unfiltered πθ0\pi_{\theta_0} samples and from Qwen2.5-7B-Instruct responses rank held-out KL with the exact same Spearman correlation (0.81, vs. 0.73 for Euclidean parameter distance), and both provide comparable protection in downstream SFT (tables below).

Safety anchor in math SFT (3 seeds; Appendix D.6)Rank correlation with held-out KL (306 models)SORRY ↓HarmBench ↓
Unfiltered θ0\theta_0 samples0.8133.114.2
Qwen2.5-7B-Instruct responses0.8133.614.2
Math anchor in refusal SFT (Appendix Table 13)GSM8K ↑p vs. refusal SFTSORRY ↓
Unfiltered θ0\theta_0 samples44.40.00911.8
Correctness-filtered θ0\theta_0 samples44.50.00811.8
GSM8K reference solutions44.70.00614.5

Refusal SFT alone: GSM8K 40.9, SORRY 3.0. Unfiltered and filtered θ0\theta_0 anchors yield virtually identical GSM8K performance (50 vs. 51 discordant items, McNemar p=1.0p=1.0; Appendix D.3).

Revision: §2.3, §4.1; Appendix B.2, C.2 (Table 8), D.3, and D.6.

Thank you again for your constructive feedback; we hope these results strengthen your assessment of the paper.

Thank you for the detailed responses and additional experiments. These address most of my concerns.

I find the comparison between post-optimizer CFP and the loss-level Fisher penalty useful. However, still not sure whether the difference between the two methods is statistically significant. The MMLU scores in the 7B PPO experiments are quite close. I am keeping my score at 3.

Reviewer_HQp8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the prolem of capability forgetting in LLM alignment via RLHF, based on the observation that the reference KL divergence penalty fails to properly preserve the capbability. To address this problem, the authors propose a new regularization method, CFP, using Fisher sensitivity as a heuristic and steer the policy optimization towards less Fisher sensitive directions. In the experiments, both of DPO and PPO safeguarded by CFP retens better or similarly good capabilities over their naive counterparts, based on a diverse group of benchmarks including math, coding, language understanding, and safety.

Critical strengths
  1. This paper is clear and easy to read.
  2. The proposed solution adding Fisher-sensitivity as an auxiliary regularization mechanism is intuitive and sound.
Critical weaknesses
  1. This paper presents like an incremental work of "The Geometry of Alignment Collapse" [1], providing empirical evidence that supprots the Fisher information analysis on RLHF alignment collapse.
  2. The positioning of this work in the literature remains unclear. While this paper presents a new method, but how is CFP better than previous alignment tax mitigation methods requires an explicit discussion in your introduction, plus an empirical comparison in your experiments.
  3. The claim that CFP retains the capabilities is too strong, when it comes the case of learning conflict capabilities. In your experiments, Figure 3 (right), both of CFP on and off results demonstrate a compromise in the Olympiad Bench results comparing to the base model.

[1] Springer, Max, et al. "The geometry of alignment collapse: When fine-tuning breaks safety." arXiv preprint arXiv:2602.15799 (2026).

Questions for authors
  1. When we apply multiple fisher-sensitity constraints, can we still optimize LLM alignment in terms of another capability?
  2. How is capability forgetting different from the well-studied alignment tax problem in the literature?
5 comments
AuthorsOpenReview ↗

Response to Reviewer HQp8 (1/2)

We thank Reviewer HQp8 for finding the paper clear and the mechanism intuitive. Below we position CFP relative to [1] (Springer et al., 2026) and prior alignment-tax methods, evaluate retention under conflicting objectives, and answer both questions.

W1. Relation to [1]. [1] provides an empirical diagnosis showing how loss-landscape curvature steers fine-tuning into safety-sensitive directions, and explicitly notes that its findings "motivate the development of curvature-aware methods" without proposing an algorithm. CFP provides this concrete algorithmic mechanism: it measures sensitivity using the initial policy's output Fisher on the protected capability, and after every optimizer step applies the closed-form proximal correction zt=(I+λF^)−1(Δt+dt0)z_t=(I+\lambda\hat F)^{-1}(\Delta_t+d_t^0) to the cumulative displacement from θ0\theta_0. Crucially, CFP corrects the accumulated displacement Δt+dt0\Delta_t+d_t^0 rather than only the current step dt0d_t^0. The quadratic increment Q(Δt+dt0)−Q(Δt)=Δt⊤F^dt0+12(dt0)⊤F^dt0Q(\Delta_t+d_t^0)-Q(\Delta_t)=\Delta_t^\top\hat Fd_t^0+\tfrac12(d_t^0)^\top\hat Fd_t^0 contains a cross term Δt⊤F^dt0\Delta_t^\top\hat Fd_t^0 that a stepwise correction ignores; under CFP, drift along every sensitive direction remains bounded independently of the training horizon (Proposition 2). Empirically, over five epochs of math SFT, the stepwise correction's held-out KL triples while CFP's decreases.

Math SFT, 1BStepwise correctionCFP
Target of correctioncurrent step dt0d_t^0 onlycumulative displacement Δt+dt0\Delta_t+d_t^0
Cross term Δt⊤F^dt0\Delta_t^\top\hat Fd_t^0ignoredcorrected at every step
Drift along sensitive directionsunboundedbounded for any horizon (Proposition 2)
Held-out KL, epoch 1 → 5 (3 seeds; Figure 3b)0.006 → 0.0180.0055 → 0.0024
HarmBench, epoch 1 → 5 (Appendix D.5)16.8 → 20.014.2–16.5 (stable)

Revision: §1 presents [1] as the motivating diagnosis, and §2.3 builds on its per-capability Fisher; §3.2 (Proposition 2), §4.5, and Appendices B.5 and D.5 provide the theoretical bound and multi-epoch experiments.

W2. Positioning and comparison. We have added both a taxonomy of prior work (§1, §1.1) and empirical comparisons against eleven retention baselines (§4.4); CFP is the only method that keeps all four safety metrics at or below initial θ0\theta_0 levels. Prior methods replay data or outputs (Replay, DER++, LwF), regularize outputs toward θ0\theta_0 (KL to θ0\theta_0, Token-wise SFT), alternate alignment and fine-tuning steps (Lisa), project gradients or restrict updates (A-GEM, NSPO, LoRA), or interpolate weights post-hoc (model averaging); the closest parameter-space baseline, L2-SP, penalizes Euclidean distance across all parameters uniformly. In contrast, CFP corrects the cumulative displacement in the protected capability's Fisher metric. Across five-seed math SFT, every baseline that retains most of the GSM8K gain either suffers a safety regression or induces severe over-refusal on XSTest (table below). Notably, CFP's protection is not due to smaller parameter updates: at matched GSM8K (47.2), replacing CFP's Fisher metric with uniform displacement scaling (F=IF=I) raises SORRY-Bench from 33.6 to 46.1 and HarmBench from 15.0 to 28.5 (Table 4); L2-SP produces a 37× smaller weight displacement than CFP (0.040.04 vs. 1.411.41) yet incurs a 2.4× larger held-out safety KL (0.0120.012 vs. 0.0050.005); and at matched GSM8K, model averaging exhibits 6.3 points higher harmful compliance even though CFP permits a 2× larger weight displacement (1.641.64 vs. 0.830.83; Figure 3a, Appendix Table 18).

Method (math SFT, 5 seeds; Table 2, App. Table 15)MechanismGSM8K ↑SORRY ↓HarmBench ↓WildJailbreak ↓XSTest ↓Largest regression vs. θ0\theta_0
Initial θ0\theta_0initial model46.435.015.027.210.0(reference)
SFTno protection51.574.562.057.64.0HarmBench +47.0
CFP, λ=100\lambda=100Cumulative Fisher correction47.233.615.026.79.8none
L2-SPEuclidean anchoring47.435.918.628.79.0HarmBench +3.6
LisaAlternating safety steps52.532.913.934.512.2WildJailbreak +7.3
KL to θ0\theta_0 on anchorsOutput regularization51.133.616.238.913.1WildJailbreak +11.7
Replay / DER++Experience replay51.1 / 51.220.5 / 24.14.8 / 6.813.0 / 14.724.0 / 23.6XSTest +14.0 / +13.6
A-GEM / NSPOGradient projection51.5 / 52.249.0 / 43.129.0 / 27.851.9 / 39.17.8 / 7.9WildJailbreak +24.7 / HarmBench +12.8
LoRA / LwFLow rank / distillation53.4 / 51.249.8 / 70.536.9 / 58.241.0 / 55.57.0 / 4.9HarmBench +21.9 / +43.2

Token-wise SFT (HarmBench +46.2) is reported in Appendix Table 15; model averaging is compared at matched GSM8K in Appendix Table 18.

Revision: §1 and §1.1 (taxonomy); §4.4 (Tables 2–3); Appendix D.4 (Table 15) and D.8 (Table 18).

AuthorsOpenReview ↗

Response to Reviewer HQp8 (2/2)

W3. Retention when the objective conflicts with the protected capability. Our new experiments directly test this conflicting regime: CFP preserves the protected capability during training, while λ\lambda smoothly controls the trade-off with the target objective. In math SFT, task training strongly conflicts with safety—unprotected SFT more than doubles harmful compliance, whereas CFP keeps SORRY-Bench and HarmBench at θ0\theta_0 levels across all five seeds. In the reverse direction, safety training degrades mathematical reasoning, and CFP recovers GSM8K from 40.9 to 44.4 after refusal SFT (p=0.009p=0.009) and from 43.5 to 46.3 after safety RLHF (p=0.018p=0.018), while retaining 79% of the RLHF safety improvement on SORRY-Bench. Furthermore, when combining four capability anchors at λ=10\lambda=10, CFP retains the full GSM8K gain of math SFT (49.6 vs. 49.6) while restoring IFEval from 32.7 to 49.4 and eliminating 73% of the SORRY-Bench degradation (see Q1 below).

Objective → protected capabilityθ0\theta_0Unprotected+ CFP
Math SFT → safety, SORRY ↓ (5 seeds; Table 2)35.074.533.6
Math SFT → safety, HarmBench ↓ (5 seeds; Table 2)15.062.015.0
Refusal SFT → GSM8K ↑ (App. Table 13)46.440.944.4
Safety RLHF → GSM8K ↑ (App. Table 12)46.443.546.3

Varying λ\lambda traces a more favorable Pareto frontier rather than simply sacrificing the target task: increasing λ\lambda transitions CFP from full GSM8K acquisition (λ=10\lambda=10) to strict safety preservation (λ=100\lambda=100), achieving lower harmful compliance than model averaging at every matched GSM8K level (§4.5, Figure 3a).

Math SFT (Appendix Table 18)GSM8K ↑SORRY ↓
Initial θ0\theta_0 / Unprotected SFT45.3 / 49.634.3 / 69.3
CFP, λ=10\lambda=10 vs. model averaging, α=0.25\alpha=0.2550.0 vs. 49.841.4 vs. 47.7
CFP, λ=100\lambda=100 vs. model averaging, α=0.1\alpha=0.147.4 vs. 46.733.6 vs. 39.1

Regarding the OlympiadBench result in Figure 3 (right) of the submission: that experiment used a safety-only anchor (containing no math prompts) to protect safety, and its OlympiadBench difference falls within the 95% confidence interval (Appendix D.2 of the submission; Table 21 in Appendix E.5 of the revision). Because CFP protects the capabilities represented in its anchor, our math-retention experiments (refusal SFT and safety RLHF) use a math anchor, whereas safety-retention experiments use a safety anchor.

Q1. Multiple Fisher constraints. Yes. §2.3 defines the composite Fisher F0=∑iωiFiF_0=\sum_i\omega_iF_i, yielding the multi-anchor update (I+∑iλiF^i)−1u(I+\sum_i\lambda_i\hat F_i)^{-1}u. Proposition 1 applies directly to the sum, ensuring that parameter directions with low sensitivity across all anchors remain unconstrained for the new task. Combining four anchors (safety, code, translation, writing) during math SFT at λ=10\lambda=10 retains the full GSM8K gain (49.6) while restoring instruction following (IFEval) and function calling (BFCL); at λ=100\lambda=100, all protected metrics return to their initial θ0\theta_0 levels.

Math SFT, four anchors (Appendix D.7, Table 17)GSM8K ↑IFEval ↑BFCL ↑SORRY ↓
Initial θ0\theta_045.350.149.134.3
SFT49.632.719.469.3
CFP, λ=10\lambda=1049.649.448.843.9
CFP, λ=100\lambda=10046.150.148.932.0

Q2. Forgetting and the alignment tax. Capability forgetting is the capability-degradation component of the alignment tax (Askell et al., 2021; Lin et al., 2024), generalized in two respects. First, the degradation occurs on prompts associated with prior capabilities that lie outside the current training distribution—and are therefore not evaluated by the training loss or standard on-policy reference-KL penalties—which is why CFP computes its Fisher anchor on those external prompts. Second, the same degradation occurs in both directions across post-training stages: alignment training can erode reasoning (lowering GSM8K under safety RLHF and refusal SFT), while downstream fine-tuning can erode safety (raising harmful compliance under math SFT and math GRPO, as studied in [1]). The revised paper (§1, Table 1) explicitly connects both phenomena and demonstrates that CFP mitigates both with the same mechanism.

We hope these revisions address your concerns, and we are happy to clarify anything further.

Thank you for the response and additional experiments. The revised positioning and the expanded baseline comparisons addressed several of my earlier concerns.

Meanwhile, my remaining concern is about the significance of the contribution. The methodological contribution remains a modest extension of established Fisher-weighted anchoring. Besides, the broad preservation narrative remains stronger than the demonstrated retention–learning trade-offs.

I therefore maintain my score of 2.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2)

We thank all reviewers for their careful reading and constructive feedback. The reviews shared two primary concerns: (1) the original experiments showed little capability degradation, making the protective benefit of CFP difficult to assess, and (2) the submission lacked matched controls to isolate the role of the Fisher geometry. The revision addresses both with new experiments on Llama-3.2-1B-Instruct—where unprotected training causes severe capability loss—and additional controls on Qwen2.5-7B-Instruct.

Main results.

  1. CFP prevents substantive capability forgetting in both directions. Math SFT more than doubles harmful compliance, whereas CFP keeps every safety metric at or below its initial level across all five seeds; conversely, refusal SFT and safety RLHF significantly degrade GSM8K, and CFP restores it (Table G1).
  2. Protection comes from the Fisher geometry rather than smaller weight updates. At matched GSM8K (47.2), uniform scaling of the displacement (CFP with F=IF=I) leaves SORRY-Bench at 46.1 (vs. 33.6 for CFP) and HarmBench at 28.5 (vs. 15.0), despite a 3× smaller weight displacement (∥Δθ∥=0.46\lVert\Delta\theta\rVert = 0.46 vs. 1.411.41). Similarly, L2-SP produces a 37× smaller displacement (0.040.04) yet incurs a 2.4× larger held-out KL (0.0120.012 vs. 0.0050.005; Tables 2 and 4). After 200 steps of 7B math GRPO, a single post-hoc CFP projection to 25% of the displacement norm reduces SORRY-Bench from 36.8 to 33.9—below θ0\theta_0 (34.3; p=0.019p=0.019)—whereas uniform scaling to the exact same norm stalls at 35.7, not significantly below the unprotected GRPO model (Appendix E.13, Table 30). Across 306 fine-tuned models, the anchor Fisher quadratic form Δθ⊤F^Δθ\Delta\theta^\top\hat F\Delta\theta ranks held-out KL with a Spearman correlation of 0.81, compared to 0.73 for Euclidean parameter distance (Appendix D.6).
  3. Correcting the cumulative displacement keeps drift bounded. Over five epochs of math SFT (three seeds), the held-out KL of a stepwise Fisher correction triples (0.006→0.0180.006 \to 0.018), whereas that of CFP decreases (0.0055→0.00240.0055 \to 0.0024), consistent with Proposition 2 (Figure 3b, Appendix D.5).
  4. CFP is the only method among eleven retention baselines that keeps all four safety metrics at or below θ0\theta_0 levels in five-seed math SFT (Table G3). Across its entire λ\lambda range, CFP also achieves lower harmful compliance than model averaging at matched GSM8K (e.g., SORRY-Bench 41.4 vs. 47.7 at GSM8K 50.0 vs. 49.8), even though CFP permits a 2× larger weight displacement (1.641.64 vs. 0.830.83; Figure 3a, Appendix Table 18).

Table G1. Protected metric before training, after unprotected training, and with CFP (Table 1; p-values from §4.3 and Appendix D.2–D.3).

RegimeProtected metricθ0\theta_0Unprotected+ CFPTarget with CFP
Math SFT, 1B (5 seeds)SORRY ↓ / HarmBench ↓35.0 / 15.074.5 / 62.033.6 / 15.0GSM8K 47.2 (SFT 51.5)
Refusal SFT, 1BGSM8K ↑46.440.9 (p=5×10−5p=5\times10^{-5})44.4 ( p=0.009p=0.009 )SORRY 11.8 (SFT 3.0)
Safety RLHF, 1BGSM8K ↑46.443.5 (p=0.014p=0.014)46.3 ( p=0.018p=0.018 )SORRY 11.4 (RLHF 5.0)
Math GRPO, 7B (post-hoc)SORRY ↓ / HarmBench ↓34.3 / 3.536.8 / 5.033.9 ( p=0.019p=0.019 ) / 3.0MATH 75.3 (GRPO 75.8)

p-values: exact McNemar tests on paired items, comparing the unprotected model against θ0\theta_0, and CFP against the unprotected model. Every 1B anchor uses 128 training-split prompts disjoint from all evaluation sets; the SFT anchors use one unfiltered πθ0\pi_{\theta_0} sample per prompt.

Table G2. Safety RLHF on Llama-3.2-1B-Instruct at step 64 (Appendix D.2, Table 12). Stronger KL regularization (β=0.2\beta=0.2) and L2-SP also restore GSM8K, but sacrifice most of the reward and safety gains.

MethodReward ↑GSM8K ↑SORRY ↓XSTest over-refusal ↓
Initial θ0\theta_010.446.435.010.0
RLHF, β=0.04\beta=0.0433.543.55.036.0
RLHF + CFP, λ=10\lambda=1029.346.311.422.4
RLHF, β=0.2\beta=0.220.545.920.513.6
RLHF + L2-SP20.146.315.215.2

Reward: mean over the last five steps (θ0\theta_0: first step). CFP restores GSM8K (p=0.018p=0.018 vs. RLHF), retains 79% of the SORRY-Bench reduction and 82% of the reward gain, and cuts RLHF over-refusal from 36.0 to 22.4; its SORRY-Bench compliance is significantly lower than both β=0.2\beta=0.2 and L2-SP (p<10−3p<10^{-3}).

AuthorsOpenReview ↗

General Response (2/2)

Table G3. Math SFT on Llama-3.2-1B-Instruct, mean of five seeds (Table 2 and Appendix Table 15). Ten of the eleven baselines are shown below; the eleventh, model averaging, interpolates toward θ0\theta_0 after training and is compared at matched GSM8K in Figure 3a and Appendix Table 18.

MethodGSM8K ↑SORRY ↓HarmBench ↓WildJailbreak ↓XSTest ↓Held-out KL ↓Largest regression vs. θ0\theta_0
Initial θ0\theta_046.435.015.027.210.00(reference)
SFT51.574.562.057.64.00.066HarmBench +47.0
CFP, λ=100\lambda=10047.233.615.026.79.80.005none
L2-SP47.435.918.628.79.00.012HarmBench +3.6
Lisa52.532.913.934.512.20.190WildJailbreak +7.3
KL to θ0\theta_0 on anchors51.133.616.238.913.10.033WildJailbreak +11.7
Replay51.120.54.813.024.00.982XSTest +14.0
DER++51.224.16.814.723.60.747XSTest +13.6
A-GEM51.549.029.051.97.80.060WildJailbreak +24.7
NSPO52.243.127.839.17.90.030HarmBench +12.8
LoRA53.449.836.941.07.00.030HarmBench +21.9
LwF51.270.558.255.54.90.040HarmBench +43.2
Token-wise SFT50.474.461.258.73.70.064HarmBench +46.2

Where each change appears in the revision.

ConcernReviewersRevision
Forgetting beyond answer formattingazH3, Ro2s§4.2–4.3 (Figure 2, Table 1); Appendix D.2–D.4
Comparison with retention methodsHQp8§1, §1.1; §4.4 (Tables 2–3); Appendix D.4, D.8
Matched controls (metric, cumulative form, proximal step)Ro2s, azH3§4.5 (Table 4, Figure 3); Appendix D.5, E.4, E.8
θ0\theta_0-sampled anchors and held-out KLazH3, Ro2s§2.3, §4.1; Appendix B.2, C.2, D.1, D.6
Multiple anchors and retention under conflicting objectivesHQp8§4.3, §4.5 (Figure 3a); Appendix D.7–D.8

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.