- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies capability degradation during reinforcement learning from human feedback (RLHF), arguing that conventional reference-policy KL regularization on alignment prompts may insufficiently protect behaviors associated with specific pretrained capabilities. The authors propose Cumulative Fisher Protection (CFP), which constrains cumulative parameter displacement along directions estimated to be sensitive to selected capabilities.
CFP constructs an output-Fisher approximation from calibration prompts and response log-probability gradients evaluated at the initial policy. Given the cumulative displacement proposed by a base optimizer, it applies a Fisher-weighted proximal correction:
The paper derives a closed-form solution and a low-rank implementation using the Woodbury identity. In its main experiments, the authors use diagonal Fisher approximations, implementing a loss-level Fisher penalty for PPO and a post-optimizer proximal correction for DPO.
Experiments are conducted on Qwen2.5-7B-Instruct, using PKU-SafeRLHF for alignment and evaluating mathematical reasoning, coding, general knowledge, and safety. The results show substantial reductions in Fisher-weighted parameter displacement. For example, mathematics-anchored PPO reduces the corresponding quadratic drift by approximately 67% at step 64. The authors also report partial recovery of OlympiadBench performance under PPO, recovery of MATH accuracy under DPO, and substantial preservation of safety improvements.
The empirical evidence, however, is less conclusive than these point estimates suggest, particularly regarding statistically reliable capability retention and the connection between the implemented calibration estimates and the theoretically defined Fisher metric.
Critical strengths
The paper identifies a meaningful limitation of conventional alignment regularization that protecting behavior on alignment prompts is not necessarily equivalent to protecting capabilities on their relevant input distributions.
The proposed correction provides an explicit mechanism for controlling accumulated displacement relative to initialization, rather than only modifying individual update directions. Its closed-form solution is straightforward to analyze, and the formulation offers a useful connection between capability-sensitive geometry and optimizer-level regularization.
Critical weaknesses
- The experimental calibration procedure does not support the claimed output-Fisher interpretation
A central premise of CFP is that its empirical matrix approximates
However, Table 3 reveals that the actual calibration responses are selected differently:
- Mathematics: Official OlympiadBench reference solutions, rather than responses sampled from the initial policy.
- Code: Initial-policy solutions selected according to whether they pass official tests.
- Safety: PPO checkpoint responses selected according to whether they are judged refusals.
These procedures induce response distributions different from . As a result, their score second moments generally do not equal the initial-policy Fisher information and cannot automatically be interpreted as the Hessian of the reference-policy KL.
Importantly, the mathematical proximal correction remains well defined for these matrices. The problem is the claimed connection between the empirical metric and the theoretically motivated capability-preservation geometry.
The mismatch between the defined and implemented quantities makes it difficult to determine whether the reported effects arise from the claimed Fisher geometry or from supervised response selection and more general parameter regularization.
- The experiments do not establish statistically reliable capability preservation or an advantage over existing regularization methods
The primary empirical claim is that CFP protects capabilities that would otherwise degrade during alignment. However, the evaluated setting exhibits relatively limited capability regression, and the evidence for improved retention is weak.
Addationally, the paper does not include the critical matched comparisons needed to isolate Fisher geometry from simpler regularization effects. The authors themselves identify these experiments as planned work in Appendix F.4.
Without significant baseline forgetting, reliable improvements across independent training runs, and appropriate matched controls, the results cannot convincingly attribute capability retention to cumulative output-Fisher protection. The evidence currently supports suppression of the selected quadratic displacement metric more strongly than it supports the broader behavioral claim.
Questions for authors
- Section 2 defines the Fisher using responses sampled from the initial policy, but Appendix C.2 describes reference solutions, correctness-filtered solutions, and later-policy refusals.
Can the authors clarify how these empirically constructed matrices should be interpreted as approximations to the initial-policy output Fisher? In particular, can the authors provide a comparison using genuinely initial-policy-sampled responses, together with measurements of held-out reference-policy KL on the protected capability distribution?
- Can the authors demonstrate statistically reliable capability retention in a setting where the unregularized alignment baseline exhibits a significant capability decline?
Ideally, this should use disjoint calibration and evaluation data, multiple independent training seeds, and comparisons at matched training checkpoints and alignment performance. It would also be useful to distinguish genuine task-correctness improvements from output-format changes.
- Can the authors provide controlled comparisons against simpler alternatives, particularly a standard diagonal Fisher penalty, cumulative L2 anchoring or uniform displacement scaling, and a stepwise Fisher correction, with coefficients selected under comparable alignment performance?
Additionally, can the authors clarify whether the main PPO results depend on the exact proximal correction or on the explicitly regularized Adam objective?
4 comments
Response to Reviewer Ro2s (2/3)
W2/Q2. Reliable retention under a significant decline. We added three experimental settings in which unprotected training causes a clear, statistically significant capability decline, using disjoint calibration and evaluation splits, comparisons at identical training steps, and format-agnostic answer scoring. In math SFT, the result holds consistently across all five seeds: every unprotected seed raises SORRY-Bench to 69.3–78.0, whereas every CFP seed stays at 33.2–33.9. In the reverse direction, refusal SFT and safety RLHF significantly degrade GSM8K (from 46.4 to 40.9, , and to 43.5, ), and CFP recovers performance to 44.4 () and 46.3 (). GSM8K is evaluated with a format-agnostic parser (Qwen2.5-Math), and under lenient matching (counting responses as correct when they derive the right answer before repeating a prompt number at the end), the refusal-SFT drop widens to 6.4 points while CFP's recovery remains highly significant (). With four anchors at , math SFT retains its full GSM8K gain (49.6 vs. 49.6) while restoring IFEval from 32.7 to 49.4 and eliminating 73% of the SORRY-Bench increase (). Finally, at matched GSM8K in safety RLHF, standard regularizers sacrifice substantially more of the alignment objective: a larger KL penalty () and L2-SP reach rewards of only 20.5 and 20.1 (vs. 29.3 for CFP) with significantly higher harmful compliance (SORRY-Bench 20.5 and 15.2 vs. 11.4, ).
| Llama-3.2-1B, disjoint splits | Unprotected | + CFP | Statistical evidence | |
|---|---|---|---|---|
| Math SFT → safety (5 seeds; Table 2) | SORRY 35.0, HarmBench 15.0 | 74.5, 62.0 | 33.6, 15.0 | all 5 seeds: CFP 33.2–33.9 vs. SFT 69.3–78.0 |
| Refusal SFT → GSM8K ↑ (App. Table 13) | 46.4 | 40.9 () | 44.4 | recovery |
| Safety RLHF → GSM8K ↑, step 64 (App. Table 12) | 46.4 | 43.5 () | 46.3 | recovery |
| Math SFT, four anchors, (App. Table 17) | GSM8K 45.3, IFEval 50.1, SORRY 34.3 | 49.6, 32.7, 69.3 | 49.6, 49.4, 43.9 | full GSM8K gain retained |
Response to Reviewer Ro2s (3/3)
Q3. Controlled comparisons. We evaluated all four controlled ablations in 1B math SFT at matched GSM8K (~47.2); removing any single component of CFP degrades safety retention (Table 4).
| 1B math SFT (Table 4) | Seeds | GSM8K ↑ | SORRY ↓ | HarmBench ↓ | WildJailbreak ↓ | |
|---|---|---|---|---|---|---|
| CFP, | 5 | 47.2 | 33.6 | 15.0 | 26.7 | 1.41 |
| Uniform scaling () | 5 | 47.2 | 46.1 | 28.5 | 35.1 | 0.46 |
| L2-SP | 5 | 47.4 | 35.9 | 18.6 | 28.7 | 0.04 |
| Stepwise Fisher correction | 3 | 46.7 | 33.6 | 16.8 | 28.5 | 1.56 |
| Fisher penalty in the loss, | 5 | 47.0 | 39.3 | 20.3 | 32.5 | 0.78 |
| Fisher penalty in the loss, | 5 | 46.5 | 34.2 | 15.1 | 26.5 | 0.18 |
- The Fisher metric. Uniform scaling differs from CFP only by setting ; at matched GSM8K, it leaves SORRY-Bench 12.5 points higher and HarmBench 13.5 points higher, and L2-SP similarly leaves a HarmBench regression (+3.6), even though both baselines move the weights much less than CFP.
- The cumulative form. The stepwise Fisher correction matches CFP on SORRY-Bench after one epoch, but underperforms on HarmBench and WildJailbreak. Over five epochs, its held-out KL triples () and its HarmBench score rises from 16.8 to 20.0, whereas CFP's held-out KL decreases (; Figure 3b, Appendix D.5). This directly reflects Proposition 2: under CFP, drift along each eigendirection of satisfies for any horizon , whereas stepwise corrections have no horizon-independent bound.
- The post-optimizer proximal step. Adding the Fisher penalty to the loss matches CFP's safety retention only when its coefficient is increased 100× (), because Adam preconditions and clips the penalty gradient together with the task gradient; at that strength, it over-constrains task-relevant directions ( vs. , reducing GSM8K to 46.5 vs. 47.2).
Regarding the 7B PPO results: the submission's 7B PPO runs applied the Fisher penalty in the loss prior to Adam (as documented in Appendix C.1, whereas the Table 1 caption of the submission inadvertently mislabelled them as post-optimizer). In the revision, we run both formulations side by side in the exact same 7B PPO setup using identical anchors (Appendix E.4, Table 20). Unprotected PPO exhibits a statistically significant 0.30-point drop on MMLU (); exact post-optimizer CFP eliminates this drop across all three anchors (+0.31, +0.28, and +0.21 points over PPO, ; Table 21), whereas the loss-level penalty does so only with (+0.25, ; Appendix E.8, Table 26).
Revision: §3.1–3.2, §4.5 (Table 4, Figure 3); Appendix B.5, B.8, D.5, E.4, E.5, and E.8.
We hope these new controls fully address your concerns, and we are happy to clarify any further questions.
Dear Reviewer Ro2s,
We apologize for writing once more, as we understand this is a busy period. With the discussion period closing soon, we wanted to send a brief reminder of our three-part response above.
We hope it addresses everything you asked and suggested: (1) true unfiltered -sampled anchors paired with held-out forward-KL evaluations and an explicit distinction from target-conditioned empirical Fishers (§2.3, Table 8, Appendix D.6); (2) statistically reliable, format-agnostic capability retention across three new regimes with severe bidirectional forgetting, disjoint splits, and five seeds (§4.2–4.3, Tables 1–3); and (3) all four matched controls—uniform displacement scaling (F=I), L2-SP, five-epoch stepwise Fisher correction, and loss-level Fisher penalty—alongside side-by-side proximal vs. loss-level runs in the 7B PPO setup (§4.5, Table 4, Figure 3, Tables 20–21, 26).
If the response resolves your questions, we would be grateful if you could confirm this or update your assessment. If anything remains unclear, we would be glad to clarify it in the time remaining.
Thank you again for your time and for a thorough review that substantially improved the paper.
Response to Reviewer Ro2s (1/3)
We thank Reviewer Ro2s for the rigorous review. We have added all the requested experimental controls in the revised paper.
W1/Q1. Initial-policy Fisher versus selected responses. We agree with this distinction and address it in two ways: (1) explicitly clarifying the theoretical interpretation of selected-response matrices, and (2) adding true πθ0-sampled anchors paired with held-out KL evaluations. First, the 7B anchors in the submission are positive semidefinite empirical score second moments evaluated on selected responses rather than unbiased estimates of the initial-policy Fisher. Proposition 1 holds for any positive semidefinite matrix, so they define valid capability-aware metrics, though only πθ0-sampled anchors correspond to the exact Hessian of the θ0-KL (§2.3, Appendix B.2, C.2). Second, our new 1B SFT anchors use one unfiltered πθ0 sample on each of 128 training prompts disjoint from all evaluation splits. In five-seed math SFT, CFP with this πθ0-sampled anchor reduces held-out forward KL from θ0 (on 330 disjoint prompts; Appendix D.1) from 0.066 to 0.005 while keeping SORRY-Bench at 33.6 and HarmBench at 15.0. Anchors constructed from Qwen2.5-7B-Instruct responses provide equivalent protection (SORRY-Bench 33.6 vs. 33.1, HarmBench 14.2 for both; three seeds), and across 306 fine-tuned models both quadratic forms rank held-out KL with a Spearman correlation of 0.81, compared to 0.73 for Euclidean parameter distance (Appendix D.6).
Crucially, this protection is not generic weight-space regularization: baselines that constrain parameter movement much more tightly still suffer larger safety regressions. In 1B math SFT at matched GSM8K, uniform displacement scaling (F=I) and L2-SP both remain much closer to θ0 in Euclidean norm than CFP, yet both leave substantial safety regressions; L2-SP also incurs a 2.4× higher held-out KL (0.012 vs. 0.005).
The same holds at 7B scale. After 200 steps of math GRPO on Qwen2.5-7B-Instruct, applying a single post-hoc CFP projection to shrink the displacement norm to 25% reduces SORRY-Bench below θ0 (33.9 vs. 34.3), whereas uniform shrinkage to the exact same norm does not.
p-value against step 200; in Table 30, the uniform retraction at 7B is not significantly below step 200.