acceptodds
Under review as a conference paper at ICLR 2027

Spectrally Broad, Behaviorally Compressible: Functional Cores of Safety-Tuning Updates and What Downstream Fine-Tuning Does to Them

Abstract

Safety tuning reliably reduces large language models harmfulness, but the geometry of the parameter change it induces, and how that geometry relates to the loss of safety under later fine-tuning, remain only partly understood. We investigate the underlying geometric structure of safety tuning and examine how downstream fine-tuning erodes alignment, both through a single object: the functional core of the safety update. We show that the safety update is spectrally broad but behaviorally compressible. Moreover, we find that this functional concentration appears early and persists even as the underlying subspace continues to evolve. We then follow the core through downstream fine-tuning in the same safety-derived coordinates. Its small-energy in-core displacement opposes the learned safety direction, and signed interventions establish the behavioral relevance of these coordinates. We then use the discovered core as a correction space: a current safety-memory loss guides minimal adjustments to downstream updates. This improves endpoint safety while largely retaining the downstream utility gain. Together, these findings connect the acquisition, erosion, and correction of safety through a learned functional core.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Safety Tuning, Large Language Models, Parameter Space Interventions

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bQQpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the parameter change induced by safety tuning and introduces the concept of “functional core”. Specifically, the authors decompose each updated parameter matrix into rank-one components (dyads) and use a small set of leading components to construct a low-dimensional subspace. Through paired retain/remove interventions, they show that this subspace captures only a modest fraction of the update energy while carrying most of the learned safety gain. The paper further examines how the functional core emerges and evolves during safety tuning and analyzes the opposing displacement accumulated within the core during downstream fine-tuning. It then uses the discovered core as a correction space, where a current safety-memory loss guides minimal adjustments to downstream updates.

Critical strengths

The paper addresses a practically relevant and important problem. It is well organized and clearly written. It examines how parameter updates induced by safety tuning carry safety behavior and how downstream fine-tuning erodes this behavior. It also explores how the identified structure can be used to mitigate safety degradation.

Critical weaknesses
  1. The paper shows that parameter updates induced by safety tuning are “spectrally broad but behaviorally compressible.” However, it remains unclear whether this functional compression is specific to safety updates. The paper validates this phenomenon across different safety-training objectives but does not perform the same spectral decomposition and retain/remove interventions on ordinary task fine-tuning updates. Without such comparisons, it is difficult to determine whether the identified structure is specific to safety learning or reflects a more general property of neural network fine-tuning.
  2. The scope of evaluation remains relatively limited. The main safety results are based primarily on AdvBench prompts, while downstream utility is measured mainly by the token-averaged NLL of Dolly responses. Although these metrics support the paper’s main analysis, they do not fully demonstrate the method’s generalization to other types of attacks or downstream capabilities. The evaluation would be strengthened by including additional safety datasets or jailbreak settings, as well as assessments of general knowledge, reasoning, and code generation.
  3. Although the proposed method mitigates the safety degradation caused by fine-tuning while preserving most of the downstream performance gains, its absolute safety performance remains notably worse than that of SafeLoRA and is still far from the before FT safety level. In addition, the baselines included in Table 3 are not sufficiently comprehensive and omit several recent methods, such as AsFT and Safety at One Shot for protection during fine-tuning, and SafeDelta and Panacea for post-fine-tuning protection.
  4. The paper lacks an explicit efficiency analysis. The authors should report the total runtime and GPU memory usage of the proposed method and compare its efficiency with that of the baselines under a unified experimental setting to better assess its practicality and scalability.
  5. The paper contains unresolved cross-references, as both Sections 3.3 and 4 refer to “Appendix ??” instead of the corresponding appendix sections.
Questions for authors
  1. Does the same functional compression emerge in ordinary task fine-tuning updates under the proposed retain/remove analysis?
  2. Can the authors broaden the safety and utility evaluations and compare with more recent baselines under a unified setting?
  3. What are the runtime, GPU memory usage, and additional training overhead of the proposed method compared with the baselines?
6 comments

We thank the reviewer for their careful reading and constructive feedback, and address each point below.

W1 / Q1. Is functional compression specific to safety learning?

Safety tuning has stronger removal effects than nine of the ten ordinary-training conditions at the same eight-dyad-per-matrix budget. Compression also occurs outside safety, as stated in Section 2, Lines 139–142: “without being exclusive to safety.”

We compare three dense safety objectives and ten ordinary conditions, with three seeds each and a projector fitted to each run's update. Effects normalize safety-risk reduction or ordinary held-out NLL improvement:

Training conditionUpdate energyRecoveryRemoval
Dense fixed refusal42.92%98.77%99.26%
Dense harm-weighted UL + refusal43.09%98.76%99.18%
Dense raw-harm PG30.59%91.16%88.56%
Ordinary: nine non-Alpaca conditions, range19.05–41.39%83.71–93.92%49.62–75.92%
Ordinary: Alpaca20.27%100.90%94.28%

Controls span Dolly, HH-helpful, GSM8K, Alpaca, OASST, classification, summarization, and continued language modeling, from pretrained and instruction-tuned starts.

Functional concentration is strong across safety objectives and task-dependent in ordinary learning.

Revision location: Please see Fig. 3 (p. 6) in the revised manuscript for all ten ordinary-training dyad-budget curves; Table 2 (p. 6) reports the numerical comparisons.

W4 / Q3. Runtime and memory

Core correction has a modest memory overhead and a measured 32.5% step-time overhead. On an A40 in FP32, three-seed means are:

Downstream procedurems/update ↓Peak allocated GiB ↓
Vanilla197.176.387
LoX, k=8199.216.388
LoX, k=32199.366.388
SafeInstr202.707.029
AsFT213.336.612
Ours261.196.554
Panacea438.447.697

The additional memory is 0.168 GiB (2.6%), and fitting the fixed basis costs approximately nine seconds once. Ours uses the least peak memory among the four online protection procedures, supporting practical use in this setting.

Revision location: See Table 30 (p. 35) for costs and profiling details.

W5. Unresolved appendix references

Thank you for the careful reading. We apologize for the broken links. Revision location: The corrected links now point to Appendix B.3 (p. 14) for pooled spectral energy and Appendix D (p. 15) for the objective comparison in the revised manuscript. Both sections were already present in the submission.

W2–W3 / Q2. Broader evaluation, requested baselines, and the safety–utility trade-off

Safety and task learning must be assessed jointly. We added all requested baselines—AsFT, Safety at One Shot, SafeDelta, and Panacea—and evaluated HEx-PHI, HarmBench, GSM8K, MMLU, and Qwen adaptive GCG (one seed, ten behaviors). Downstream rows are three-seed means; safety columns are Granite risk.

GPT-2 methodAdvBench ↓HEx-PHI ↓HarmBench ↓Dolly NLL ↓GSM8K % ↑MMLU % ↑
Aligned model0.07010.29570.15152.98712.1223.07
Vanilla0.79290.86630.78682.61051.3922.99
Ours0.13810.66040.31812.61291.2922.99
SafeLoRA, corrected pair0.08820.33610.19262.97032.1023.06
LoX, k=80.42610.74360.52752.61411.5422.98
LoX, k=320.41270.72430.49052.61661.5222.99
SafeInstr0.61650.81190.73822.60861.5722.99
AsFT0.08040.44270.25952.84811.6923.05
One Shot0.77050.86350.78672.61061.3422.99
SafeDelta0.07040.29500.15162.98412.1223.06
Panacea0.78900.85420.78732.96280.6822.94
Safety SFT0.90400.88430.85702.65481.3922.91
Vaccine0.89920.87420.84822.66481.3423.00
Booster0.90720.87880.86092.63771.2923.72

Ours retains 99.36% of Vanilla's Dolly improvement while reducing risk and lies on the GPT-2 mean-endpoint Pareto frontier for HarmBench risk and Dolly NLL (see Fig. 17 (p. 35)). Corrected SafeLoRA achieves lower risk but retains 4.44% of that improvement. We apologize for the submitted SafeLoRA alignment-pair mismatch: it used the safety-only increment instead of the pretrained-to-aligned difference. With the corrected pair, the original sampled-64 evaluation gives Granite/Guard2 risk .02308/.01215 and NLL 2.97035 (revised Table 4); the benchmark table above reports the broader common greedy evaluation separately.

A numerical safety–utility comparison. Using the GPT-2 endpoints underlying Fig. 17 (p. 35), let RR denote HarmBench Granite risk and LL Dolly NLL. Subscripts A,V,mA,V,m denote the aligned model, Vanilla, and method mm. Define

s_m=\frac{R_V-R_m}{R_V-R_A},\qquad T_m(w)=w s_m+(1-w)u_m.$$ Here $u_m$ is the fraction of Vanilla's task improvement retained, and $s_m$ is the fraction of its safety degradation prevented. Larger $w$ prioritizes safety; smaller $w$ prioritizes retaining the Dolly NLL improvement. At equal weighting ($w=.5$): | Method | Utility retained (%) ↑ | Degradation prevented (%) ↑ | $100T_m(.5)$ ↑ | |---|---:|---:|---:| | **Ours** | 99.36 | 73.78 | **86.57** | | LoX, k=32 | 98.38 | 46.63 | 72.51 | | LoX, k=8 | 99.05 | 40.82 | 69.94 | | AsFT | 36.91 | 83.00 | 59.96 | | SafeInstr | 100.51 | 7.64 | 54.07 | | SafeDelta | 0.80 | 99.99 | 50.39 | | SafeLoRA | 4.44 | 93.54 | 48.99 | **Ours maximizes this weighted score among all 14 evaluated mean endpoints for safety weights ($w$) from approximately 0.017 to 0.790.** Its advantage therefore holds across a broad range of safety–utility preferences, not only equal weighting. **With no weighting choice, ours also has the lowest AdvBench, HEx-PHI, and HarmBench risk among methods retaining at least 99% of Vanilla's Dolly improvement; the same holds at 95%.** >**Revision location:** Table 28 (p. 34) reports the full comparison. **The method also tests a practical implication of our analysis:** correcting downstream updates within the fixed core using a current safety-memory loss reduces degradation while retaining task learning. **This supports the core’s functional role in downstream safety, connecting Sections 4–6 to a behavioral intervention.**

Thank you for the detailed response and the additional experiments. However, I still have the following two concerns.

First, regarding efficiency, ms/update and peak GPU memory reflect the per-update computational and memory overhead, but they are insufficient to characterize the overall computational cost of the method. The authors should report the complete end-to-end runtime under a unified experimental setting.

Second, the paper primarily uses GPT-2 as the evaluation model. In the newly added experiments, its accuracy on GSM8K is only 1%–2%, indicating that the model itself has limited task capability. Therefore, these results do not provide sufficient evidence that the proposed method preserves the model’s reasoning and general capabilities. Meanwhile, the paper’s claims regarding the preservation of task utility are still mainly based on Dolly NLL, which primarily measures how well the model fits the Dolly reference responses and does not adequately represent broader downstream capabilities.

Therefore, I will maintain my original score.

Reviewer_aaWcOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper studies the geometry of safety-tuning updates in LLMs and their interaction with downstream fine-tuning. It introduces the functional core, a low-dimensional subspace identified through retain/remove interventions. The paper shows that safety updates have broad spectral support but concentrated behavioral effects, and investigates how this core develops, how downstream updates interact with it. The problem addresses an important problem. However, the robustness of the behavioral evaluation, and the generlity of the proposed mechanism require further validation.

Critical strengths

The paper studies an important problem in LLM alignment, i.e., understanding which parts of a safety-tuning update contribute to behavioral safety and how subsequent fine-tuning affects those components. Connecting parameter-space geometry to behavioral interventions is more informative than relying exclusively on weight norms or spectral statistics.

Critical weaknesses

The detailed comments are described as follows:

  1. The main dense-update experiments focus on a single GPT-2-based model. The primary results also depend on adapting a specific set of attention projection matrices while freezing the remaining parameters. It is therefore unclear whether the reported functional compression generalizes across architectures, model scales, safety-tuning methods.
  2. Presentation issue: Two cross-references are missing: (Section 3.3, p. 4)"…not an average of per-layer ranks (Appendix ??)";“Both criteria remain satisfied at all larger tested budgets (Appendix ??)”. It is recommended to carefully check all the citations of the appendix.
  3. The paper compares its method with several existing approaches. However, some approaches use different starting checkpoints, while others interevene at different stages of the training. More comparisons should use the same initialization batches, optimizer, and etc.
  4. The method selects the top-kk singular components in each layer and concatenates them into a single global basis, rather than maintaining a separate basis for each layer. It is unclear whether this coarse-grained design is more effective than a layer-wise approach. The authors should compare these two strategies to justify the design choice and evaluate their impact on safety preservation and downstream fine-tuning.
Questions for authors

The detailed comments are described as follows:

  1. The main dense-update experiments focus on a single GPT-2-based model. The primary results also depend on adapting a specific set of attention projection matrices while freezing the remaining parameters. It is therefore unclear whether the reported functional compression generalizes across architectures, model scales, safety-tuning methods.
  2. Presentation issue: Two cross-references are missing: (Section 3.3, p. 4)"…not an average of per-layer ranks (Appendix ??)";“Both criteria remain satisfied at all larger tested budgets (Appendix ??)”. It is recommended to carefully check all the citations of the appendix.
  3. The paper compares its method with several existing approaches. However, some approaches use different starting checkpoints, while others interevene at different stages of the training. More comparisons should use the same initialization batches, optimizer, and etc.
  4. The method selects the top-kk singular components in each layer and concatenates them into a single global basis, rather than maintaining a separate basis for each layer. It is unclear whether this coarse-grained design is more effective than a layer-wise approach. The authors should compare these two strategies to justify the design choice and evaluate their impact on safety preservation and downstream fine-tuning.
5 comments

We thank the reviewer for their careful reading and constructive feedback.

W1 / Q1. Architectures, model scales, trainable parameters, and safety objectives

Dense cross-model evidence is already in the submission. Introduction Lines 87–91 explicitly identify GPT-2, Qwen, and Llama dense updates; Lines 253–255 state the Qwen/Llama energy–behavior result. The dense training scopes are specified at Lines 953–957 (Qwen) and 1304–1311 (Llama). Tables 14 and 18 give their endpoint interventions. We extend this evidence with Qwen3B, Phi, broader trainable parameters, and matched dense objectives:

Model / training conditionDyads/matrixEnergyRecoveryRemoval
GPT-2: dense (in paper)828.56%97.94%94.97%
Qwen2.5-0.5B6460.32%99.91%58.92%
Qwen2.5-0.5B12876.14%99.99%81.70%
Qwen2.5-1.5B (in paper)1629.45%107.16%97.33%
Qwen2.5-3B6451.12%97.85%66.93%
Qwen2.5-3B12865.11%98.17%135.70%
Llama-2-7B (in paper)6436.46%94.2%73.6%
Llama-2-7B (in paper)12849.21%97.2%81.1%
Phi-3.5-mini813.51%107.14%120.46%
GPT-2: attention + MLP1628.85%99.82%92.51%
GPT-2: dense fixed refusal842.92%98.77%99.26%
GPT-2: dense harm-weighted UL + refusal843.09%98.76%99.18%
GPT-2: dense raw-harm PG830.59%91.16%88.56%

Bold: both effects ≥90%.

Parameter scope. The attention+MLP experiment trains all 48 attention/MLP matrices, totaling 84.9M entries. Attention-only adaptation has precedent in LoRA (Hu et al., 2021, §4.2), and attention+MLP adaptation in QLoRA (Dettmers et al., 2023, Appendix D). These are established adaptation scopes, not a module selection unique to our analysis. We test these scopes using dense matrix updates; embeddings, normalization parameters, and biases remain frozen.

Safety objectives. The submitted comparison of three objectives already appears at Lines 247–252 and in Appendix D, Table 5, Lines 756–761, using rank-32 LoRA. We now compare the same three objectives with dense updates.

Together, these experiments establish functional concentration across model families and scales, safety objectives, and trainable-matrix scopes.

Revision location: Table 2 (p. 6) in the revised manuscript summarizes these extensions. Please also see Fig. 3 (p. 6) for the safety-objective and ordinary-training dyad-budget curves.

W2 / Q2. Missing appendix references

Thank you for the careful reading. We apologize for the broken references. The intended sections were already present: Appendix B.3 for pooled spectral energy and Appendix D for the objective comparison.

Revision location: The revised manuscript now links correctly to Appendix B.3 (p. 14) and Appendix D (p. 15).

W3 / Q3. Matching initialization, batches, and optimizer

An identical downstream starting checkpoint cannot be imposed on every method since the methods targeting different stage of training. The appropriate controls depend on the stage:

StageMethodsIdentical source/input
During downstream FTVanilla, ours, SafeInstr, AsFT, PanaceaSafety checkpoint θA\theta_A
After downstream FTSafeLoRA, SafeDelta, One ShotThe same Vanilla endpoint θD(s)\theta_D^{(s)}
Before downstream FTLoXθA\theta_A before extrapolation
During safety trainingSafety SFT, Vaccine, BoosterPre-alignment checkpoint θ0\theta_0

Within these groups, we hold the shared checkpoints and downstream batches, dense-QKV optimizer, and 450-update budget fixed. LoX extrapolation and alignment-stage training intentionally change the actual downstream starting weights. Figure 1, Table 1, Section 7, and Appendix I describe these stages and protocols; before-FT risk is reported in submitted Table 3.

Thus, the comparison uses identical inputs wherever the methods permit them, while preserving each method's defining intervention.

Revision location: Appendix J.3 (p. 34) of the revised manuscript describes these checkpoint and training controls.

W4 / Q4. Does the global basis discard layer-wise information?

We think you misunderstood the role of the global notation: layer-wise and global projection are mathematically equivalent in our construction. Section 3.2 and Appendix B.1 place each matrix's basis in disjoint parameter coordinates. If QℓQ_\ell contains that matrix's vectorized singular dyads,

P_C=Q_CQ_C^\top=\operatorname{diag}(Q_1Q_1^\top,\ldots,Q_LQ_L^\top).$$ Consequently, $$P_Cu=[P_1u_1;\ldots;P_Lu_L].$$ This is **exactly** what separate layer-wise projections compute. >**Revision location:** The revised Section 3.2 and Appendix B.1 (p. 13) state this block-diagonal identity explicitly.

Dear Reviewer aaWc,

Thank you once more for the time and care you have put into reviewing this submission. I greatly appreciate your thoughtful feedback and valuable contribution.

Now that the authors have replied to your comments, could you please review their responses and engage with them in the discussion when convenient?

Thank you again for your continued engagement and support!

Best wishes,

Your AC

Dear Reviewer aaWc,

As the discussion period closes shortly, we wanted to check whether our response has addressed your concerns, and whether any concerns remain that we could clarify.

In particular, regarding W4: since the concatenated basis is block-diagonal, the global and layer-wise projections are the same operator, so the two strategies give identical results by construction. We have made this explicit in the revised Section 3.2 and Appendix B.1. We would appreciate knowing whether this explanation was clear.

If our responses have resolved your concerns, we would be grateful if you would consider updating your assessment.

Reviewer_fUcWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper analyses the effects of LoRA based safety fine-tuning of Large Language Models (LLMs), and the downstream task fine-tuning after this stage. The paper found a functional core of the safety update and observed that the downstream fine-tuning leads to opposing displacement within the core. Based on the findings, a mitigation strategy was derived.

Critical strengths
  • The paper presented intensively analysis of the effects of a post-training setting where the model is safety fine-tuning then task performance.
  • The mitigation balances between the safety and NLL in the particular task Dolly.
Critical weaknesses
  • In section 6 (or in Line 097-098), the paper states that “downstream fine-tuning accumulates opposing displacement with the core”. Could the author discuss whether this observed phenomenon is the general behaviour after safety tuning, or it would depend on the downstream task that was used?
  • Minors: some places has blank in Appendix: notation “??” appears in Line 196 and Line 251; and in Line 190: “its” —> “is”.
Questions for authors
  • Could the author explain further the “harm-increasing motion” (Line 344)?
4 comments

We thank the reviewer for their careful reading and constructive feedback, and address each point below.

W1. Does opposing drift depend on the downstream task?

Opposing drift is not confined to Dolly: its sign recurs across downstream tasks, while its magnitude depends on the task. The submitted paper already reports negative qq for all 66 checkpoint observations across 11 tasks (Appendix F.2, Lines 816–825; 33 runs, two checkpoints each). Tables 16 and 20 report the corresponding Qwen and Llama results. With the safety projector fitted before downstream training,

q_t=\frac{\langle c_t,a\rangle}{\|a\|^2}.$$ Thus $q_t<0$ means that the downstream update accumulates a component opposing the safety-derived direction $a$. The direction and projector remain fixed as the downstream task changes. On GPT-2, we tested Alpaca, OASST, and GSM8K with three seeds per task, starting from the same safety endpoint and using the same fixed core: | Downstream task | Mean endpoint $q$ | $\cos(c,a)$ | $q<0$ | |---|---:|---:|---:| | Alpaca | −.1535 | −.9431 | **3/3** | | OASST | −.1276 | −.9112 | **3/3** | | GSM8K | −.0370 | −.5259 | **3/3** | **All nine endpoints show opposing drift, including the instruction-following and mathematical-reasoning tasks.** > **Revision location:** Table 27 (p. 33) in the revised manuscript reports these endpoints. Please see Fig. 16 (p. 33) for the signed-coordinate trajectories. A related but distinct question is whether **ordinary training (not safety-related) updates themselves** exhibit functional compression. That comparison fits each task's own update basis and measures recovery/removal of held-out task-NLL gain. Across three seeds per condition, the three dense safety objectives erase **88.56–99.26%** of their safety gain at eight dyads per matrix, compared with **49.62–75.92%** of task-NLL gain for nine ordinary-training conditions. Alpaca reaches **94.28%**, showing that strong compression also occurs outside safety. **Safety has stronger removal effects than nine of the ten ordinary conditions at the same dyad budget.** See our response to **bQQp W1** for the full comparison. >**Revision location:** Please see Fig. 3 (p. 6) in the revised manuscript for the dyad-budget–recovery/removal curves; Table 2 (p. 6) reports the numerical comparisons. ### Q1. What does “harm-increasing motion” mean? **The downstream update optimizes Dolly response likelihood; “harm-increasing” describes its effect on a separate behavioral safety risk.** After safety tuning, we use AdamW to minimize supervised response-token cross-entropy on Dolly. This objective makes the model assign higher probability to the dataset's reference answers. We record the resulting parameter displacement $u$ between saved checkpoints. We then examine what this task-driven movement does to safety. Here, $R(\theta)$ is the expected **continuous Granite harm score** of responses generated by the current model on fixed harmful prompts, averaged over those prompts. Thus, reducing the Dolly loss and reducing $R$ are different objectives: an update that better fits ordinary task responses can also make the model more likely to generate harmful responses when queried with harmful prompts. Using the checkpoint at the start of each recorded interval, we estimate $\widehat g\approx\nabla R(\theta)$ from the current model's sampled responses, their harm scores, and their log-probability gradients. This risk gradient is used only for analysis; the downstream updates are driven by the Dolly loss. The connection to *“harm-increasing”* is $$R(\theta+u)-R(\theta)\approx\widehat g^\top u.$$ For the core component, $\widehat g^\top P_Cu>0$ means that **the part of the Dolly-trained displacement lying in the safety core is locally predicted to increase expected harm**. The term's magnitude estimates that component's contribution to the local risk change. We therefore use “harm-increasing motion” to describe a measured conflict between task training and current safety sensitivity. Section 6.2 and Appendix F.3 define the estimator; submitted Table 7 reports the measurements. ### W2. Missing appendix references and the sentence at Line 190 **Thank you for the careful reading. We apologize for the broken references and unclear wording.** The links at Lines 196 and 251 should point to Appendix B.3, “Pooling spectral energy,” and Appendix D, “LoRA Endpoint Cores across Objectives.” Both are repaired in the revision. At Line 190, $c$ is the in-core downstream displacement, $r$ its orthogonal-complement component, and $q=\langle c,a\rangle/\|a\|^2$ the signed coefficient along $a$. >**Revision location:** The corrected links now point to Appendix B.3 (p. 14) and Appendix D (p. 15) in the revised manuscript.

Dear Reviewer fUcW,

Thank you once more for the time and care you have put into reviewing this submission. I greatly appreciate your thoughtful feedback and valuable contribution.

Now that the authors have replied to your comments, could you please review their responses and engage with them in the discussion when convenient?

Thank you again for your continued engagement and support!

Best wishes,

Your AC

Thank the authors for your detailed response. My current comments are addressed, and I will keep the current evaluation.

Thank you for your time and for confirming that your concerns have been addressed. We truly appreciate your thoughtful feedback.

Comments on the paper

2 comments

We expanded the experiments to address four central concerns:

  • Generality beyond GPT-2 and one safety objective. Building on the submitted dense Qwen/Llama evidence, we added Qwen2.5-3B, Phi-3.5-mini, attention-plus-MLP training, and three matched dense safety objectives. Keep/remove interventions reproduce functional concentration across these extensions. Table 2 (p. 6) in the revised manuscript summarizes these comparisons.

  • Safety versus ordinary learning. Ten ordinary-training controls, with three seeds each, show that compression also occurs outside safety. At the same eight-dyad-per-matrix budget, all three safety objectives have stronger normalized removal effects than nine of ten ordinary conditions, comparing safety-risk gain with task-NLL gain. Please see Fig. 3 (p. 6) in the revised manuscript for the full dyad-budget comparison.

  • Downstream task dependence. New experiments on Alpaca, OASST, and GSM8K all show opposing core drift across three seeds per task, using the same fixed safety projector. This extends the submitted multi-task evidence: the phenomenon is not confined to Dolly. The revised manuscript reports endpoints in Table 27 (p. 33) and trajectories in Fig. 16 (p. 33).

  • Method evaluation and verification of the analysis. We added all four requested baselines, expanded safety/capability evaluation, corrected the SafeLoRA alignment pair, and measured computational costs. On GPT-2, core correction retains 99.36% of Vanilla's Dolly improvement while reducing risk, lying on the empirical Pareto frontier of the tested mean endpoints for HarmBench Granite risk and Dolly NLL. The method also provides intervention-based support for both analyses: the core learned during safety tuning is functionally meaningful, and using it as the correction space can mitigate downstream safety degradation. Please see Fig. 17 (p. 35) in the revised manuscript for the trade-off; Table 28 (p. 34), Table 29 (p. 34) and Table 30 (p. 35) report the comparisons and costs.

These experiments directly address generality, safety specificity, task dependence, and empirical validation. New material is marked red, with the central comparisons in Section 4 (pp. 5–6) and supporting results in Appendix J (pp. 33–35). The responses also clarify the equivalence of layer-wise and global projection and the checkpoint controls across intervention stages.

We apologize for the broken appendix references. The links to the existing Appendices B.3 and D are now repaired.

AuthorsOpenReview ↗

Functional Safety Cores: Our Contributions and Supporting Evidence from the Rebuttal

Thank you to the reviewers and AC. The rebuttal strengthens the connection from safety-core formation to downstream safety erosion and corrective intervention:

\begin{gathered} \underset{\text{Sections 4–5: early emergence and function}}{ \underbrace{\text{Safety tuning}\\;\longrightarrow\\;\text{functional core}} } \xrightarrow{\text{downstream FT}} \underset{\text{Section 6}}{ \underbrace{\text{Opposing drift and safety erosion}} } \\\\[4pt] \underset{\text{method design and intervention evidence}}{\Updownarrow} \\\\[4pt] \underset{\text{Section 7: intervention evidence}}{ \underbrace{\text{Core-guided correction}} } \end{gathered} }$$ - **Safety tuning → functional core (Sections 4–5).** **Added during rebuttal: Qwen2.5-3B, Phi-3.5-mini, attention+MLP training, and three dense safety objectives**, extending the submitted GPT-2/Qwen/Llama evidence for functional concentration. See our response to **aaWc, W1/Q1**. We also added **ten ordinary-training controls, three seeds each**. At eight dyads per matrix, all three safety objectives have stronger normalized removal effects than nine of the ten ordinary conditions. **Compression also occurs outside safety, but is particularly strong in safety tuning.** See our response to **bQQp, W1/Q1**. - **Opposing drift and safety erosion (Section 6).** **Added Alpaca, OASST, and GSM8K:** all nine task/seed endpoints oppose the learned safety direction, extending the drift evidence beyond Dolly. See our response to **fUcW, W1**. - **Core-guided correction (Section 7).** **Expanded to Qwen1.5B and Phi, three separately trained tasks, and six protection baselines:** ours ranks **second on average** under equal-weight safety–utility scoring, with **1.02% training-time overhead and 0.0210 GiB additional peak memory** using a precomputed core (construction excluded). These interventions further corroborate the core's behavioral relevance during adaptation. See our **follow-up response to bQQp, Sections 1–3**, for costs, capability evaluation, and rankings. **An open question is what drives functional-core formation.** We hypothesize that temporally coherent components of training gradients accumulate into behaviorally concentrated directions; establishing this mechanism remains future work. We have corrected the broken appendix references and marked the revised evidence in red.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.