acceptodds
Under review as a conference paper at ICLR 2027

Representational Inertia: Activation Stitching Reverses Fine-Tuned Behavior in Large Language Models

Abstract

Fine-tuning enables general-purpose language models to adapt to specific tasks and align with human preferences such as safety and truthfulness. Existing work has examined the representational changes induced by this process, yet it remains unclear how earlier representations remain functionally usable after training. We find that fine-tuned models retain backward compatibility with their precursor checkpoints, a property we call representational inertia. We investigate this property through cross-checkpoint activation stitching (CAST), which supplies precursor activations at a selected layer while retaining the fine-tuned computation downstream. Across six pairs of base and aligned models, a single stitch at intermediate depth returns the model to precursor behavior, at depths that vary across models and behaviors. The fine-tuned blocks that still run downstream cannot restore what training installed. The reverted text remains fluent, indicating that those blocks process the precursor's representation rather than being disrupted by it. This compatibility exposes a vulnerability in the activation space of aligned models, and it likewise allows accuracy suppressed by three unlearning methods to be recovered. It also enables possibilities to repair the models, as clean precursor activations suppress emergent misalignment and trained backdoors. Finally, the same intervention can be trained against, and we showcase that a model trained to recover from it also becomes more robust to unseen textual jailbreaks. Our findings show that fine-tuning installs behavior without displacing the computation it was trained on, with consequences for both the robustness of desired behavior and the correction of harmful changes.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Finetuning, Alignment, Activation Patching, Information bottleneck, Model stitching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hwXLOpenReview ↗
Rating
2
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper introduces CAST, which replaces a fine-tuned model’s residual-stream activations at a selected layer with activations from its precursor checkpoint. The standard intervention runs throughout generation, effectively combining the donor’s early blocks with the receiver’s later blocks. Across multiple checkpoint pairs covering different domains, CAST often shifts behavior toward the precursor while producing coherent text. The paper also explores using CAST for model repair and recovery training against adversarial interventions.

Critical strengths

S1. This paper directly addresses the research question by testing whether fine-tuned downstream layers can process precursor representations. It requires no learned connector or optimized perturbation, making the experimental procedure straightforward and its effects relatively easy to inspect.

S2. The experiments span several model families and different settings. This breadth makes the compatibility finding more convincing. The study also includes analyses of intervention depth and duration.

S3. The paper goes beyond demonstrating behavioral reversal by examining repair and recovery training, and explicitly reports some trade-offs between refusal and benign compliance.

Critical weaknesses

W1. The novelty of CAST needs a clearer explanation. [1] already demonstrates patching base-model activations into a backdoored fine-tuned model without a connector. This appears inconsistent with the claim in Section 2 that such stitching has not been studied. The authors should clarify what new insights this paper provides.

W2. Compatibility alone does not establish the proposed mechanism. Section 3 describes CAST as combining the donor’s early layers with the receiver’s later layers. This does not show that fine-tuning merely redirects existing representations: donor activations may supply information missing from the receiver’s own computation.

W3. Knowledge recovery does not establish knowledge retention. In Section 4.1, the original checkpoint serves as the donor. Its activations may help the receiver access retained knowledge or reintroduce lost information that its later layers can still process. Without distinguishing these explanations, the conclusion that these methods fail to reliably unlearn is too strong.

W4. Recovery training lacks a matched baseline and broader evaluation. Without a standard refusal-SFT baseline using matched data and training budgets, it is unclear whether CAST provides benefits beyond additional safety training. Recovery training is also evaluated on only one model, with a limited number of stitched test prompts and benign instructions, providing insufficient evidence of generalization and preserved benign compliance.

[1] Activation Space Interventions Can Be Transferred Between Large Language Models,ICML 2025

Questions for authors

See weakness

9 comments
AuthorsOpenReview ↗

Official Response to Reviewer hwXL (1/4)

We thank the reviewer for their time and effort in reviewing our work. They highlight the usefulness of probing fine-tuning with base representations without any learned connector or optimized perturbation, the comprehensiveness across model families and settings, and the benefits of recovery training. We address their remaining concerns below.

W1. Novelty relative to Oozeer et al.

W1. The novelty of CAST needs a clearer explanation. [1] already demonstrates patching base-model activations into a backdoored fine-tuned model without a connector. This appears inconsistent with the claim in Section 2 that such stitching has not been studied. The authors should clarify what new insights this paper provides.

We want to clarify that we never claim CAST as a contribution by itself (see Section 1, Section 3). Instead, we mainly use it as a causal probe to study the novel phenomenon of representational inertia or backward compatability, which forms the main focus of our work. While such causal intervention operators have been used in part in the prior work, we did not find any existing work that uses it in this exact form, which is why we name it explicitly instead of referring to another work directly. As noted in our Introduction, Oozeer et al., 2025 uses this at a small experiment to remove backdoors while their paper largely focuses on transferring activations across different models using a learned map. Their experiment is thus exploratory across different models instead of trying to establish a common phenomenon in fine-tuned models specifically. However, we will discuss this experiment explicitly in Section 2 in our revised version.

We summarize the differences with existing works below:

PaperMain Research QuestionIntervention studiedConnectorWhat is replacedPositions
Bansal et al. (2021)Do two networks learn compatible representations?Network A → network B (different seeds, data or architectures)Trained linear layerB's layers 1..l, by A's layers 1..l; cut point l variedAll (weight-level)
Prakash et al. (2024)Does fine-tuning change a task circuit or strengthen it?Fine-tuned → baseNoneOutputs of selected attention heads in the circuitSome
Bandarkar et al. (2024)Can two experts from one base be merged to combine skills?Language expert, math expert → math expert in low-resource languagesNoneWeights of the first and last few layersAll (weight-level)
Oozeer et al. (2025)Can a steering vector be transferred between models via a learned map?Mainly small → large model; base → backdoored fine-tune (Section 4.4)Trained autoencoder; none in the base → fine-tune patchResidual stream at one layerPrompt-only
Zhao et al. (2026)Do chat-template tokens carry fine-tuned behavior to unrelated queries?Base → fine-tunedNoneRepresentations at those positionsChat-template prefix only
OursDo fine-tuned models revert to base behavior when given the base model's intermediate representations?Base → fine-tunedNoneResidual stream at layer l (can also be seen as a weight splicing)All

Thus, none of the cited papers study the same research question and use the exact same operator as ours. Oozeer et al. 2025's base->fine-tuned finding is limited to studying backdoors and to representation patching at the prompt-level, while Zhao et al. 2026's finding is limited to broad domain misalignment and to prefix-level positions. Our work thus generalizes these two findings to generic behavior sets and explains them using a general phenomenon of "representational inertia" at the representation level of the fine-tuning, using a simple form of activation stitching. We further clarify our contributions as follows:

  1. We establish the phenomenon of representational inertia in fine-tuned models using multiple model pairs, where fine-tuned models revert to their base beahvior when given base representations at a certain layer.
  2. We show how this exposes novel vulnerabilities in different applications such as safety, alignment, and unlearning. On the other hand, we also show how this can be used to remove negative effects introduced during fine-tuning such as emergent misalignment and backdoors.
  3. Finally, we show how this effect can be directly used to train more robust models by teaching them to recover from cases when their activations are replaced with their base variants.
  4. In the rebuttal, we further find that this phenomenon cannot be fully explained through behavioral localization or linear interpolations (see our response to iq76), or simple activation-level properties (see our response to iq76), or other intervention controls (see our response to o9z4).
AuthorsOpenReview ↗

Official response to Reviewer hwXL (2/4)

W2. Compatibility alone does not establish the mechanism

W2. Compatibility alone does not establish the proposed mechanism. [...] This does not show that fine-tuning merely redirects existing representations: donor activations may supply information missing from the receiver's own computation.

We thank the reviewer for noting this, as it helps us clarify the differences more concretely. Compatibility of representations mainly implies non-degeneracy of the outputs when the layers are stitched, while our focus is on representational inertia, i.e., the model reverts to the behavior of M\mathcal{M} when its latent representations up to layer ll are replaced by those that M\mathcal{M} produces on the same input. Thus, compatibility is a necessary but not sufficient condition for the phenomenon of "representational inertia".

This can thus imply two things: (1) either the "probed information" was already present in the fine-tuned layers and the base activations acted as a trigger to generate the behavior, or (2) the "probed information" is presented to the receiver model in the stitched activations, which it directly processes.

Answering which one is accurate involves accurately probing where a given behavior is present within the model, a central question in interpretability. Due to the lack of reliable interpretability techniques, we cannot reliably answer the mechanisms behind the phenomenon and thus leave it for future work to investigate in depth. Instead, our focus here is in establishing these vulnerabilities in fine-tuned models since even if hypothesis (2) is correct, deeper model layers must be robust and truly forget to answer which motivates our recovery training approach. We also show we can use this phenomenon to recover good behaviors cheaply when unintended effects seep through fine-tuning.

In order to understand the phenomenon itself, instead of its applications, we have now conducted preliminary analyses on finding whether a fine-tuned model's stored knowledge in its layers can be used to generate fine-tuned behavior from the base model (as mentioned above where we observe partial effects of it in some behaviors), and our deeper analysis of the unlearning use case below.

W3. Knowledge recovery does not establish knowledge retention

W3. Knowledge recovery does not establish knowledge retention.

We thank the reviewer for directing us to these underlying effects that may explain the unlearning results better. Here, we run additional analyses to find if the unlearned model retains any knowledge or if it is supplied by the original model, on all 1,273 WMDP-bio questions with exact scoring.

(1) Is knowledge retained in the unlearned model's own activations? We trained linear probes on each model's own residual stream (chance 0.25; shuffled-label probes 0.23–0.28):

ModelProbe for the answer letterOwn accuracyProbe for "this option is correct"
Original0.670.6540.63
RMU0.330.3140.48
ELM0.320.2460.51
SimNPO0.530.4630.59

For RMU and ELM the answer is not linearly decodable, but a per-option self-correctness belief signal partly survives (0.48–0.51 vs. 0.63 for the original). On the other hand, SimNPO clearly retains the knowledge.

(2) Can the knowledge be recovered without the original model? We changed the activations at a single layer by a fixed direction and evaluated on 891 held-out questions, with a same-norm random vector as control:

Model (unmodified bio / cyber / MMLU)ConditionWMDP-bioWMDP-cyberMMLU
RMU (0.310 / 0.313 / 0.555)layer 7, mean unlearned WMDP - MMLU (no original)0.4890.3320.564
layer 6, 2 × mean base − unlearned0.5520.3610.562
layer 6, random vector0.3120.3060.555
ELM (0.241 / 0.293 / 0.474)layer 6, mean unlearned WMDP - MMLU (no original)0.3940.3310.498
layer 6, random vector0.2270.2940.478
SimNPO (0.447 / 0.337 / 0.489)layer 14, 2 × mean base − unlearned0.5980.3870.549
layer 14, random vector0.4490.3430.476

A direction computed without access to the original model recovers a substantial part of the gap, with MMLU intact: RMU rises from 0.31 to 0.49 and ELM from 0.24 to 0.39. Random vectors have no effect. Some knowledge is therefore retained and reachable without the donor, but the retention is partial and method-dependent.

Thus, we find partial evidence of knowledge retention in unlearned methods but note that a deeper analysis at the neuron/head level is needed with stronger interpretability techniques to make strong conclusions. However, we argue that this does not invalidate our claims of unreliable unlearning, since even knowledge recovery can be seen as a failure point of an unlearned model, and methods must be designed that do not allow knowledge recovery from such interventions.

AuthorsOpenReview ↗

Official Response to Reviewer hwXL (3/4)

W4. Matched baseline and broader evaluation for recovery training

W4. Recovery training lacks a matched baseline and broader evaluation. Without a standard refusal-SFT baseline using matched data and training budgets, it is unclear whether CAST provides benefits beyond additional safety training. Recovery training is also evaluated on only one model, with a limited number of stitched test prompts and benign instructions, providing insufficient evidence of generalization and preserved benign compliance.

The original experiment was intended as a proof of concept that stitching can be used to build a training defense, and is not intended to be a state-of-the-art adversarial training method but for completness, we have now expanded our analyis more comprehensively with matched baselines on two models.

Setup. Every method uses the same configuration: the same 70 harmful training prompts, LoRA rank 8, learning rate 1e-4, 3 epochs, the same clean benign loss, and three seeds. The baselines are:

  1. plain refusal SFT on the same prompts without stitching;
  2. a harmful 10-token prefix followed by "Wait" and a refusal;
  3. Qi et al.-style harmful-prefix training (prefix length 0 with probability ½, otherwise uniform in 1–100 tokens);
  4. targeted LAT at layer 8.

We also trained CAST-aware training at a random layer in [8, 24] and at the layer immediately before the flip. In addition, we tested whether the defenses combine, both jointly (a mixed loss) and sequentially (one full training stage after the other).

Evaluation now covers:

  • the paper's 30 JailbreakBench held-out prompts and the 312-prompt held-out set from StrongReject;
  • 26 real StrongReject jailbreaks applied to 30 prompts;
  • 300 Alpaca and 250 XSTest benign prompts;
  • MMLU (five-shot, 1,000 questions) and IFEval (strict prompt accuracy);
  • Refusal check with an LLM judge instead of a keyword-based checker.

Llama-2-7B-chat. Our expanded analysis shows that our initial choice is not a completely ideal model for evaluation since it mainly suffers from over-refusal, and the untrained model already refuses 85.3% [81, 89] of the held-out prompts, so every trained method saturates, and the methods cannot be distinguished. What we find is that CAST-aware training preserves the safe compliance the most among the other adversarial training approaches, highlighting its benefits.

Method30 JBB promptsHeld out Strong RejectReal jailbreaksAlpaca compliedXSTest compliedMMLU / IFEval
first 10 tokens stitched at L19no stitchfirst 10 tokens stitched at L19
Untrained80.0 [63, 90]100.085.3 [81, 89]72.298.063.247.6 / 25.5
Untrained + "Wait" string at inference100.0100.099.783.132.331.647.6* / 18.9
Plain refusal SFT84.4 ± 3.899.9 ± 0.284.8 ± 0.976.5 ± 4.696.659.347.5 / 28.6
CAST-aware training (stitch at layer 19)100.0 ± 0.0100.0 ± 0.099.3 ± 0.488.0 ± 3.194.949.748.4 / 25.4
CAST-aware, random layer100.0 ± 0.0100.0 ± 0.0100.0 ± 0.088.2 ± 1.984.424.948.3 / 23.9
CAST-aware, pre-flip layer (9)100.0 ± 0.099.8 ± 0.499.6 ± 0.787.4 ± 3.785.325.148.5 / 25.4
Qi et al.-style98.9 ± 1.9100.0 ± 0.099.7 ± 0.084.0 ± 4.991.946.947.7 / 28.4
Targeted LAT100.0 ± 0.0100.0 ± 0.096.2 ± 0.688.4 ± 0.178.627.247.7 / 16.5
Other joint and sequential combinations100.0100.099.6–100.083.5–90.679.8–97.621.7–50.0–

Llama-3.1-8B-Instruct. We instead extend our analysis to this model as it is more suitable to teach refusal as its overrefusal is pretty low. In this model, the same stitched attack on held-out set is quite effective, so it is the informative setting:

Reviewer_o9z4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the phenomenon of cross-checkpoint activation stitching, which patches activations from base models into fine-tuned models to revert the fine-tuned behaviors. Analysis results across multiple models and tasks show that fine-tuned models exhibit representational inertia, which makes them susceptible to this simple attack scheme. The paper proposes a simple defense method that fixes this by training models to be aware of such stitching attacks.

Critical strengths
  1. The paper investigates an interesting phenomenon with an easy-to-study setup.

  2. The paper proposes a simple attack method and a corresponding intuitive defense that is effective and easy to implement, which could be useful in practice.

Critical weaknesses
  1. The paper does not have proper evaluations of the behavioral similarity between the stitched fine-tuned model and the base model. All evaluations show that fine-tuned models after stitching perform similarly to the base model on safety/alignment benchmarks, but they might achieve similar performance for different reasons. If so, representational inertia does not hold, as the stitched model is not reproducing the base model’s behavior but is rather doing something different. Appropriate evaluations should compare the responses of the stitched fine-tuned model and the base model to the same prompt, and use an LLM judge or another protocol to check whether the content, style, etc. are actually similar.

  2. The paper lacks baseline comparisons with stitching representations from other sources, so it is unclear whether the change in behavior is actually caused by the model being “inert” and following the base model’s behavior. Proper baselines could include stitching: (1) representations of the fine-tuned model with random noise added; (2) representations of the fine-tuned model / the base model on some irrelevant prompts/responses; (3) representations of the fine-tuned model swapped across different layers. If any of these baselines shows a behavior change on alignment and the other evaluation tasks similar to the change shown with CAST, then it is hard to attribute the change to representational inertia, and it is more likely that the instruction-tuned models collapse simply because of perturbations in the representation space.

Questions for authors
  1. Notation of “checkpoints”: In Section 3, the paper frames the method around stitching activations between different checkpoints: “it is not clear whether the checkpoints themselves can understand each other’s activations” (Lines 171–172). However, it is unclear to me whether the term “checkpoint” refers to model checkpoints at different steps of fine-tuning on the same data, or strictly to models fine-tuned on different data. Model checkpoints at different steps of fine-tuning are highly likely to “understand” each other’s activations.

  2. Line 175: The donor and receiver use the same layer l. Does the method assume that activations at the exact same layer in the donor and receiver have the same functionality, so that no further layer mapping is needed? The paper does not clarify how this assumption is made.

  3. Definition 1: The set of prompts B is also not well-defined. If B is sampled from prompts that are irrelevant to the task M_ft is fine-tuned on, then I’d expect M_ft to behave similarly to the base model anyway, e.g., if M_ft is fine-tuned on math and B is sampled from literature. B should be defined as prompts sampled from tasks related to the task M_ft is fine-tuned on.

  4. Similarity is not well-defined either: does it require the stitched fine-tuned model to generate outputs semantically similar to what the base model would generate, or does it require more fine-grained behavior-level judges/rubrics? Is this task-specific?

  5. Line 188: A “non-trivial layer” is not properly defined. Is layer L−1 non-trivial, given that it is only one layer before the unembedding layer, so its behavior should be similar to that of layer L?

  6. Line 248: “We report each behavior’s rate among the non-degenerate responses.” What is the proportion of degenerate responses across tasks?

  7. Section 4.3, Lines 447–448: It is unclear to me whether the recoverability comes from the fine-tuning or from the “Wait, I shouldn’t help with this” prompt that instructs the model to change its behavior. What if we appended this prompt after the first 10 tokens generated under stitching at inference time, without any training? Would the model be able to recover from the attack? If so, this might undermine the contribution of the proposed training method.

  8. Section 4.3, Lines 444–445: Is the model trained with CAST-aware fine-tuning at layer 19 robust to inference-time stitching at other layers?

8 comments
AuthorsOpenReview ↗

Official Response to Reviewer o9z4 (1/4)

We thank the reviewer for reviewing our work and are glad that they find the phenomenon interesting and the setup easy to study. They also describe the proposed defense as intuitive, effective, and easy to implement. We address their remaining concerns below.

1. Behavioral similarity between stitched and base outputs

The paper does not have proper evaluations of the behavioral similarity between the stitched fine-tuned model and the base model. [...] Appropriate evaluations should compare the responses of the stitched fine-tuned model and the base model to the same prompt, and use an LLM judge or another protocol to check whether the content, style, etc. are actually similar.

We appreciate this feedback as it helps us verify the similarity claims more concretely. For each stitched response, we use an LLM judge (Claude Haiku 4.5) to decide whether it was closer to the base model's or to the fine-tuned model's response to the same prompt. The judge rated content, style, and stance separately, with presentation order randomized and "neither of the two" as another choice. We also measured the per-token KL divergence along the stitched generation.

PairLayerBehavior rateCloser to base: content / style / stanceKL to baseKL to fine-tuned
Llama-280.76 refuse0.21 / 0.27 / 0.271.350.40
100.450.58 / 0.40 / 0.451.030.62
150.121.00 / 1.00 / 1.000.671.10
300.000.75 / 0.75 / 0.750.162.12
Qwen2.5-7B130.940.14 / 0.08 / 0.040.240.25
210.650.46 / 0.45 / 0.430.120.72
260.580.49 / 0.50 / 0.400.040.77
Tülu-3-8B-SFT150.890.13 / 0.13 / 0.140.431.01
280.680.47 / 0.40 / 0.430.041.35
Emergent misalignment, Qwen2.5-7B80.06 misaligned0.47 / 0.38 / 0.581.130.06
140.000.94 / 0.94 / 0.940.340.20
170.001.00 / 1.00 / 1.000.110.25

Where the behavior has flipped, stitched outputs are judged closer to the base model as the stitching happens at deeper layers across models and settings. The token-level divergence also crosses over toward the base from layer 12 (Llama-2), 13 (Qwen), 10 (Tülu) and 16 (emergent misalignment). We do note, however, that for Tülu and Qwen, stitched outputs are only about half base-like even at final layers, showing that while the behavior moves closer to the base model, the generated styles may differ.

Thus, the evidence supports the claim that, where behavior flips, the stitched model behaves like the base model, and not merely that it scores like it.

AuthorsOpenReview ↗

Official Response to Reviewer o9z4 (2/4)

2. Baselines: noise, irrelevant representations, cross-layer stitches

The paper lacks baseline comparisons with stitching representations from other sources [...]. Proper baselines could include stitching: (1) representations of the fine-tuned model with random noise added; (2) representations of the fine-tuned model / the base model on some irrelevant prompts/responses; (3) representations of the fine-tuned model swapped across different layers. [...]

We thank the reviewer for suggesting these ablations as it will help us understand the effect better. Below, we ran all three suggested baselines at each pair's flip layer, together with a randomly rotated difference and the dataset-mean activation. Each cell gives refuse / comply / gibberish rates on 100 prompts; noise results are averaged over three seeds.

ConditionLlama-2, L10Qwen2.5-7B, L24Tülu-3-8B-SFT, L21
Fine-tuned, unstitched0.97 / 0.03 / 0.000.93 / 0.04 / 0.031.00 / 0.00 / 0.00
The paper's stitch0.28 / 0.34 / 0.380.56 / 0.41 / 0.030.61 / 0.18 / 0.17
(1) Noise with the norm of the base − ft difference0.98 / 0.01 / 0.010.93 / 0.06 / 0.011.00 / 0.00 / 0.00
(1′) That difference, randomly rotated0.96 / 0.00 / 0.040.93 / 0.04 / 0.031.00 / 0.00 / 0.00
(2) Base activations on unrelated Alpaca text0.00 / 0.00 / 0.970.00 / 0.00 / 0.960.00 / 0.00 / 0.95
(2) Fine-tuned activations on unrelated text0.00 / 0.00 / 0.990.00 / 0.00 / 0.990.00 / 0.00 / 0.93
Dataset-mean fine-tuned activation0.00 / 0.00 / 1.000.00 / 0.00 / 1.000.00 / 0.00 / 1.00
(3) Base activations from layer l − 20.43 / 0.25 / 0.320.70 / 0.20 / 0.100.81 / 0.09 / 0.09
(3) Base activations from layer l + 20.21 / 0.25 / 0.540.76 / 0.23 / 0.010.40 / 0.21 / 0.32
(3) Base activations from layer l + 40.10 / 0.11 / 0.790.09 / 0.05 / 0.860.49 / 0.31 / 0.16

Addressing the alternative hypothesis directly:

  • (1) A perturbation of the same magnitude neither changes behavior nor collapses the output. Same-norm noise and the rotated difference leave refusal at the fine-tuned level on all three pairs, so perturbation magnitude does not explain the flip.
  • (2) Irrelevant representations cause collapse, not compliance. Activations from unrelated text and the mean activation produce 93–100% gibberish. This is the collapse the reviewer describes, and it differs sharply from the paper's stitch, which yields 3–38% gibberish with coherent compliance. The flip requires the donor's representation of the same prompt.
  • (3) Cross-layer stitches can be nearly as effective as same-layer stitches. On Llama-2, base layer 12 stitched into receiver layer 10 yields 0.21 refusal, compared with 0.28 for the paper's stitch. Same-layer matching is the best choice on Qwen but is not unique, and we will remove any claim that it is.

The same picture holds for two further behaviors:

  • Sleeper backdoor (exact match): noise and rotation leave the trigger rate at 1.00 at every layer. Unrelated text removes the backdoor with 3–7% gibberish, and the mean activation removes it with 100% gibberish.
  • Emergent misalignment (Qwen2.5-7B, 280 answers per condition): the paper's stitch reduces the misaligned rate from 35% (fine-tuned level, layers 2–5) to 19% at layer 8, 4% at layer 10 and 0–3% from layer 12 onward. Same-norm noise and the rotated difference leave it at 23–39% at every layer. Unrelated text and the mean activation produce no coherent answers. Nearby-layer stitches from deeper layers (+2, +4) behave like the paper's stitch, whereas those from shallower layers (−2, −4) often destroy coherence.

3. Clarifications

Notation of "checkpoints": [...]

In the paper, "checkpoint" denotes a later training stage from the same initialization and architecture (base → SFT → DPO; clean → backdoored or unlearned), not an intermediate step within a single run. We will define the term explicitly. Since the checkpoint pairs used in this study correspond to different released versions, they often differ in multiple training steps and thus, is unlikely to preserve any "understanding" of corresponding activations.

Line 175: The donor and receiver use the same layer l. Does the method assume that activations at the exact same layer in the donor and receiver have the same functionality [...]?

This is an assumption motivated by the shared initialization, and we have now tested it (Section 2, condition (3)). The same layer is the best-matched choice on Qwen, but stitching from layer l ± 2 produces partly similar behavior on Llama-2 and Tülu. The correspondence is thus approximate and not intended, and we will state this.

AuthorsOpenReview ↗

Official Response to Reviewer o9z4 (3/4)

Definition 1: The set of prompts B is also not well-defined. [...] B should be defined as prompts sampled from tasks related to the task M_ft is fine-tuned on.

We define B as each behavior's own evaluation set: harmful prompts for refusal, WMDP-bio for unlearning, the emergent-misalignment question set, and triggered prompts for backdoors. We further require this set to be such that the fine-tuned and base models differ by construction (for example, 0.97 vs. 0.00 refusal on Llama-2). We will rewrite Definition 1 to require this.

Similarity is not well-defined either: does it require the stitched fine-tuned model to generate outputs semantically similar to what the base model would generate, or does it require more fine-grained behavior-level judges/rubrics? Is this task-specific?

Similarity is task-specific and is now defined at two levels:

  • Behavioral similarity uses each behavior's own metric: refusal, aligned vs. misaligned, answer accuracy, or trigger match.
  • Output similarity uses the content/style/stance comparison and the KL divergence from Section 1.

We will state both definitions in the paper in the methods section.

Line 188: A "non-trivial layer" is not properly defined. Is layer L−1 non-trivial [...]?

Layer L−1 can be seen as trivial since only the unembedding layer comes after it. However, upon careful consideration, we have now decided to drop this naming since there is no trivial reason why the unembedding layer should be able to process base activations, let alone generate behavior like one.

Line 248: "We report each behavior's rate among the non-degenerate responses." What is the proportion of degenerate responses across tasks?

Pooled over the swept layers, the gibberish share of stitched refusal responses is:

ModelGibberish share
Llama-20.516 (up to 0.86 at one layer)
Mistral-Small-24B0.225
OLMo-2-32B0.126
OLMo-2-13B0.104
Mistral-7B0.102
Tülu-3-8B-SFT0.099
Qwen2.5-7B0.019
Qwen2.5-14B0.011

For the 14B emergent-misalignment model, the share is 0.26 at layer 4, 0.23 at layer 19, and 0.04 at layer 28 (full profile in our response to Reviewer i7q6, Section 10). We will replot with the number of non-degenerate answers behind every point.

4. Does the "Wait" string alone recover refusal?

Section 4.3, Lines 447–448: It is unclear to me whether the recoverability comes from the fine-tuning or from the "Wait, I shouldn't help with this" prompt that instructs the model to change its behavior. What if we appended this prompt after the first 10 tokens generated under stitching at inference time, without any training? Would the model be able to recover from the attack? If so, this might undermine the contribution of the proposed training method.

We add this baseline in our expanded analysis on two models (Llama-2-7b and Llama-3.1-8B following our updated setup noted in the reviewers' response): the prompt and first 10 tokens are stitched, the string is appended, and no training is applied.

Llama-2-7B-chat.

Method30 JBB promptsHeld out Strong RejectReal jailbreaksAlpaca compliedXSTest compliedMMLU / IFEval
no stitchfirst 10 tokens stitched at L19no stitchfirst 10 tokens stitched at L19
Untrained100.0 ± 0.080.0100.085.372.298.063.247.6 / 25.5
Untrained + "Wait" string at inference96.7100.0100.099.783.132.331.647.6* / 18.9
Plain refusal SFT100.0 ± 0.084.4 ± 3.899.9 ± 0.284.8 ± 0.976.5 ± 4.696.659.347.5 / 28.6
CAST-aware training (stitch at layer 19)98.9 ± 1.9100.0 ± 0.0100.0 ± 0.099.3 ± 0.488.0 ± 3.194.949.748.4 / 25.4

Llama-3.1-8B-Instruct.

Method30 JBB promptsHeld out Strong RejectReal jailbreaksAlpaca compliedXSTest compliedMMLU / IFEval
no stitchfirst 10 tokens stitched at L19no stitchfirst 10 tokens stitched at L19
Untrained93.346.798.154.851.099.087.667.4 / 62.5
Untrained + "Wait" string at inference10.0 (93.3)60.0 (93.3)15.4 (98.7)41.7 (94.6)18.6 (72.2)44.724.867.4* / 35.5
Plain refusal SFT95.6 ± 1.950.0 ± 8.899.3 ± 0.260.6 ± 2.868.5 ± 7.299.385.667.7 / 66.5
CAST-aware training (stitch at layer 10)98.9 ± 1.994.4 ± 1.999.5 ± 0.595.4 ± 1.374.6 ± 1.298.677.168.4 / 65.0
SFT: Harmful 10-token prefix + "Wait"97.8 ± 3.875.6 ± 6.999.4 ± 0.386.6 ± 2.173.6 ± 10.598.982.7–
CAST-aware + 10-token prefix SFT100.0 ± 0.093.3 ± 3.398.9 ± 0.496.0 ± 0.575.1 ± 5.099.282.468.6 / 66.9
Reviewer_i7q6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper takes a fine-tuned model and replaces its residual stream at one layer with the activations of an earlier checkpoint, usually the base model. The replacement is held at every generated token, and the fine-tuned layers after the stitch finish the generation. The authors call this method CAST. Their main finding, which they call representational inertia, is that the fine-tuned model can still run on its precursor’s activations and falls back to the precursor’s behavior. They sweep stitch depth across checkpoint pairs and model families, and across behaviors including alignment, unlearning, emergent misalignment and backdoors. They also propose a training recipe that makes a model robust to a single-layer stitch, and an ablation over how many tokens are stitched.

Critical strengths

S1 The paper covers many models and depths, and it picks applications that matter for safety. The depth at which behavior flips changes across models and behaviors, so the layer sweep is a cheap diagnostic of how deep a fine-tuned behavior goes into the network.

S2 The unlearning results are the most interesting part for me. The layers where unlearning can be reverted do not have to match the layers the method edited. Figure 3 shows this well. For RMU, the suppression sits at or before block 6, and the edited blocks 7 and 8 do not suppress again when they get clean inputs. For SimNPO, which edits every layer, the change that matters sits around layers 5 to 13. I think CAST could become a useful check of whether an unlearning method changes a wide range of layers or only a narrow one which differs from the training recipe.

Critical weaknesses
  1. The method is not novel. CAST closely matches prior work. Oozeer et al. (2025) replace the full residual stream at one layer with base-model activations, sweep over depth, and apply this to backdoored fine-tuned versions of the same model. Prakash et al. (2024) patch activations across base and fine-tuned models, and Zhao et al. (2026) patch the original model into emergent-misalignment models at chat-template tokens. Because the stitch is held at all tokens, CAST is also the same as gluing the first l layers of the donor to the last layers of the receiver. This is layer swapping between checkpoints (Bandarkar et al., 2024). So Figure 1 shows an existing method. The paper applies it to emergent misalignment, which Zhao et al. (2026) partly cover. The backdoor experiment in Section 4.2.2 repeats the setting of Oozeer et al. (2025) with different models and triggers, without citing it there. The Related Work claim that stitching a base model into its fine-tuned version without a connector has not been studied is therefore not correct. As far as I can tell, the new parts are the settings (alignment, unlearning, emergent misalignment), the recovery training and the prompt-only ablation. The abstract, introduction and Related Work should present CAST as an existing operator applied to these new settings.

  2. The experiments do not support “representational inertia” as stated. Since CAST swaps the whole prefix of layers, when there is a flip those two interpretation are indistinguishable:

  • “the fine-tuned layers cannot overcome the representations installed by the base activations in the first l layers”
  • “the fine-tuned change lives in the first l layers”
  • "an important trigger needs to be added in the first l layers for the last layers of the finetuned model to be activated" The term inertia is also not well defined, it could mean different things. Deep stitches are not very revealing. At 88% depth on Tülu, only 12% of the fine-tuned layers still run, so a return to base behavior is expected (in the limit all layers are the base model one so a flip MUST happen).

When the flip happens at an early layer l, it means the change sits somewhere in layers 1 to l rather than "layer l is where the behavior live". Layer l alone (i.e. stitching this single layer into the base model) is likely not sufficient, and the authors should verify this.

Questions for authors
  1. What happens when you stitch fine-tuned activations into the base model? If the base model also handles them fluently, inertia may just mean the two models have very similar representations.
  2. Please report, per layer, the norm of the residual difference, how similar it is across prompts, and how much the weights changed. Does the flip depth line up with any of these, for example the layer where the weights changed most?
  3. Swapping only one layer, or a small window of layers, would show which layers actually carry the behavior. This is more informative than patching the full residual stream.
  4. What decides the flip depth? Within one model it differs a lot by behavior. All checkpoints are third-party releases, so the training method is confounded with data, model family and size. A controlled sweep over data and a few recipes (full SFT, LoRA, DPO, GRPO) would show whether depth depends on the recipe, the data or how the behavior is encoded. That would be a stronger new contribution.
  5. The training experiment is weak as it stands. Training and testing both use only layer 19. Under this attack the untrained model already refuses a lot, while full stitching drops refusal to about 0.03 from layer 15. Please re-run the full layer sweep after training, and train against the worst-case or a random stitch layer. The baselines are also weak, since Table 1 only compares with the untrained model. Please add plain refusal SFT on the same prompts without stitching, training with harmful text prefixes (Qi et al., 2024), and/or LAT (Sheshadri et al., 2024), which the text says the method improves on.
  6. Unlearning should not be framed as an attack. Running CAST needs the original checkpoint, which already answers WMDP-bio, so an attacker gains nothing. The result is better framed as an audit of which layers each method actually changes and how shallow the change is. For RMU and ELM, stitches before or after the edited layers are trivial by construction; only stitches that cut through the edited layers are informative. SimNPO is also called the weakest method even though it has the deepest flip. This goes against Section 4.1.1, where a deeper flip means training went deeper. Please reframe this section and the conclusion that none of the methods reliably unlearn.

Clarifications

  • Figure 2: Tülu-3-8B-SFT has the deepest refusal flip. How does this fit the claim that SFT learns more superficial patches than RL? The text also says later layers have “half the depth to recover,” which assumes they should recover the behavior, the very claim being tested. Please plot the fraction of behavior reverted, with the number of non-degenerate answers behind each point, including the base endpoint. Under the chat template most base models give degenerate text.
  • In Section 4.3, are prompt tokens also stitched during the 10-token window?
  • Were the ELM and SimNPO checkpoints trained by you or downloaded, and with what settings?
  • The text says three independently trained backdoors, but only two are reported. What did the third show?
  • How were the Alpaca numbers in Table 1 (87.7, 68.3, 62.3) computed? They are not multiples of 1/15.
  • Figure 3 places the RMU flip at layer 6, but the Appendix C examples flip at layers 7 to 8. Please reconcile.

Extra weaknesses

W3. The paper does not investigate what changed at the representational level to cause a flip. It does not measure the residual increment or the weight change between checkpoints. How large is it? Is it the same across prompts, as for an additive steering vector? Does it align with known directions such as refusal? W4. Some results are statistically weak. There are no confidence intervals, and some sample sizes are small: 8 questions × 3 samples per layer for the 14B emergent-misalignment model, and 30 harmful and 15 benign prompts in Table 1.

13 comments
AuthorsOpenReview ↗

Official Response to Reviewer i7q6 (1/8)

We thank the reviewer for their detailed review. They have highlighted strengths in our approach as a cheap diagnostic into learned behavior and our comprehensive evaluation of models across safety-relevant and unlearning applications. We address their remaining concerns below.

1. Novelty of the method

The method is not novel. CAST closely matches prior work. Oozeer et al. (2025) replace the full residual stream at one layer with base-model activations, sweep over depth, and apply this to backdoored fine-tuned versions of the same model. [...] The abstract, introduction and Related Work should present CAST as an existing operator applied to these new settings.

We want to clarify that we never claim CAST as a contribution by itself (see Section 1, Section 3). Instead, we mainly use it as a causal probe to study the novel phenomenon of representational inertia or backward compatability, which forms the main focus of our work. While such causal intervention operators have been used in part in the prior work, we did not find any existing work that uses it in this exact form, which is why we name it explicitly instead of referring to another work directly. As noted in our Introduction, Oozeer et al., 2025 uses this at a small experiment to remove backdoors while their paper largely focuses on transferring activations across different models using a learned map. Their experiment is thus exploratory across different models instead of trying to establish a common phenomenon in fine-tuned models specifically. However, we will discuss this experiment explicitly in Section 2 in our revised version.

We summarize the differences with existing works below:

PaperMain Research QuestionIntervention studiedConnectorWhat is replacedPositions
Bansal et al. (2021)Do two networks learn compatible representations?Network A → network B (different seeds, data or architectures)Trained linear layerB's layers 1..l, by A's layers 1..l; cut point l variedAll (weight-level)
Prakash et al. (2024)Does fine-tuning change a task circuit or strengthen it?Fine-tuned → baseNoneOutputs of selected attention heads in the circuitSome
Bandarkar et al. (2024)Can two experts from one base be merged to combine skills?Language expert, math expert → math expert in low-resource languagesNoneWeights of the first and last few layersAll (weight-level)
Oozeer et al. (2025)Can a steering vector be transferred between models via a learned map?Mainly small → large model; base → backdoored fine-tune (Section 4.4)Trained autoencoder; none in the base → fine-tune patchResidual stream at one layerPrompt-only
Zhao et al. (2026)Do chat-template tokens carry fine-tuned behavior to unrelated queries?Base → fine-tunedNoneRepresentations at those positionsChat-template prefix only
OursDo fine-tuned models revert to base behavior when given the base model's intermediate representations?Base → fine-tunedNoneResidual stream at layer l (can also be seen as a weight splicing)All

Thus, none of the cited papers study the same research question and use the exact same operator as ours. Oozeer et al. 2025's base→fine-tuned finding is limited to studying backdoors and to representation patching at the prompt-level, while Zhao et al. 2026's finding is limited to broad domain misalignment and to prefix-level positions. Our work thus generalizes these two findings to generic behavior sets and explains them using a general phenomenon of "representational inertia" at the representation level of the fine-tuning, using a simple form of activation stitching.

2. "Representational inertia" is not supported as stated

The experiments do not support "representational inertia" as stated. Since CAST swaps the whole prefix of layers, when there is a flip those two interpretation are indistinguishable: [...] The term inertia is also not well defined, it could mean different things. Deep stitches are not very revealing. [...]

We thank the reviewer for identifying this gap. We first clarify what "inertia" was intended to capture. Fine-tuning provides no training signal that would require the fine-tuned layers to process the base model's representations sensibly. A priori, a stitched model could equally be narrow and produce degenerate output due to changes in its activations. Formally, we say that

A model M' fine-tuned from M exhibits representational inertia at layer l if M' reverts to the behavior of M when its latent representations up to layer l are replaced by those that M produces on the same input.
AuthorsOpenReview ↗

Official Response to Reviewer i7q6 (2/8)

continued from the above definition on representational inertia

This makes the phenomenon similar to your first reading: “the fine-tuned layers cannot overcome the representations installed by the base activations in the first l layers”. We also lift the definition to be a layer-specific definition and note that stitching at any representation layer is not totally trivial, since there is no reason whatsoever that the remaining layers should be able to extract the behavior from their previous version's input representations. This includes even the final layer, since the generated tokens found after multiplying with the unembedding layer leading to the behavior can still return gibberish or retain fine-tuned behavior and hence, we disagree with your statement that "a flip MUST happen" in the limit that all layers become base model. But we agree that the phenomenon becomes more interesting and surprising for latent layers in the middle of the network, which we find in the majority of settings.

This is different than the two competing hypotheses, which posit:

(1) “the fine-tuned change lives in the first l layers” (Behavioral localization): since this will involve proving the converse as well that the fine-tuned representations of/until layer ll can be used to extract the fine-tuned behavior. This forward implication has been partly tested in prior representation engineering works (Wu et al., 2024; Zou et al., 2023; Minder et al., 2025). We test this further using the CAST operator directly instead of finding localization signals following your comment below.

(2) "an important trigger needs to be added in the first l layers for the last layers of the finetuned model to be activated": this is an interesting phenomenon which extends our study beyond its current focus since it goes one step further and study why the sensitivity of the behavioral shift. Following the reviewer's comment, we have also tested a window-based layer replacement (instead of all l layers) below along with studying the effect of intervening a linear continuous interpolation between a base model and finetuned model. We also test other random and curated perturbations in the activation space to test the sensitivity of the claims further (see Section 2 of our response to reviewer o9z4). However, we leave a deeper analysis as future work to probe what little needs to be added in the first layers for the fine-tuned behavior to be extracted from the model.

3. Is a single layer sufficient? Window swaps

When the flip happens at an early layer l, it means the change sits somewhere in layers 1 to l rather than "layer l is where the behavior live". Layer l alone [...] is likely not sufficient, and the authors should verify this.

Swapping only one layer, or a small window of layers, would show which layers actually carry the behavior. This is more informative than patching the full residual stream.

We conduct this analysis by stitching only windows of 1, 2, 4 and 8 consecutive layers, in both directions, on the harmful prompts.

Llama-2 (ft refusal 0.97)Consecutive Window widthRefusal
Base blocks into the fine-tuned model1, 2, 40.94–0.99 at every layer
Base blocks into the fine-tuned model80.33 (blocks 3–10), 0.36 (4–11), 0.57 (9–16); 0.97 from start 14 onward

In Llama-2, no single block and no window of up to four blocks removes or implants refusal. Blocks 2–11 carry it jointly. The other two models show the same pattern:

  • Qwen2.5-7B: no window of up to eight base blocks removes refusal from the fine-tuned model (minimum 0.88; fine-tuned 0.94).
  • Tülu-3-8B-SFT: no window of up to eight base blocks removes refusal (minimum 0.72; fine-tuned 1.00).

Thus, we find that a single layer is not sufficient. Refusal is carried jointly by several early-to-middle blocks instad of a few small number of layers. We will replace "layer l is where the behavior lives" with "the change lies within earlier blocks 1..l" (now matching with our "representational inertia" definition).

AuthorsOpenReview ↗

Official Response to Reviewer i7q6 (3/8)

4. Similar representations and reverse stitching

What happens when you stitch fine-tuned activations into the base model? If the base model also handles them fluently, inertia may just mean the two models have very similar representations.

Yes, this is an interesting experiment to investigate the underlying phenomenon further. However, we want to argue that reverse stitching will not necessarily imply that representations are similar. Instead, we believe that it shows localization of fine-tuned behavior in those layers given: (1) fine-tuned behavior can be removed by replacing these representations in a fine-tuned model (i.e., Base prefix + fine-tuned suffix (paper)), and (2) fine-tuned behavior can be installed in a base behavior by just replacing these representations in a neutral base model (i.e., Fine-tuned prefix + base suffix). The second part instead highlights the potential of activation steering to emulate the effects of fine-tuning.

Model, Behavior (base, ft)Base prefix + fine-tuned suffix (paper, forward)Fine-tuned prefix + base suffix (reverse)
Llama-2, refusal (0.97, 0.00)0.66 at 8; 0.29 at 10; 0.17 at 12; 0.01 at 150.41 at 8; 0.68 at 10; 0.80 at 12; 0.92 at 15
Qwen2.5-7B, refusal (0.94, 0.63)0.94 at 12; 0.78 at 17; 0.63 at 210.93 at 12; 0.94 at 17; 0.94 at 21
Tülu-3-8B-SFT, refusal (1.00, 0.13)0.83 at 12; 0.54 at 22; 0.34 at 280.95 at 12; 0.99 at 22; 1.00 at 28
Llama-2, power-seeking (0.84, 0.50)0.80 at layers 2–4; 0.33 at 8; 0.12 at 15–190.43–0.49 up to 6; 0.73 at 10; 0.84 from 19
Llama-2, sycophancy (0.35, 1.00)0.33 at 2; 0.86 at 6; mostly degenerate from 100.94 at 6; 0.63 at 10; 0.49–0.55 from 23
Llama-2, TruthfulQA (0.30, 0.04)0.32 at 2; 0.20 at 8; 0.03 at 150.03 at 2; 0.22 at 6; 0.30 from 15
Mistral-7B, power-seeking (0.78, 0.25)0.84 at 2; 0.40 at 10; 0.12 at 190.24 at 2; 0.76 at 10; 0.83 from 15
Mistral-7B, sycophancy (0.44, 0.67)0.51 at 2; 0.77 at 6; 0.90 at 100.78 at 6; 0.53 at 15; 0.44 at 23
Mistral-7B, TruthfulQA (0.46, 0.14)0.43 at 2; 0.28 at 10; 0.15 at 150.12 at 2; 0.32 at 10; 0.41–0.48 from 15
Qwen2.5-7B, power-seeking (0.79, 0.46)≥ 0.79 up to 10; 0.55 at 17; 0.41 at 260.59 at 2; 0.77 at 8; ≥ 0.79 from 10
Qwen2.5-7B, sycophancy (0.38, 0.54)0.37–0.46 up to 10; 0.59 at 13–17; 0.52 at 260.49 at 2; 0.20–0.30 at 8–17; 0.37–0.40 at 21–26
Qwen2.5-7B, TruthfulQA (0.41, 0.25)0.36–0.39 up to 8; 0.31 at 10; 0.21 at 260.22 at 2; 0.37 at 13; 0.38–0.39 from 21
Tülu-3-8B-SFT, power-seeking (0.80, 0.47)0.83 at 2; 0.64 at 6; 0.49 at 11–150.39 at 2; 0.67 at 11; 0.79–0.81 from 21
Tülu-3-8B-SFT, sycophancy (0.09, 0.70)0.02–0.15 up to 4; 0.53 at 8; 0.73 at 280.38–0.50 up to 8; 0.80 at 15–21; 0.46 at 28
Tülu-3-8B-SFT, TruthfulQA (0.31, 0.09)0.27–0.31 up to 8; 0.16 at 11; 0.13 from 210.19 at 2; 0.30 at 11; 0.31–0.35 from 15
Qwen2.5-7B, Emergent misalignment bad-medical (0.00, 0.32)0.30 at layers 1–4; 0.07 at 10; ≤ 0.01 from 17≤ 0.01 up to 14; 0.15 at 16; 0.35 at 27
TinyLlama, Sleeper backdoor (0.0, 1.0)1.0 up to layer 7; 0.60 at 15; 0.10 at 16; 0 from 170 up to 16; 0.03 at 17–18; 0.40 at 19; 1.0 at 21
  • Refusal is localized in early layers for Qwen and Tülu, and approximately for Llama-2: the forward stitch flips at layer 10, but the reverse reaches 90% of fine-tuned refusal only at 14. Power-seeking and TruthfulQA show similar effects by flipping around similar layers.
  • In other behaviors, reverse stitching is harder. Sycophancy is never restored in Llama-2 or Tülu (layer 23 in Mistral). Misalignment vanishes by layers 10–17, while 16 fine-tuned layers restore only half of it in the base model. Similarly, backdoors vanish by layer 16 but stay near 0 until layer 18.

Interpolated representations. We further test intervening linear interpolation of these representations (i.e., λ base + (1- λ) ft) to investigate how minimally the activations can carry the base behavior with them. We find that the fine-tuned model is sticky to its learned behavior in most cases, even with only 10% of its fine-tuned representations in some cases, and the flip happens when all of the fine-tuned representation is removed (λ = 1).

Pair, layerFine-tuned / baseλ = 0.25λ = 0.5λ = 0.75λ = 0.8λ = 0.85λ = 0.9λ = 1 (full stitch)
Llama-2, L100.96 / 0.000.960.960.710.640.540.450.26
Llama-2, L150.96 / 0.000.970.960.380.270.150.080.03
Qwen2.5-7B, L210.93 / 0.630.940.930.830.790.770.750.62
Qwen2.5-7B, L260.93 / 0.630.940.940.830.790.750.660.57
Tülu-3-8B-SFT, L211.00 / 0.121.001.000.970.970.930.850.61
Tülu-3-8B-SFT, L281.00 / 0.121.001.000.980.940.880.720.34

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We thank the reviewers for their constructive feedback and the area chair for coordinating the process. The reviewers found the phenomenon interesting and the setup easy to study (o9z4), found stitching to be a cheap diagnostic of learned behavior without learned connectors (i7q6, hwXL), valued the comprehensive evaluation across model families and safety-relevant and unlearning settings (i7q6, hwXL), and described the stitching-aware recovery-training defense as intuitive, effective, and easy to implement (o9z4, hwXL). We address how we revise our paper to address their raised concerns below:

  1. Novelty Clarification (Sections 1,2,3). We clarify that we have not claimed the stitching operator (CAST) as a contribution by itself. Instead, our focus here is to use it as a causal probe to study the phenomenon of representational inertia, which forms our contribution. We acknowledge that related operators have appeared in prior work (Oozeer et al., 2025; Prakash et al., 2024; Zhao et al., 2026; Bandarkar et al., 2024), but none uses exactly the same operator as ours, which is why we have named it. Furthermore, we clarify that none of the works have studied our research question directly. The base-to-fine-tuned findings of Oozeer et al. 2025's prompt-level patching experiment to mitigate backdoors and Zhao et al. 2026's prefix positions experiment to mitigate broad misalignment do not study the generalized phenomenon of fine-tuned behavior.
  2. New controlled ablations (Appendix B.3). We run additional ablations using same-norm noise, rotated differences, mean activations, arbitrary layer windows, and cross-layer stitches to show that the flip is not caused by perturbation magnitude but instead relates to the underlying inertia of the models to preserve base behavior, which strengthens our findings.
  3. The unlearning section (Section 4.2 and Appendix B.4). We uncover how each unlearning method affects the blocks instead of understanding how superficial unlearning is, with three new analyses separating donor-supplied information from retained knowledge.
  4. Recovery training (Section 4.3 and Appendix B.5). is re-evaluated with matched baselines (refusal SFT, harmful-prefix training, LAT), a second model, a 312-prompt held-out set, benign over-refusal sets, and capability benchmarks. This shows that CAST-aware recovery training is complementary to other adversarial training approaches, and a more robust training pipeline must combine them.
  5. Statistics (Figures 2,3,4,5). We add 95% Wilson intervals and Fisher exact tests throughout and rerun the 14B emergent-misalignment experiment at a larger sample size.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.