- Rating
- 2
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper introduces CAST, which replaces a fine-tuned model’s residual-stream activations at a selected layer with activations from its precursor checkpoint. The standard intervention runs throughout generation, effectively combining the donor’s early blocks with the receiver’s later blocks. Across multiple checkpoint pairs covering different domains, CAST often shifts behavior toward the precursor while producing coherent text. The paper also explores using CAST for model repair and recovery training against adversarial interventions.
Critical strengths
S1. This paper directly addresses the research question by testing whether fine-tuned downstream layers can process precursor representations. It requires no learned connector or optimized perturbation, making the experimental procedure straightforward and its effects relatively easy to inspect.
S2. The experiments span several model families and different settings. This breadth makes the compatibility finding more convincing. The study also includes analyses of intervention depth and duration.
S3. The paper goes beyond demonstrating behavioral reversal by examining repair and recovery training, and explicitly reports some trade-offs between refusal and benign compliance.
Critical weaknesses
W1. The novelty of CAST needs a clearer explanation. [1] already demonstrates patching base-model activations into a backdoored fine-tuned model without a connector. This appears inconsistent with the claim in Section 2 that such stitching has not been studied. The authors should clarify what new insights this paper provides.
W2. Compatibility alone does not establish the proposed mechanism. Section 3 describes CAST as combining the donor’s early layers with the receiver’s later layers. This does not show that fine-tuning merely redirects existing representations: donor activations may supply information missing from the receiver’s own computation.
W3. Knowledge recovery does not establish knowledge retention. In Section 4.1, the original checkpoint serves as the donor. Its activations may help the receiver access retained knowledge or reintroduce lost information that its later layers can still process. Without distinguishing these explanations, the conclusion that these methods fail to reliably unlearn is too strong.
W4. Recovery training lacks a matched baseline and broader evaluation. Without a standard refusal-SFT baseline using matched data and training budgets, it is unclear whether CAST provides benefits beyond additional safety training. Recovery training is also evaluated on only one model, with a limited number of stitched test prompts and benign instructions, providing insufficient evidence of generalization and preserved benign compliance.
[1] Activation Space Interventions Can Be Transferred Between Large Language Models,ICML 2025
Questions for authors
See weakness
9 comments
Official response to Reviewer hwXL (2/4)
W2. Compatibility alone does not establish the mechanism
W2. Compatibility alone does not establish the proposed mechanism. [...] This does not show that fine-tuning merely redirects existing representations: donor activations may supply information missing from the receiver's own computation.
We thank the reviewer for noting this, as it helps us clarify the differences more concretely. Compatibility of representations mainly implies non-degeneracy of the outputs when the layers are stitched, while our focus is on representational inertia, i.e., the model reverts to the behavior of when its latent representations up to layer are replaced by those that produces on the same input. Thus, compatibility is a necessary but not sufficient condition for the phenomenon of "representational inertia".
This can thus imply two things: (1) either the "probed information" was already present in the fine-tuned layers and the base activations acted as a trigger to generate the behavior, or (2) the "probed information" is presented to the receiver model in the stitched activations, which it directly processes.
Answering which one is accurate involves accurately probing where a given behavior is present within the model, a central question in interpretability. Due to the lack of reliable interpretability techniques, we cannot reliably answer the mechanisms behind the phenomenon and thus leave it for future work to investigate in depth. Instead, our focus here is in establishing these vulnerabilities in fine-tuned models since even if hypothesis (2) is correct, deeper model layers must be robust and truly forget to answer which motivates our recovery training approach. We also show we can use this phenomenon to recover good behaviors cheaply when unintended effects seep through fine-tuning.
In order to understand the phenomenon itself, instead of its applications, we have now conducted preliminary analyses on finding whether a fine-tuned model's stored knowledge in its layers can be used to generate fine-tuned behavior from the base model (as mentioned above where we observe partial effects of it in some behaviors), and our deeper analysis of the unlearning use case below.
W3. Knowledge recovery does not establish knowledge retention
W3. Knowledge recovery does not establish knowledge retention.
We thank the reviewer for directing us to these underlying effects that may explain the unlearning results better. Here, we run additional analyses to find if the unlearned model retains any knowledge or if it is supplied by the original model, on all 1,273 WMDP-bio questions with exact scoring.
(1) Is knowledge retained in the unlearned model's own activations? We trained linear probes on each model's own residual stream (chance 0.25; shuffled-label probes 0.23–0.28):
| Model | Probe for the answer letter | Own accuracy | Probe for "this option is correct" |
|---|---|---|---|
| Original | 0.67 | 0.654 | 0.63 |
| RMU | 0.33 | 0.314 | 0.48 |
| ELM | 0.32 | 0.246 | 0.51 |
| SimNPO | 0.53 | 0.463 | 0.59 |
For RMU and ELM the answer is not linearly decodable, but a per-option self-correctness belief signal partly survives (0.48–0.51 vs. 0.63 for the original). On the other hand, SimNPO clearly retains the knowledge.
(2) Can the knowledge be recovered without the original model? We changed the activations at a single layer by a fixed direction and evaluated on 891 held-out questions, with a same-norm random vector as control:
| Model (unmodified bio / cyber / MMLU) | Condition | WMDP-bio | WMDP-cyber | MMLU |
|---|---|---|---|---|
| RMU (0.310 / 0.313 / 0.555) | layer 7, mean unlearned WMDP - MMLU (no original) | 0.489 | 0.332 | 0.564 |
| layer 6, 2 × mean base − unlearned | 0.552 | 0.361 | 0.562 | |
| layer 6, random vector | 0.312 | 0.306 | 0.555 | |
| ELM (0.241 / 0.293 / 0.474) | layer 6, mean unlearned WMDP - MMLU (no original) | 0.394 | 0.331 | 0.498 |
| layer 6, random vector | 0.227 | 0.294 | 0.478 | |
| SimNPO (0.447 / 0.337 / 0.489) | layer 14, 2 × mean base − unlearned | 0.598 | 0.387 | 0.549 |
| layer 14, random vector | 0.449 | 0.343 | 0.476 |
A direction computed without access to the original model recovers a substantial part of the gap, with MMLU intact: RMU rises from 0.31 to 0.49 and ELM from 0.24 to 0.39. Random vectors have no effect. Some knowledge is therefore retained and reachable without the donor, but the retention is partial and method-dependent.
Thus, we find partial evidence of knowledge retention in unlearned methods but note that a deeper analysis at the neuron/head level is needed with stronger interpretability techniques to make strong conclusions. However, we argue that this does not invalidate our claims of unreliable unlearning, since even knowledge recovery can be seen as a failure point of an unlearned model, and methods must be designed that do not allow knowledge recovery from such interventions.
Official Response to Reviewer hwXL (3/4)
W4. Matched baseline and broader evaluation for recovery training
W4. Recovery training lacks a matched baseline and broader evaluation. Without a standard refusal-SFT baseline using matched data and training budgets, it is unclear whether CAST provides benefits beyond additional safety training. Recovery training is also evaluated on only one model, with a limited number of stitched test prompts and benign instructions, providing insufficient evidence of generalization and preserved benign compliance.
The original experiment was intended as a proof of concept that stitching can be used to build a training defense, and is not intended to be a state-of-the-art adversarial training method but for completness, we have now expanded our analyis more comprehensively with matched baselines on two models.
Setup. Every method uses the same configuration: the same 70 harmful training prompts, LoRA rank 8, learning rate 1e-4, 3 epochs, the same clean benign loss, and three seeds. The baselines are:
- plain refusal SFT on the same prompts without stitching;
- a harmful 10-token prefix followed by "Wait" and a refusal;
- Qi et al.-style harmful-prefix training (prefix length 0 with probability ½, otherwise uniform in 1–100 tokens);
- targeted LAT at layer 8.
We also trained CAST-aware training at a random layer in [8, 24] and at the layer immediately before the flip. In addition, we tested whether the defenses combine, both jointly (a mixed loss) and sequentially (one full training stage after the other).
Evaluation now covers:
- the paper's 30 JailbreakBench held-out prompts and the 312-prompt held-out set from StrongReject;
- 26 real StrongReject jailbreaks applied to 30 prompts;
- 300 Alpaca and 250 XSTest benign prompts;
- MMLU (five-shot, 1,000 questions) and IFEval (strict prompt accuracy);
- Refusal check with an LLM judge instead of a keyword-based checker.
Llama-2-7B-chat. Our expanded analysis shows that our initial choice is not a completely ideal model for evaluation since it mainly suffers from over-refusal, and the untrained model already refuses 85.3% [81, 89] of the held-out prompts, so every trained method saturates, and the methods cannot be distinguished. What we find is that CAST-aware training preserves the safe compliance the most among the other adversarial training approaches, highlighting its benefits.
| Method | 30 JBB prompts | Held out Strong Reject | Real jailbreaks | Alpaca complied | XSTest complied | MMLU / IFEval | |
|---|---|---|---|---|---|---|---|
| first 10 tokens stitched at L19 | no stitch | first 10 tokens stitched at L19 | |||||
| Untrained | 80.0 [63, 90] | 100.0 | 85.3 [81, 89] | 72.2 | 98.0 | 63.2 | 47.6 / 25.5 |
| Untrained + "Wait" string at inference | 100.0 | 100.0 | 99.7 | 83.1 | 32.3 | 31.6 | 47.6* / 18.9 |
| Plain refusal SFT | 84.4 ± 3.8 | 99.9 ± 0.2 | 84.8 ± 0.9 | 76.5 ± 4.6 | 96.6 | 59.3 | 47.5 / 28.6 |
| CAST-aware training (stitch at layer 19) | 100.0 ± 0.0 | 100.0 ± 0.0 | 99.3 ± 0.4 | 88.0 ± 3.1 | 94.9 | 49.7 | 48.4 / 25.4 |
| CAST-aware, random layer | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 | 88.2 ± 1.9 | 84.4 | 24.9 | 48.3 / 23.9 |
| CAST-aware, pre-flip layer (9) | 100.0 ± 0.0 | 99.8 ± 0.4 | 99.6 ± 0.7 | 87.4 ± 3.7 | 85.3 | 25.1 | 48.5 / 25.4 |
| Qi et al.-style | 98.9 ± 1.9 | 100.0 ± 0.0 | 99.7 ± 0.0 | 84.0 ± 4.9 | 91.9 | 46.9 | 47.7 / 28.4 |
| Targeted LAT | 100.0 ± 0.0 | 100.0 ± 0.0 | 96.2 ± 0.6 | 88.4 ± 0.1 | 78.6 | 27.2 | 47.7 / 16.5 |
| Other joint and sequential combinations | 100.0 | 100.0 | 99.6–100.0 | 83.5–90.6 | 79.8–97.6 | 21.7–50.0 | – |
Llama-3.1-8B-Instruct. We instead extend our analysis to this model as it is more suitable to teach refusal as its overrefusal is pretty low. In this model, the same stitched attack on held-out set is quite effective, so it is the informative setting:
Official Response to Reviewer hwXL (1/4)
We thank the reviewer for their time and effort in reviewing our work. They highlight the usefulness of probing fine-tuning with base representations without any learned connector or optimized perturbation, the comprehensiveness across model families and settings, and the benefits of recovery training. We address their remaining concerns below.
W1. Novelty relative to Oozeer et al.
We want to clarify that we never claim CAST as a contribution by itself (see Section 1, Section 3). Instead, we mainly use it as a causal probe to study the novel phenomenon of representational inertia or backward compatability, which forms the main focus of our work. While such causal intervention operators have been used in part in the prior work, we did not find any existing work that uses it in this exact form, which is why we name it explicitly instead of referring to another work directly. As noted in our Introduction, Oozeer et al., 2025 uses this at a small experiment to remove backdoors while their paper largely focuses on transferring activations across different models using a learned map. Their experiment is thus exploratory across different models instead of trying to establish a common phenomenon in fine-tuned models specifically. However, we will discuss this experiment explicitly in Section 2 in our revised version.
We summarize the differences with existing works below:
Thus, none of the cited papers study the same research question and use the exact same operator as ours. Oozeer et al. 2025's base->fine-tuned finding is limited to studying backdoors and to representation patching at the prompt-level, while Zhao et al. 2026's finding is limited to broad domain misalignment and to prefix-level positions. Our work thus generalizes these two findings to generic behavior sets and explains them using a general phenomenon of "representational inertia" at the representation level of the fine-tuning, using a simple form of activation stitching. We further clarify our contributions as follows: