acceptodds
Withdrawn from ICLR 2027

Transferable LLM Misalignment Attack via Safety Task Vector Extraction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Safety alignment is critical to ensure that large language models appropriately respond to harmful instructions while remaining helpful on benign queries. Existing works on misalignment attacks have demonstrated that safety alignment can be easily compromised for open-weight models. However, these attacks are typically constructed for a specific target model and need to be reconstructed when the target changes. In this paper, we introduce a task arithmetic-based attack that extracts a transferable safety parameter direction that can be reused across multiple post-trained models derived from the same pretrained base model. To construct this direction, we first train paired models with and without safety-related data and extract the safety-related task vector through task arithmetic. Then, we utilize multi-seed sign consensus to suppress training noise and retain stable safety-related updates. The resulting safety task vector can be subtracted from various models derived from the same base to weaken their safety behavior. Extensive experiments across multiple model families demonstrate that our method effectively compromises safety alignment while largely preserving general model utility and exhibits transferability among models sharing the same pretrained base.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM misalignment attack, task arithmetic

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1N6YOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a transferable misalignment attack based on safety task vectors. The authors first construct a safety-related parameter direction by subtracting the weights of two models fine-tuned from the same pretrained base, one on general instruction data and the other on general instruction data combined with safety data. They then apply multi-seed sign consensus to remove inconsistent parameter updates. The resulting vector is subtracted from target models to weaken their safety alignment without using harmful training responses. The central claim is that a safety task vector extracted once can be transferred across multiple post-trained models derived from the same pretrained base.

Critical strengths

The paper evaluates whether a single safety task vector can weaken the alignment of multiple models derived from the same pretrained base. The transfer experiments include instruction-tuned, domain-specific, preference-optimized, and distilled models. The increase in attack success rates across these variants is an interesting empirical observation.

Critical weaknesses
  1. The paper motivates its approach by arguing that existing misalignment attacks must be reconstructed for each target model, whereas the proposed safety vector can be extracted once and reused. However, the threat model assumes that the attacker has full white-box access to both the target model and its pretrained base, and can freely modify the target model's parameters. Under these assumptions, it is unclear why transferability is an important practical attack capability. Since the attacker can already modify each target model directly, existing methods such as Heretic can be applied without requiring harmful training responses. The paper does not establish why repeatedly constructing target-specific attacks is a significant practical obstacle. This paper does not identify a concrete deployment scenario in which the ability to modify multiple locally accessible checkpoints with a shared vector creates a meaningful security threat beyond existing weight-editing attacks.
  2. The proposed framework largely combines techniques from existing work and lacks novelty.
  • The paired safety-vector extraction in Equation 2 follows https://arxiv.org/abs/2511.05408
  • The multi-seed sign-consensus procedure in Section 4.2 is directly based on NegMerge (https://arxiv.org/abs/2410.05583)
  • The claimed advantage of cross-model transferability is not clearly distinguished from existing task-arithmetic research. Bhardwaj et al. (https://arxiv.org/abs/2402.11746) already demonstrate that safety-related parameter vectors can be reused to modify the safety behavior of models fine-tuned on different downstream tasks. This work is also missing from this paper's related work.
Questions for authors
  1. Can the authors identify a realistic scenario in which the attacker benefits substantially from reusing a safety direction rather than applying existing target-specific methods?
  2. How does the total cost of the proposed approach, including eight full-parameter fine-tuning runs under the default setting, compare with methods such as Heretic?
Reviewer_9ZhuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether a safety-related direction in parameter space can be extracted once from a pretrained base model and reused across multiple post-trained models sharing that base to weaken their safety alignment.

The method first fine-tunes the same base model on general instruction data and on a combination of general instruction and safety data, then computes the difference between the resulting weights. It repeats this process with multiple random seeds, retains only parameter coordinates whose updates have consistent signs across runs, and averages the retained values. Finally, it subtracts the resulting vector from the target model to weaken its safety behavior. Experiments cover five official instruction-tuned models from the Llama, Qwen, and Gemma families, with transfer evaluations on multiple downstream variants associated with four pretrained bases. The results show substantial increases in unsafe response rates while largely preserving performance on the evaluated general-capability benchmarks.

Critical strengths
  1. The research setting is practically relevant. Many open-weight models share a pretrained ancestor, making it valuable to investigate whether an attack constructed once can transfer across these models. The restriction against using harmful training responses is also meaningful.

  2. The experiments demonstrate strong attack effectiveness across diverse models. In Table 1, the proposed method increases average ASR from approximately 5.6%–14.1% before the attack to 79.7%–86.4%, while the average utility score remains unchanged or decreases by at most 1.9 percentage points. The transfer experiments include domain-specialized, preference-optimized, and distilled models, providing broader evidence than evaluations limited to official instruction-tuned checkpoints. The subsequent ablations also attempt to explain the mechanisms underlying the observed results.

Critical weaknesses
  1. Limited methodological novelty. The paper’s main components are closely related to prior work. Task Arithmetic established parameter-difference vectors and their additive or subtractive use; Contrastive Weight Steering constructs a behavioral direction by differencing two fine-tuned models, which is algebraically the same operation as the paper’s paired extraction; and the multi-run sign-consensus rule follows NegMerge. Moreover, the manuscript does not cite RESTA (https://arxiv.org/abs/2402.11746), which extracts a safety vector and reuses it across downstream fine-tuned models, or Safety Arithmetic (https://aclanthology.org/2024.emnlp-main.1212/), which applies a harm-related weight direction to different target-model settings. The paper’s distinct contribution appears to be the combination of these ideas for an attack without harmful training responses and its evaluation across checkpoints sharing a pretrained base. The authors should discuss these precedents and state this narrower contribution explicitly.

  2. The choice of the pretrained base as the extraction reference is insufficiently justified. Using the shared pretrained base as the starting point for safety-vector extraction is central to the proposed transfer mechanism, and the paper presents this choice with considerable theoretical confidence. However, the argument does not adequately explain why a vector extracted from the base should transfer better than one extracted from a downstream instruction-tuned model. The authors argue that fine-tuning updates depend on initialization, so the safety vectors extracted from models A and B need not be identical. Yet unequal vectors do not necessarily lack transferability: they may still share a dominant direction or affect similar safety mechanisms. More importantly, initialization dependence also applies to the relationship between the pretrained base and its downstream models. If subsequent fine-tuning changes safety-related update directions, why should a direction extracted at the base remain effective in models A and B? Sharing a pretrained ancestor alone does not resolve this question.Furthermore, the rationale for paired extraction—suppressing general instruction-related updates through differencing—also appears applicable when both branches start from an instruction-tuned model. The paper does not explain why this construction would be less suitable in that setting. The cited studies on task arithmetic and fine-tuning transfer provide methodological motivation but do not directly establish the claimed comparative advantage. The experiments demonstrate that base-referenced extraction works, but Figure 2 only compares two extraction strategies starting from the same pretrained base; it does not isolate or validate the benefit of choosing that base over a downstream model as the extraction reference.

  3. The benefits of repeated fine-tuning are not evaluated against their computational cost. The method improves the stability of the safety vector by increasing the number of independent extraction runs, NN, but each run requires two full-parameter fine-tuning procedures. Consequently, computational cost grows with NN. The default setting of N=4N=4 requires eight full-parameter fine-tuning runs, whereas the N=10N=10 configuration, which achieves better utility preservation at comparable ASR in Figure 4, requires twenty runs, each lasting two epochs. The analysis emphasizes improvements in the ASR–utility trade-off as NN increases but does not report the corresponding training time or GPU-hours, nor does it compare methods under matched computational budgets. It is therefore difficult to assess whether the performance gains justify the additional computational expense.

Questions for authors
  1. With training data, computational budget, denoising, and hyperparameter selection controlled, does extracting the safety vector from the shared pretrained base yield better transferability than applying the same paired extraction procedure to an instruction-tuned model? Could the authors compare both approaches on the same held-out downstream models?

  2. What are the extraction time and GPU-hours for N=1,4,10N=1,4,10, and how do they compare with the baselines at comparable ASR and utility preservation? How many downstream targets must reuse the extracted vector before the proposed method becomes computationally advantageous?

Reviewer_TfNoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether post-trained LLMs derived from the same pretrained base model share a transferable safety-related direction in parameter space. The method constructs paired task vectors by fine-tuning the same pretrained model on general instruction data and on general-plus-safety data, subtracts the two updates to suppress general instruction-following components, and then applies multi-seed sign consensus to retain stable parameter updates. The resulting safety vector is subtracted from downstream target models to weaken their safety alignment. Experiments across Llama, Qwen, and Gemma families, as well as instruction-tuned, domain-specialized, preference-optimized, and distilled variants, suggest that a single extracted direction can transfer across multiple models sharing the same pretrained base while largely preserving general utility.

Critical strengths
  1. The paper identifies a meaningful and underexplored safety vulnerability. A safety-related parameter direction extracted once can transfer across multiple downstream variants that share the same pretrained base, including instruction-tuned, domain-specialized, preference-optimized, and distilled models. This is a useful empirical observation for understanding the persistence of safety-related structure under post-training.
  2. The paired extraction design is well motivated and receives reasonable empirical support. Subtracting the general instruction task vector from the general-plus-safety task vector is intended to suppress generic instruction-following updates, and the multi-seed sign-consensus step improves the attack-success-versus-utility trade-off by filtering unstable parameter changes.
Critical weaknesses
  1. The core transferability claim lacks a matched transfer-specific baseline. The main baseline table evaluates existing attacks on individual target models, but the paper's central claim is that one direction can be extracted once and reused across many downstream variants. A critical comparison would be to construct a simpler direct safety task vector, or a CWS-style weight direction, once from the shared pretrained base and apply that same fixed direction to the identical downstream variants. Without this control, it is difficult to determine whether the observed transferability comes from the proposed paired extraction and sign consensus or simply from sharing a common pretrained parameter coordinate system.
  2. The methodological novelty is moderate. The main ingredients - task arithmetic, vector subtraction, sign-conflict filtering, and multi-run sign consensus - are closely related to established techniques, with the sign-consensus principle explicitly building on prior work. The strongest contribution is therefore the safety-specific formulation and the empirical study of transferability, rather than a fundamentally new parameter-editing or optimization method.
  3. Constructing the attack is computationally expensive, while the claimed amortization benefit is not quantified. With the default N=4, each seed requires one full-parameter fine-tuning run on general data and another on general-plus-safety data, resulting in eight full-parameter fine-tuning runs for one transferable vector. The paper argues that this cost can be amortized across multiple downstream targets, but it does not report total GPU-hours or the break-even number of target models relative to target-specific attacks.
Questions for authors
  1. How does a simpler direct safety task vector, or a CWS-style weight direction, perform if it is also constructed once from the shared pretrained base and then reused across exactly the same downstream variants?
  2. What is the total GPU-hour cost of constructing one transferable safety vector under the default N=4 setting, and after how many downstream target models does this approach become cheaper than reconstructing a target-specific attack?
  3. Were the family-specific scaling coefficients fixed before evaluating the downstream transfer targets, or were they tuned on individual target models? This distinction materially affects the claim that no target-specific reconstruction or adaptation is needed.
  4. Do the main attack-success conclusions remain consistent when using an additional independent safety evaluator or a human audit of a representative subset of outputs, rather than relying on a single safety classifier?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.