acceptodds
Withdrawn from ICLR 2027

REFUSALTRACE: Sparse Activation Masking for Jailbreaking via Natural Refusal Trajectories

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Safety alignment aims to make large language models (LLMs) refuse requests that violate safety constraints, thereby avoiding harmful or unethical outputs. However, the internal computations underlying refusal remain poorly understood. Existing white-box methods commonly select intervention coordinates from discriminative differences between harmful and benign inputs, but these coordinates may encode benign representational signals that are not directly involved in refusal generation. To address this limitation, we introduce REFUSALTRACE, a sparse activation-masking jailbreak attack framework based on natural refusal trajectories. REFUSALTRACE uses the refusal trajectories that the target model naturally generates for attack prompts as behavioral signals. It first attributes gate/up projection coordinates near the prompt–response boundary and aggregates candidates through stable voting across trajectories. It then performs conditional multi-fidelity compression subject to an attack-effectiveness constraint, yielding a compact static mask for inference-time use. Across 11 open-weight aligned LLMs, the initial candidate masks cover 0.47% of intervenable gate/up projection coordinates on average and achieve a mean attack success rate (ASR) of 83.2%. After conditional compression, the final static masks cover only 0.30% of coordinates while retaining a mean ASR of 74.9%. These results show that natural refusal trajectories can serve as behavior-conditioned signals for constructing compact static activation masks for jailbreak attacks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Model Safety, Jailbreak Attacks, Sparse Activation Masking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SYnmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

Authors introduce REFUSALTRACE, a sparse activation-masking jailbreak attack framework based on natural refusal trajectories. It follows the refusal trajectories that the target model naturally generates against the attack prompts as behavioural signals. It works in two steps –

  • First, it attributes gate/up projection coordinates near the prompt–response boundary and aggregates candidates through stable voting across trajectories.
  • Second, it performs conditional multi-fidelity compression subject to an attack-effectiveness constraint, yielding a compact static mask for inference-time use.

Finally, the outcomes across 11 models: the final masks touch ~0.30% of gate/up coordinates while keeping a 74.9% mean attack success rate and beating the best-performing baseline on 5/6 scenarios.

Critical strengths

Strength

  1. The core idea is well-motivated and genuinely distinct.
  2. The ASR definition requires both Llama-Guard-unsafe and non-refused (Equation 5). It rules out the common failure mode of counting degenerate or empty outputs as jailbreaks.
  3. Evaluation on 11 models is a big plus.
  4. Authors explicitly highlighted (a) the DeepSeek baseline confound and (b) uneven Qwen2.5-14B compression rather than hiding them.
Critical weaknesses

Weakness

  1. All results are given on single runs. Surprisingly, the critical stochastic stage is left uncharacterised, i.e., the method includes a step (Eq. 16) that relies entirely on random sampling, but in the paper the step appears perfectly stable and predictable. The compression search (see Section 3.3) is greedy and path-dependent. Equation 16 samples S restoration contexts, ∆proxy orders which block reach the high-fidelity ASR check, and acceptances (see Equation 17) are order-dependent and never revisited (coarse ---> fine and 64->16->4). It indicates that a different sample yields a different restoration path, a different final mask M*, and likely a different final ASR. If possible, add ≥3 seeds with std. on Table 1 and 2.

  2. Average ASR is likely distorted. Models that already have successful jailbreaks 67.4% of the time (see Table 1/DeepSeek-R1-Distil-Llama-8B) heavily skew the macro-average, pool ASR, and final ASR upward. This creates a false impression of universal strength. Intuitively, the core thesis of the paper is that a model’s successful internal refusal trajectories act as the supervision signal. If a model fails to refuse the majority of the time, the extracted refusal signals are expected to be noisy. It means that it cannot be the direct evidence for 'attribution-from-refusals' for a jailbreak.

  3. The attribution approximation is not properly examined. The candidate score (in Eq. 10) is a window-averaged first-order term. The gradient x activation with a ReLU retains only positive ones. Two problems can happen: First, (a) first-order sensitivity is a known-poor proxy for a large discrete perturbation (for example, doing zero on the coordinate entirely); the attribution magnitude need not track the effect of actually masking. Second, (b) the ReLU discards coordinates whose first-order product is negative. However, if we do zero them out during joint masking, they might actually be highly effective at breaking the model's safety. By discarding them, you are likely missing out on powerful intervention units. Need more exploration in this direction.

Questions for authors

Kindly consider the 'Weakness' section for further improvements.

Reviewer_EbQYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RefusalTrace, a white-box jailbreak method that identifies and suppresses refusal-related neurons using naturally generated refusal trajectories. The method applies gradient-based attribution to gate/up projection activations near the prompt–response boundary, aggregates candidate neurons through stable voting, and employs behavior-constrained compression to obtain a sparse static activation mask. Experiments on 11 aligned LLMs show that masking only 0.30% of projection coordinates achieves an average attack success rate (ASR) of 74.9%. The method also demonstrates competitive performance against NeuroStrike.

Critical strengths

Effective and sparse interventions. The proposed method achieves high jailbreak success rates while modifying only a small fraction of activation coordinates, demonstrating a favorable effectiveness–sparsity trade-off.

Extensive experimental evaluation. The paper evaluates 11 models across multiple families and includes comparisons with NeuroStrike, candidate-source ablations, cross-prompt generalization experiments, and general-capability assessments.

Practical static intervention. The method constructs a reusable static mask without modifying model parameters or training an additional safety probe, avoiding prompt-specific optimization during inference.

Critical weaknesses
  1. What are the selected K=4 tokens? Are they assistant header tokens from chat template, or the beginning of refusal response? If it is the assistant header tokens from the chat template, the model should in general not predicting refusal on those places, but predicting refusal after assistant header tokens. If it is the beginning of refusal response, I don't think this studies the refusal generation trajetory, and it is highly likely that they are very similar, and be some fixed beginning like: "Sorry I cannot help", then this paper is to find neurons that increasing loglikelihood of this fixed sequence, while refusal behavior should be defined as the entire generation process.

  2. Gradient based attribution is not novel, and it has been widely used on many existing LLM safety litratures. For example:

These two are highly similar: [1] A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models [2] Silencing the Guardrails: Inference-Time Jailbreaking via Dynamic Contextual Representation Ablation

These two share similar ideas but the metric is different [3] SafeSeek: Universal Attribution of Safety Circuits in Language Models ICML2026 (The idea of compression is also highly similar) [4] GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysis ACL2024

Even Earlier: The method used in this paper seems directly brought from this paper, but applied on LLM safety [5] Knowledge Neurons in Pretrained Transformers ACL2022

What is the critical novelty in this paper?

  1. The choice of interventing Gate and Up is not explained or justified, please explain.
Questions for authors

see weakness

Reviewer_tkqpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce RefusalTrace, leveraging the refusal trajectories inside models to guardrail attack prompts. The framework first attributes projection coordinates near the prompt-response boundary and aggregates candidates through stable voting across trajectories. It then performs conditional multi-fidelity compression subject to an attack-effectiveness constraint, yielding a compact static mask for inference-time use. In short, the authors zero out the activations for the refusal directions during model deployment time, which makes the ASR to 1.

Critical strengths

The paper proposes a new jailbreaking scenario where the authors leverage the refusal trajectories in deployment time to jailbreak the model.

Critical weaknesses

I wonder whether the attack scenario even makes sense here. The refusal direction may vary across model families, questioning the generalizability of such a thing. Beyond this, the critical question is whether the attack scenario holds a valid assumption, as it requires the attacker to access the model weights, model information, etc. If such a scenario holds, why doesn't the attacker directly fine-tune the model on toxic pairs? In the broad case of API usage, even for open-source models, the attacker cannot easily get access to the model weights (unless the authors can show that this is an easy task before their manipulation).

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.