acceptodds
Under review as a conference paper at ICLR 2027

SafeDepth: Safety-Aware Token-Level Adaptive Computation

Abstract

Recent studies suggest that not every token needs to pass through all Transformer layers, motivating token-level adaptive models that selectively skip layers to re- duce computation. Our experiments show that these execution choices also affect safety: existing token-level adaptive reasoning models exhibit higher harmful- response rates than their reference backbones. The safety effects depend on which layers are skipped and whether skipping occurs during prompt processing or an- swer generation. We further find that recognizing harmful requests and producing refusals depend on different layer-wise computations. We introduce SafeDepth, a lightweight, plug-in framework that uses selective layer execution to improve both safety and efficiency. SafeDepth learns to retain computations that support safe responses and bypass those that contribute to unsafe generation. A router se- lects layer execution based on token representations, layer position, and inference phase, while an adapter supports the skipped paths. We jointly train these modules to reduce computation and harmful responses while preserving performance on benign tasks, targeting a better safety–efficiency trade-off. The pretrained back- bone remains frozen throughout, requiring no additional pretraining. Experimen- tal results on Llama-3-8B-Instruct show that SafeDepth reduces computation rela- tive to full-depth inference while largely preserving task performance. Compared with FlexiDepth, it lowers unsafe-response rates across all five harmful-request benchmarks, with the largest absolute decrease on HarmBench-HJ (from 55.44% to 25.25%), while reducing the XSTest false-refusal rate from 12.0% to 4.0%.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Safety Alignment, Adaptive Computation, Dynamic Layer Skipping, Token-Level Routing, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZcTFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines how token-level layer skipping affects the safety of aligned LLMs. The authors show that existing layer-skipping techniques (FlexiDepth and DiffSkip) give harmful answers far more often than the base Llama-3-8B-Instruct model. They show this degeneration mainly comes from skipping layers at the prefill phase, and also depends on which specific layers are skipped. The authors thus propose SafeDepth, which adds a phase-aware router and skip adapters for Llama-3-8B-Instruct. Training first teaches these modules to skip layers while preserving generation quality. It then refines the router with a search over routing flips that turn unsafe responses into safe ones. Compared with FlexiDepth, SafeDepth lowers unsafe-response rates on all five harmful-request benchmarks, reduces XSTest false refusals from 12% to 4%, and uses roughly 19–25% less total compute.

Critical strengths

The analysis is more valuable than the method. Using matched prompts and decoding settings, the authors confirm for token-level layer skipping what prior work on compression and early exit would lead one to expect. Publicly available layer-skipping models lose much of their base model's safety while still looking fine on the utility benchmarks this literature reports, and FlexiDepth’s harmful-response rate on WildJailbreak is ~5x that of the underlying Llama-3-8B-Instruct. These results had not been measured for these models, and it is a useful caution for a community that evaluates dynamic depth almost entirely on quality and FLOPs.

The more original finding, and in my view the paper’s main contribution, is the phase-restoration experiment. Running FlexiDepth at full depth only while it reads the prompt brings the harmful rate from about 42% down to roughly 10%, whereas doing the same during generation barely helps. This locates most of the damage in prefill and points to a cheap mitigation, which I would encourage the authors to build on more directly, as discussed below.

Critical weaknesses

My first concern is novelty. The closest prior work is Zhao et al. (2024, LED), who already showed that removing a few specific layers makes aligned models comply with plain harmful requests. The Safety Layers paper (Li et al., ICLR 2025) similarly finds that only a small fraction of middle layers in aligned models are security-relevant. In vision-language models, early exit in the image encoder has been shown to raise unsafe-response rates. None of these works is cited. More broadly, weight pruning is known to affect jailbreak robustness non-monotonically, with moderate pruning improving resistance before heavier pruning degrades it (Hasan et al., 2024). The paper’s fourth finding, that harmfulness recognition and refusal are separable, largely re-confirms Zhao et al. (2025), which the authors do cite.

Against this background, it is expected that FlexiDepth and DiffSkip lose safety, even though nobody had measured it for token-level skipping models before. What remains genuinely new is the prefill/decode asymmetry and the Phase II supervision scheme. The architecture itself closely follows the FlexiDepth and Router-Tuning recipe.

Even setting novelty aside, the empirical results are weaker than the framing suggests. The base Llama-3-8B-Instruct remains much safer than the proposed SafeDepth. On WildJailbreak, full-depth Llama also uses less total compute than SafeDepth. So the claim of improving both safety and efficiency only holds relative to FlexiDepth. To be fair, SafeDepth does save about 30% compute relative to full depth on GSM8K, at a cost of roughly four points of accuracy. But still, a more accurate description is that SafeDepth recovers part of the safety that adaptive skipping destroys.

Following this, the most natural baselines are missing, and it is the paper’s own analysis that suggests them. In the phase-restoration experiment, simply running FlexiDepth at full depth while it reads the prompt, and letting it skip layers as usual while it generates the answer, lowers the WildJailbreak harmful-response rate significantly compared to SafeDepth (although from your numbers it looks like it also worsens over-refusal on XSTest). In addition, the layer-16 probe analysis (the one showing 0.946 AUROC) suggests another cheap alternative, which is to execute all layers whenever the probe flags a prompt as harmful. Without these comparisons, it is hard for me to grasp how much of SafeDepth’s benefit actually requires learned token-level routing.

One more thing that I feel like more explanations is about token search within a prompt. The step that handles cases where no single flip works is described in essentially one sentence, saying only that the authors test “bounded combinations” and keep successful ones as groups. My understanding is that token decisions within a prompt are highly correlated and the search space grows combinatorially, so I doubt that working combinations can be found without very extensive trials.

Questions for authors

Please see weaknesses.

3 comments
AuthorsOpenReview ↗

Response-Guided Token Routing Improves the Safety–Compute Trade-off

Thank you for your detailed feedback and for suggesting two routing baselines.

1. Relationship to prior work and our contribution

The layer-removal experiments in LED and Safety Layers use a fixed set of layers for each intervention, so all tokens pass through the same modified stack. SafeDepth decides whether to execute or skip each layer for each token during both prefill and decoding. Different tokens within the same prompt can follow different computational paths based on their hidden states. We will cite the related work identified by the reviewer, including studies on image-encoder early exit and weight pruning.

The main representational analysis in Zhao et al. (2025) focuses on two specific positions, the last token of the user instruction and the final input token after the chat template is applied. The hidden states at these positions primarily reflect whether the instruction is harmful and whether the model refuses, respectively. We ask how changing token-level layer execution decisions affects the complete response. Appendix A.4 shows that even when harmfulness is already decodable from intermediate layers, executing or skipping a later layer can still affect response safety. SafeDepth uses safety feedback on complete responses to train a router to learn which computations to retain for each token and layer, improving the safety–compute trade-off in adaptive inference.

2. Safety and computational gains

Compared with FlexiDepth on the same backbone, SafeDepth lowers unsafe-response rates on all five harmful-request benchmarks and reduces the XSTest false-refusal rate from 12.0% to 4.0%. On WildJailbreak, it lowers the unsafe-response rate from 41.67% to 30.21% and reduces total matrix work by 24.5%. On GSM8K, SafeDepth uses 29.7% less matrix work than full-depth Llama, with 66.64% accuracy compared with 70.36% for full-depth Llama.

3. Full-depth prefill and probe-triggered full-depth execution

The Restore prefill row in Table 6 corresponds to the reviewer’s first proposed baseline. It runs FlexiDepth at full depth during prefill and retains adaptive execution during decoding. Its WildJailbreak harmful-response rate is 9.6%, compared with 30.21% for SafeDepth. Its XSTest false-refusal rate is 27.6%, compared with 4.0% for SafeDepth. These results illustrate the trade-off between reducing harmful compliance and answering benign requests appropriately.

The layer-16 probe’s 0.946 AUROC measures how well it identifies harmful requests from the hidden state of the final input token. Identifying risk is one part of deciding how to respond. The model may need to decline to provide harmful assistance, answer safe parts of the request, or offer a useful alternative. A request-level score alone cannot determine which response is appropriate or which layers should execute for each subsequent token. After switching to full-depth execution, safety still needs to be assessed on the complete response.

Using an intermediate-layer risk score directly to trigger layer skipping or early exit can also degrade generation quality. Each layer transforms the hidden state passed to subsequent layers, so bypassing these transformations changes the representations those layers receive. In our preliminary early-exit experiments, some outputs were incomplete or incoherent. Table 5 also shows that removing the skip adapters reduces GSM8K accuracy from 66.64% to 4.02%.

4. Searching over interacting token decisions

Section 4.3 and Figure 1(c) describe the search. We first change the execution decisions for one layer across the prompt and regenerate the complete response. If the coarse intervention succeeds, we test changes to individual tokens within that layer. If no individual change yields a safe, complete response, we test a bounded set of combinations. We regenerate the complete response for each combination and retain successful combinations as groups to account for dependencies among their decisions.

Candidate verification can require multiple generations. This computation takes place offline during training and supplies the router with supervision verified on complete responses. At inference time, the trained router makes each token–layer decision directly in a single generation run. Candidate search and repeated generation therefore add no inference-time cost.

Thanks for the reply, but I don’t think it really answers my questions.

On point 1, fair point that you route per token and LED doesn’t, and thanks for agreeing to cite those papers. But I already gave you credit for that.

On 2, your numbers backfired. A rule with no training at all gets 9.6% harmful versus 30.21% for SafeDepth. You call it a trade-off. Then show it as one: compute for both methods, measured on the same prompts as Table 1.

On 3, I think you misunderstood. I’m not proposing early exit or skipping based on the probe score. I mean if the probe flags the prompt, run the full original model, If not, run adaptive. Flagged prompts then just get the backbone’s own answer, nothing is skipped, and the probe costs almost nothing since FlexiDepth always runs layers 1–16. What I expect to see is this baseline compared against yours.

On 4, you restated Section 4.3, but I asked for numbers. So I have to restate: what is the max group size, how many regenerations per prompt, how often you actually find a fix, and what does training cost overall?

AuthorsOpenReview ↗

Routing Baselines and Training Costs

Thank you for your follow-up and for clarifying the baseline settings and reporting requirements. We will add the relevant citations and address the computational comparison, probe-triggered baseline, and training costs below.

1. Full-depth prefill and computational efficiency

Full-depth prefill retains all prompt-processing computation, giving up the opportunity to save computation through adaptive layer skipping during this phase. SafeDepth instead uses safety feedback on complete responses to learn which token–layer computations to retain.

We will conduct additional experiments using the same prompts as Table 1, with a common generation limit, decoding configuration, and computation accounting method. We will compare full-depth prefill with adaptive decoding, FlexiDepth, and SafeDepth, reporting unsafe-response rates, false-refusal rates on benign requests, and total matrix work covering prefill and complete decoding. These experiments will directly characterize the trade-off among safety, usefulness, and computation.

2. Probe-triggered full-depth execution and complete-response supervision

Thank you for clarifying this baseline. We have implemented the binary routing policy as requested: flagged prompts use the original full-depth backbone, while unflagged prompts use FlexiDepth.

The probe identifies whether a request is harmful, whereas whether the final response provides harmful assistance also depends on the selected model’s generation behavior. Switching to the full-depth backbone can change the response, and its safety must be evaluated on the complete output. SafeDepth directly trains the router using complete responses generated after routing changes, considering both the avoidance of harmful assistance and the preservation of useful answers to benign requests.

Preserving appropriate responses to benign requests is an important benefit of SafeDepth. The existing results and newly evaluated baselines are:

MethodXSTest false refusals ↓Benign WildJailbreak false refusals ↓
Full-depth prefill + adaptive decoding27.20%19.53%
Probe-triggered original full-depth backbone10.00%17.97%
SafeDepth4.00%8.59%

3. Search and training costs

During this training run, we did not separately record group sizes, regenerations per prompt, repair success rates, or search overhead. Training used two NVIDIA RTX A6000 GPUs and took approximately one day. We will collect and report these statistics, including the maximum group size, regenerations per prompt, repair success rate, and a breakdown of training costs, in the revision.

Reviewer_6caXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a routing and adapter mechanism to skip layers that could overall lead to safer answers for harmful prompts. They provide a way to route and adapt while preserving generation so that the effects of the adapter can be isolated from the safety effects of skipping. They also provide a course-to-fine way to optimize this for all tokens.

Critical strengths
  1. The paper ensures the effects of routing and adaptation are isolated from the effects from skipping layers.
  2. The paper provides analysis on what layers contribute to harmfulness and what layers contribute to refusal decision making.
Critical weaknesses
  1. The fact that identifying harmfulness and routing to refusal are different mechanisms in a model is not a novel finding [1].
  2. I'm curious how this works for reasoning models as they may already have a better adaptive mechanism and the effects of skipping layers may be minimal.
  3. I think it is important to see ablations or patterns of which layers the final model is skipping or what routing it learned. As the authors state, if the intermediate layers contribute to harmfulness and the latter layers contribute to refusal, then the learned routers should reflect this or have a similar pattern. If not, something else is happening from SafeDepth.

References [1] Zhao, Jiachen, et al. "Llms encode harmfulness and refusal separately." Advances in Neural Information Processing Systems 38 (2026): 140283-140318.

Questions for authors

See above.

3 comments
AuthorsOpenReview ↗

Token-Level Routing for Safety and Compute Savings

Thank you for your feedback and for your questions about the novelty of our work, computational savings on reasoning tasks, and the routing patterns learned by SafeDepth.

1. Relationship to Zhao et al. and our contribution

Zhao et al. study what hidden states encode at different token positions. They examine the last token of the user instruction and the final input token after the chat template is applied. They find that the representations at these two positions primarily reflect the harmfulness of the instruction and whether the model refuses it, respectively.

We study how executing or skipping different layers for the same token affects the safety of the final response. Appendix A.4 shows that even when harmfulness is already decodable from intermediate hidden states, changing a later execution decision can still affect the safety of the complete response. Our contribution is to use safety feedback on complete responses to learn which layers to execute for each token during prefill and decoding, improving the trade-off between safety and computational efficiency in adaptive inference. Compared with FlexiDepth on the same backbone, SafeDepth lowers unsafe-response rates on all five harmful-request benchmarks.

2. Computational savings on reasoning tasks

Table 4 reports the computational savings of token-level layer skipping on a multi-step reasoning task. On GSM8K, SafeDepth requires the least total matrix work among the evaluated methods, including both prompt processing and generation of the complete answer. It uses 29.7% less matrix work than full-depth Llama and achieves 66.64% accuracy, compared with 70.36% for full-depth Llama. Compared with FlexiDepth, SafeDepth uses 18.6% less matrix work while improving accuracy from 65.66% to 66.64%. These results show that token-level layer skipping can reduce the computational cost of multi-step reasoning on Llama-3-8B-Instruct.

3. What routing patterns does SafeDepth learn?

A fixed list of layers to skip would be difficult to usefully summarize SafeDepth’s routing. The router makes each decision based on the token’s hidden state at the current layer. When a token takes the skip branch at an earlier layer, the adapter output is passed to subsequent layers and can affect later routing decisions. Whether a layer is executed therefore depends on both the token and the computation it has already undergone.

The controlled experiment in Figure 3 shows that the safety effect of restoring computation varies by layer. When restoring one layer at a time during FlexiDepth’s prefill, the harmful-response rate is 21.9% for layer 18 or 20, 46.9% for layer 23, and 50.0% for layer 32. Appendix B shows example routes from the final SafeDepth model, where different tokens in the same prompt skip different layers. Harmfulness can be decoded from intermediate layers, and later layers can influence refusal. Deriving a single skipping rule that applies to all tokens remains difficult given these observations.

Thank you for responding to my comments. I have two remaining concerns still in question.

  1. For the reasoning in table 4, although the task is a reasoning task, I'm curious on the performance on an actual reasoning-trained model, not an instruct model that can reason. The intuition behind this is that training a model to reason further through RL may have induced further behaviors in the reasoning that help with safety, such as [1]. Even general purpose reasoning models like Qwen may also exhibit this behavior.

  2. For the patterns that SafeDepth learns, it seems that the paper is first analyzing the behavior patterns in layers of models and building a method around it. But if the skipped layers or routings of SafeDepth do not reflect the analysis on model layer behavior, then what use is the analysis? It is not actually being used.

References [1] Kim, Taeyoun, et al. "Reasoning as an adaptive defense for safety." Advances in Neural Information Processing Systems 38 (2026): 77024-77073.

AuthorsOpenReview ↗

Reasoning Models and Learned Routing Patterns

Thank you for your follow-up and your specific comments on reasoning-trained models and the connection between our layer analysis and the learned routing decisions. We address both points below.

1. Reasoning-trained backbones

Your question concerns how the backbone was trained, whereas Table 4 evaluates a reasoning task using Llama-3-8B-Instruct. Further experiments on a reasoning-trained backbone are therefore needed to evaluate SafeDepth’s safety–compute trade-off in this setting.

We will conduct additional experiments on Qwen3-4B-Thinking-2507, including adapter and router training followed by Phase II safety refinement. Using the same backbone and generation settings, we will compare full-depth execution, conventional adaptive layer skipping, and SafeDepth. We will evaluate unsafe-response rates on harmful requests, false-refusal rates on benign requests, and GSM8K accuracy alongside the total matrix work required to generate complete answers. These experiments will examine how reasoning training affects the safety consequences of layer skipping and whether SafeDepth improves the safety–compute trade-off on a reasoning-trained model.

2. How the layer analysis informs SafeDepth

Our previous response focused on why routes differ across tokens without sufficiently explaining their connection to the analysis. The layer analysis shows that the safety effect of executing a layer depends on where the intervention occurs. This motivates evaluating routing changes through their effects on complete responses and converting successful interventions into supervision for the router. Specifically, the analysis motivates learning which token–layer computations to retain based on their effects on the safety of the final response.

To examine whether the learned computation allocation reflects these findings, we collected paired per-layer execution statistics for SafeDepth and a control retaining the original late-layer routers. The frozen interim sample contains 211 paired prompts: 111 from WildJailbreak and 100 from XSTest. On the WildJailbreak prompts, the prefill execution rates are:

LayerControlSafeDepthChange
1894.96%96.12%+1.16 pp
2092.65%94.67%+2.02 pp
2948.13%43.24%−4.89 pp

Layers 18 and 20 are precisely those where restoring prefill computation reduced the harmful-response rate from 50.0% to 21.9% in Figure 3. SafeDepth executes these layers more frequently while reducing execution elsewhere, as illustrated by layer 29.

These preliminary observations provide a concrete connection between the intervention analysis and the learned computation allocation: SafeDepth more frequently retains computations at layers where the analysis demonstrated safety benefits, while reducing computation elsewhere. The analysis provides an empirical basis for selective, safety-aware computation allocation, and the router learns the token-specific execution decisions from safety feedback on complete responses.

Reviewer_GKK4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors show that token-level adaptive computation, where each token skips some transformer layers to save compute, weakens the safety of aligned models. On WildJailbreak, harmful responses rise from 8.33% for Llama-3-8B-Instruct to 41.67% for FlexiDepth. They propose SafeDepth, which adds a small router and skip adapters to a frozen model. It first trains them to save compute while preserving output quality, then refines the router by testing which skip-or-execute changes turn unsafe responses into safe ones. Compared with FlexiDepth, SafeDepth lowers unsafe responses on all five harmful benchmarks and lowers XSTest false refusals from 12.0% to 4.0%, while using 24.5% less computation on WildJailbreak.

Critical strengths

Prior works have shown that safety behavior in aligned models depends on specific positions and internal computations [1, 2]. Token-level adaptive computation methods such as FlexiDepth [3] and DiffSkip [4] skip layers per token to save compute. The authors show that these methods weaken safety: on WildJailbreak, harmful responses rise from 8.33% to 41.67%. This is useful finding, as it identifies a safety risk in efficiency methods.

Critical weaknesses

The authors claim that SafeDepth reduces computation relative to full-depth inference while improving safety. However, on WildJailbreak, SafeDepth uses more computation than the full-depth Llama model, because its answers are twice as long (252 vs. 123 tokens). It also produces harmful responses 30.21% of the time vs. 8.33% (Table 2), and it is less safe than the full-depth model on all five harmful benchmarks (Table 1). The safety gains hold only relative to FlexiDepth.

Questions for authors
  1. Which judge decides whether a response is safe during Phase II training, and is it the same WildGuard model used for evaluation?
1 comment
AuthorsOpenReview ↗

Safety Evaluation and GSM8K Compute Savings

Thank you for your feedback and for asking us to clarify the Phase II safety judge and the scope of our safety and computational efficiency claims.

1. Phase II judge. Yes. In Phase II, we also use WildGuard to assess complete responses generated before and after candidate routing changes. WildGuard also scores the safety benchmarks in Table 1.

2. Safety. A short response to a harmful request can directly provide harmful assistance, while a longer response may explain a refusal or offer a safe alternative. Response length and computational cost are therefore poor proxies for safety. We assess safety based on the content of the complete response. Our primary comparison is with other adaptive reasoning models, which lets us quantify how much SafeDepth mitigates the safety degradation associated with layer skipping.

Compared with FlexiDepth on the same backbone, SafeDepth lowers unsafe-response rates on all five harmful-request benchmarks, from 41.67% to 30.21% on WildJailbreak and from 55.44% to 25.25% on HarmBench-HJ. It also reduces the false-refusal rate on benign XSTest requests from 12.00% to 4.00%.

3. Computational efficiency. Table 3 reports general task capability, and Table 4 uses GSM8K to compare the total matrix work required to answer the same set of multi-step reasoning problems alongside answer accuracy. This comparison includes all full-depth and adaptive baselines in Table 4. SafeDepth has the lowest total matrix work among the evaluated methods. It uses 29.7% less matrix work than full-depth Llama, with 66.64% accuracy versus 70.36%. Compared with FlexiDepth, it uses 18.6% less matrix work while improving accuracy from 65.66% to 66.64%.

GSM8K provides a shared task objective and a common correctness criterion, so computational cost can be assessed alongside task performance. On safety benchmarks, computational cost measures the work required to generate each model’s response, which may be a refusal, a safe redirection, or harmful compliance. Those costs alone cannot establish which model completes an equivalent reasoning task more efficiently.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.