- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper examines how token-level layer skipping affects the safety of aligned LLMs. The authors show that existing layer-skipping techniques (FlexiDepth and DiffSkip) give harmful answers far more often than the base Llama-3-8B-Instruct model. They show this degeneration mainly comes from skipping layers at the prefill phase, and also depends on which specific layers are skipped. The authors thus propose SafeDepth, which adds a phase-aware router and skip adapters for Llama-3-8B-Instruct. Training first teaches these modules to skip layers while preserving generation quality. It then refines the router with a search over routing flips that turn unsafe responses into safe ones. Compared with FlexiDepth, SafeDepth lowers unsafe-response rates on all five harmful-request benchmarks, reduces XSTest false refusals from 12% to 4%, and uses roughly 19–25% less total compute.
Critical strengths
The analysis is more valuable than the method. Using matched prompts and decoding settings, the authors confirm for token-level layer skipping what prior work on compression and early exit would lead one to expect. Publicly available layer-skipping models lose much of their base model's safety while still looking fine on the utility benchmarks this literature reports, and FlexiDepth’s harmful-response rate on WildJailbreak is ~5x that of the underlying Llama-3-8B-Instruct. These results had not been measured for these models, and it is a useful caution for a community that evaluates dynamic depth almost entirely on quality and FLOPs.
The more original finding, and in my view the paper’s main contribution, is the phase-restoration experiment. Running FlexiDepth at full depth only while it reads the prompt brings the harmful rate from about 42% down to roughly 10%, whereas doing the same during generation barely helps. This locates most of the damage in prefill and points to a cheap mitigation, which I would encourage the authors to build on more directly, as discussed below.
Critical weaknesses
My first concern is novelty. The closest prior work is Zhao et al. (2024, LED), who already showed that removing a few specific layers makes aligned models comply with plain harmful requests. The Safety Layers paper (Li et al., ICLR 2025) similarly finds that only a small fraction of middle layers in aligned models are security-relevant. In vision-language models, early exit in the image encoder has been shown to raise unsafe-response rates. None of these works is cited. More broadly, weight pruning is known to affect jailbreak robustness non-monotonically, with moderate pruning improving resistance before heavier pruning degrades it (Hasan et al., 2024). The paper’s fourth finding, that harmfulness recognition and refusal are separable, largely re-confirms Zhao et al. (2025), which the authors do cite.
Against this background, it is expected that FlexiDepth and DiffSkip lose safety, even though nobody had measured it for token-level skipping models before. What remains genuinely new is the prefill/decode asymmetry and the Phase II supervision scheme. The architecture itself closely follows the FlexiDepth and Router-Tuning recipe.
Even setting novelty aside, the empirical results are weaker than the framing suggests. The base Llama-3-8B-Instruct remains much safer than the proposed SafeDepth. On WildJailbreak, full-depth Llama also uses less total compute than SafeDepth. So the claim of improving both safety and efficiency only holds relative to FlexiDepth. To be fair, SafeDepth does save about 30% compute relative to full depth on GSM8K, at a cost of roughly four points of accuracy. But still, a more accurate description is that SafeDepth recovers part of the safety that adaptive skipping destroys.
Following this, the most natural baselines are missing, and it is the paper’s own analysis that suggests them. In the phase-restoration experiment, simply running FlexiDepth at full depth while it reads the prompt, and letting it skip layers as usual while it generates the answer, lowers the WildJailbreak harmful-response rate significantly compared to SafeDepth (although from your numbers it looks like it also worsens over-refusal on XSTest). In addition, the layer-16 probe analysis (the one showing 0.946 AUROC) suggests another cheap alternative, which is to execute all layers whenever the probe flags a prompt as harmful. Without these comparisons, it is hard for me to grasp how much of SafeDepth’s benefit actually requires learned token-level routing.
One more thing that I feel like more explanations is about token search within a prompt. The step that handles cases where no single flip works is described in essentially one sentence, saying only that the authors test “bounded combinations” and keep successful ones as groups. My understanding is that token decisions within a prompt are highly correlated and the search space grows combinatorially, so I doubt that working combinations can be found without very extensive trials.
Questions for authors
Please see weaknesses.
3 comments
Thanks for the reply, but I don’t think it really answers my questions.
On point 1, fair point that you route per token and LED doesn’t, and thanks for agreeing to cite those papers. But I already gave you credit for that.
On 2, your numbers backfired. A rule with no training at all gets 9.6% harmful versus 30.21% for SafeDepth. You call it a trade-off. Then show it as one: compute for both methods, measured on the same prompts as Table 1.
On 3, I think you misunderstood. I’m not proposing early exit or skipping based on the probe score. I mean if the probe flags the prompt, run the full original model, If not, run adaptive. Flagged prompts then just get the backbone’s own answer, nothing is skipped, and the probe costs almost nothing since FlexiDepth always runs layers 1–16. What I expect to see is this baseline compared against yours.
On 4, you restated Section 4.3, but I asked for numbers. So I have to restate: what is the max group size, how many regenerations per prompt, how often you actually find a fix, and what does training cost overall?
Routing Baselines and Training Costs
Thank you for your follow-up and for clarifying the baseline settings and reporting requirements. We will add the relevant citations and address the computational comparison, probe-triggered baseline, and training costs below.
1. Full-depth prefill and computational efficiency
Full-depth prefill retains all prompt-processing computation, giving up the opportunity to save computation through adaptive layer skipping during this phase. SafeDepth instead uses safety feedback on complete responses to learn which token–layer computations to retain.
We will conduct additional experiments using the same prompts as Table 1, with a common generation limit, decoding configuration, and computation accounting method. We will compare full-depth prefill with adaptive decoding, FlexiDepth, and SafeDepth, reporting unsafe-response rates, false-refusal rates on benign requests, and total matrix work covering prefill and complete decoding. These experiments will directly characterize the trade-off among safety, usefulness, and computation.
2. Probe-triggered full-depth execution and complete-response supervision
Thank you for clarifying this baseline. We have implemented the binary routing policy as requested: flagged prompts use the original full-depth backbone, while unflagged prompts use FlexiDepth.
The probe identifies whether a request is harmful, whereas whether the final response provides harmful assistance also depends on the selected model’s generation behavior. Switching to the full-depth backbone can change the response, and its safety must be evaluated on the complete output. SafeDepth directly trains the router using complete responses generated after routing changes, considering both the avoidance of harmful assistance and the preservation of useful answers to benign requests.
Preserving appropriate responses to benign requests is an important benefit of SafeDepth. The existing results and newly evaluated baselines are:
| Method | XSTest false refusals ↓ | Benign WildJailbreak false refusals ↓ |
|---|---|---|
| Full-depth prefill + adaptive decoding | 27.20% | 19.53% |
| Probe-triggered original full-depth backbone | 10.00% | 17.97% |
| SafeDepth | 4.00% | 8.59% |
3. Search and training costs
During this training run, we did not separately record group sizes, regenerations per prompt, repair success rates, or search overhead. Training used two NVIDIA RTX A6000 GPUs and took approximately one day. We will collect and report these statistics, including the maximum group size, regenerations per prompt, repair success rate, and a breakdown of training costs, in the revision.
Response-Guided Token Routing Improves the Safety–Compute Trade-off
Thank you for your detailed feedback and for suggesting two routing baselines.
1. Relationship to prior work and our contribution
The layer-removal experiments in LED and Safety Layers use a fixed set of layers for each intervention, so all tokens pass through the same modified stack. SafeDepth decides whether to execute or skip each layer for each token during both prefill and decoding. Different tokens within the same prompt can follow different computational paths based on their hidden states. We will cite the related work identified by the reviewer, including studies on image-encoder early exit and weight pruning.
The main representational analysis in Zhao et al. (2025) focuses on two specific positions, the last token of the user instruction and the final input token after the chat template is applied. The hidden states at these positions primarily reflect whether the instruction is harmful and whether the model refuses, respectively. We ask how changing token-level layer execution decisions affects the complete response. Appendix A.4 shows that even when harmfulness is already decodable from intermediate layers, executing or skipping a later layer can still affect response safety. SafeDepth uses safety feedback on complete responses to train a router to learn which computations to retain for each token and layer, improving the safety–compute trade-off in adaptive inference.
2. Safety and computational gains
Compared with FlexiDepth on the same backbone, SafeDepth lowers unsafe-response rates on all five harmful-request benchmarks and reduces the XSTest false-refusal rate from 12.0% to 4.0%. On WildJailbreak, it lowers the unsafe-response rate from 41.67% to 30.21% and reduces total matrix work by 24.5%. On GSM8K, SafeDepth uses 29.7% less matrix work than full-depth Llama, with 66.64% accuracy compared with 70.36% for full-depth Llama.
3. Full-depth prefill and probe-triggered full-depth execution
The Restore prefill row in Table 6 corresponds to the reviewer’s first proposed baseline. It runs FlexiDepth at full depth during prefill and retains adaptive execution during decoding. Its WildJailbreak harmful-response rate is 9.6%, compared with 30.21% for SafeDepth. Its XSTest false-refusal rate is 27.6%, compared with 4.0% for SafeDepth. These results illustrate the trade-off between reducing harmful compliance and answering benign requests appropriately.
The layer-16 probe’s 0.946 AUROC measures how well it identifies harmful requests from the hidden state of the final input token. Identifying risk is one part of deciding how to respond. The model may need to decline to provide harmful assistance, answer safe parts of the request, or offer a useful alternative. A request-level score alone cannot determine which response is appropriate or which layers should execute for each subsequent token. After switching to full-depth execution, safety still needs to be assessed on the complete response.
Using an intermediate-layer risk score directly to trigger layer skipping or early exit can also degrade generation quality. Each layer transforms the hidden state passed to subsequent layers, so bypassing these transformations changes the representations those layers receive. In our preliminary early-exit experiments, some outputs were incomplete or incoherent. Table 5 also shows that removing the skip adapters reduces GSM8K accuracy from 66.64% to 4.02%.
4. Searching over interacting token decisions
Section 4.3 and Figure 1(c) describe the search. We first change the execution decisions for one layer across the prompt and regenerate the complete response. If the coarse intervention succeeds, we test changes to individual tokens within that layer. If no individual change yields a safe, complete response, we test a bounded set of combinations. We regenerate the complete response for each combination and retain successful combinations as groups to account for dependencies among their decisions.
Candidate verification can require multiple generations. This computation takes place offline during training and supplies the router with supervision verified on complete responses. At inference time, the trained router makes each token–layer decision directly in a single generation run. Candidate search and repeated generation therefore add no inference-time cost.