acceptodds
Under review as a conference paper at ICLR 2027

Where Does Layer-Skip Fragility Live? A Matched-Harness Decomposition of Training-Free Adaptive Layer Skipping

Abstract

Training-free adaptive layer skipping — executing only a subset of transformer layers per token, chosen at decode time by a cheap online signal — is an attractive route to cheaper inference on memory-bound accelerators. In a matched harness on Llama-3.2-1B/3B we ask where skip cost lives and whether a training-free signal can harvest it. Findings are negative for the method, informative for the mechanism. Trailing top-layer early exit is catastrophic (NLL 2.15→5.46 at a 14% layer skip on 3B); interior block skipping at the same budget is several times cheaper. The per-token oracle — skipping exactly the cheapest 25% of positions by a block-skip cost probe — does not upper-bound causal policies (on 3B it is slightly worse than rate-matched random skipping), because per-layer skip costs are sub-additive: summing per-layer costs mis-ranks positions for block skips. Output confidence, the most commonly proposed training-free gate signal, correlates with true skip cost at only r=0.07–0.14 and buys no reproducible advantage over rate-matched random skipping: on 3B it is significantly better than random on one evaluation set (−0.015 nats) and worse on another (+0.003 nats) — a sign flip. A faithful re-implementation of SkipDecode’s actual structure — bypassing lower layers, with a monotonically decreasing entry depth — is far worse than trailing early exit at matched budget (+6.0 vs. +3.3 nats over dense at K=2 on 1B): the depth axis is even more asymmetric than the skip-position axis. Because KV caches are per-layer, a layer skipped at every step can be removed from the decode loop entirely: we measure batch-1 speedups of up to 3.5× (1B) and 1.7× (3B) from true layer omission on A100-40GB — uniform dropping, the least adaptive policy, is the one that converts to wall-clock time.

Primary area:
foundation or frontier models, including LLMs
Keywords:
layer skipping, early exit, inference efficiency, LLM decoding, negative results, KV cache

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_EsLMOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates the sensitivity of pretrained Llama-3.2-1B and 3B to training-free layer skipping, separating the choice of which layers to skip from the choice of which token positions to skip them at. Using a common evaluation harness, it compares prefix bypass, trailing early exit, interior-block skipping, and uniformly spaced layer removal, alongside random selection, confidence and residual-norm gates, and a cost-based oracle heuristic. Evaluation primarily uses teacher-forced NLL on WikiText-2 and Alpaca-format prompts. The authors report that sensitivity varies substantially across depth, whereas the tested token-adaptive policies offer little consistent benefit over random selection. They attribute this behavior to interactions among skipped layers and weak correlations between gate signals and skip costs. Separate decoding experiments measure speedups from static layer omission.

Critical strengths

The problem is relevant to efficient inference, and the layer-wise sensitivity measurements provide some useful observations.

Critical weaknesses
  1. The main conclusion is not adequately supported. The oracle is defined inconsistently across sections and is explicitly acknowledged to be a heuristic rather than an upper bound. Its failure to outperform random selection therefore does not establish that token adaptivity has negligible headroom. Sub-additivity across layers alone does not justify this inference.

  2. The presentation is poorly balanced, and the empirical scope is too limited for the conclusions. The paper devotes substantial space to related work and repeatedly restates its conclusions, while key experimental definitions and table labels remain inconsistent. Evaluation covers only two small models from the same family, three seeds, and teacher-forced NLL rather than downstream generation performance. Although many of these limitations are acknowledged, the broad conclusions about training-free adaptive skipping remain insufficiently qualified.

Questions for authors

What exact intervention and ranking score produced the oracle results? What evidence supports the claimed lack of adaptive headroom independently of this heuristic’s failure?

1 comment

Thank you for the author's response. In my view, this work does not meet the acceptance criteria for ICLR; I stand by my original assessment.

Reviewer_dwhuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies training-free, decode-time layer skipping on Llama-3.2-1B and 3B. Skipping is implemented with forward hooks that replace a layer’s output with its input at flagged positions. Quality is measured as teacher-forced NLL on wikitext-2, with a secondary Alpaca prompt set. The paper is framed as an informative negative result. The paper shows that skipping the lowest layers is catastrophic, trailing exit is also catastrophic, and interior skipping is cheaper.

Critical strengths

The paper presents a meaningful negative result with usefull controlled comparison. the paper motivation is post-hoc.

Critical weaknesses

While the paper presents some negative findings, it fails to provide a novel solution for the problem.

  1. The oracle is inconsistently defined, and its failure does not establish that token-level adaptivity has little potential.
  2. The SkipDecode baseline appears to omit warmup layers and reverse the original computation schedule.
  3. The proposed explanation based on sub-additive costs is not sufficiently validated.
  4. Testing only two previous-token signals does not support broad conclusions about training-free gating.
  5. Evaluation is limited to two small models from one family and a narrow set of datasets.
  6. Inconsistent skip rates, baseline labels, and seed descriptions reduce reproducibility.
Questions for authors
  1. How exactly is the oracle computed?
  2. Why does an ineffective oracle imply limited potential for adaptive token selection?
  3. Does your SkipDecode baseline preserve warmup layers and follow the original scheduling direction?
  4. What direct evidence supports sub-additivity as the cause of the gates’ failure?
  5. Would current-token signals perform better than the previous-token signals tested?
  6. Do the conclusions hold across other model families, larger models, and generation tasks?
Reviewer_JC7aOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates training-free adaptive layer skipping on Llama-3.2-1B and 3B models. It compares different skipped-layer structures and token-selection strategies under the same executed-layer ratio, evaluates teacher-forced NLL, and separately measures actual wall-clock speedups from skipping layers. The main conclusion is that choosing which layers to skip matters a lot, while the tested token-adaptive signals do not show a clear advantage over simple random selection.

The research question is interesting and relevant, and a well-verified negative result would be useful for the community. However, I have a few major concerns:

  1. Inconsistencies in how the oracle is defined, as well as potential issues in reproducing the claimed SkipDecode baseline, make key parts of the main argument less convincing.

  2. The quality and speedup measurements are reported separately, which does not clearly establish a real practical trade-off between speed and performance during actual generation.

Critical strengths

The rate-matched random baseline is a helpful control. It checks if the proposed token-selection signals actually improve quality compared to just skipping fewer layers. This makes the main finding—that confidence and residual signals give little consistent gain—quite informative. However, this result alone does not fully prove that token-adaptive skipping as a whole has no potential.

Critical weaknesses
  1. Weak oracle experiment: Sections 3.2 and 6.2 calculate skip costs differently. More importantly, the failure of one heuristic ranker does not prove that token adaptivity lacks potential—it only shows that the specific tested signals were ineffective.

  2. Inconsistent SkipDecode baseline: Original SkipDecode keeps bottom warmup layers, but this paper skips starting from Layer 0. Skipping the earliest layers causes severe degradation and creates KV cache issues. If the implementation differs from the original, the comparison is invalid.

Questions for authors
  1. Oracle clarification: What exact score was used for the oracle row in Table 2—directly measured joint block-skip costs, summed individual-layer costs, or isolated-position intervention costs? Sections 3.2, 6.2, and 7 describe different procedures. Since this ranker is not a true upper bound, what evidence actually supports the conclusion that token adaptivity has negligible headroom?

  2. SkipDecode implementation clarification: Does your baseline keep the original method's bottom warmup layers? Does the schedule actually decrease the number of bypassed layers over time, and if so, how are historical KV entries supplied when later tokens re-enter layers that were previously skipped? Please clarify the actual implemented behavior rather than potential notation errors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.