acceptodds
Withdrawn from ICLR 2027

AVERT: Average-Depth Variable Execution via Request-Local Token Routing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Layer skipping can reduce large language model inference costs, but a request budget does not specify how to allocate layer execution across generated tokens. We introduce AVERT, which treats a continuous request budget as a target for mean executed depth. At each dynamic layer, a request-local router uses the current hidden state and that request’s budget to decide whether to execute or skip the Transformer block, allowing token depths to vary without coupling decisions across requests. Skipped tokens bypass the whole Transformer block, omit their layer-local key/value (K/V) entries, and use a lightweight residual-repair adapter. A single checkpoint serves requests with different budgets in the same batch. Across three LLaMA backbones, six quality metrics, and 13 requested budgets, AVERT is best or tied for best in 206 of 234 comparisons. Across homogeneous and mixed-budget batches and five off-grid targets, the mean executed-layer fraction for each target differs from the requested budget by at most 1.15 percentage points. When splitting requests by budget leaves batches underfilled, mixing budgets reduces decode time per output token (TPOT) by 4.3–21.2%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Efficient Inference, Dynamic Layer Skipping, Budget-Conditioned Inference, Continuous-Valued Request Budgets, Request-Average Depth Control, Token-Adaptive Computation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8cq8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes AVERT, a framework for budget-aware dynamic layer skipping in LLM inference. It introduces a request-average depth objective that allows different tokens to execute different numbers of Transformer layers under a shared request-level budget. The framework combines budget-conditioned progressive routing, skip compensation adapters, and compact KV caches. Experiments evaluate generation quality, budget calibration, and inference efficiency across multiple LLaMA models.

Critical strengths
  1. The request-average budget formulation is intuitive and allows more flexible computation allocation across tokens than fixed per-token depth constraints.
  2. Supporting continuous, request-specific budgets within a single checkpoint is practically appealing for heterogeneous LLM serving workloads.
  3. The evaluation covers multiple models, benchmarks, budget levels, and efficiency metrics, with useful ablations of the key components.
Critical weaknesses
  1. Limited methodological novelty. Most components, including token-wise routing, skip compensation, and KV omission, are inherited from existing approaches. The main contribution is the request-average budget objective, which appears to be a relatively straightforward modification of existing budget regularization.
  2. Lack of strict request-level budget control. The router does not track accumulated computation or remaining budget during inference. The reported calibration errors are computed over groups of requests rather than individual requests, leaving the reliability of per-request budget enforcement unclear.
  3. Insufficient explanation of improvements over Vanilla. AVERT sometimes outperforms the full-compute Vanilla baseline (e.g., Table 2, Figure 2 and Figure 4), particularly on IFEval. Since the models undergo different training procedures, it is unclear whether these improvements result from dynamic routing or training effects. A same-checkpoint, forced-full-depth ablation is needed.
  4. Limited evidence of practical serving benefits. The efficiency experiments rely on synchronized PyTorch decoding without online continuous batching or dynamic arrivals. Moreover, prefill remains fully executed, and the reported peak memory savings are relatively modest.
  5. Insufficient analysis of dynamic routing overhead. Token-level routing introduces irregular execution patterns and requires per-layer indexing, gather/scatter operations, and additional Router and SCA computations. These overheads may substantially offset the theoretical speedup from layer skipping. Although end-to-end TPOT is reported, a fine-grained latency breakdown of routing, SCA, gather/scatter, and dynamic attention is missing.
  6. Minor Issues
    1. The paper should generally prefer \citep{} for parenthetical citations rather than \citet{}, reserving the latter for citations that are grammatically part of a sentence.
    2. The manuscript contains excessive textual descriptions of experimental results, making the presentation dense and reducing overall readability. A more concise presentation would improve clarity.
    3. Numerical values in Table 2 should include leading zeros (e.g., 0.278 instead of .278) for better readability and consistency.
Questions for authors

Please refer to weaknesses.

Reviewer_4steOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

AVERT lets each request specify how much Transformer computation it can use, then distributes that computation unevenly across generated tokens. Tokens that appear to need more computation may execute more layers, while other tokens execute fewer layers. Only the average number of executed layers within that request is trained to match the requested budget. The authors support their claims using model metrics on text generation tasks. At the level of novelty in the training objective, I find this paper to be quite modest nevertheless.

Critical strengths
  1. The main research problem studied in this paper is genuinely relevant and the method present has potential to be useful (despite its limited novelty).

  2. The experiments and ablations presented in the paper demonstrate the usefulness of their method.

Critical weaknesses
  1. AVERT combines useful components, but the experimental comparison omits the methods closest to its principal claims. L2A (End to End dynamic sparsity by Chen et al, 2026) already provides one-checkpoint continuous budget conditioning with hidden-state-dependent layer gates; FlexiDepth (adaptive layer skipping by Luo et al, 2025) provides token-specific interior-layer routing and the skip adapter adopted by AVERT; and SkipGPT (each token is one of a kind, ICML 2025) already controls average computation across token-level routing decisions. None of these are evaluated. The closest included method, BUDDY, is independently reimplemented without its Taylor prior and with changed training duration, budget sampling, prompt-processing policy, and adaptation settings; it underperforms substantially even when all layers execute. The current experiments therefore do not establish whether AVERT’s gains come from request-average accounting, continuous budget conditioning, compact cache omission, or differences in baseline implementation.

  2. Clear AI slop in the writing which makes it hard to follow a simple and potentially, elegant idea. I don't see any need to obfuscate simple concepts with complicated AI slop terms E.g. The prose repeatedly renames layer use as “executed depth,” “realized activation,” and “all-layer activation,” while naming ordinary batching “budget-fragmentation execution.” Labels such as “request-local physical decode design,” “non-nested selections,” and “progressive routing” make ordinary routing, layer selection, and batching unnecessarily difficult to follow.

Questions for authors
  1. Any specific reason the authors restricted to text generation tasks alone? The improvements presented in the paper can work with text classification tasks for example.

  2. The compute penalty is fixed at 10 without training curves, convergence tests, gradient measurements, or experiments showing how this value affects quality and budget accuracy. The experiments cannot separate damage caused by executing fewer layers from damage caused by adding the compute penalty to the training objective. Adding these diagnostics for your proposed method AVERT and baselines could offer new insights to a reader. Otherwise there is very little to take away from this paper.

  3. Please add comparison with missing baselines like L2A, FlexiDepth, SkipGTP, etc. If you think that these are not comparable baselines, please discuss it in the paper.

  4. Most of the presented results are point estimates. Main tables report point estimates. Appendix R gives standard deviations only for AVERT on one model and three targets, without matched uncertainty for baselines. Please consider adding uncertainty estimates over multiple runs.

  5. I also believe your buddy implement is not exactly replicating the method discussed in their paper e.g. you skipped the taylor layer importance, and their lora settings are very different from AVERT. This could be potentially one of the reason why at the full-compute target, where buddy executes all layers, the reimplementation already performs much worse than the dense reference on several metrics. For LLaMA-3.1-8B, its Prometheus score is 3.29, compared with 3.66 for the dense reference and 3.65 for AVERT. Its GSM8K accuracy is 17.6 percent, compared with 42.8 percent and 42.5 percent.

Reviewer_qqjwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a way to target for execution depth per request. As clearly described in Figure 1 and Table 1, this work, to some degree, leverages existing token-wise layer skipping techniques, with novel combination: per-request budget coupled with request average depth. And main contribution also comes from this premise. The paper thoroughly investigates the quality and performance.

Critical strengths

The main strength is that the proposed method gives control of layer depth per request. This keeps mean depth in control to some degree, which may beneficial in certain scenarios. Also this paper investigates the outcome in various angles with proper depth.

Critical weaknesses

No critical weakness. However, I have a few concerns regarding the comparison w.r.t. existing methods and how to implement this in real-serving scenarios.

Questions for authors
  • Could you elaborate on how you can implement the method in aggregated (mixed-mode) serving, with a serving engine, such as vLLM? No need to fully implement this but just want to know your thoughts. And with mixed-mode serving I assume the improvement will shrink compared to decode only cases?
  • For disaggregated serving, this method will improve decode side, which is good. However, we need to match the throughput of prefill and decode, otherwise either prefill or decode side will be waiting for the other one. The throughput of AVERT-enabled decode can vary, so throughput matching can be very tricky, and that would hurt the performance. Could you elaborate on this point as well?
  • Could you elaborate more on why per-request budget target method is good? There are numerous other methods to control the per-user interactivity, and I would like to understand the benefit of per-request mean-depth budget matching with respect to other methods.
  • In figure 2, I think BUDDY method should have very similar accuracy compared to vanilla at budget=1 (every layer executing)? At least similar to AVERT if it were trained fairly? And if we just imagine the curve starting from the same point from vanilla, similar to AVERT, in some cases BUDDY may perform better.
  • Same figure: some BUDDY graphs are noisy, which also causes concern if the reimplementation was fair. Please comment on why we are seeing noisy result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.