- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes AVERT, a framework for budget-aware dynamic layer skipping in LLM inference. It introduces a request-average depth objective that allows different tokens to execute different numbers of Transformer layers under a shared request-level budget. The framework combines budget-conditioned progressive routing, skip compensation adapters, and compact KV caches. Experiments evaluate generation quality, budget calibration, and inference efficiency across multiple LLaMA models.
Critical strengths
- The request-average budget formulation is intuitive and allows more flexible computation allocation across tokens than fixed per-token depth constraints.
- Supporting continuous, request-specific budgets within a single checkpoint is practically appealing for heterogeneous LLM serving workloads.
- The evaluation covers multiple models, benchmarks, budget levels, and efficiency metrics, with useful ablations of the key components.
Critical weaknesses
- Limited methodological novelty. Most components, including token-wise routing, skip compensation, and KV omission, are inherited from existing approaches. The main contribution is the request-average budget objective, which appears to be a relatively straightforward modification of existing budget regularization.
- Lack of strict request-level budget control. The router does not track accumulated computation or remaining budget during inference. The reported calibration errors are computed over groups of requests rather than individual requests, leaving the reliability of per-request budget enforcement unclear.
- Insufficient explanation of improvements over Vanilla. AVERT sometimes outperforms the full-compute Vanilla baseline (e.g., Table 2, Figure 2 and Figure 4), particularly on IFEval. Since the models undergo different training procedures, it is unclear whether these improvements result from dynamic routing or training effects. A same-checkpoint, forced-full-depth ablation is needed.
- Limited evidence of practical serving benefits. The efficiency experiments rely on synchronized PyTorch decoding without online continuous batching or dynamic arrivals. Moreover, prefill remains fully executed, and the reported peak memory savings are relatively modest.
- Insufficient analysis of dynamic routing overhead. Token-level routing introduces irregular execution patterns and requires per-layer indexing, gather/scatter operations, and additional Router and SCA computations. These overheads may substantially offset the theoretical speedup from layer skipping. Although end-to-end TPOT is reported, a fine-grained latency breakdown of routing, SCA, gather/scatter, and dynamic attention is missing.
- Minor Issues
- The paper should generally prefer
\citep{}for parenthetical citations rather than\citet{}, reserving the latter for citations that are grammatically part of a sentence. - The manuscript contains excessive textual descriptions of experimental results, making the presentation dense and reducing overall readability. A more concise presentation would improve clarity.
- Numerical values in Table 2 should include leading zeros (e.g.,
0.278instead of.278) for better readability and consistency.
- The paper should generally prefer
Questions for authors
Please refer to weaknesses.