- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper explores 4 methods to optimize model inference by either skipping layers or exiting early. The paper formulates and defines each method, implements them on 2 Qwen3 model sizes, and discusses the results. Results show either moderate speedups and/or large drops in accuracy.
Critical strengths
- Provided a description and formulation of 4 layer skipping and early exit methods, including methods that are lossless, lossy, training-free, require training, etc.
- implemented search method and/or training method for various layer skipping / early exit methods
- Lines 429 to 431: attempt to make insights into the generated text, and not just metrics
Critical weaknesses
-
Solution implemented on small models only each on a couple of tasks. I understand amount of compute available to a lab may be a challenge, but since this is an analysis of inference solutions with lightweight training, I believe showing results on larger model sizes, differenbt model families, and more tasks, are needed to get published in a top-tier conference rather than a workship.
-
The results were negative ("real speedups (1.08–1.33×) but accuracy well below the periodic-step methods". This could be due to Qwen3 models being more difficult to speedup with layer skipping compared to other models like Llama (like "vSkipper: Translating Dynamic Layer Skipping into LLM Serving Gains" has shown). However, the paper did not attempt to explore why the results were negative. It would have been stronger if it found that models of different sizes or different families are more suitable to to layer skipping.
-
Unfortunately, the paper's writing style requires substantial revision, for example:
-
title is awkwardly long. It could have been shortened to something like "Analysis of layer skipping an early exit approaches", but listing names of a couple of approaches and - for some reason - categorizing the analysis of another couple of approaches as "supplemental analysis" makes no sense to me.
-
Calling the approach an audit is awkward. Running three-seeds is beneficial but mentioning that in the Abstract is awkward.
-
Usually abstracts don't include citations, but this paper has around 7 citations in the Abstract
-
Lines 059 to 062 sounded highly convoluted and Claudish
-
Issues with results:
- Some results seem to indicate a bug in running the experiments (see "Questions for Authors")
- Presenting the same results as both table and graph (e.g., Table 2 and Figure 1) in the main body of the paper is an overkill. The table could go to the Appendix
- Some of the results are not surpising but the manuscript discusses them in detail, e.g., there is no surprise in the drop of accuracy of ConfLayers as it is a lossy approach while vanilla and SWIFT are lossless, but it is dicussed in details in lines 261 to 266
- In some tables it does not add entries to compare with vanilla baselines that have no layer skipping, e.g., Table 3 and Table 4
Questions for authors
- Table 2: Very strange results for SWIFT. SWIFT is supposed to be lossless, so it should have the same accuracy as the baseline (i.e., "Vanilla" column). Instead results show it is more accurate for Qwen 0.5B (30.3% vs. 18.0%) and less accurate for 1.5B (30.7% vs. 41.3%). Why is that the case?
- Lines 309 and 310: please provide citation(s) for this claim: "Every prior cost comparison involving SWIFT in this line of work understated its true inference speed by folding search cost into the reported number."
- Table 4: It doesn't make sense that accuracy of the same layer skipping approach to get 13.7% accuracy on 0.5B model size but almost 0% on 1.5B model. I believe the authors need to do further investigation and debugging.