acceptodds
Under review as a conference paper at ICLR 2027

Exits Need Only Classify: Efficient Early Exiting for Vision Transformers

16 views

Abstract

Anytime Early Exit (EE) Vision Transformers (ViT) adapt inference to image difficulty along an accuracy-speedup Pareto frontier. Transformer-based exits are the most accurate option but incur excessive per-exit computation and parameter overheads, so the field has turned to cheaper, weaker exits. We show this overhead is avoidable, based on one key insight: unlike an encoder layer that refines tokens for later layers, an EE classifier has one job—to pool and summarize global class information. Three optimizations follow directly. First, because only the [CLS]-token is relevant for image classification, an exit can compute the [CLS]-token update exactly at a fraction of a full encoder update's cost. Second, image patches that are less relevant to classification can be pruned, trading small accuracy losses for additional speedup. Third, because all exits have this one objective, carefully constructed weight sharing allows for reduced exit parameters with small accuracy loss. We instantiate these in Class-only ANytime image Transformers (CANiT) on a frozen ViT backbone. On ImageNet-1k, CANiT improves accuracy over prior EE work by 0.88–3.35 percentage points at matched speedup, with similar gains across three additional datasets. Ablations isolate the contributions of [CLS]-token-only computation, patch pruning, and weight sharing.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Early Exit Networks, AnyTime Networks, Patch Pruning, Token Pruning, Vision Transformers

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VGL9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents approaches that could further optimize the early exit framework for vision transformers. It argues that for the said task only the classification token would matter, so the relevant parts of the attention computation and update should be restricted to it while the computations related to patches with low influence on it could be pruned to further improve the computing efficiency, and builds CANiT with these principles in mind. It also presents CANiT-P as a more parameter efficient version through low rank adaptation and weight sharing. They claim that their approaches would be Pareto dominant compared to the previous ones in early exiting, primarily due to the reduction in computation overhead while keeping the accuracy level on par. The paper then evaluates the method on several datasets and provides ablation studies to examine the contributions of individual components.

Critical strengths
  1. The research question is real and well motivated, and this paper serves as an innovative exploration of methods that would help greatly cut down compute overheads for ViT classification tasks.
  2. The method is tested on several different datasets and compared against competing methods, providing evaluation beyond a single benchmark, with substantial ablation studies that help interpret the method.
Critical weaknesses
  1. In the Pareto accuracy reporting for matched FLOPs it appears that the optimized accuracy value is obtained via a sweep on different exit thresholds and pruning rates, where the number could be inflated if the final reporting is not done on a separate evaluation set. Furthermore, as many of the direct comparison numbers on accuracy are quoted from other works it might be helpful to have a uniform standard/practice on accuracy reporting.
Questions for authors
  1. Can you give more information on whether the accuracy number was obtained from testing on separate evaluation sets? If not can a separate measurement be done on a split out set and reexamine the Pareto claim?
2 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for their feedback and concerns.

Regarding the evaluation strategy for Pareto frontiers, accuracy numbers were optimized and evaluated on the same set of samples following established practice of prior literature [1-2]. Moreover, the search is coarse, with thresholds in steps of 0.1 and just 9 pruning rates, so overfitting on a sample size of 50K images is limited. However, we agree that providing additional results on a separate evaluation set would enhance the results. We had planned to include those results, plus our own local training of EnViT and LGViT, to produce the Pareto values. Unfortunately, during the response period we lost access to our compute and storage in a security incident. The additional experiments and an explicit statement of the procedure for matching cross paper numbers, will be included in future revisions.

[1] Dong et al. EnViT: enhancing the performance of early-exit vision transformers via exit-aware structured dropout-enabled self-distillation. AAAI ‘26

[2] Wang et al. Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition. NeurIPS ‘21

Reviewer_VGL9OpenReview ↗

respond

Thanks for the detailed responses and clarifications. They have addressed most of my concerns. I keep my rating of Weak acceptance

Reviewer_Kd7rOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CANiT combines early exiting with token pruning for Vision Transformers. Its exit heads update only the CLS token, while their attention scores guide patch pruning for samples that continue. CANiT-P reduces added parameters through weight sharing and per-exit LoRA. Experiments on four classification datasets include component ablations and batched throughput measurements.

Critical strengths
  1. For a single-layer Transformer exit head, patch-token outputs are unnecessary for the CLS prediction. Omitting these updates retains the head's aggregation over all input tokens while reducing exit FLOPs by approximately 84%. Table 5 supports the expected preservation of accuracy. This makes Transformer exit heads more practical without requiring a change to their prediction function.

  2. Reusing attention scores from a head trained for immediate classification is a reasonable way to guide token pruning without adding a separate scoring network. Table 6 isolates this signal with early exiting disabled and shows lower accuracy loss than backbone-attention or random pruning at matched pruning rates. This provides direct evidence for the choice of pruning signal.

Critical weaknesses

Key details of configuration selection and dynamic batched inference remain unclear, and the manuscript provides no code link for verification. I would consider raising my score if the following concerns are addressed.

1. Baseline comparability and attribution of gains

Comparisons against combined baselines are missing. CANiT reduces computation in two ways: confident samples exit and skip subsequent backbone layers, while samples that continue have patch tokens removed using the exit head's attention scores. Its overall gains therefore come from combining early exiting with token pruning. Comparisons against methods using either technique alone demonstrate the complete system's performance, but do not establish an advantage over straightforward combinations of existing techniques.

Although the paper includes ablations that disable pruning or compare pruning signals, two further controls would be useful: retaining early exiting in CANiT while using backbone CLS attention for pruning, and adding simple token pruning to a representative baseline such as EnViT. These comparisons should use consistent settings and account for the cost of scoring and filtering tokens.

Configuration search and data separation are unclear. Appendix B.2 jointly sweeps exit thresholds and pruning rates to select configurations for different compute budgets. It does not specify which dataset is used for selection, whether that dataset is separate from the final evaluation set, or whether thresholds are optimized independently for each exit.

The baseline methods also adjust thresholds, but it is unclear whether their search spaces and selection procedures are comparable to CANiT's. Please explain how the configuration for each reported point is selected and whether the baselines receive comparable optimization.

The contribution of the exit-head design is difficult to isolate. CANiT uses a simplified Transformer layer exit head and freezes the backbone during exit-head training, achieves substantially better results than several existing methods. Table 5 establishes computational savings over a full Transformer exit head, but does not directly establish an advantage over existing efficient attention heads, such as LGViT's design based on spatial pooling.

Most baseline results are taken from EnViT, whose backbone is described as ViT-B/16, whereas CANiT uses fine-tuned DeiT3-B backbones in the non-ImageNet experiments. The authors use similar final classification accuracy to justify this choice, but early-exit performance depends on intermediate representations, the different pretrained weights and training recipes may affect these representations even when final accuracy is similar. Furthermore, the published ImageNet accuracies of the ViT-B at 81.78% and the DeiT-B checkpoint at 81.98% are closer to EnViT's 81.63% than DeiT3-B's 83.8% is, leaving the rationale for selecting DeiT3-B unclear.

The current comparisons therefore do not isolate the contribution of the exit-head design from differences in the backbones or training and threshold-search procedures. Please compare CANiT against existing efficient attention or MLP exit heads using the same backbone checkpoint and consistent training and threshold-calibration procedures, with token pruning disabled.

2. Throughput gains with large batches need further explanation

Table 3 reports throughput speedups relatively close to FLOP speedups on an A40 GPU at batch size 256. However, average per-sample FLOPs do not capture a central limitation of batched early exiting: remaining samples traverse deeper layers in progressively smaller batches. The batch finishes only after its deepest remaining computation completes, and later-stage costs need not decrease proportionally with sample counts. This can substantially reduce throughput gains. Token pruning reduces computation but does not itself resolve the loss of GPU utilization from shrinking batches.

Appendix B.3 describes tensor indexing and torch.compile with dynamic shapes. These can reduce overhead and avoid shape-related recompilation, but do not establish how efficiently smaller batches execute. Per-stage sample counts and execution times would help quantify utilization losses and filtering overhead, clarifying the contribution of early exiting relative to pruning.

The measurement protocol adds uncertainty. Appendix B.3 states that each batch has exit proportions similar to the full dataset, controlling stage workloads that would vary under random batching. Whether this affects throughput remains unclear. Please explain how batches are formed, whether exit decisions are determined in advance, and whether this information or preprocessing simplifies timed execution.

Validation should compare controlled and random batches under identical settings and measure sensitivity to initial batch size. Timing ablations should evaluate early exiting and pruning separately and together, using the same compilation settings for all variants and the backbone baseline.

Questions for authors
  1. Which dataset is used to select thresholds and pruning rates in Appendix B.2, and is it separate from the final evaluation set? Please clarify whether thresholds are optimized independently for each exit, whether a configuration is selected separately for each reported point, and whether the baselines use comparable search procedures.

  2. Do the advantages in accuracy and computational cost remain when CANiT retains early exiting but uses backbone CLS attention for pruning, or when simple token pruning is added to methods such as EnViT?

  3. With the same backbone checkpoint and training recipe, and with token pruning disabled, how does CANiT compare with existing efficient attention exit heads such as those in LGViT?

  4. How are the exit proportions in each batch controlled, and does this procedure use exit information obtained in advance? Please compare random batches with the controlled setting and report the number of remaining samples and execution time at each stage to explain the actual speedups as the effective batch size decreases.

  5. Could the authors release the source code?

3 comments
AuthorsOpenReview ↗

Author Response 1/2

We thank the reviewer for their feedback, concerns, and suggestions. We had planned (and have the necessary code) to collect additional experimental results for this response. These included experiments with a reimplementation of prior baselines (LGViT and EnViT), a better calibration/evaluation data split, and additional throughput experiments.Unfortunately, during the response period we lost access to our compute and storage due to a security incident. Despite this limitation, we answer each concern as fully as we can with existing results (below). The additional experiments will be included in future revisions.

W1. Baseline comparability and attribution of gains.

  • Combined baselines. As discussed in the “Pruning Signal” (section 4.3) of our ablation and Table 6, the CANiT’s [CLS] score is the superior pruning signal and maintains a higher accuracy than the backbone’s [CLS] attention, with the difference widening at higher pruning rates (for example at κ=0.30\kappa = 0.30 on DeiT-S: 75.78% against 73.95% for backbone attention and 71.31% for random). These results speak directly to the proposed control because an MLP-exit , such as EnViT’s, has no exit attention of its own, so “EnViT + token pruning” must use the backbone’s {CLS] attention. In other words, it must use the comparatively weaker signal from Table 6. Figure 4 isolates pruning’s contribution to the frontier (see CANiT with no pruning vs. pruning). Nevertheless, these results focus on pruning alone and elide combined operation of pruning and early exiting. While we are unable to provide a comparison at this moment, we will include both of the proposed controls in future iterations.
  • Configuration search. Regarding search/evaluation split, we followed established practice of prior literature [1-2] , calibrating and reporting results on the validation set without an explicit calibration split. For each pruning rate, thresholds are optimized together on the validation set and we report the combined best performance across surveyed pruning rates. To the best of our knowledge, this process is the standard evaluation protocol of prior early exit methods (LGViT and EnViT), so their published curves are selected comparably. The search grid is coarse, which limits overfitting across 50K images, but we agree that a held-out split would enhance the results. Additionally, we had planned on using 50% validation set as the calibration data and the rest as validation and we will report these results in a future revision.
  • Contributions of exit-head design. Table 1 holds backbone and recipe fixed and varies only the head (MLP-EE, ViT-EE, CANiT), and Table 5 shows CANiT matches ViT-EE’s accuracy at about 16% of its exit FLOPs. LGViT’s head on the same backbone is missing and will be added. Due to unavailability of prior checkpoints and limited reproducibility details, we were unable to reproduce EnViT’s performance in our preliminary exploration of the field and instead opted to utilize their reported numbers, which favor the baseline. On ImageNet-1k the backbones are within 0.17 points (81.80 vs. 81.63); we will add DeiT-B runs for the non-ImageNet datasets.

Q1. See the answer for “Configuration search” above.

Q2. See “Combined Baselines” above.

Q3. See “Contributions of exit-head design” above.

[1] Dong et al. EnViT: enhancing the performance of early-exit vision transformers via exit-aware structured dropout-enabled self-distillation. AAAI ‘26

[2] Wang et al. Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition. NeurIPS ‘21

AuthorsOpenReview ↗

Author Response 2/2

W2. Throughput gains.

  • The reviewer’s reading of Appendix B.3 is correct. In the timed runs, the number of samples removed at each exit was fixed to the validation-set average for that configuration rather than computed online from confidence. We ensure that removal uses data-dependent indexing under torch.compile by marking the batch index as dynamic (via torch._dynamo.mark_dynamic) and verify data-dependent indexing through the absence of recompilation when batch size or fraction change and by the lack CPU–GPU synchronization observed under older PyTorch. Thus, the shrinking batch is executed rather than elided. This measures mean-batch workload but not per-batch variation.

    Based on your feedback, we were planning to replace it with online confidence-based exiting over the full validation set in random batches of 256. That would allow us to report the per-exit sample counts and per-exit execution time, with and without pruning and under identical compilation for all baseline variants. One issue we had was overcoming CPU bottlenecks, but we had worked out how to fix that before we lost our compute access. Because every batch of 256 carries samples to the final layer, the deepest-sample effect that the reviewer describes is present in the current results (in the original submission). However, we commit to presenting these results in the next revision of the manuscript.

Q4. See W2

I’m sorry to hear about the security incident, and I appreciate you taking the time to respond under these circumstances. Since the additional results are not yet available, I’ll maintain my original score.

Reviewer_YYtCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CANiT, an early-exit framework for frozen Vision Transformers. Its exit classifiers update only the [CLS] token, use exit-attention scores for optional patch pruning, and share weights with per-exit LoRA in a parameter-efficient variant. Experiments on four image-classification datasets evaluate accuracy–speedup trade-offs and the contributions of these components.

Critical strengths
  1. The central observation is clear and practically useful. Patch-token updates within an exit can be omitted without changing its [CLS]-based prediction. Table 5 shows that this reduces exit FLOP overhead from 7.3 to 1.2 GFLOPs, with no significant accuracy difference over five runs.

  2. The evaluation includes useful hardware and component-level controls. CANiT shows favorable accuracy–FLOP trade-offs on four datasets, with full Pareto curves on three. On an A40 GPU, a 1.75× FLOP speedup corresponds to a measured 1.49× batched-throughput speedup. Separate ablations examine pruning signals and weight sharing.

Critical weaknesses
  1. The design's applicability and independent advantage over simpler exits remain uncertain. Lossless [CLS]-only computation relies on a dedicated class token, and cheaper pooling exits or reduced-width attention exits are not tested at matched parameters and FLOPs. CANiT-P still adds 12.1% parameters on DeiT-S, while peak inference memory and single-image latency are not reported.

  2. The cross-paper comparisons are not fully controlled. Prior baselines use different backbones: on Flowers-102, the reported backbone accuracies are 97.67% versus 96.65% (Table 2). Food-101 results at 2.99× and 2.20× speedup are not matched operating points, and the matching or interpolation procedure for the reported Pareto-curve gains is unclear. The threshold and pruning-rate search is described, but a calibration split separate from the reported validation evaluation is not specified.

Questions for authors
  1. How does CANiT compare with simpler global-pooling exits, such as mean- or attention-pooling followed by an MLP, and with reduced-width attention exits at matched parameter and FLOP budgets?

  2. Since prior backbone checkpoints are unavailable, can the authors re-implement a strong baseline (e.g., EnViT or LGViT) using their own backbone and training recipe, clarify how gains are matched or interpolated along the Pareto curves, and calibrate thresholds and pruning rates on a held-out split separate from the reported evaluation data?

2 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for their feedback, concerns, and suggestions. We had planned (and have the necessary code) to collect additional experimental results for this response. These included experiments with global pooling exits, reduced-width attention exits, and a reimplementation of prior baselines (LGViT and EnViT). Unfortunately, during the response period we lost access to our compute and storage due to a security incident. Despite this limitation, we answer each concern as fully as possible with existing results (below). The additional experiments will be included in future revisions.

W1. Design Limitations

  • As explicitly stated in section 5, the reliance on the [CLS]-token is a limitation of our approach. An alternative solution for architectures without the dedicated class token would be to globally-pool all tokens and treat the pooled token as the stand in for the [CLS]-token in our architecture. We plan to explore such solutions in future work.
  • Additionally, we planned on comparing our exit head architecture in performance, FLOPs, and memory to alternative exit architectures using the same training recipe and backbone. Unfortunately, we were unable to do so during the response period due to the reason stated above. We commit to including comparisons, single-image latency, and peak memory, in a future revision. Based on our current data, MLP-EE & ViT-EE (Table 1) utilize different exit head architectures but were trained with the same backbone and training recipe. The results show that an MLP exit on the same frozen backbone reaches 17.03% at exit 1 versus 67.26% for the Transformer exit. Mean-pooling and reduced -width attention heads would lie between those two points.

W2. Cross Paper Comparisons

  • Due to unavailability of prior checkpoints and limited reproducibility details, we were unable to reproduce EnViT’s performance in our preliminary exploration of the field and instead opted to utilize their reported numbers. It is important to note that such an evaluation setup favors the baseline. On ImageNet-1k, our DeiT-B backbone is within 0.17 points of EnViT’s and CANiT’s accuracy gains (0.88-3.35 points) are beyond this difference at every matched FLOP speedup. We will add this explicitly to the paper.
  • Regarding the calibration split, we followed established practice of prior literature [1-2] which calibrates and reports results on the validation set without an explicit calibration split. The search grid is coarse (stepping thresholds in .1 increments and using just 9 pruning rates), which limits how much overfitting at a sample size of 50K images, but we agree a held-out split would be more informative. Nevertheless, we had planned to collect additional results using 50% validation set as the calibration data and the rest as validation and will report this in a future revision.

Q1. Please refer to W1.

Q1. Please refer to W2.

[1] Dong et al. EnViT: enhancing the performance of early-exit vision transformers via exit-aware structured dropout-enabled self-distillation. AAAI ‘26

[2] Wang et al. Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition. NeurIPS ‘21

Thank you for the detailed responses and clarifications. The comparison between MLP-EE and ViT-EE provides useful context, but does not replace the requested comparisons with simpler pooling and reduced-width attention exits. The cross-paper comparisons also remain insufficiently controlled, particularly given the backbone differences across datasets. Moreover, using the same validation set for threshold and pruning-rate selection and final evaluation leaves my concern about Pareto-frontier reliability unresolved. The parameter overhead, peak inference memory, and single-image latency of CANiT-P also remain unquantified. I therefore retain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.