- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes DeclareViT, an early-exit framework for Vision Transformers that aims to decouple early classification from the backbone’s feature-refinement pathway. The paper identifies a potential [CLS] role coupling problem in conventional early-exit ViTs, where the same intermediate [CLS] representation is used both for early prediction and as an input to subsequent backbone layers. The authors argue that this shared role can create conflicting optimization objectives and gradient interference between early-exit and final classification losses. To address this issue, DeclareViT introduces dedicated, learnable exit queries that read intermediate backbone features through one-way cross-attention without modifying the backbone token stream. The model is trained in two stages: first, the backbone and final classifier are trained using the full-depth objective; subsequently, these components are frozen and only the exit-query branches are optimized using a combination of ground-truth cross-entropy and knowledge distillation from the final classifier. The authors evaluate the approach on CIFAR-100, ImageNet-1K, and Food-101 using ViT, DeiT, and Swin backbones. Across these settings, DeclareViT reports improved accuracy–efficiency trade-offs, with gains in both MAC-based and wall-clock speed-up over the evaluated early-exit baselines.
Critical strengths
A> The paper explores the extent of contribution of CLS in both feature representation and classification label identification.
B> The combination of independent queries, one-way cross-attention, and frozen-backbone training provides a particularly strong form of decoupling. Importantly, the method does not merely stop gradients at the exit interface; it also trains the exit branches against a stationary backbone representation.
C> The paper presents explores a variety of backbones and datasets. The ablations presented also helped with understanding the impact of the intermediate components of the proposed architecture.
Critical weaknesses
A> Since there are multiple early exit evaluation substeps for DeclareViT, it might be possible that other SOTA methods (like [1]) take lesser wall clock inference time when using a higher batch size, say 32, instead of 1. From a deployment standpoint, this might affect performance.
B> The overall novelty of the method is limited. DeclareViT combines existing early exit, query/cross attention methods. The core architectural motivation and mechanism substantially overlap with Dynamic Perceiver and Branchy Net.
C>The cost and necessity of the two-stage training procedure are insufficiently justified. There are multiple methods as [1] that can be trained in just one step. This might also affect the scalability of the method - especially if it were used in some downstream task as object detection. [1] Li, Y., Hu, J., Wen, Y., Evangelidis, G., Salahi, K., Wang, Y., ... & Ren, J. (2023, October). Rethinking vision transformers for mobilenet size and speed. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 16843-16854). IEEE.
Questions for authors
The paper is an interesting read. However, I have the following clarification requests: A> Please differentiate your novel contributions beyond BranchyNet and Dynamic Perceiver.
B> Please add some single-stage image classification comparisons to your results. There are multiple opensource models available on HuggingFace.
2 comments
Response to Reviewer BCvX (2/2): Question B and Weakness C
Reply to Question B: Comparison with single-stage models
We thank the reviewer for this suggestion. Following the suggestion, we add two publicly available single-stage models, DeiT-S and EfficientFormerV2-L [1], and report each of them on its own and with DeclareViT attached in Table R8.
Static efficient architectures and early exiting improve efficiency in complementary ways. A static model fixes one computation budget for every input, whereas early exiting adapts the computation of a given model to each input. DeclareViT can therefore be attached to these models as well, without retraining them: in both cases we use the public checkpoint unchanged and train only the exit branches.
DeiT-S uses the same reader as DeiT-B, with exits at blocks {2, 4, 6, 8, 10}. EfficientFormerV2-L is a hierarchical hybrid whose first two stages are purely convolutional, so we use the exit-local reader with five exits in stages 3 and 4, at global blocks {13, 17, 22, 27, 31} of 35. On DeiT-S, DeclareViT gives average speed-ups of 1.44× and 1.94× at a loss of 0.8 and 4.3 points. On EfficientFormerV2-L, the exit-local reader of our Swin variant attaches just as directly and lowers the average cost from 2.55 to 1.76 GMAC at 80.97% accuracy. The loss at the 2.00× target is larger here, which is expected: [1] was searched to remove redundancy, its two convolutional stages offer no usable exit, and at this budget 64% of the samples must leave at the first two exits.
The EfficientFormerV2 adaptation and these results are included in the revised paper (Appendix A.2, page 15; Appendix B.4, page 17; Table 10, page 19).
Table R8. Single-stage models and the same models with DeclareViT on ImageNet-1K using seed 2024. Operating points are selected on the calibration split at the 1.50× and 2.00× targets. GMAC is the average cost per image. The first Acc. (%) and GMAC columns are for the single-stage model on its own; the columns marked @1.50× and @2.00× are with DeclareViT at that target.
| Backbone | #Params | Acc. (%) | GMAC | Acc. (%) @1.50× | GMAC @1.50× | S_MAC @1.50× | Acc. (%) @2.00× | GMAC @2.00× | S_MAC @2.00× |
|---|---|---|---|---|---|---|---|---|---|
| ViT-B | 86.6M | 81.77 | 17.56 | 81.77 | 12.21 | 1.44× | 79.90 | 9.05 | 1.94× |
| DeiT-B | 87.3M | 83.38 | 17.66 | 82.74 | 12.32 | 1.43× | 78.75 | 9.11 | 1.94× |
| Swin-B | 87.8M | 83.59 | 15.43 | 83.35 | 10.88 | 1.42× | 80.57 | 8.13 | 1.90× |
| DeiT-S | 22.4M | 81.21 | 4.62 | 80.43 | 3.21 | 1.44× | 76.89 | 2.38 | 1.94× |
| EfficientFormerV2-L | 26.3M | 83.36 | 2.55 | 80.97 | 1.76 | 1.45× | 73.66 | 1.29 | 1.98× |
Reply to Weakness C: Cost and necessity of two-stage training
We thank the reviewer for pointing this out.
Cost. Stage I is ordinary full-depth training or fine-tuning of the backbone, and it is not needed when a model trained on the target dataset already exists; then only Stage II, which trains the exit branches, is run. [1] itself illustrates this: for Table R8 we used its public checkpoint unchanged and ran only Stage II. On 4×H100 GPUs, Stage II takes approximately 8 hours on ImageNet-1K. For CIFAR-100, Stages I and II each take approximately 30 minutes, while for Food-101, each stage takes approximately 50 minutes. The training costs are included in the revised paper (Appendix A.3, page 15).
Necessity. Table 4(d) demonstrates the advantages of our two-stage training over single-stage alternatives. Compared with joint training, our approach preserves higher full-depth accuracy and achieves a better dynamic accuracy–efficiency trade-off.
Scalability. The two-stage protocol allows exit branches to be added to an existing trained backbone without modifying its parameters or representations. This makes DeclareViT compatible with reusing pretrained or task-specific backbones for downstream applications, including object detection.
[1] Yanyu Li, Ju Hu, Yang Wen, Georgios Evangelidis, Kamyar Salahi, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Rethinking vision transformers for MobileNet size and speed. In IEEE/CVF International Conference on Computer Vision (ICCV), pp. 16843–16854, 2023.
We hope that our response has addressed the reviewer's concerns. We are happy to answer further questions.
Response to Reviewer BCvX (1/2): Weakness A, Weakness B & Question A
Dear reviewer BCvX,
Thank you for your review and for acknowledging the strong form of decoupling and the variety of backbones and datasets in our evaluation. We have addressed your concerns as follows.
Reply to Weakness A: Batched inference
Thank you for raising this. In terms of computation, yes, and at any batch size: EfficientFormerV2-L [1] needs 2.55 GMAC per image, whereas DeclareViT on ViT-B needs about 9.8 GMAC on average (17.56 GMAC for ViT-B at 1.79× S_MAC, the operating point of Table R7). This is a comparison between backbones. Early exiting answers a different question: how much of a given model's cost can be saved without changing the model. The two therefore combine: attached to EfficientFormerV2-L itself, DeclareViT lowers its average cost by a further 1.45× at a loss of 2.4 points (Table R8, Reply to Question B).
Following your suggestion, we have measured batched inference at batch sizes of {4, 8, 16, 32, 64} for all early-exit methods (Table R7), using the setup of Appendix C. For batched inference, we adapted every method with in-batch compaction (samples that exit are removed from the batch) and cross-batch regrouping (the remaining samples are merged with those from other batches).
DeclareViT is the fastest early-exit method at B=32 and B=64 and on par with DMPO at B=16, whereas SDN, BYOT, COSEE and DMPO are faster at B=4, and COSEE and DMPO at B=8. Larger batches therefore favor DeclareViT. This trend comes from the exit sub-steps you point out. The baselines add computation only at their exits, whereas DeclareViT also updates its exit queries after each of the first ten blocks. Each update has a fixed cost per call that does not depend on the batch size. In a small batch, this cost is shared by only a few images; in a larger batch, it is shared by more images. From a deployment standpoint, the sub-steps therefore matter only at small batches, and at the batch size you suggest (B=32), DeclareViT is the fastest early-exit method.
These results and the discussion are included in the revised paper (Appendix C, page 20; Table 12, page 21).
Table R7. Batched inference on ImageNet-1K with ViT-B/16 (identical to Table R4 in our response to reviewer mke6). Each batch-size cell gives the measured speed-up over ViT at the same batch size and, in parentheses, the time per image in ms. Bold marks the best early-exit method in each column.
Reply to Weakness B & Question A: Novelty vs. BranchyNet and Dynamic Perceiver
We thank the reviewer for the positive assessment and constructive comments. We agree that early exiting and query-based cross-attention are established techniques. Our key contribution lies in decoupling exit branches from the backbone in both forward and backward propagation: exit queries only read backbone features without modifying them, while exit losses do not update the pretrained backbone.
BranchyNet is the conventional coupled design that jointly trains exit classifiers and the shared backbone, allowing exit losses to influence deeper representations. Dynamic Perceiver separates a classification branch, but its cross-attention is bidirectional and the model is trained end-to-end with all exit losses, so early classification still shapes the features that deeper layers receive. In contrast, DeclareViT preserves the original backbone representations and full-depth predictions. This difference is measured in our ablations: letting exit tokens interact with the backbone token stream (token injection, Table 4(a)) gives 79.82%/1.46×, against 80.55%/1.79× for the read-only queries of DeclareViT. Furthermore, DeclareViT can be directly attached to publicly available pretrained checkpoints without retraining the backbone. We provide a more detailed comparison with BranchyNet, Dynamic Perceiver, Dyn-Adapter, VQT and SynQT in Table R1 of our response to reviewer mke6 (Weakness 1).