acceptodds
Under review as a conference paper at ICLR 2027

DeclareViT: Decoupling Early Classification from Feature Refinement with Exit Query for Efficient ViTs

21 views

Abstract

Vision Transformers achieve remarkable recognition performance but incur substantial computational cost during inference, limiting their use in latency- and compute-constrained deployment scenarios. Early-exit inference offers a promising solution by allowing easy samples to terminate at intermediate layers. However, existing approaches typically couple early classification with the shared backbone, requiring intermediate representations to support both early classification and subsequent feature refinement. Under joint training, the corresponding optimization objectives jointly update shared backbone parameters, which can lead to conflicting gradient directions. To address this issue, we introduce DeclareViT, a framework that decouples early classification from feature refinement. Each exit uses a dedicated query to read intermediate features through one-way cross-attention without modifying the backbone token stream. The query branches are optimized for early classification while the backbone and final classifier remain frozen. Across ViT, DeiT, and Swin on CIFAR-100, ImageNet-1K, and Food-101, DeclareViT consistently establishes a state-of-the-art accuracy–efficiency trade-off: It achieves – multiply–accumulate (MAC)-based speed-up with only a – percentage-point accuracy drop. Wall-clock measurements on CPU and GPU further demonstrate practical inference acceleration. Code is available at https://anonymous.4open.science/r/declarevit.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Early Exit, Vision Transformer, Edge AI, Efficient Inference, Adaptive Computation, Intermediate Representation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_BCvXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DeclareViT, an early-exit framework for Vision Transformers that aims to decouple early classification from the backbone’s feature-refinement pathway. The paper identifies a potential [CLS] role coupling problem in conventional early-exit ViTs, where the same intermediate [CLS] representation is used both for early prediction and as an input to subsequent backbone layers. The authors argue that this shared role can create conflicting optimization objectives and gradient interference between early-exit and final classification losses. To address this issue, DeclareViT introduces dedicated, learnable exit queries that read intermediate backbone features through one-way cross-attention without modifying the backbone token stream. The model is trained in two stages: first, the backbone and final classifier are trained using the full-depth objective; subsequently, these components are frozen and only the exit-query branches are optimized using a combination of ground-truth cross-entropy and knowledge distillation from the final classifier. The authors evaluate the approach on CIFAR-100, ImageNet-1K, and Food-101 using ViT, DeiT, and Swin backbones. Across these settings, DeclareViT reports improved accuracy–efficiency trade-offs, with gains in both MAC-based and wall-clock speed-up over the evaluated early-exit baselines.

Critical strengths

A> The paper explores the extent of contribution of CLS in both feature representation and classification label identification.

B> The combination of independent queries, one-way cross-attention, and frozen-backbone training provides a particularly strong form of decoupling. Importantly, the method does not merely stop gradients at the exit interface; it also trains the exit branches against a stationary backbone representation.

C> The paper presents explores a variety of backbones and datasets. The ablations presented also helped with understanding the impact of the intermediate components of the proposed architecture.

Critical weaknesses

A> Since there are multiple early exit evaluation substeps for DeclareViT, it might be possible that other SOTA methods (like [1]) take lesser wall clock inference time when using a higher batch size, say 32, instead of 1. From a deployment standpoint, this might affect performance.

B> The overall novelty of the method is limited. DeclareViT combines existing early exit, query/cross attention methods. The core architectural motivation and mechanism substantially overlap with Dynamic Perceiver and Branchy Net.

C>The cost and necessity of the two-stage training procedure are insufficiently justified. There are multiple methods as [1] that can be trained in just one step. This might also affect the scalability of the method - especially if it were used in some downstream task as object detection. [1] Li, Y., Hu, J., Wen, Y., Evangelidis, G., Salahi, K., Wang, Y., ... & Ren, J. (2023, October). Rethinking vision transformers for mobilenet size and speed. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 16843-16854). IEEE.

Questions for authors

The paper is an interesting read. However, I have the following clarification requests: A> Please differentiate your novel contributions beyond BranchyNet and Dynamic Perceiver.

B> Please add some single-stage image classification comparisons to your results. There are multiple opensource models available on HuggingFace.

2 comments
AuthorsOpenReview ↗

Response to Reviewer BCvX (1/2): Weakness A, Weakness B & Question A

Dear reviewer BCvX,

Thank you for your review and for acknowledging the strong form of decoupling and the variety of backbones and datasets in our evaluation. We have addressed your concerns as follows.

Reply to Weakness A: Batched inference

Thank you for raising this. In terms of computation, yes, and at any batch size: EfficientFormerV2-L [1] needs 2.55 GMAC per image, whereas DeclareViT on ViT-B needs about 9.8 GMAC on average (17.56 GMAC for ViT-B at 1.79× S_MAC, the operating point of Table R7). This is a comparison between backbones. Early exiting answers a different question: how much of a given model's cost can be saved without changing the model. The two therefore combine: attached to EfficientFormerV2-L itself, DeclareViT lowers its average cost by a further 1.45× at a loss of 2.4 points (Table R8, Reply to Question B).

Following your suggestion, we have measured batched inference at batch sizes of {4, 8, 16, 32, 64} for all early-exit methods (Table R7), using the setup of Appendix C. For batched inference, we adapted every method with in-batch compaction (samples that exit are removed from the batch) and cross-batch regrouping (the remaining samples are merged with those from other batches).

DeclareViT is the fastest early-exit method at B=32 and B=64 and on par with DMPO at B=16, whereas SDN, BYOT, COSEE and DMPO are faster at B=4, and COSEE and DMPO at B=8. Larger batches therefore favor DeclareViT. This trend comes from the exit sub-steps you point out. The baselines add computation only at their exits, whereas DeclareViT also updates its exit queries after each of the first ten blocks. Each update has a fixed cost per call that does not depend on the batch size. In a small batch, this cost is shared by only a few images; in a larger batch, it is shared by more images. From a deployment standpoint, the sub-steps therefore matter only at small batches, and at the batch size you suggest (B=32), DeclareViT is the fastest early-exit method.

These results and the discussion are included in the revised paper (Appendix C, page 20; Table 12, page 21).

Table R7. Batched inference on ImageNet-1K with ViT-B/16 (identical to Table R4 in our response to reviewer mke6). Each batch-size cell gives the measured speed-up over ViT at the same batch size and, in parentheses, the time per image in ms. Bold marks the best early-exit method in each column.

MethodAcc. (%)S_MACB=4B=8B=16B=32B=64
ViT-B/1681.771.00×1.00× (7.66)1.00× (6.91)1.00× (6.26)1.00× (5.93)1.00× (5.78)
SDN79.201.63×1.55× (4.95)1.57× (4.40)1.58× (3.95)1.59× (3.73)1.60× (3.62)
BYOT79.531.64×1.55× (4.94)1.57× (4.39)1.59× (3.94)1.59× (3.72)1.60× (3.61)
ViT-EE78.561.36×1.31× (5.85)1.32× (5.21)1.33× (4.69)1.34× (4.44)1.34× (4.30)
LGViT79.211.36×1.18× (6.50)1.17× (5.90)1.16× (5.37)1.16× (5.11)1.16× (4.97)
COSEE79.381.66×1.58× (4.87)1.60× (4.32)1.61× (3.88)1.62× (3.67)1.62× (3.56)
DMPO80.521.68×1.58× (4.87)1.62× (4.26)1.64× (3.81)1.65× (3.60)1.66× (3.48)
DeclareViT (Ours)80.551.79×1.48× (5.17)1.57× (4.40)1.64× (3.82)1.68× (3.53)1.71× (3.38)

Reply to Weakness B & Question A: Novelty vs. BranchyNet and Dynamic Perceiver

We thank the reviewer for the positive assessment and constructive comments. We agree that early exiting and query-based cross-attention are established techniques. Our key contribution lies in decoupling exit branches from the backbone in both forward and backward propagation: exit queries only read backbone features without modifying them, while exit losses do not update the pretrained backbone.

BranchyNet is the conventional coupled design that jointly trains exit classifiers and the shared backbone, allowing exit losses to influence deeper representations. Dynamic Perceiver separates a classification branch, but its cross-attention is bidirectional and the model is trained end-to-end with all exit losses, so early classification still shapes the features that deeper layers receive. In contrast, DeclareViT preserves the original backbone representations and full-depth predictions. This difference is measured in our ablations: letting exit tokens interact with the backbone token stream (token injection, Table 4(a)) gives 79.82%/1.46×, against 80.55%/1.79× for the read-only queries of DeclareViT. Furthermore, DeclareViT can be directly attached to publicly available pretrained checkpoints without retraining the backbone. We provide a more detailed comparison with BranchyNet, Dynamic Perceiver, Dyn-Adapter, VQT and SynQT in Table R1 of our response to reviewer mke6 (Weakness 1).

AuthorsOpenReview ↗

Response to Reviewer BCvX (2/2): Question B and Weakness C

Reply to Question B: Comparison with single-stage models

We thank the reviewer for this suggestion. Following the suggestion, we add two publicly available single-stage models, DeiT-S and EfficientFormerV2-L [1], and report each of them on its own and with DeclareViT attached in Table R8.

Static efficient architectures and early exiting improve efficiency in complementary ways. A static model fixes one computation budget for every input, whereas early exiting adapts the computation of a given model to each input. DeclareViT can therefore be attached to these models as well, without retraining them: in both cases we use the public checkpoint unchanged and train only the exit branches.

DeiT-S uses the same reader as DeiT-B, with exits at blocks {2, 4, 6, 8, 10}. EfficientFormerV2-L is a hierarchical hybrid whose first two stages are purely convolutional, so we use the exit-local reader with five exits in stages 3 and 4, at global blocks {13, 17, 22, 27, 31} of 35. On DeiT-S, DeclareViT gives average speed-ups of 1.44× and 1.94× at a loss of 0.8 and 4.3 points. On EfficientFormerV2-L, the exit-local reader of our Swin variant attaches just as directly and lowers the average cost from 2.55 to 1.76 GMAC at 80.97% accuracy. The loss at the 2.00× target is larger here, which is expected: [1] was searched to remove redundancy, its two convolutional stages offer no usable exit, and at this budget 64% of the samples must leave at the first two exits.

The EfficientFormerV2 adaptation and these results are included in the revised paper (Appendix A.2, page 15; Appendix B.4, page 17; Table 10, page 19).

Table R8. Single-stage models and the same models with DeclareViT on ImageNet-1K using seed 2024. Operating points are selected on the calibration split at the 1.50× and 2.00× targets. GMAC is the average cost per image. The first Acc. (%) and GMAC columns are for the single-stage model on its own; the columns marked @1.50× and @2.00× are with DeclareViT at that target.

Backbone#ParamsAcc. (%)GMACAcc. (%) @1.50×GMAC @1.50×S_MAC @1.50×Acc. (%) @2.00×GMAC @2.00×S_MAC @2.00×
ViT-B86.6M81.7717.5681.7712.211.44×79.909.051.94×
DeiT-B87.3M83.3817.6682.7412.321.43×78.759.111.94×
Swin-B87.8M83.5915.4383.3510.881.42×80.578.131.90×
DeiT-S22.4M81.214.6280.433.211.44×76.892.381.94×
EfficientFormerV2-L26.3M83.362.5580.971.761.45×73.661.291.98×

Reply to Weakness C: Cost and necessity of two-stage training

We thank the reviewer for pointing this out.

Cost. Stage I is ordinary full-depth training or fine-tuning of the backbone, and it is not needed when a model trained on the target dataset already exists; then only Stage II, which trains the exit branches, is run. [1] itself illustrates this: for Table R8 we used its public checkpoint unchanged and ran only Stage II. On 4×H100 GPUs, Stage II takes approximately 8 hours on ImageNet-1K. For CIFAR-100, Stages I and II each take approximately 30 minutes, while for Food-101, each stage takes approximately 50 minutes. The training costs are included in the revised paper (Appendix A.3, page 15).

Necessity. Table 4(d) demonstrates the advantages of our two-stage training over single-stage alternatives. Compared with joint training, our approach preserves higher full-depth accuracy and achieves a better dynamic accuracy–efficiency trade-off.

Scalability. The two-stage protocol allows exit branches to be added to an existing trained backbone without modifying its parameters or representations. This makes DeclareViT compatible with reusing pretrained or task-specific backbones for downstream applications, including object detection.

[1] Yanyu Li, Ju Hu, Yang Wen, Georgios Evangelidis, Kamyar Salahi, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Rethinking vision transformers for MobileNet size and speed. In IEEE/CVF International Conference on Computer Vision (ICCV), pp. 16843–16854, 2023.

We hope that our response has addressed the reviewer's concerns. We are happy to answer further questions.

Reviewer_ieapOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DeclareViT, an early-exit framework that separates intermediate classification from backbone feature refinement. Independent exit queries read intermediate features through cross-attention, while a two-stage procedure freezes the trained backbone and final classifier before optimizing the exit branches. The method preserves full-depth predictions and is evaluated across ViT, DeiT, and Swin on three datasets, with accuracy-efficiency curves, controlled ablations, and GPU/CPU latency measurements.

Critical strengths
  1. The read-only exit queries provide a practical way to add early prediction to a frozen backbone while preserving its full-depth predictions. Under the same frozen backbone and training objective, Table 3 shows that the proposed reader improves on the Transformer-encoder head with fewer parameters, supporting the value of the reader design.
  2. The evaluation covers nine dataset-backbone combinations and includes GPU/CPU timing, three-seed results, and calibration-targeted budget comparisons. Tables 4(d) and 11(d) hold the query architecture, KD, and exit-training budget fixed and show benefits from gradient isolation and then freezing the backbone. These comparisons provide substantive evidence for the method beyond comparisons with external baselines.
Critical weaknesses
  1. The contribution of decoupling needs to be distinguished more clearly from the exit-reader design. The controlled training ablations support the protocol, but do not isolate gradient-direction conflict from other effects of changing backbone optimization and making the features stationary. Moreover, the LGViT implementation already freezes the backbone and final classifier in its second stage (Appendix E). The query reader appears to be the more specific contribution, while the additional methodological insight attributed to decoupling remains less clearly established.
  2. The component ablations support an improved accuracy-efficiency trade-off, but the robustness of these gains is unclear. In Table 4(a), the query reader achieves nearly the same accuracy as pooling while improving S_MAC from 1.48x to 1.79x. Query accumulation also improves both accuracy and S_MAC on ViT, whereas Table 11(e) supports the non-accumulative design adopted for Swin. However, multi-seed results are not reported for these component comparisons. Repeated comparisons at matched computational budgets would help establish how consistently the reader and accumulation contribute.
Questions for authors
  1. What methodological contribution does the decoupling protocol make beyond the frozen-backbone stage already used by LGViT? Please distinguish the evidence for the query reader and stationary features from evidence that gradient-direction conflict specifically explains the performance gap.
  2. Do the improvements in Tables 4(a) and 4(b) persist across seeds and at matched S_MAC budgets? Please report paired comparisons with the pooling and one-shot readouts. Is the benefit of accumulation also observed on DeiT?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ieap (1/2): Weakness 1 & Question 1

Dear reviewer ieap,

Thank you for your careful review and for acknowledging the value of the reader design and of our controlled comparisons. We have addressed your concerns as follows.

Reply to Weakness 1 & Question 1: Decoupling vs. LGViT; separating the evidence

We thank the reviewer for pointing out this lack of clarity. We agree that the contributions of the query-reader architecture and the decoupled training protocol should be distinguished more clearly. We now first explain what the protocol contributes beyond the frozen second stage of LGViT, and then separate the evidence for the query reader and stationary features from the evidence for gradient-direction conflict.

Contribution of the decoupling protocol beyond LGViT. LGViT freezes the backbone only in its second stage. In its first stage the backbone and the exits are trained alternately, and its official implementation (TRAIN_STRATEGY=alternating_weighted) back-propagates the exit losses into the backbone, so the backbone it freezes has already been reshaped by exit supervision. In our protocol the backbone is trained with the full-depth objective only and never receives an exit loss at any stage. Table R5 shows what this is worth with the reader held fixed: letting exit losses update the backbone lowers every exit, the final classifier, and the dynamic accuracy at matched S_MAC. The protocol also makes the exits portable: they can be attached to an existing pretrained backbone without retraining or modifying it, and its full-depth accuracy is fully preserved. The comparison with LGViT's two-stage training is included in the revised paper (Section 2, page 3).

Separating the evidence.

Table R5. Training protocols with ViT-B/16 on ImageNet-1K, seed 2024. All rows share the reader, initialization, exit objective, distillation settings and exit-training budget. Columns @2 to @10 and Final give the per-exit accuracy (%); Dynamic 1.75× and Dynamic 2.00× give the dynamic accuracy (%) at matched S_MAC.

Protocol@2@4@6@8@10FinalDynamic 1.75×Dynamic 2.00×
Joint-attached55.8567.5374.0077.6980.5381.3979.4177.24
Joint-detached60.2569.9475.5978.6780.7481.5080.1678.91
Two-stage (Ours)60.1070.0875.7179.0381.2081.7780.7079.53
Δ exit gradients blocked+4.40+2.41+1.59+0.98+0.21+0.11+0.75+1.67
Δ backbone frozen−0.15+0.14+0.12+0.36+0.46+0.27+0.54+0.62

(1) Query reader. As the reviewer notes under Critical Strengths, Table 3 compares the heads on the same frozen backbone with the same exits and training objective, and the reader improves on the Transformer-encoder head with fewer parameters. No variant sends gradients to the backbone and all see fixed features, so these differences are due to the reader architecture alone.

(2) Gradient isolation (Joint-attached → Joint-detached). Joint-attached and Joint-detached use the same reader and both keep training the backbone, so features are non-stationary in both. The only difference is whether exit losses update the backbone. This ablation therefore shows that gradient isolation by itself improves performance.

(3) Frozen backbone (Joint-detached → Two-stage). Exit gradients are blocked in both setups, so that we can isolate the performance gain from stationary features with this ablation.

On gradient-direction conflict specifically. We agree with the reviewer. Comparison (2) shows that letting the exit losses update the backbone is harmful, and this result alone justifies isolating the backbone. It does not show that direction conflict is the mechanism: the stop-gradient removes the entire exit gradient, so it also removes magnitude effects and the additional feature drift that these updates cause, and the ablation cannot tell which of them causes the accuracy loss. Our conflict analysis is consistent with the ablation but does not resolve this either. We have therefore revised the claim. The revised paper (Section 3.2, page 4) now presents the conflict analysis as the motivation for isolating the backbone and as a diagnostic that indicates when exit losses are likely to interfere, not as a proven causal account. The method and the results of Comparison (2) are unchanged.

AuthorsOpenReview ↗

Response to Reviewer ieap (2/2): Weakness 2 & Question 2

Reply to Weakness 2 & Question 2: Robustness of component ablations

We thank the reviewer for pointing out the need to establish the robustness of the component improvements. To address this concern, we extend the pooling and one-shot readout comparisons from Tables 4(a) and 4(b) to three seeds (2024, 2025, and 2026) at matched calibration-targeted computational budgets. Table R6 compares pooling, one-shot, and accumulative readouts on ViT, as well as one-shot and accumulative readouts on DeiT.

The nearly identical accuracy in Table 4(a) reflects different operating points with substantially different computational costs (1.48× vs. 1.79× S_MAC). At comparable accuracy, our reader requires 17.3% fewer MACs. More importantly, Table R6 confirms its advantage under matched computational budgets: at the 1.50× and 2.00× calibration targets, our accumulative reader outperforms pooling on ViT by 1.05 and 5.88 percentage points, and one-shot reading by 0.42 points at both targets. The improvements also extend to DeiT (+0.18 and +0.53 points).

Regarding Swin, its computational overhead becomes substantial for Swin's high-resolution features. In the first Swin stage, updating five queries over 56×56 tokens costs approximately 1.09 GMAC per block, exceeding the cost of a backbone block. Therefore, Swin retains the same read-only reader and frozen backbone but adopts exit-local reading to avoid repeated cross-attention. As shown in Table 11(e) (Table 13(e) in the revised paper, page 21), this non-accumulative design achieves a better overall accuracy–efficiency trade-off despite lower shallow-exit accuracy.

Table R6. Readout ablations on ImageNet-1K at calibration targets 1.50× and 2.00×. Top-1 accuracy (%) is reported as the mean ± sample standard deviation over seeds 2024, 2025, and 2026, together with the mean realized S_MAC.

Readout1.50× target: Acc. (%)1.50× target: S_MAC2.00× target: Acc. (%)2.00× target: S_MAC
ViT-B
Pooling readout80.76 ± 0.051.45×74.03 ± 0.051.93×
One-shot81.39 ± 0.061.44×79.49 ± 0.021.92×
Accumulative (Ours)81.81 ± 0.051.44×79.91 ± 0.121.94×
DeiT-B
One-shot82.56 ± 0.021.44×78.28 ± 0.051.92×
Accumulative (Ours)82.74 ± 0.081.43×78.81 ± 0.061.94×

We hope that our response has addressed the reviewer's concerns. We are happy to answer further questions.

Reviewer_ieapOpenReview ↗

Response to Rebuttal

The authors’ comprehensive numerical results have addressed my concerns overall. It would be even better if the claims in the remaining sections—which do not align with the new results—could be revised accordingly. I am raising my score to 3.

AuthorsOpenReview ↗

Thank You and Further Revisions

Dear Reviewer ieap,

We sincerely thank you for your constructive suggestions and for raising your score.

Following your suggestion, we have revised the paper accordingly, with all changes marked in blue in the updated PDF. In the Abstract and the first contribution of the Introduction, we now present gradient conflict as a motivation and a diagnostic. In Section 5.3, we describe the conflict analyses in Figures 2 and 5 as complementary diagnostics rather than a controlled comparison. In Section 5.5, we state the evidence for the query reader, gradient isolation, and the frozen backbone separately.

Thank you again for your valuable feedback. We are happy to address any further questions.

Best regards,

The authors

Reviewer_mke6OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DeclareViT, an early-exit framework designed to resolve optimization interference and representation competition in multi-exit Vision Transformers. The authors argue that conventional designs overburden the intermediate CLS token by forcing it to simultaneously act as an early classifier and an intermediate representation for subsequent feature refinement, leading to conflicting optimization gradients. To address this, DeclareViT decouples early classification from the backbone by maintaining dedicated, read-only exit queries outside the main token stream. These queries progressively extract intermediate representations via one-way cross-attention and are discarded once their assigned exit prediction is made. A two-stage training strategy is adopted, freezing the fully trained backbone and optimizing only the query branches with ground-truth and distillation objectives. Evaluations across ViT, DeiT, and Swin on ImageNet-1K, CIFAR-100, and food-101 demonstrate favorable accuracy-efficiency trade-offs in terms of both MACs and batch-1 wall-clock latency on CPU and GPU.

Critical strengths
  • The motivation is well-substantiated through a quantitative gradient-conflict analysis on standard jointly-trained early-exit ViTs, showing that directional conflicts between shallow and deep objectives persist even at late training stages.
  • The experimental evaluation is thorough and transparent, covering three distinct architectures and three datasets, supported by multi-seed runs, explicit analytical MAC accounting, and actual wall-clock latency measurements on both CPU and GPU.
  • The ablation studies effectively isolate the contributions of the architectural components, parameter-sharing strategies, and the two-stage training regime, proving that the performance gains do not merely stem from freezing the backbone.
Critical weaknesses
  • The technical novelty is somewhat incremental, as the core mechanism, using external learnable queries with one-way cross-attention to read from a frozen ViT backbone, is heavily inspired by existing parameter-efficient transfer learning methods like VQT and SynQT, as well as prior decoupled early-exit frameworks like Dynamic Perceiver and Dyn-Adapter.
  • The proposed progressive query accumulation mechanism fails to generalize to hierarchical vision transformers; for Swin, the computational overhead of attending to high-resolution early stages forces the authors to abandon accumulation in favor of a simple one-shot local readout, which undermines the generality of the core architectural claim.
  • In the main comparison, the representative operating points are selected post hoc from the final evaluation Pareto frontier, which is methodologically flawed as it uses test-split statistics for model configuration selection, even though Appendix B.1 partially mitigates this with calibration-targeted tables.
  • The framework introduces a substantial parameter overhead (~19% increase, from 90M to 107M on ViT-B/16 even after aggressive parameter sharing) and incurs a 13.6% MAC overhead for difficult samples that reach the final layer, which limits its appeal in memory-constrained edge deployments.
  • Wall-clock latency is only evaluated at a batch size of 1. In batched inference scenarios, dynamic early exiting requires additional handling or sample gathering that often negates theoretical MAC speed-ups on GPUs, an aspect entirely omitted from the discussion.
Questions for authors

Please refer to the weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer mke6 (1/3): Weaknesses 1 and 2

Dear reviewer mke6,

Thank you for your thorough review and for acknowledging our motivation, evaluation and ablation studies. We have addressed your concerns as follows.

Reply to Weakness 1: Novelty vs. VQT, SynQT, Dynamic Perceiver and Dyn-Adapter

We agree that DeclareViT is inspired by existing methods. Our contribution is not the query mechanism itself, but how it is used to remove the optimization interference between early classification and feature refinement in multi-exit ViTs. None of the methods mentioned by the reviewer provides an early-exit interface that is read-only in both directions: the exit branch never writes into the backbone's token stream, and no exit loss ever updates the backbone at any training stage. As a result, the pretrained backbone and its full-depth prediction remain unchanged. Table R1 summarizes the differences.

DeclareViT versus decoupled early-exit methods. Dynamic Perceiver feeds its latent branch, i.e., early-exit branch back into the feature branch and trains both end-to-end with all exit losses. Dyn-Adapter freezes the backbone weights but trains in-block adapters with the exit losses, so exit supervision still changes the features that deeper layers receive. In both cases the full-depth model differs from the original one.

DeclareViT versus PETL methods. VQT and SynQT target parameter- and memory-efficient transfer learning, not early exiting. VQT adds a few query tokens to every layer and concatenates their outputs from all layers with the final [CLS] token as the input of one linear classifier. SynQT passes one query from block to block and feeds the features extracted at all blocks into one classifier, whose aggregation weights are computed from the output of the last block. In both methods, the classifier needs the last block, so the whole backbone must always run. Both also read through frozen pretrained attention weights. DeclareViT keeps one query state per exit, makes each exit prediction from the backbone blocks executed up to that exit, and learns its own exit-side projections. The comparison with VQT and SynQT is included in the revised paper (Section 2, page 3).

Table R1. Comparison with related early-exit and query-based PETL methods.

MethodEarly exitsRead-only forwardBackbone isolated in trainingFull-depth model keptQuery state across depthReads backbone features via
Early-exit methods
BranchyNet / SDN✓✓✗✗none (one head per exit)side-branch head
Dynamic Perceiver✓✗✗✗one latent code shared by all exitstwo-way cross-attention
Dyn-Adapter✓✓✗✗noneadapters inside blocks + early heads
Query-based PETL methods
VQT✗✓✓✓new queries at every layer, outputs concatenatedfrozen backbone projections
SynQT✗✓✓✓one query passed block to blockfrozen backbone projections
DeclareViT✓✓✓✓one per exit, accumulated up to that exitlearned exit-side projections

Reply to Weakness 2: Accumulation on Swin

We thank the reviewer for raising this important question. We agree that progressive query accumulation introduces substantial computational overhead on Swin. However, adopting an exit-local reader is a deliberate design choice based on the accuracy–efficiency trade-off, rather than a limitation of DeclareViT.

To further support this choice, we provide additional results in Table 11(e) (Table 13(e) in the revised paper, page 21). Although accumulation improves shallow-exit accuracy, updating five active queries over 3,136 tokens incurs approximately 1.09 GMAC per block, exceeding the cost of a Swin-B backbone block. Consequently, the accumulative variant achieves 81.74% accuracy at 1.38× speed-up, compared with 82.71% at 1.58× for the exit-local design.

More importantly, DeclareViT's core contribution lies in decoupling early classification from backbone feature refinement through read-only exit queries and gradient-isolated training, rather than progressive accumulation. The Swin variant fully preserves this mechanism, demonstrating its applicability across backbone architectures.

AuthorsOpenReview ↗

Response to Reviewer mke6 (2/3): Weakness 3

Reply to Weakness 3: Post-hoc operating-point selection

We thank the reviewer for this point and for noting that the calibration-targeted tables in Appendix B.1 partially address the concern. We agree that the representative points in Table 1 are selected post-hoc from the final-evaluation Pareto frontier, as disclosed in Appendix A.4.

This follows the reporting style of prior early-exit work such as LGViT and EnViT, to allow a compact side-by-side comparison of all methods at one representative operating point each.

We report the calibration-targeted results with three seeds at the 1.50×, 2.00× and 2.50× targets. The results are in Table R2 below. Across the 27 evaluated settings, DeclareViT achieves the highest mean accuracy in 25, ranking second only to DMPO in the remaining two. These three-seed results are included in the revised paper (Appendix B.1, page 17; Table 8, page 18), where they replace the single-seed results of the original submission.

Table R2. Top-1 accuracy (%) at calibration-targeted S_MAC. Operating points are selected on the calibration split and applied unchanged to the final evaluation split. Each cell gives accuracy with the realized S_MAC in parentheses, as mean ± std over three seeds. Bold: best mean accuracy. †EnViT: single operating points reported in the original paper, not calibration-targeted and not ranked. BYOT, ViT-EE and COSEE are omitted here for space; the full table is included in the revised paper (Table 8, page 18).

MethodCIFAR-100 1.50×CIFAR-100 2.00×CIFAR-100 2.50×ImageNet-1K 1.50×ImageNet-1K 2.00×ImageNet-1K 2.50×Food-101 1.50×Food-101 2.00×Food-101 2.50×
ViT
SDN91.43±0.23 (1.50±0.02×)87.09±0.57 (1.99±0.03×)79.53±0.84 (2.49±0.04×)80.70±0.14 (1.46±0.00×)74.31±0.31 (1.95±0.00×)66.09±0.39 (2.45±0.01×)90.92±0.15 (1.58±0.03×)88.42±0.23 (2.14±0.02×)82.75±0.21 (2.71±0.02×)
LGViT91.02±0.03 (1.51±0.00×)83.85±0.35 (2.02±0.01×)74.83±0.47 (2.49±0.01×)75.61±1.36 (1.47±0.00×)61.89±2.33 (1.99±0.00×)51.93±2.71 (2.49±0.00×)90.05±0.25 (1.59±0.01×)85.59±0.58 (2.14±0.01×)79.67±0.99 (2.58±0.01×)
DMPO91.30±0.95 (1.50±0.02×)88.06±3.91 (1.99±0.03×)82.67±6.16 (2.54±0.02×)81.85±0.08 (1.46±0.00×)78.39±0.12 (1.95±0.00×)72.39±0.18 (2.51±0.00×)90.87±0.19 (1.58±0.05×)90.16±0.38 (2.12±0.03×)87.32±0.40 (2.71±0.01×)
EnViT†–89.23 (2.05×)––80.45 (1.72×)––89.15 (2.20×)–
Ours91.75±0.13 (1.52±0.10×)90.76±0.28 (1.98±0.05×)87.64±0.31 (2.49±0.05×)81.81±0.05 (1.44±0.00×)79.91±0.12 (1.94±0.00×)76.35±0.19 (2.45±0.00×)91.18±0.15 (1.73±0.30×)91.20±0.04 (2.17±0.05×)89.61±0.09 (2.76±0.04×)
DeiT
SDN89.03±0.23 (1.50±0.01×)84.08±0.13 (1.99±0.02×)76.99±0.25 (2.48±0.03×)79.19±0.12 (1.47±0.00×)72.83±0.36 (1.97±0.00×)64.94±0.42 (2.48±0.01×)89.37±0.18 (1.58±0.03×)86.27±0.12 (2.15±0.01×)79.88±0.23 (2.72±0.00×)
LGViT87.62±0.18 (1.51±0.00×)78.51±0.37 (2.02±0.01×)68.20±1.03 (2.50±0.01×)76.49±0.55 (1.46±0.00×)64.22±1.05 (2.00±0.00×)54.73±1.11 (2.50±0.00×)87.99±0.38 (1.59±0.01×)83.74±0.78 (2.14±0.01×)77.94±0.95 (2.58±0.01×)
DMPO89.12±0.44 (1.49±0.02×)84.49±1.73 (1.99±0.04×)77.15±2.83 (2.55±0.04×)80.13±0.05 (1.46±0.00×)74.42±0.43 (1.96±0.00×)66.57±0.65 (2.53±0.00×)89.63±0.26 (1.56±0.02×)86.65±1.16 (2.12±0.02×)80.70±1.55 (2.72±0.01×)
Ours89.49±0.07 (1.47±0.05×)86.59±0.10 (1.98±0.06×)81.36±0.12 (2.49±0.04×)82.74±0.08 (1.43±0.00×)78.81±0.06 (1.94±0.00×)73.40±0.05 (2.46±0.01×)90.19±0.13 (1.57±0.05×)89.45±0.15 (2.15±0.05×)86.16±0.05 (2.75±0.02×)
Swin
SDN87.42±0.27 (1.49±0.01×)77.31±0.37 (1.95±0.01×)64.74±0.69 (2.52±0.02×)79.78±0.02 (1.44±0.00×)76.47±0.08 (1.91±0.00×)70.40±0.32 (2.41±0.00×)91.77±0.11 (1.56±0.01×)87.89±0.21 (2.12±0.01×)79.91±0.20 (2.77±0.02×)
LGViT89.98±0.08 (1.43±0.00×)82.39±0.35 (1.88±0.01×)73.57±0.43 (2.36±0.03×)81.96±0.05 (1.42±0.00×)76.29±0.16 (1.89±0.00×)68.68±0.05 (2.41±0.00×)93.37±0.06 (1.50±0.01×)90.51±0.12 (1.99±0.00×)85.24±0.09 (2.53±0.01×)
DMPO91.49±0.15 (1.47±0.01×)86.99±0.45 (1.99±0.02×)80.39±0.18 (2.48±0.02×)81.62±0.04 (1.43±0.00×)80.24±0.13 (1.94±0.00×)76.54±0.28 (2.44±0.01×)92.84±0.11 (1.54±0.01×)91.60±0.06 (2.13±0.01×)88.01±0.16 (2.65±0.01×)
Ours92.03±0.04 (1.45±0.00×)89.11±0.30 (1.95±0.02×)84.20±0.57 (2.40±0.04×)83.37±0.02 (1.42±0.00×)80.52±0.04 (1.90±0.00×)76.23±0.19 (2.37±0.00×)93.66±0.02 (1.53±0.00×)93.36±0.02 (2.07±0.01×)91.91±0.08 (2.60±0.01×)
AuthorsOpenReview ↗

Response to Reviewer mke6 (3/3): Weaknesses 4 and 5

Reply to Weakness 4: Parameter and MAC overhead

We thank the reviewer for pointing this out. We agree that the default configuration adds parameters and final-path MACs, which matters for memory-constrained deployment. With parameter sharing, DeclareViT already has fewer parameters than the Transformer-encoder and CNN exit heads while giving a better accuracy–efficiency trade-off (Table 3), and sharing itself removes 22.5% of the parameters at marginal cost (Table 4(c)).

A lightweight configuration. To address memory-constrained deployment, we further add a lightweight configuration, DeclareViT-Lite, with a reduced query dimension (D_q=384), narrower K/V projections (768→384) and an FFN expansion ratio of 2. As shown in Table R3, it reduces the additional parameters from 20.4M to 7.0M and the final-path MAC overhead from 13.6% to 6.7%, at a modest accuracy cost at the same budgets. This configuration is included in the revised paper (Appendix D.6, page 22; Table 14, page 22).

The standard configuration therefore suits latency-bound deployment where weight memory is available, and DeclareViT-Lite suits memory-constrained deployment.

Table R3. Standard vs. lightweight DeclareViT with ViT-B/16 on ImageNet-1K using seed 2024. Overheads are relative to the original ViT-B/16. Operating points are selected on the calibration split as in Table 8.

Configuration#Params (extra)Final-path GMAC (overhead)1.50× target: Acc. (%)S_MAC2.00× target: Acc. (%)S_MAC
ViT-B/1686.6M (–)17.56 (–)81.77 (full depth, 1.00×)
DeclareViT107.0M (+20.4M)19.96 (+13.6%)81.771.44×79.901.94×
DeclareViT-Lite93.5M (+7.0M)18.74 (+6.7%)81.591.44×78.631.93×

Reply to Weakness 5: Batched inference latency

Thank you for raising this. We have now measured batched inference at batch sizes of {4, 8, 16, 32, 64} for all methods in Table R4. The experimental setup is the same as in the paper (Appendix C). For batched inference, we adapted every method with in-batch compaction (samples that exit are removed from the batch) and cross-batch regrouping (the remaining samples are merged with those from other batches). The measured time includes the overheads the reviewer mentions: the exit decisions with their GPU-to-CPU synchronization, the gathering of the remaining samples, and the regrouping.

Results. Batching does not cancel the MAC savings. DeclareViT is 1.48× to 1.71× faster than ViT at the same batch size. At B=32 and B=64 it is the fastest early-exit method and also the most accurate. At B=16 it is on par with DMPO. At B=4 it is behind SDN, BYOT, COSEE and DMPO, and at B=8 behind COSEE and DMPO.

The gap at small batches comes from the query branch: the baselines add computation only at their exits, whereas DeclareViT also updates its exit queries after each of the first ten blocks. Each update has a fixed cost per call that does not depend on the batch size. In a small batch, this cost is shared by only a few images; in a larger batch, it is shared by more images. As a result, the speed-up of DeclareViT increases from B=4 to B=64. These results and the discussion are included in the revised paper (Appendix C, page 20; Table 12, page 21).

Table R4. Batched inference on ImageNet-1K with ViT-B/16. Each batch-size cell gives the measured speed-up over ViT at the same batch size and, in parentheses, the time per image in ms. Bold marks the best early-exit method in each column.

MethodAcc. (%)S_MACB=4B=8B=16B=32B=64
ViT-B/1681.771.00×1.00× (7.66)1.00× (6.91)1.00× (6.26)1.00× (5.93)1.00× (5.78)
SDN79.201.63×1.55× (4.95)1.57× (4.40)1.58× (3.95)1.59× (3.73)1.60× (3.62)
BYOT79.531.64×1.55× (4.94)1.57× (4.39)1.59× (3.94)1.59× (3.72)1.60× (3.61)
ViT-EE78.561.36×1.31× (5.85)1.32× (5.21)1.33× (4.69)1.34× (4.44)1.34× (4.30)
LGViT79.211.36×1.18× (6.50)1.17× (5.90)1.16× (5.37)1.16× (5.11)1.16× (4.97)
COSEE79.381.66×1.58× (4.87)1.60× (4.32)1.61× (3.88)1.62× (3.67)1.62× (3.56)
DMPO80.521.68×1.58× (4.87)1.62× (4.26)1.64× (3.81)1.65× (3.60)1.66× (3.48)
DeclareViT (Ours)80.551.79×1.48× (5.17)1.57× (4.40)1.64× (3.82)1.68× (3.53)1.71× (3.38)

We hope that our response has addressed the reviewer's concerns. We are happy to answer further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to the Area Chair and All Reviewers

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and Reviewers mke6, ieap and BCvX for their time and for their careful and constructive feedback, which has helped us to improve the paper.

We are encouraged that the reviewers found the motivation well substantiated by the gradient-conflict analysis (mke6), the read-only exit queries a practical way to add early prediction to a frozen backbone (ieap), and the design a particularly strong form of decoupling (BCvX). We also appreciate the recognition of the thorough evaluation and the controlled ablations (all reviewers).

Responses to the main concerns and new experiments. Detailed replies are posted under each review (Tables R1–R4 for mke6, R5–R6 for ieap, R7–R8 for BCvX).

  • Novelty (mke6, BCvX). Our early-exit interface is read-only in both directions: the exit branch never writes into the backbone's token stream, and no exit loss ever updates the backbone (Table R1).
  • Decoupling vs. LGViT and gradient conflict (ieap). Table R5 separates the gain of gradient isolation from that of freezing the backbone. The conflict analysis is now presented as motivation and as a diagnostic, not as a proven causal account.
  • Robustness of the component ablations (ieap). New experiment: three-seed comparisons of pooling, one-shot and accumulative readouts at matched budgets (Table R6).
  • Accumulation on Swin (mke6). Exit-local reading is a deliberate accuracy–efficiency choice; the read-only, gradient-isolated mechanism is preserved.
  • Post-hoc operating-point selection (mke6). New experiment: three-seed calibration-targeted results in all 27 settings; DeclareViT has the highest mean accuracy in 25 (Table R2).
  • Parameter and MAC overhead (mke6). New experiment: DeclareViT-Lite reduces the additional parameters from 20.4M to 7.0M and the final-path MAC overhead from 13.6% to 6.7% (Table R3).
  • Batched inference (mke6, BCvX). New experiment: batch sizes 4 to 64 for all methods; DeclareViT is 1.48× to 1.71× faster than ViT and the fastest early-exit method at batch sizes 32 and 64 (Tables R4 and R7).
  • Single-stage models (BCvX). New experiment: DeclareViT attached to DeiT-S and EfficientFormerV2-L without retraining them (Table R8).
  • Cost of two-stage training (BCvX). Stage I is not needed when a trained model already exists; Stage II takes about 8 hours on ImageNet-1K on 4×H100 GPUs.

Revisions in the paper. All changes are marked in blue\color{blue}{\text{blue}} in the revised paper.

  • Section 2 (page 3): comparison with LGViT's two-stage training; new paragraph on query-based transfer learning (VQT, SynQT).
  • Section 3.2 (page 4): the conflict analysis is presented as a diagnostic and as motivation.
  • Sections 5 and 6 (pages 8–10): pointers to the new results; the limitations mention the lightweight configuration.
  • Appendix A.2 and A.3 (page 15): EfficientFormerV2 adaptation and training cost.
  • Appendix B.1 and Table 8 (pages 17–18): three-seed calibration-targeted results.
  • Appendix B.4 and Table 10 (pages 17 and 19): DeiT-S and EfficientFormerV2-L with DeclareViT.
  • Appendix C and Table 12 (pages 20–21): batched GPU inference.
  • Appendix D.6 and Table 14 (page 22): DeclareViT-Lite.

Table 11 of the original submission is Table 13 in the revised paper.

We hope that our responses and revisions address the concerns raised. We are happy to answer further questions.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.