acceptodds
Under review as a conference paper at ICLR 2027

Conditional Complementary Feature Learning for Early-Exit Networks

14 views

Abstract

Early-exit networks enable adaptive inference on resource-constrained devices by allowing easy inputs to terminate early while harder inputs receive additional computation. Conventional depth-wise designs attach classifiers to a shared backbone, introducing two limitations: architectural constraints, which tie early-exit capacity to backbone position, and early-late interference, where shared features must support both immediate classification and continued processing, potentially degrading early-exit accuracy under joint training. Our approach is motivated by the intuition that shared-prefix computation may contribute to these limitations. Our key insight is to separate architectural coupling from sequential execution. Rather than evaluating a single network to increasing depth, we organize inference as a sequence of complete, exit-specific subnetworks. Since earlier representations are already available when a later exit is reached, later computation can focus on learning features that complement them. Guided by this principle of conditional complementary feature learning, we introduce an early-exit framework in which each subnetwork processes the original image through a full-depth feature hierarchy. This permits narrow, full-depth early predictors and relaxes the requirement that earlier features preserve all information needed for later computation. Each later subnetwork conditions feature extraction on the stage-end features of all preceding subnetworks, while its classifier combines the current and earlier final embeddings. Earlier representations thus guide subsequent extraction while remaining directly available for prediction. We instantiate the framework with CNN backbones and evaluate it on ImageNet and embedded CPU and GPU platforms. Our models improve the accuracy-efficiency trade-off, requiring up to 2.2× fewer FLOPs than the compared early-exit baselines at matched accuracy. Comparisons with Dynamic Perceiver show gains of up to 10.2 and 14.0 percentage points in top-1 accuracy at matched FLOPs and latency, respectively, and up to 2.5× lower latency at matched accuracy. Our work offers a new perspective on early-exit design, in which additional computation is conditioned on, and complementary to, what has already been computed.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Early Exit Networks, Dynamic Inference, Efficient Neural Network, Adaptive Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bNmCOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This work proposes an early-exit architecture targeting two limitations of prior works: (1) early exits are backbone prefixes, so their capacity is tied to their position, and (2) joint training causes early-late interference. In this paper, each exit gets its own complete, full-depth subnetwork, executed sequentially. Later subnetworks are conditioned on earlier subnetworks' stage-end features, and each classifier also takes the final embeddings from all earlier exits. ImageNet experiments with MobileNetV3/EfficientNetV2 show improved accuracy-efficiency trade-offs, mainly at low budgets, with latency measured on a Raspberry Pi 4 and a Jetson Xavier.

Critical strengths
  • The paper addresses a practical problem and correctly identifies limitations of prior early-exit designs.
  • Low-budget results are strong (exit 1: 73.4% at 207 MFLOPs vs. 64.9% at 225 MFLOPs for DP).
Critical weaknesses
  1. The proposed design has its own weakness. Reaching exit k requires executing k full-depth subnetworks in sequence, so the critical path grows roughly as k × depth. This cost is hidden on CPUs but becomes significant on GPUs. Table 8 already hints at it: at near-identical final-exit FLOPs (696 vs. 683 M), the Raspberry Pi 4 latency is on par with DP (148.9 vs. 147.7 ms), while the Xavier GPU latency is worse (13.4 vs. 11.7 ms). This problem is likely to be more pronounced on desktop or server GPUs with greater parallelism. Evaluation on such platforms is needed to demonstrate the method's actual latency.
  2. The claim that DP leaves early-late interference unaddressed does not seem appropriate, since DP's main goal was to address this problem by separating the feature and classification branches. Furthermore, this paper does not clearly address the interference problem either. Earlier subnetworks still receive gradients from all later losses, as acknowledged in Sec. 3.5. The evidence offered in Table 1 is an accuracy gap between standalone and joint training, which can arise from many factors and does not by itself demonstrate interference or its reduction.
  3. DP's Forward Knowledge Transfer (FKT) module already forwards earlier exits' outputs to later classifiers, although it forwards predictions rather than embeddings. An ablation comparing forwarding predictions with forwarding embeddings is needed to show that the proposed classifier feature reuse offers a meaningful improvement.
Questions for authors
  1. How do the reproduced DP models' per-exit FLOPs and accuracy compare with the original DP paper? Which baseline does the +6.3 annotation in Fig. 3 refer to, and does DP have measured points near 0.15 GFLOPs?
  2. In Table 3, why is z_2 so weak for the classifier-only and matched-depth variants (54.5% and 60.7%) despite coming from a full-depth subnetwork? Were they trained to convergence?
4 comments

We thank the reviewer for the detailed review and constructive suggestions. We apologize for the delayed response, as the additional experiments required several new full training runs. Below, we provide new GPU measurements and analysis, clarify the DP comparisons, and identify claims that need more precise wording. We will incorporate these results and clarifications into the revised manuscript.

W1: Sequential full-depth subnetworks may incur greater latency on GPUs; evaluation on desktop or server GPUs is needed.

Following the reviewer’s suggestion, we evaluated both matched model pairs on an RTX 3090 and an RTX 6000 Ada. The results confirm a final-exit latency penalty, but also show that our accuracy-latency advantage persists at low and intermediate budgets.

We use TensorRT 11.4, FP32, batch size one, identical build settings for both methods, and the timing protocol of Appendix A.3. Cumulative GPU-compute latencies (ms) through each exit, obtained by summing measured per-exit times, are:

GPUModelExit 1Exit 2Exit 3Exit 4
RTX 3090Ours (MBV3)0.7091.4082.1212.837
DP (MBV3)1.4082.3592.5642.713
Ours (EffV2)0.8441.6612.4913.328
DP (RegNet)1.3352.3342.5082.619
RTX 6000 AdaOurs (MBV3)0.4810.9741.4671.960
DP (MBV3)1.0651.7281.8481.933
Ours (EffV2)0.5801.1501.7292.307
DP (RegNet)0.9751.7041.8251.898

Our first exits are around 1.6-2.2x faster than DP’s across the two matched pairs and GPUs. However, when every image reaches the final exit, our MBV3 model is around 1.4-4.6% slower than MBV3-based DP, and our EffV2 model is 21.5-27.1% slower than RegNet-based DP. The latency advantage therefore depends on where inference terminates.

Practical efficiency also depends on average inference latency across inputs. We therefore evaluate accuracy at matched average-latency budgets, using the measured GPU latencies above as exit costs and the same threshold-calibration procedure used in our Raspberry Pi 4 and Xavier evaluations. Each cell reports top-1 accuracy (%) as ours / DP, with the average-latency budget in parentheses. Average costs are within 0.011 ms of the stated budgets. “–” indicates that DP’s first exit exceeds the budget. The results are shown below:

GPUModel pairOur first-exit budgetDP’s first-exit budgetIntermediate budgetDP’s final-exit budget
RTX 3090MBV3 / MBV373.39 / – (0.71 ms)78.97 / 64.92 (1.41 ms)79.16 / 78.04 (2.00 ms)79.16 / 79.21 (2.71 ms)
EffV2 / RegNet74.93 / – (0.84 ms)79.83 / 68.81 (1.34 ms)80.57 / 79.81 (2.00 ms)80.56 / 80.35 (2.62 ms)
RTX 6000 AdaMBV3 / MBV373.39 / – (0.48 ms)79.14 / 64.92 (1.07 ms)79.16 / 78.31 (1.50 ms)79.16 / 79.21 (1.93 ms)
EffV2 / RegNet74.93 / – (0.58 ms)80.00 / 68.81 (0.98 ms)80.59 / 79.96 (1.50 ms)80.56 / 80.35 (1.90 ms)

Our models already achieve strong accuracy at budgets below the corresponding DP models’ minimum inference latencies. They remain more accurate at DP’s first-exit latency and the intermediate budgets, and achieve comparable accuracy at DP’s final-exit latency.

To summary, our models achieve improved accuracy at low and intermediate average-latency budgets, with higher latency when inference reaches the final exit. We will add these results and clarify the trade-off in the revised manuscript.

W2: DP already addresses interference, and Table 1 does not by itself establish reduced interference.

We agree that addressing interference is a central contribution of DP, as acknowledged at L82-84 in the manuscript. Our phrase “the underlying coupling remains” refers to residual interaction, but we recognize that the contrast could obscure our acknowledgment of DP’s contribution. Table 1 examines a complementary question: how well the first exit preserves its accuracy under joint training, whereas DP’s motivation emphasizes protecting later-exit performance from early supervision. We will clarify this distinction in the revision.

Our architectural contribution is the conditional complementary-learning design, which replaces the shallow, position-constrained exit predictors of prefix-based approaches with thin, full-depth subnetworks. Each subnetwork has direct image access, while stage-wise conditioning and embedding reuse allow later exits to build on earlier representations. Earlier features are therefore no longer the sole source of information for later computation, relaxing the requirement that they preserve everything needed downstream. Although later losses still update earlier subnetworks through reuse paths, as acknowledged in Sec. 3.5, the architectural objective is to provide strong early predictors without restricting their depth to their exit position, while allowing later exits to benefit from their representations.

Table 1 evaluates this accuracy retention: our first exit loses 1.16 percentage points under joint training, compared with 3.05 for DP, under the evaluated training recipes. This supports better preservation of first-exit accuracy despite the remaining cross-exit interaction. We agree, however, that the accuracy gap alone does not isolate interference from other effects of joint optimization. We will therefore revise causal wording such as “confirms that the gains arise from … reduced interference” and instead present better first-exit accuracy retention under joint training as an empirical benefit of the proposed architecture.

W3: DP’s FKT already forwards earlier predictions; an ablation comparing prediction and embedding reuse is needed.

We agree that DP’s FKT already enables prediction-level reuse, and our reproduced DP models include this module. Following the reviewer’s suggestion, we compared three classifier-reuse variants on our matched MobileNetV3 model, keeping stage-wise conditioning and the training recipe unchanged. Their compute differs by less than 1%.

The prediction-reuse variant follows FKT’s linear projection, BN, and ReLU transformation before concatenation with the current embedding. It uses predictions from all earlier exits, matching the history available to our embedding-reuse variant.

Classifier inputExit 1Exit 2Exit 3Exit 4
No classifier reuse74.2676.6177.7678.19
FKT-style prediction reuse74.6477.3078.3478.72
Embedding reuse (ours)73.3977.7078.7079.17

Values are ImageNet top-1 accuracy (%). Embedding reuse improves every later exit by 0.36-0.45 percentage points over prediction reuse, while prediction reuse better preserves first-exit accuracy. We also calibrate exit thresholds for each variant using the same budgeted-classification protocol. Across the approximately 0.21-0.70 GFLOPs range, prediction reuse leads below about 0.26 GFLOPs, while embedding reuse leads over the remaining higher-budget range. We therefore select embedding reuse for its stronger later-exit performance and advantage across most of the evaluated budget range.

We will add this ablation and explicitly discuss DP’s FKT in the revised manuscript.

Q1: How do the reproduced DP models compare with the published results? What does +6.3 in Fig. 3 refer to, and does DP have results near 0.15 GFLOPs?

First, we clarify the source of the DP results: Fig. 3 uses curves digitized from the DP paper, while Fig. 4 and Tables 1 and 8 use our reproductions of MobileNetV3-based model 4 and RegNet-based model 4. Sec. 4.1 does not make this distinction clear, and we will correct it.

Reproduction details and agreement. Published curves alone are insufficient for our controlled comparisons: calibrating exit thresholds requires per-exit predictions across the calibration dataset, evaluating budgeted accuracy requires the corresponding validation predictions, and hardware measurements require executable models. The published family plots also expose only part of each target model’s curve. Checkpoints for these two configurations were not publicly available; we requested them from the authors via GitHub and email but did not receive a response. We therefore reproduced both models using DP’s official implementation.

We use DP’s official implementation and the model configurations in its Appendix Tables 7 and 6, respectively. Training follows the published recipes, including self-distillation and 600 epochs for MobileNetV3 and 300 for RegNet. We use a global batch size of 1024, distributed across four GPUs with 256 images each rather than the repository’s eight GPUs with 128 each; thus, the global batch is unchanged, but the per-device BatchNorm batch size differs. We enable AMP for both the DP reproductions and our own models.

Our reproduced per-exit FLOPs and accuracies are reported in Table 8, with FLOPs calculated using the same counter as our models. DP does not provide a complete per-exit breakdown for these exact configurations, so we compare their budgeted accuracy-FLOPs curves with the identifiable model-4 segments in the published family plots. Over the available segments, our MobileNetV3 reproduction is approximately 0.1-0.5 percentage points below the published curve at 0.34-0.64 GFLOPs. For RegNet, the difference is approximately -0.21 to +0.12 percentage points at 0.98-1.42 GFLOPs (reproduced minus published). These comparisons use interpolation within the available digitized ranges.

The +6.3 annotation. This compares our model-family curve with DP’s published MobileNetV3 family curve at approximately 0.143 GFLOPs: 74.3% versus approximately 68.0% accuracy. DP therefore has a published curve extending to this budget; this is its leftmost available plotted point, not an identified first-exit result. Our value comes from the budgeted-classification evaluation calibrated using the protocol in Sec. 4.1. The +6.3 annotation in Fig. 3 compares model families, whereas the +10.2 annotation in Fig. 4 compares individual models with approximately matched first- and final-exit FLOPs.

Q2: Why is z_2 weak in the classifier-only and matched-depth variants? Were they trained to convergence?

All variants were trained for 300 epochs with the same cosine schedule and training recipe as our full model. Their late-training trajectories stabilize alongside the converged full model, without narrowing the accuracy gap. The linear and MLP probes also converge. These observations suggest that the weak standalone z_2 results are not due to incomplete training.

During training, z_2 is not supervised for classification on its own. The exit 2 classifier receives [z_1,z_2], and its loss supervises their joint prediction. Thus, a full-depth subnetwork need not produce an independently strong classifier representation. Indeed, the native exit 2 accuracies are 76.16% and 76.30% for the classifier-only and matched-depth variants, respectively, despite their low standalone z_2 probe accuracies.

The variants differ in where and which earlier features are available. Classifier-only reuse supplies earlier embeddings to the classifier but does not condition the feature-extraction body. Matched-depth conditioning supplies earlier features at corresponding depths in the body. Our stage-end conditioning instead uses the mature representations at each reused resolution in the body. Because the earlier subnetwork has already finished the stage, these features can guide the later subnetwork from the beginning of its corresponding stage. The aim is to let later extraction build on completed earlier processing and learn features that improve prediction alongside the retained embeddings. Table 3 supports this rationale: stage-end conditioning produces both the strongest standalone z_2 and the largest gain from combining z_1 and z_2. We will clarify this interpretation and the convergence in the revised manuscript.

We thank the reviewer again for the thoughtful feedback. We hope our responses address the concerns and would be grateful for their consideration in the final assessment. We would be happy to clarify any remaining questions.

W2 follow-up: direct measurement of early-late gradient conflict

To complement the accuracy retention comparison in Table 1, we have directly measured gradient alignment between the first and later exit objectives using the same matched four-exit MobileNetV3 checkpoints. We evaluate both models on the same 100 disjoint ImageNet validation batches, each containing 128 randomly shuffled images, with deterministic preprocessing in FP32 evaluation mode. For each batch, we compute

g_1 = grad_{theta_shared} (L_1)

g_late = grad_{theta_shared} (L_2 + L_3 + L_4)

where {theta_shared} includes parameter tensors reached by both objectives within each model. We concatenate their gradients and compute the cosine similarity. A batch is considered conflicting when this cosine is negative.

"Common CE" applies equally weighted hard-label cross-entropy loss. We additionally evaluate DP with its native exit weights and self-distillation loss.

Model and measurement objectiveMean gradient cosineConflicting batches
Ours, common CE0.6870/100
DP, common CE0.29221/100
DP, native weights + KD0.24526/100

Under common CE, our model shows substantially higher gradient alignment, with no conflicting batches observed. This pattern also holds for the individual later exits: mean exit 1 versus exit 2/3/4 cosines are 0.738/0.606/0.570 for ours and 0.265/0.321/0.203 for DP. At the evaluated checkpoints, later-exit gradients are better aligned with the first-exit objective in our model under the stated measurement conditions. This provides gradient-level evidence consistent with the better first-exit accuracy retention in Table 1. We will include these results and the measurement details in the revision.

We thank the reviewer for motivating this analysis and hope it addresses the concerns.

Reviewer_xSvPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, author provides conventional early-exit networks, which attach classifiers to a shared backbone. they point out two primay issues

  1. structural constraints that there ar e shallow representations in the early exits,
  2. interference from the share features

therefore, they proposed conditional complementary feature learning, which assigned independent and full subnetwork to each exit and provides a connection between subsequent features. the paper also take a result 10.2% accuracy up,

Critical strengths

The paper offers the three design choices, which is independent subnetworks, stage-wise conditioning, and the reuse of features in classifiers. This is not merely an empirical collection of techniques; rather, a consistent design rationale permeates the entire approach. The experimental design is particularly rigorous and fair. Furthermore, the authors independently verify whether FLOPs accurately reflect real-world hardware efficiency with Raspberry Pi 4 and the Jetson AGX Xavier.

Critical weaknesses

In my opinion, the evaluation is very limited (only apply ImageNet classification), Although the author argued that the propsed mehod can provide adaptive inference under a variable computational budget. but the paper does not invesdtigate whether these benefits extend to dense prediction task. (we can noit generalize your propsoed methods)

Another problem is scomputational costs, may be the cost increases quadratically with the number of exit point. the small number of exit will be managable, but a large number of exit requires more cachen memory and costs associated with feature fusion.

Questions for authors
  1. Regarding the comparison of interference in Table 1, has the potential for self-distillation within certain models been analyzed?
  2. In Figure 3, the MobileNetV3 baseline curve is used for comparison; have you verified that the chosen baseline represents the optimal or best-performing version of the model?
4 comments

We thank the reviewer for the thoughtful review and constructive feedback. We apologize for the delayed response: the additional experiments required multiple new full training runs. We address each point below and will incorporate the new results into the revision.

W1: Evaluation is limited to ImageNet classification; do the adaptive inference benefits extend to dense prediction?

We agree that evaluation beyond classification is important. We extended our framework to object detection and trained a 3-exit detector on COCO to obtain results within the rebuttal period. Each exit is a complete, independently parameterized RTMDet detector that processes the original image. Each of them provides the multi-scale features needed for detection. In contrast, an intermediate depth-wise exit would require additional adaptation to supply the full feature hierarchy. We extend stage-wise conditioning to the backbone, neck, and head, and reuse earlier exits’ spatial head features separately for classification and box regression. Unlike Dynamic Perceiver (DP)’s detection experiment, which uses its model as a RetinaNet backbone without early exiting, our model supports a complete detection prediction at every exit.

All exits are trained jointly from scratch on COCO for 300 epochs using the sum of their detection losses, without the ImageNet-pretrained backbone used in the official RTMDet-s recipe. The remaining training settings follow the official recipe except for using a global batch size of 128 instead of 256 with a linearly scaled learning rate, and BF16 mixed precision. The results are as follows:

Exit 1Exit 2Exit 3
Cumulative GFLOPs (640×640)10.0321.0032.93
COCO val2017 box AP40.645.747.6

A single jointly trained detector provides three operating points between 10 and 33 GFLOPs, with AP increasing from 40.6 to 47.6. GFLOPs count one multiply-add as one FLOP and include all preceding exits and feature-reuse operations. We will extend the evaluation to four exits and include the results and implementation details in the revised manuscript.

W2: The cost may grow quadratically with the number of exits, with more cache memory and fusion overhead.

The reviewer is correct that all-to-all feature reuse introduces a cumulative fusion cost that grows quadratically with the number of exits. However, this does not preclude practical efficiency: our four-exit models achieve up to a 2.5x speedup at matched accuracy with reuse overhead included (Fig. 4), and reuse accounts for only 3.4-7.8% of cumulative FLOPs in the evaluated 4-exit and 6-exit models. The retained feature cache grows linearly and is below 2 MiB for these models at batch size one in FP32. Latency overhead is hardware-dependent, with the 6-exit measurements showing more substantial growth on Raspberry Pi 4. We show the results below.

FLOPs. We isolate the reuse overhead by comparing each subnetwork with the same subnetwork after removing both stage-wise conditioning and classifier feature reuse. The results are as follows:

ModelPer-exit FLOPs overhead relative to the reuse-free subnetwork, exits 2-KReuse share of cumulative FLOPs at the final exit
MBV3, 4 exits (Table 8)5.1 / 7.9 / 10.7%5.2%
EffV2, 4 exits (Table 8)3.5 / 5.4 / 7.2%3.4%
MBV3, 6 exits (Model 3, Table 4)5.3 / 8.2 / 11.1 / 14.0 / 16.9%7.8%

Latency. For the four-exit models, we similarly compare each later subnetwork against its reuse-free counterpart (same for exit 2-4), using the same devices and settings as the corresponding full-model measurements. Desktop-GPU settings are provided in our response to Reviewer bNmC, W1.

PlatformMBV3: reuse-freeMBV3: overhead at exits 2-4EffV2: reuse-freeEffV2: overhead at exits 2-4
Raspberry Pi 428.26 ms20.5 / 32.0 / 42.6%46.51 ms15.9 / 27.2 / 32.8%
Jetson AGX Xavier3.08 ms8.6 / 11.5 / 15.3%3.25 ms8.2 / 13.0 / 18.2%
RTX 30900.654 ms6.9 / 9.0 / 9.5%0.764 ms6.9 / 8.7 / 9.5%
RTX 6000 Ada0.458 ms7.6 / 7.8 / 7.8%0.538 ms6.1 / 7.6 / 7.5%

We additionally measure latency overhead for the 6-exit MBV3 model used in the FLOPs comparison above, following the same procedure. Similarly, exits 2-6 use a common reuse-free baseline.

PlatformMBV3, 6 exits: reuse-freeOverhead at exits 2-6
Raspberry Pi 426.831 ms20.7 / 33.2 / 44.9 / 61.1 / 64.2%
Jetson AGX Xavier2.630 ms9.1 / 14.1 / 18.1 / 22.9 / 25.8%
RTX 30900.630 ms7.8 / 10.3 / 11.5 / 14.3 / 14.5%
RTX 6000 Ada0.461 ms6.0 / 8.2 / 8.4 / 10.3 / 8.7%

The latency overhead depends on both the platform and the number of exits. It remains below 15% per later subnetwork on the tested desktop GPUs, but grows more substantially on Xavier and Raspberry Pi 4 as additional exits are introduced. These percentages describe the overhead of individual subnetworks, rather than cumulative inference latency. The CPU results identify a practical limitation of all-to-all reuse at larger exit counts.

Memory. We cache each earlier exit’s stage-end features at four spatial resolutions (56x56, 28x28, 14x14, and 7x7) and its final embedding. The final exit’s features need not be retained for reuse. For one image at 224x224 in FP32, the maximum retained cache is 1.17 MiB for the four-exit MBV3 model, 1.53 MiB for the four-exit EffV2 model, and 1.89 MiB for the six-exit MBV3 model. These numbers describe the feature cache rather than total inference memory. For fixed subnetwork dimensions, the cached features from each additional exit contribute a fixed amount, resulting in linear growth with the number of exits.

These results support the practicality of our design in the evaluated settings while identifying fusion overhead as a limitation when scaling to more exits on Raspberry Pi 4 CPU. Restricting both conditioning and classifier feature reuse to a fixed number of recent exits could bound the retained cache and per-exit fusion cost, although its accuracy-efficiency trade-off remains to be evaluated. We will include these measurements and discuss this extension in the revised manuscript.

Q1: Has self-distillation been analyzed in the interference comparison of Table 1?

The DP joint model in Table 1 follows its official training recipe, including self-distillation; the DP standalone predictor and our models in that table are trained without it (Sec. 4.4 and Appendix A.2). DP’s published ablation reports that self-distillation improves first-exit accuracy from 57.3% to 61.8% on its RegNetY-400MF-based model. This suggests that self-distillation may reduce DP’s joint-versus-standalone accuracy gap, although its effect in our Table 1 configuration has not been isolated. Retraining DP without self-distillation requires 600 epochs and is not feasible within the rebuttal period.

We are also training our model with DP’s self-distillation to assess whether it can benefit from the same technique. We will report the result here when training completes.

Q2: Is the MobileNetV3 baseline in Fig. 3 the best-performing version?

Fig. 3 includes both MobileNetV3-based and RegNet-based DP models, using the upper envelopes of their published per-model accuracy-FLOPs curves. MobileNetV3-based DP provides the best accuracy-FLOPs performance among DP models over the compute range it covers, while RegNet-based DP extends coverage to higher budgets. Our comparison therefore includes DP’s strongest published operating points across these ranges.

The two DP curves in Fig. 3 are digitized from the original paper. Sec. 4.1 incorrectly suggests that we reproduced them; our DP reproductions were used only for the controlled comparisons in Fig. 4 and Tables 1 and 8. We apologize for the confusion and will correct the text.

We hope these results address the reviewer's concerns, and we would be grateful if the reviewer would consider them in the final assessment.

Q1 follow-up: self-distillation and table 1

The self-distillation experiment mentioned in our previous response has now completed. We trained our matched MobileNetV3 model using DP’s self-distillation formulation: the final exit serves as the teacher for earlier exits with detached teacher predictions, with weight 0.5 assigned to the summed KL-divergence losses and equal weights to the cross-entropy losses across all exits. All other settings are unchanged. The results are as follows:

TrainingExit 1Exit 2Exit 3Exit 4
Without self-distillation73.38877.71078.71079.172
With self-distillation73.38677.68278.50678.938

Self-distillation leaves exits 1-2 essentially unchanged and slightly reduces accuracy at exits 3-4. The first-exit gap to our standalone predictor therefore remains approximately 1.16 percentage points, preserving the accuracy-retention result in Table 1.

We retain the original cross-entropy-only objective and will include this ablation alongside the Table 1 discussion in the revision. We thank the reviewer again for raising this question.

Reviewer_NkwDOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose an architecture for early-exit processing designed to learn features that are complimentary to those of previous stages. This is done by re-introducing the input at each stage, and connecting the outputs of layer i for previous stages to the input of layer i of the current stage (c.f. figure 1). This is done to avoid reducing the effective capacity of early stages while still leveraging their results to learn complimentary features at later stages.

The approach is intuitive and effective. Results show consistent gains in the accuracy vs. flop/latency tradeoff relative to other methods (figure 3, figure 4). Furthermore they show that first-exit accuracy comes close to standalone accuracy (Table 1), and that their "stage-end" sharing is more effective than several other sharing options (Table 2,3).

Critical strengths
  • An intuitive and effective approach to early exit processing, that to my knowledge is novel.
  • The paper is well written and the experiments are extensive, and even include accuracy vs. latency measurements on both GPU and Rasberry PI 4 CPU.
Critical weaknesses
  • The experiments so far investigate the framework only for stage-wise CNNs, as the authors acknowledge, but the performance of several ViT models is compared (c.f. figure 3)
Questions for authors
  • It would be interesting and informative to somewhere depict the stage termination profiles (in %) as flops/latency is traded off for accuracy, for both the proposed approach and DP.
  • Would self-distillation further improve the results?
3 comments

We thank the reviewer for the constructive feedback. We apologize for the delayed response, the additional experiments required new full training runs. We address each point below and will incorporate the new results into the revision.

W1: The framework is evaluated only on stage-wise CNNs, yet Fig. 3 includes several ViT models.

The ViT methods in Fig. 3 (DVT, CF-ViT, ThinkingViT) are published early-exit results on ImageNet in this compute range, digitized from the original papers. As in the Dynamic Perceiver paper, which compares its models with DVT and CF-ViT, we compare our models against prior works regardless of backbone family.

To test whether our framework applies to a Transformer backbone, we evaluate it on PVTv2, a hierarchical vision Transformer and measure the effect of joint training on first-exit accuracy. PVTv2 has a four-stage pyramid so our design carries over directly:

  • Each exit owns a complete PVTv2 backbone that processes the original image (Sec. 3.2).
  • Stage-wise conditioning follows Eqs. 2-3 (Sec. 3.3): the projection P is a token-wise linear layer, equivalent to the 1x1 convolution in our CNN models, and T is the attention branch of the first Transformer block of each stage. As in our CNN models, the fused tokens enter the residual branch while the local tokens stay on the shortcut.
  • Classifier feature reuse follows Sec. 3.4 except that each exit exports its pooled, normalized backbone output directly, without a learned embedding transformation.

We trained a 3-exit model and a standalone first-exit control with the same recipe: from scratch for 300 epochs on ImageNet, following the official PVTv2 recipe. The global batch size is changed to 384 to fit our GPUs and the learning rate is linearly scaled. We chose 3 exits instead of 4 so that results can be obtained within the rebuttal period. The standalone control has exactly the same architecture of exit 1. The following results are preliminary since we have not tuned the configuration for the multi-exit setting:

Exit 1Exit 2Exit 3
Cumulative GFLOPs1.162.363.59
Top-1 (%), joint training74.1678.1079.36
Top-1 (%), exit 1 trained standalone74.97––

Accuracy increases to 78.10% and 79.36% at exits 2 and 3, while the first exit is 0.81 points below the standalone baseline. These preliminary results demonstrate that our framework can be applied to hierarchical Transformers. We will extend this experiment and include the results in the final version.

Extending the framework to plain ViTs which have no resolution stages would require defining conditioning over groups of blocks rather than over stages, and we leave this to future work.

Q1: Stage-termination profiles as FLOPs/latency is traded for accuracy, for ours and DP.

For both methods, thresholds are calibrated with the same protocol, following MSDNet (Sec. 4.1). Under this protocol, the share of images terminating at each exit is set by the target budget and the per-exit costs. The tables below report top-1 accuracy and the percentage of validation images terminating at each exit. Achieved budgets are within 1.5% of the targets. The models considered are the matched MobileNetV3 pair that produce the curves (a) (b) (c) in Fig. 4, and the metrics are shown as ours/DP. At 12 ms on the Jetson AGX Xavier, DP runs all images to its final exit (11.7 ms).

FLOPs:

Budget (GFLOPs)Top-1 (%)Exit 1Exit 2Exit 3Exit 4
0.378.20 / 73.9460.4 / 71.125.0 / 21.010.3 / 6.24.3 / 1.8
0.479.14 / 78.3234.9 / 40.927.3 / 27.721.2 / 18.716.6 / 12.7
0.579.17 / 79.1316.7 / 19.421.3 / 22.727.2 / 26.634.8 / 31.3
0.679.16 / 79.224.0 / 4.210.0 / 10.224.7 / 24.961.3 / 60.9

Latency on the Jetson AGX Xavier:

Budget (ms)Top-1 (%)Exit 1Exit 2Exit 3Exit 4
476.78 / –77.8 / –17.5 / –3.9 / –0.9 / –
678.88 / –47.8 / –27.4 / –15.8 / –9.1 / –
879.13 / 77.0325.9 / 54.525.3 / 26.424.7 / 12.824.1 / 6.2
1079.16 / 79.1411.0 / 18.217.5 / 22.127.7 / 26.943.9 / 32.8
1279.16 / 79.211.8 / 0.06.1 / 0.020.9 / 0.071.2 / 100.0

Latency on the Raspberry Pi 4:

Budget (ms)Top-1 (%)Exit 1Exit 2Exit 3Exit 4
4074.44 / –93.8 / –5.8 / –0.4 / –0.0 / –
6078.38 / 69.7557.8 / 85.625.7 / 12.411.4 / 1.85.1 / 0.3
8079.14 / 78.0635.0 / 44.927.3 / 27.621.2 / 17.016.5 / 10.5
10079.16 / 79.1018.3 / 21.622.2 / 23.726.9 / 26.132.6 / 28.6
12079.16 / 79.237.0 / 6.913.7 / 13.626.8 / 26.852.4 / 52.7

"–": DP cannot reach the budget, because its first exit alone takes 6.1 ms on the Xavier and 55.0 ms on the Raspberry Pi 4 (Table 8).

Our advantage is most pronounced at low budgets, this is because our first exit is more accurate, and our earlier exits are also cheaper, allowing more images to proceed beyond exit 1 under the same budget. Our model also supports latency/compute budgets below DP's first-exit cost. At higher budgets, both models reach similar accuracies but ours reaches it earlier. We will add the complete profiles including the EfficientNetV2/RegNet pair on all hardware to the appendix.

Q2: Would self-distillation further improve the results?

We are training our matched MobileNetV3 model with self-distillation using DP's formulation: the last exit serves as the teacher for all earlier exits with detached teacher predictions, weight 0.5 assigned to the summed KL-divergence losses and equal weights to the cross-entropy losses across all exits. All other settings are unchanged. The experiment is still running, so we cannot yet quantify its effect on our model. In DP's ablation, self-distillation primarily improves the first exit (+4.5 points), with smaller changes at exits 2–4 (+0.2, +0.6, and -0.1 points). Since self-distillation only modifies the training objective, it is compatible with our framework. We will report the results once they become available.

We thank the reviewer again for the helpful suggestions.

Q2 followup: self-distillation results

The self-distillation experiment mentioned in our previous response has now completed. We trained our matched MobileNetV3 model using DP’s self-distillation formulation: the final exit serves as the teacher for earlier exits with detached teacher predictions, with weight 0.5 assigned to the summed KL-divergence losses and equal weights to the cross-entropy losses across all exits. All other settings are unchanged. The results are as follows:

TrainingExit 1Exit 2Exit 3Exit 4
Without self-distillation73.38877.71078.71079.172
With self-distillation73.38677.68278.50678.938

Self-distillation leaves exits 1-2 essentially unchanged and slightly reduces accuracies at exits 3-4. We therefore retain the original cross-entropy-only objective.

We thank the reviewer again for suggesting this experiment, and we will include this ablation in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.