- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This work proposes an early-exit architecture targeting two limitations of prior works: (1) early exits are backbone prefixes, so their capacity is tied to their position, and (2) joint training causes early-late interference. In this paper, each exit gets its own complete, full-depth subnetwork, executed sequentially. Later subnetworks are conditioned on earlier subnetworks' stage-end features, and each classifier also takes the final embeddings from all earlier exits. ImageNet experiments with MobileNetV3/EfficientNetV2 show improved accuracy-efficiency trade-offs, mainly at low budgets, with latency measured on a Raspberry Pi 4 and a Jetson Xavier.
Critical strengths
- The paper addresses a practical problem and correctly identifies limitations of prior early-exit designs.
- Low-budget results are strong (exit 1: 73.4% at 207 MFLOPs vs. 64.9% at 225 MFLOPs for DP).
Critical weaknesses
- The proposed design has its own weakness. Reaching exit k requires executing k full-depth subnetworks in sequence, so the critical path grows roughly as k × depth. This cost is hidden on CPUs but becomes significant on GPUs. Table 8 already hints at it: at near-identical final-exit FLOPs (696 vs. 683 M), the Raspberry Pi 4 latency is on par with DP (148.9 vs. 147.7 ms), while the Xavier GPU latency is worse (13.4 vs. 11.7 ms). This problem is likely to be more pronounced on desktop or server GPUs with greater parallelism. Evaluation on such platforms is needed to demonstrate the method's actual latency.
- The claim that DP leaves early-late interference unaddressed does not seem appropriate, since DP's main goal was to address this problem by separating the feature and classification branches. Furthermore, this paper does not clearly address the interference problem either. Earlier subnetworks still receive gradients from all later losses, as acknowledged in Sec. 3.5. The evidence offered in Table 1 is an accuracy gap between standalone and joint training, which can arise from many factors and does not by itself demonstrate interference or its reduction.
- DP's Forward Knowledge Transfer (FKT) module already forwards earlier exits' outputs to later classifiers, although it forwards predictions rather than embeddings. An ablation comparing forwarding predictions with forwarding embeddings is needed to show that the proposed classifier feature reuse offers a meaningful improvement.
Questions for authors
- How do the reproduced DP models' per-exit FLOPs and accuracy compare with the original DP paper? Which baseline does the +6.3 annotation in Fig. 3 refer to, and does DP have measured points near 0.15 GFLOPs?
- In Table 3, why is z_2 so weak for the classifier-only and matched-depth variants (54.5% and 60.7%) despite coming from a full-depth subnetwork? Were they trained to convergence?
4 comments
W2: DP already addresses interference, and Table 1 does not by itself establish reduced interference.
We agree that addressing interference is a central contribution of DP, as acknowledged at L82-84 in the manuscript. Our phrase “the underlying coupling remains” refers to residual interaction, but we recognize that the contrast could obscure our acknowledgment of DP’s contribution. Table 1 examines a complementary question: how well the first exit preserves its accuracy under joint training, whereas DP’s motivation emphasizes protecting later-exit performance from early supervision. We will clarify this distinction in the revision.
Our architectural contribution is the conditional complementary-learning design, which replaces the shallow, position-constrained exit predictors of prefix-based approaches with thin, full-depth subnetworks. Each subnetwork has direct image access, while stage-wise conditioning and embedding reuse allow later exits to build on earlier representations. Earlier features are therefore no longer the sole source of information for later computation, relaxing the requirement that they preserve everything needed downstream. Although later losses still update earlier subnetworks through reuse paths, as acknowledged in Sec. 3.5, the architectural objective is to provide strong early predictors without restricting their depth to their exit position, while allowing later exits to benefit from their representations.
Table 1 evaluates this accuracy retention: our first exit loses 1.16 percentage points under joint training, compared with 3.05 for DP, under the evaluated training recipes. This supports better preservation of first-exit accuracy despite the remaining cross-exit interaction. We agree, however, that the accuracy gap alone does not isolate interference from other effects of joint optimization. We will therefore revise causal wording such as “confirms that the gains arise from … reduced interference” and instead present better first-exit accuracy retention under joint training as an empirical benefit of the proposed architecture.
W3: DP’s FKT already forwards earlier predictions; an ablation comparing prediction and embedding reuse is needed.
We agree that DP’s FKT already enables prediction-level reuse, and our reproduced DP models include this module. Following the reviewer’s suggestion, we compared three classifier-reuse variants on our matched MobileNetV3 model, keeping stage-wise conditioning and the training recipe unchanged. Their compute differs by less than 1%.
The prediction-reuse variant follows FKT’s linear projection, BN, and ReLU transformation before concatenation with the current embedding. It uses predictions from all earlier exits, matching the history available to our embedding-reuse variant.
| Classifier input | Exit 1 | Exit 2 | Exit 3 | Exit 4 |
|---|---|---|---|---|
| No classifier reuse | 74.26 | 76.61 | 77.76 | 78.19 |
| FKT-style prediction reuse | 74.64 | 77.30 | 78.34 | 78.72 |
| Embedding reuse (ours) | 73.39 | 77.70 | 78.70 | 79.17 |
Values are ImageNet top-1 accuracy (%). Embedding reuse improves every later exit by 0.36-0.45 percentage points over prediction reuse, while prediction reuse better preserves first-exit accuracy. We also calibrate exit thresholds for each variant using the same budgeted-classification protocol. Across the approximately 0.21-0.70 GFLOPs range, prediction reuse leads below about 0.26 GFLOPs, while embedding reuse leads over the remaining higher-budget range. We therefore select embedding reuse for its stronger later-exit performance and advantage across most of the evaluated budget range.
We will add this ablation and explicitly discuss DP’s FKT in the revised manuscript.
Q1: How do the reproduced DP models compare with the published results? What does +6.3 in Fig. 3 refer to, and does DP have results near 0.15 GFLOPs?
First, we clarify the source of the DP results: Fig. 3 uses curves digitized from the DP paper, while Fig. 4 and Tables 1 and 8 use our reproductions of MobileNetV3-based model 4 and RegNet-based model 4. Sec. 4.1 does not make this distinction clear, and we will correct it.
Reproduction details and agreement. Published curves alone are insufficient for our controlled comparisons: calibrating exit thresholds requires per-exit predictions across the calibration dataset, evaluating budgeted accuracy requires the corresponding validation predictions, and hardware measurements require executable models. The published family plots also expose only part of each target model’s curve. Checkpoints for these two configurations were not publicly available; we requested them from the authors via GitHub and email but did not receive a response. We therefore reproduced both models using DP’s official implementation.
We use DP’s official implementation and the model configurations in its Appendix Tables 7 and 6, respectively. Training follows the published recipes, including self-distillation and 600 epochs for MobileNetV3 and 300 for RegNet. We use a global batch size of 1024, distributed across four GPUs with 256 images each rather than the repository’s eight GPUs with 128 each; thus, the global batch is unchanged, but the per-device BatchNorm batch size differs. We enable AMP for both the DP reproductions and our own models.
Our reproduced per-exit FLOPs and accuracies are reported in Table 8, with FLOPs calculated using the same counter as our models. DP does not provide a complete per-exit breakdown for these exact configurations, so we compare their budgeted accuracy-FLOPs curves with the identifiable model-4 segments in the published family plots. Over the available segments, our MobileNetV3 reproduction is approximately 0.1-0.5 percentage points below the published curve at 0.34-0.64 GFLOPs. For RegNet, the difference is approximately -0.21 to +0.12 percentage points at 0.98-1.42 GFLOPs (reproduced minus published). These comparisons use interpolation within the available digitized ranges.
The +6.3 annotation. This compares our model-family curve with DP’s published MobileNetV3 family curve at approximately 0.143 GFLOPs: 74.3% versus approximately 68.0% accuracy. DP therefore has a published curve extending to this budget; this is its leftmost available plotted point, not an identified first-exit result. Our value comes from the budgeted-classification evaluation calibrated using the protocol in Sec. 4.1. The +6.3 annotation in Fig. 3 compares model families, whereas the +10.2 annotation in Fig. 4 compares individual models with approximately matched first- and final-exit FLOPs.
Q2: Why is z_2 weak in the classifier-only and matched-depth variants? Were they trained to convergence?
All variants were trained for 300 epochs with the same cosine schedule and training recipe as our full model. Their late-training trajectories stabilize alongside the converged full model, without narrowing the accuracy gap. The linear and MLP probes also converge. These observations suggest that the weak standalone z_2 results are not due to incomplete training.
During training, z_2 is not supervised for classification on its own. The exit 2 classifier receives [z_1,z_2], and its loss supervises their joint prediction. Thus, a full-depth subnetwork need not produce an independently strong classifier representation. Indeed, the native exit 2 accuracies are 76.16% and 76.30% for the classifier-only and matched-depth variants, respectively, despite their low standalone z_2 probe accuracies.
The variants differ in where and which earlier features are available. Classifier-only reuse supplies earlier embeddings to the classifier but does not condition the feature-extraction body. Matched-depth conditioning supplies earlier features at corresponding depths in the body. Our stage-end conditioning instead uses the mature representations at each reused resolution in the body. Because the earlier subnetwork has already finished the stage, these features can guide the later subnetwork from the beginning of its corresponding stage. The aim is to let later extraction build on completed earlier processing and learn features that improve prediction alongside the retained embeddings. Table 3 supports this rationale: stage-end conditioning produces both the strongest standalone z_2 and the largest gain from combining z_1 and z_2. We will clarify this interpretation and the convergence in the revised manuscript.
We thank the reviewer again for the thoughtful feedback. We hope our responses address the concerns and would be grateful for their consideration in the final assessment. We would be happy to clarify any remaining questions.
W2 follow-up: direct measurement of early-late gradient conflict
To complement the accuracy retention comparison in Table 1, we have directly measured gradient alignment between the first and later exit objectives using the same matched four-exit MobileNetV3 checkpoints. We evaluate both models on the same 100 disjoint ImageNet validation batches, each containing 128 randomly shuffled images, with deterministic preprocessing in FP32 evaluation mode. For each batch, we compute
g_1 = grad_{theta_shared} (L_1)
g_late = grad_{theta_shared} (L_2 + L_3 + L_4)
where {theta_shared} includes parameter tensors reached by both objectives within each model. We concatenate their gradients and compute the cosine similarity. A batch is considered conflicting when this cosine is negative.
"Common CE" applies equally weighted hard-label cross-entropy loss. We additionally evaluate DP with its native exit weights and self-distillation loss.
| Model and measurement objective | Mean gradient cosine | Conflicting batches |
|---|---|---|
| Ours, common CE | 0.687 | 0/100 |
| DP, common CE | 0.292 | 21/100 |
| DP, native weights + KD | 0.245 | 26/100 |
Under common CE, our model shows substantially higher gradient alignment, with no conflicting batches observed. This pattern also holds for the individual later exits: mean exit 1 versus exit 2/3/4 cosines are 0.738/0.606/0.570 for ours and 0.265/0.321/0.203 for DP. At the evaluated checkpoints, later-exit gradients are better aligned with the first-exit objective in our model under the stated measurement conditions. This provides gradient-level evidence consistent with the better first-exit accuracy retention in Table 1. We will include these results and the measurement details in the revision.
We thank the reviewer for motivating this analysis and hope it addresses the concerns.
We thank the reviewer for the detailed review and constructive suggestions. We apologize for the delayed response, as the additional experiments required several new full training runs. Below, we provide new GPU measurements and analysis, clarify the DP comparisons, and identify claims that need more precise wording. We will incorporate these results and clarifications into the revised manuscript.
W1: Sequential full-depth subnetworks may incur greater latency on GPUs; evaluation on desktop or server GPUs is needed.
Following the reviewer’s suggestion, we evaluated both matched model pairs on an RTX 3090 and an RTX 6000 Ada. The results confirm a final-exit latency penalty, but also show that our accuracy-latency advantage persists at low and intermediate budgets.
We use TensorRT 11.4, FP32, batch size one, identical build settings for both methods, and the timing protocol of Appendix A.3. Cumulative GPU-compute latencies (ms) through each exit, obtained by summing measured per-exit times, are:
Our first exits are around 1.6-2.2x faster than DP’s across the two matched pairs and GPUs. However, when every image reaches the final exit, our MBV3 model is around 1.4-4.6% slower than MBV3-based DP, and our EffV2 model is 21.5-27.1% slower than RegNet-based DP. The latency advantage therefore depends on where inference terminates.
Practical efficiency also depends on average inference latency across inputs. We therefore evaluate accuracy at matched average-latency budgets, using the measured GPU latencies above as exit costs and the same threshold-calibration procedure used in our Raspberry Pi 4 and Xavier evaluations. Each cell reports top-1 accuracy (%) as ours / DP, with the average-latency budget in parentheses. Average costs are within 0.011 ms of the stated budgets. “–” indicates that DP’s first exit exceeds the budget. The results are shown below:
Our models already achieve strong accuracy at budgets below the corresponding DP models’ minimum inference latencies. They remain more accurate at DP’s first-exit latency and the intermediate budgets, and achieve comparable accuracy at DP’s final-exit latency.
To summary, our models achieve improved accuracy at low and intermediate average-latency budgets, with higher latency when inference reaches the final exit. We will add these results and clarify the trade-off in the revised manuscript.