acceptodds
Under review as a conference paper at ICLR 2027

EEL: Calibrating Early-Exit Neural Networks via Hierarchical Self-Supervised Learning

Abstract

Early-exit neural networks reduce inference latency by enabling dynamic termination at intermediate exit branches. However, standard supervised training typically prioritizes the final-layer objective, leaving shallow exits with underdeveloped representations and poor calibration. To address this, we propose a Siamese hierarchical Self-Supervised Learning (SSL) paradigm for producing early-exit neural networks, that treats intermediate exits as independent representation endpoints. By applying SSL regularization symmetrically across network depth, our framework promotes useful representations at all exits independently of task-specific classification heads. The resulting multi-branch encoder is then fine-tuned with early-exit classification heads to obtain the final early-exit network. Evaluation across convolutional and Vision Transformer backbones testing both contrastive and non-contrastive objectives demonstrates that the proposed approach improves the calibration of shallow exits, while maintaining a comparable efficiency trade-off.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
inference-adaptive DNNs, self-supervised learning, SIGReg, VICReg, early-exit DNNs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_82b1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a learning strategy named EEL, a hierarchical self-supervised learning (SSL) framework for early-exit neural networks. EEL first pretrains network's intermediate representation with SSL and then performs supervised finetuning. The authors claim that the proposed EEL improves network's internal representation quality and shallow-exit calibration, while providing competitive accuracy–latency trade-offs, faster downstream convergence and potential benefits from cross-depth alignment. Experiments are conducted across multiple CNN and ViT backbones, and datasets.

Critical strengths

EEL targets at improving the reliability of intermediate predictions, which is an important issue for early-exit neural networks. The authors evaluate several backbone architectures, datasets, and both contrastive and non-contrastive SSL objectives.

Critical weaknesses

1. Limited methodological novelty and insufficient differentiation from prior work.

EEL appears to be a relatively straightforward recombination of established SSL and early-exit training techniques, rather than a substantially new methodology. In previous works, Ladder Siamese Networks (Yoshihashi et al.) already introduced intermediate SSL objectives with branch-specific projectors. Although Ladder Siamese has been acknowledged in Appendix B, the distinction mainly relies on applying the learned representations to early-exit inference rather than introducing a fundamentally different learning mechanism. Additionally, DAISY (Lin et al.) has already explored the relationship between self-supervision and dynamic early exiting, though through a different approach.

This paper fails to convince what is new in terms of the core algorithm beyond combining existing techniques. More importantly, the authors should have a conventional SSL pretraining baseline followed by identical early-exit fine-tuning so as to make their claim stronger.

2. Inconsistent experimental evidence and potentially misleading presentation of the claimed improvements.

Even when considering EEL as an empirical contribution, the experimental results doesn't consistently support the claimed improvements. First, Fig. 1 shows improved calibration, but Fig. 29 in the appendix shows that EEL produces higher Expected Calibration Error (ECE) than the supervised baseline across all four exits on ImageNet100 with ViT-B. These two figures show the opposite results. More concerningly, the caption of Fig. 29 explicitly claims better calibration despite the opposite trend seen in the figure. Similarly, Fig. 2 in the main text and Fig. 10 in the appendix give different trend on accuracy–latency trade-offs. Although these results are obtained under different experimental settings, they clearly suggest that EEL's reported improvements are not consistent. The selection of favorable results for the main text, while important negative examples are placed in the appendix, may indicate selective presentation and misleading conclusions.

Second, the faster-convergence claim is not well justified. Although fine-tuning requires fewer epochs, the proposed framework additionally requires 300 epochs of SSL pretraining. According to Appendix C.7, the total training time is generally greater than that of the supervised baseline. Therefore, faster downstream fine-tuning should not be presented as an overall training-efficiency improvement without further explanation.

Third, the contribution of cross-depth alignment is unclear. The experiments haven't demonstrated a consistent improvement on top of the 'Standard' variant, and the authors themselves acknowledge the lack of a systematic improvement. This further weakens the significance of this component.

Finally, the overall presentation of the experimental results is not sufficient. The main text contains very few figures and quantitative tables summarizing performance across the tested settings are missing. Most results are in the long appendix. It's difficult for readers to assess the consistency and statistical significance of EEL. Given that the claimed contributions are mostly empirical, these presentation issues further weaken the credibility of the conclusions.

Reference:

Yoshihashi, R., Nishimura, S., Yonebayashi, D., Otsuka, Y., Tanaka, T., & Miyazaki, T. "Ladder Siamese Network: A Method and Insights for Multi-Level Self-Supervised Learning." ICIP, 2023.

Lin, T.-Q., Lee, H.-Y., & Tang, H. "DAISY: Data Adaptive Self-Supervised Early Exit for Speech Representation Models." INTERSPEECH 2024.

Questions for authors

1 . What is the essential methodological difference between EEL and prior hierarchical SSL methods, particularly the Ladder Siamese Networks? The authors should provide direct evidence that hierarchical SSL offers benefits that are not given by conventional final-layer SSL pretraining followed by identical multi-exit fine-tuning.

2 . The authors should explain why the caption and the line plot of Fig. 29 are contradictive. Why Fig.2 has the opposite trend as in Fig. 10? The interpretation of near-zero ECE as underconfidence in Section 5.2 also needs to be clarified.

3 . How should the faster-convergence claim be interpreted when the additional SSL pretraining cost is included?

4 . What experimental evidence supports the necessity of cross-depth alignment?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 82b1

We thank the reviewer for the feedback. We address each point below, and "initial submission" line numbers refer to the originally submitted PDF.

Question 1

EEL does not aim to introduce a new hierarchical SSL objective. Its contribution is a framework using such objectives to produce a multi-branch encoder in which every exit is a standalone endpoint, evaluated under confidence-threshold early exiting and shallow-exit calibration. As discussed in Section 2.2 and Appendix B (Table 2), none of the prior hierarchical SSL methods evaluates early-exit inference.

Ladder Siamese Networks also attach per-stage projectors, but differ in purpose and mechanism: (i) their intermediate branches are training-only auxiliaries for a better final layer, downstream tasks use final-layer features and no early exit is trained or evaluated, (ii) shallow losses are down-weighted, wi=2i−Nww_i = 2^{i-N}w for N stages, whereas EEL weights all exits equally, (iii) their BYOL-based loss is asymmetric (their Eq. 5):

$ L_i = \|q_i(y^a_i) - y^b_i\|_2^2, y^k_i = p_i(\mathrm{avgpool}(z^k_i))$

where zikz^k_i is the stage-i feature of view k ∈\in a,b\\{a,b\\}, pip_i a projector and qiq_i a predictor applied to one view only. Following BYOL, view b uses a momentum encoder and stop-gradient. EEL is symmetric, with no predictor or momentum encoder, and a decoupled regularizer (Eq. 3):

Lssli=∥gi(h1i)−gi(h2i)∥22+LregiL^{i}_{\mathrm{ssl}} = \lVert g_i(h^i_1) - g_i(h^i_2) \rVert_2^2 + L^{i}_{\mathrm{reg}}

where h1ih^i_1, h2ih^i_2 are the depth-i features of both views and gig_i the shared projector. This decoupling lets EEL work with different regularizers (Sections 5.5–5.6).

DAISY keeps a pretrained HuBERT frozen and trains a linear exit branch per layer with a self-supervised loss. In contrast, EEL does not freeze the encoder: it trains the backbone with a separate SSL objective at every exit so that each depth learns a standalone representation, and then fine-tunes the exits with classification heads.

As requested, Section 5.7 and Appendix D.12 compare against final-layer SSL pretraining followed by identical multi-exit fine-tuning. With comparable or better shallow-exit calibration, EEL routes far more samples through Exit 0 (≈59% vs. ≈13% on ResNet34/CIFAR10; ≈45% vs. none on ViT-S/Galaxy10), and final-layer SSL yields a clearly dominated Pareto frontier. So the gains come from hierarchical SSL, not SSL pretraining alone.

Question 2

We thank the reviewer for spotting the incorrect Fig. 29 caption of the initial submission. The calibration benefit does not hold in every setting. Addressing the presentation concern, the new Table 1 summarizes all backbone–dataset pairs (mean ± std over three seeds for our models), including ViT-B/ImageNet100, where Supervised has lower shallow-exit ECE. Section 5.2, the corrected caption (now Fig. 28) and the Conclusion state this explicitly. Even there, our exits route ≈43% of samples through Exit 0 (vs. 9.8%), yielding higher accuracy (64.12%/63.52% vs. 60.22%) at lower latency (≈1.64 vs. 2.37 ms).

Regarding Figs. 2 and 10 of the initial submission, we agree the efficiency trade-off varies across settings, as revised Section 5.2 discusses. Following Kubaty et al. (2025) (line 377, initial submission), we report calibration and efficiency separately rather than assuming they are always correlated.

Regarding near-zero ECE, per-exit ECE is computed only over the samples that exit at that branch to reproduce actual exit behavior. For Supervised ViT-S on Galaxy10, no prediction exceeds τ = 0.9 at Exit 0, so its per-exit ECE is undefined, not low. Table 21 now reports full-test-set ECE without thresholding: this exit matches our accuracy (77.8%) but has an ECE of 0.175 vs. 0.028, indicating the problematic calibration that prevents early exiting.

Question 3

We agree and have softened the third contribution. For a single task, SSL pretraining plus fine-tuning costs more than supervised training (Appendix C.7), as the revised Conclusion states. The benefit arises when an encoder is reused: fine-tuning needs at most 50 epochs vs. 300 for a supervised model trained from scratch. Appendix E (line 2160, initial submission) shows this on three CIFAR10 sub-tasks: 150 epochs or fewer in total for our models vs. 900 for Supervised, with comparable trade-offs.

Question 4

Alignment is an exploratory component (lines 61–62, initial submission) testing whether cross-depth alignment helps. Align gives no consistent gain in calibration or efficiency and is insensitive to λ (Appendix D.15, initial submission). Section 5.7 (lines 488–489) now frames this properly as Standard suffices on its own (see also Table 22).

Reviewer_Ezu4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduced a self-supervised Siamese structure to do early termination under a “better” metric that considers training different intermediate representations. Using this multi-branch encoder provides an independent stopping criterion from the task-specific metrics. Having these “early stopped” representations, one can later finetune with a specific classification head. The authors have provided results for different ResNet and ViT structures with different early stop strategies to show the performance of the proposed method.

Critical strengths
  1. The writing of the paper is relatively clear, the paper logic is easy to follow and understand, despite some typos and minor errors.
  2. This paper tries to use the self-supervised Siamese structure to find a better way to do early termination of network training by balancing the performance and efficiency tradeoff, which could be useful for many tasks.
Critical weaknesses
  1. This paper lacks a fair comparison to the previous methods. Although the authors claimed that this is the first paper that used the multi-layer encoder for early-exit analysis, comparisons against related work are expected. The authors have analyzed the issues that exist in the previous methods, but if no direct comparisons were made, these arguments would not stand.
  2. The authors claimed that their proposed method is backbone-agnostic and regularization-agnostic. However, the current experiments did not show convincing results. The claim in the paper is also too general, maybe the authors could constrain the discussion to shallow, relatively small network structures.
  3. Again, on different network architectures, such as ResNet and ViT as shown in all the figure results, the relationship between the proposed method, whether with or without cross-depth alignment, is not very clear. I presume that the transformer structure and convolution structure, or other network structures should not be discussed all together. A better analysis strategy should be designed to better show the advantage of the proposed method. And the backbone-agnostic claim should not be made too general here.
  4. The authors have provided many results in the main paper and the supplementary material. However, no additional information is provided in those experiments. I encourage the authors to provide more informative experimental results, such as ablation studies, a wider range of architectures, etc.
Questions for authors

Please see my detailed comments above. My main concern is that the authors have claimed a general backbone-agnostic, regularization-agnostic early-exit strategy for network training, but the current version of the paper does not back up those arguments. I only saw many repeated experimental results, but no concrete comparisons to previous methods and no thorough ablation studies (only lambda choice). I encourage the authors to reorganize the experiments, include more analysis, and resubmit.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Ezu4

We thank the reviewer for the feedback. We address each point below, and "initial submission" line numbers refer to the originally submitted PDF.

First, we would like to clarify one point in the summary. Early-exit neural networks reduce inference latency by enabling dynamic termination at intermediate exit branches (lines 11-12 and 29-32, initial submission). Such models are typically trained with the joint objective described in Section 3.2. Our method does not stop training early, and this is not discussed in our submission. It is distinct from early stopping, which terminates training.

  1. Most current approaches to early-exit networks for image classification use supervised joint optimization with cross-entropy losses at each exit, which is why we chose this as our main baseline. The hierarchical SSL methods in our related work (Appendix B, Table 2) do not produce multi-branch encoders or explore early exiting. Instead, they use representations across depth to improve final-layer performance, so they cannot be compared directly under confidence-threshold exiting. We therefore validate the method with controls that isolate our contribution (Section 5.7). The first is final-layer SSL with identical fine-tuning (Appendix D.12), which shows that the gains come from the per-branch objectives rather than from SSL pretraining alone. The second is supervised training with self-distillation between exits (Appendix D.13), and the third is the cross-depth alignment ablation. Since our method changes how the encoder is pretrained rather than the exit rule, exit-policy and post-hoc calibration techniques are complementary to it (Appendix C.8). We already show this for entropy-based, patience-based and UCB-based exiting (Appendix D.9).

  2. We agree that the claim was phrased too generally. As discussed in Section 2.1, the dominant early-exit backbones in literature are CNN-based and Transformer-based, and our claim is meant in this context. We evaluate four backbones, namely ResNet18, ResNet34, ViT-S and ViT-B. We have rephrased the claim accordingly at line 51 and in the contributions list, and we now state the backbone scale as a limitation in Section 6. Similarly, "regularization-agnostic" refers to the three objectives we test. Two of them are non-contrastive (VICReg and SIGReg) and one is contrastive (NT-Xent), and all three give comparable trade-offs (Appendix D.6 and D.7).

  3. Results previously shown only as plots are now also reported numerically for each backbone and dataset pair (Table 1 and Appendix D.8), so the behaviour of CNNs and ViTs can be read separately. Section 5.2 also discusses backbone-specific exceptions, such as the weaker ViT-S frontier on CIFAR10 and the calibration result of ViT-B on ImageNet100. Cross-depth alignment is an ablation that tests whether exploiting the hierarchy yields further gains (lines 60-62, the initial submission). The newly added Section 5.7 states this explicitly. Align gives no consistent improvement, so the Standard approach is sufficient on its own.

  4. We agree that the purpose of each experiment could be clearer, and we summarize it below. The experiments cover 4 backbones, 4 training datasets plus 2 distribution-shift test sets, and 3 seeds per configuration of our method. Beyond the lambda sensitivity, we ablate the regularizer across three objectives, the placement of the SSL objective (per-branch versus final-layer) and the exit policy (maximum probability, entropy, patience and UCB). We also compare against three baselines, which are joint supervised optimization, self-distillation and final-layer SSL. The ECE and efficiency trade-off section (5.2) shows that our method reduces shallow-exit ECE while achieving comparable accuracy and latency trade-offs. The distribution shift section (5.3) tests whether these trade-offs hold under distribution shift, compared to the baseline. The representation quality and alignment section (5.4) examines the learned representations through CKA and linear probes to explain where the gains come from. The SIGReg versus VICReg section (5.5) tests whether the choice of non-contrastive regularizer matters, and finds that the trade-offs are comparable. The NT-Xent section (5.6) tests a contrastive objective instead, which also works but collapses at small batch sizes. Finally, Section 5.7 covers the additional baselines, adaptive exit policies, exit behaviour and lambda sensitivity.

Reviewer_wQUYOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2(−1)
Presentation
3
Soundness
2(−1)
Read review
Paper summary

This paper investigates whether self-supervised pretraining at multiple network depths can improve early-exit networks. Each exit of a shared backbone receives its own self-supervised objective, using VICReg or SIGReg, before the network is fine-tuned with supervised classification heads. The authors also explore an optional cross-depth alignment objective. Experiments cover ResNet18/34 and ViT-S/B across six datasets. The reported results generally show improved calibration at shallow exits, while accuracy–latency trade-offs are often comparable to or better than joint supervised training. Linear probes, kNN, CKA, and t-SNE provide additional analyses of intermediate representations. Cross-depth alignment does not yield consistent additional improvements.

Critical strengths

Clear motivation: the method encourages each exit to learn features useful for independent prediction. Broad experiments across different backbones, datasets, SSL objectives, and distribution shifts. Linear-probe and kNN results support better intermediate representations on ViT-B/ImageNet-100, with complementary CKA analysis. Lower shallow-exit ECE than joint supervised training in several settings.

Critical weaknesses

I think the connection between the two-stage training and the inference results needs to be clearer. Better intermediate features and lower ECE do not always lead to better exit decisions, so I am not fully convinced about when the proposed training actually improves efficiency.

I also have concerns about pretraining stability. The NT-Xent results show problems with small batches. It would help to explain how these pretraining limitations affect the later fine-tuning and the final early-exit performance.

The method seems more useful when fast inference is the main concern, since it requires additional training cost. For a model that will be used many times, this may be worthwhile, but the paper could make clearer when the inference savings justify the extra training.

Questions for authors

Would the method show a larger advantage with a different rule for deciding when to exit? Also, are the benefits consistent across different model architectures, or do certain types of models benefit more than others?

2 comments
AuthorsOpenReview ↗

Response to Reviewer wQUY

We thank the reviewer for the feedback. We address each point below and “initial submission” line numbers refer to the originally submitted PDF.

  1. We agree that better calibration does not necessarily lead to better exit decisions. Our manuscript already notes this, building on prior work including Kubaty et al. (line 377, initial submission), and therefore evaluates ECE and Pareto frontiers separately (lines 375-377, initial submission). Our results confirm that the two are decoupled in both directions. ViT-S/CIFAR10 is better calibrated but has a weaker frontier. ViT-B/ImageNet100 is less calibrated at shallow exits yet reaches higher accuracy at lower latency. Nevertheless, the proposed method affects both axes. To communicate this more clearly, we introduce Table 1, which reports ECE, exit fractions, accuracy and latency jointly. We also explain how poor calibration can prevent early exiting in the baselines (line 398). Where a baseline routes almost no samples to a shallow exit, its per-exit ECE is undefined. For these cases we add Table 21 (line 2432), which evaluates every exit on the full test set rather than on the routed fraction. For ViT-S/Galaxy10, the supervised Exit 0 matches our accuracy (77.8%) but is far worse calibrated. None of its predictions exceed τ = 0.9, so no samples exit there, whereas ours route 45–47% of samples through Exit 0.

  2. We chose joint optimization because it is the most widely used training regime for early-exit networks [1]. We agree that a growing body of work explores other regimes, and we make three additions:

  • Self-distillation baseline (Appendix D.13). We add a stronger baseline that combines joint optimization with self-distillation between exits. It reaches a comparable frontier, but our shallow exits remain better calibrated (Section 5.7).
  • Final-layer SSL control (Appendix D.12). This control applies SSL only at the final layer, followed by identical fine-tuning. It routes far fewer samples through Exit 0 (about 13% vs. 59% on ResNet34/CIFAR10, and none on ViT-S/Galaxy10) and yields a dominated frontier. This experiment confirms that the gains come from treating each branch as a standalone endpoint, not from SSL pretraining alone.
  • We now state that future work should compare against hierarchical/sequential training too (Sepehri et al., TNNLS 2024) (line 524).
  1. We agree that alignment is an exploratory component on top of our main method, testing whether aligning representations across depth helps (lines 61-62, initial submission). Our results show that it does not: Align gives no consistent improvement in calibration or efficiency and is insensitive to λ (Appendix D.15). The Standard method is therefore sufficient on its own. Section 5.7 (lines 488-489) now frames this explicitly as a negative ablation result.

  2. Exit criteria act only at inference time, so the trained models are unchanged. We agree, however, that exit distributions could differ substantially under adaptive criteria. We therefore evaluated entropy, patience, and threshold selection as a multi-armed bandit problem (UCB1). We ran these on the two settings suggested, ResNet34/CIFAR10 and ViT-S/Galaxy10 (Section 5.7, Appendix D.9). The advantage persists, and in several cases grows:

  • Entropy: our encoders send more samples through Exit 0 (60.0% vs. 36.0%; 47.4% vs. 0%) at 31–37% lower latency in comparison to the Supervised baseline.
  • UCB: Exit-0 ECE is lower (0.021 vs. 0.069; 0.028 vs. 0.163) at lower latency in comparison to the Supervised baseline.
  • Patience: this criterion forbids exiting at Exit 0. Latency is comparable, and Exit 1 remains better calibrated in comparison to the Supervised baseline.
  1. We add evaluations of exit behavior across Standard, Align and Supervised (Appendix D.11). As the threshold increases, the supervised baselines shift samples to deeper branches much faster than our models (Figure 2). Hard classes are on average predicted by deeper exits under all three methods. This behavior is consistent with the representation analysis. On ViT-B/ImageNet100, our features already reach near-final linear-probe accuracy at Exit 2 (Table 22), and our models route about 68% of samples through Exits 0–1 vs. 21% for Supervised (Table 1).

  2. Table 1 now reports mean ± std over three seeds for all models produced by the proposed method. In certain cases, the shallow-exit ECE gaps relative to Supervised are well beyond the seed std (e.g., ResNet34/CIFAR10 Exit 0: 0.013 ± 0.001 vs. 0.039). By contrast, the differences between Align and Standard mostly fall within one std.

[1] Haseena Rahmath P, Vishal Srivastava, Kuldeep Chaurasia, Roberto G. Pacheco, and Rodrigo S. Couto. 2024. Early-Exit Deep Neural Network - A Comprehensive Survey. ACM Comput. Surv. 57, 3, Article 75 (March 2025)

AuthorsOpenReview ↗

Response to Revised Review

We thank the reviewer for the updated review and for noting that the method lowers shallow-exit ECE with accuracy-latency trade-offs comparable to or better than joint supervised training. This reply builds on our first response (12 Nov), which addressed the six original weaknesses with new experiments (Appendices D.9 - D.13, Table 1). Line numbers marked “initial submission” refer to the originally submitted PDF.

  1. Our claims are better-calibrated shallow exits and an accuracy-latency trade-off comparable to, or better than, Supervised (contributions, lines 58–69). We do not claim that the first causes the second. As in our first response, we agree that lower ECE alone need not produce better exits (line 377, initial submission), and we showed there that the two are decoupled in both directions, which is why we report them separately. Table 1 shows that the calibration gain does not cost efficiency in six of seven settings: our latency at τ = 0.9 is 17-44% lower, with accuracy within 0.4 points on CIFAR10 and higher on Galaxy10 and ImageNet100. In the seventh (ViT-S/CIFAR10), accuracy is comparable at higher latency, as Supervised already routes 71.8% through Exit 0. One way calibration affects routing is visible on ViT-S/Galaxy10. Supervised Exit 0 matches our full-test-set accuracy (77.8%) but is poorly calibrated (ECE 0.175 vs. 0.028), so none of its predictions exceed τ = 0.9 and it routes 0% of samples, vs. 45% for ours (Table 21, from our first response). This routing gap widens with τ, matching the exit-distribution analysis in our first response (Fig. 2, Appendix D.11), and the final-layer SSL control from that response (Appendix D.12) shows that the hierarchical objective drives this routing.
  2. Both the initial submission and the revision cover this. Section 2.2 (lines 103-106) explains that contrastive objectives rely on negative samples within each batch, and Section 5.6 (lines 470–474) attributes the NT-Xent collapse at batch size 32 to too few samples per batch, recommending non-contrastive regularizers for low-resource training. The effect on the final early-exit model is also reported. Appendix D.7 shows the accuracy-latency frontiers after fine-tuning, where NT-Xent at batch 512 is comparable to VICReg (Fig. 31) and the collapsed batch-32 encoders stay far below it (Fig. 32), so fine-tuning does not recover a collapsed encoder. The limitation is therefore specific to contrastive objectives, not to our framework. Our default regularizers (VICReg, SIGReg) need no negatives. Collapse is also detectable before fine-tuning via the per-exit linear probes (Section 4.2).
  3. The paper addresses when the extra training pays off. The Conclusion (lines 527–528) states that, for a single task, our method costs more than supervised training and pays off when an encoder is reused. Appendix E shows this: one encoder adapted to three CIFAR10 sub-tasks took ≤150 fine-tuning epochs in total, vs. 900 for three supervised models, at comparable trade-offs (see also our response to Reviewer 82b1, Q3). Each new task costs ≤50 epochs instead of 300. Whether the latency savings justify the extra training depends on how many inferences a deployed model serves, which varies by application. Hence we report the measured training cost of every phase (Appendix C.7) alongside the per-sample latency and accuracy of every setting on the same GPU (Table 1, Pareto frontiers), so practitioners can weigh one against the other for their workload.
  • Q1. We ran this on ResNet34/CIFAR10 and ViT-S/Galaxy10, the two settings suggested in the original review (Section 5.7, Appendix D.9, Tables 18–20). Across all four criteria (max-probability, entropy, UCB, patience) and both settings, our shallow exits are better calibrated than Supervised, so the calibration benefit does not depend on the exit rule. Latency is also lower under every rule that lets a shallow exit fire on its own confidence: 31–37% under entropy (vs. 26–36% under max-probability), with 60.0% vs. 36.0% and 47.4% vs. 0% of samples at Exit 0, and 18–31% under UCB, where Exit-0 ECE is 3–6× lower (0.021 vs. 0.069; 0.028 vs. 0.163). Under patience = 2, which by design never exits at Exit 0, latency is comparable and Exit 1 remains better calibrated (0.006 vs. 0.080; 0.048 vs. 0.169). Across all criteria, accuracy changes by −0.7 to +1.2 points vs. Supervised.
  • Q2. The calibration benefit holds across ResNets and ViTs wherever Supervised exits early enough to compare, with the one exception of ViT-B/ImageNet100 (line 523). The efficiency benefit is largest for ViTs on Galaxy10 and ImageNet100, where supervised ViTs route only 0–9.8% of samples through Exit 0. On ViT-S/Galaxy10, latency is 34–36% lower. On ViT-B/Galaxy10, the supervised model routes no samples through Exits 0-1 and reaches 49.2% accuracy, vs. 81.4% for ours. ResNets show smaller but consistent gains: 17–28% lower latency and 1.5–3× lower Exit-0 ECE.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.