- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a learning strategy named EEL, a hierarchical self-supervised learning (SSL) framework for early-exit neural networks. EEL first pretrains network's intermediate representation with SSL and then performs supervised finetuning. The authors claim that the proposed EEL improves network's internal representation quality and shallow-exit calibration, while providing competitive accuracy–latency trade-offs, faster downstream convergence and potential benefits from cross-depth alignment. Experiments are conducted across multiple CNN and ViT backbones, and datasets.
Critical strengths
EEL targets at improving the reliability of intermediate predictions, which is an important issue for early-exit neural networks. The authors evaluate several backbone architectures, datasets, and both contrastive and non-contrastive SSL objectives.
Critical weaknesses
1. Limited methodological novelty and insufficient differentiation from prior work.
EEL appears to be a relatively straightforward recombination of established SSL and early-exit training techniques, rather than a substantially new methodology. In previous works, Ladder Siamese Networks (Yoshihashi et al.) already introduced intermediate SSL objectives with branch-specific projectors. Although Ladder Siamese has been acknowledged in Appendix B, the distinction mainly relies on applying the learned representations to early-exit inference rather than introducing a fundamentally different learning mechanism. Additionally, DAISY (Lin et al.) has already explored the relationship between self-supervision and dynamic early exiting, though through a different approach.
This paper fails to convince what is new in terms of the core algorithm beyond combining existing techniques. More importantly, the authors should have a conventional SSL pretraining baseline followed by identical early-exit fine-tuning so as to make their claim stronger.
2. Inconsistent experimental evidence and potentially misleading presentation of the claimed improvements.
Even when considering EEL as an empirical contribution, the experimental results doesn't consistently support the claimed improvements. First, Fig. 1 shows improved calibration, but Fig. 29 in the appendix shows that EEL produces higher Expected Calibration Error (ECE) than the supervised baseline across all four exits on ImageNet100 with ViT-B. These two figures show the opposite results. More concerningly, the caption of Fig. 29 explicitly claims better calibration despite the opposite trend seen in the figure. Similarly, Fig. 2 in the main text and Fig. 10 in the appendix give different trend on accuracy–latency trade-offs. Although these results are obtained under different experimental settings, they clearly suggest that EEL's reported improvements are not consistent. The selection of favorable results for the main text, while important negative examples are placed in the appendix, may indicate selective presentation and misleading conclusions.
Second, the faster-convergence claim is not well justified. Although fine-tuning requires fewer epochs, the proposed framework additionally requires 300 epochs of SSL pretraining. According to Appendix C.7, the total training time is generally greater than that of the supervised baseline. Therefore, faster downstream fine-tuning should not be presented as an overall training-efficiency improvement without further explanation.
Third, the contribution of cross-depth alignment is unclear. The experiments haven't demonstrated a consistent improvement on top of the 'Standard' variant, and the authors themselves acknowledge the lack of a systematic improvement. This further weakens the significance of this component.
Finally, the overall presentation of the experimental results is not sufficient. The main text contains very few figures and quantitative tables summarizing performance across the tested settings are missing. Most results are in the long appendix. It's difficult for readers to assess the consistency and statistical significance of EEL. Given that the claimed contributions are mostly empirical, these presentation issues further weaken the credibility of the conclusions.
Reference:
Yoshihashi, R., Nishimura, S., Yonebayashi, D., Otsuka, Y., Tanaka, T., & Miyazaki, T. "Ladder Siamese Network: A Method and Insights for Multi-Level Self-Supervised Learning." ICIP, 2023.
Lin, T.-Q., Lee, H.-Y., & Tang, H. "DAISY: Data Adaptive Self-Supervised Early Exit for Speech Representation Models." INTERSPEECH 2024.
Questions for authors
1 . What is the essential methodological difference between EEL and prior hierarchical SSL methods, particularly the Ladder Siamese Networks? The authors should provide direct evidence that hierarchical SSL offers benefits that are not given by conventional final-layer SSL pretraining followed by identical multi-exit fine-tuning.
2 . The authors should explain why the caption and the line plot of Fig. 29 are contradictive. Why Fig.2 has the opposite trend as in Fig. 10? The interpretation of near-zero ECE as underconfidence in Section 5.2 also needs to be clarified.
3 . How should the faster-convergence claim be interpreted when the additional SSL pretraining cost is included?
4 . What experimental evidence supports the necessity of cross-depth alignment?
Response to Reviewer 82b1
We thank the reviewer for the feedback. We address each point below, and "initial submission" line numbers refer to the originally submitted PDF.
Question 1
EEL does not aim to introduce a new hierarchical SSL objective. Its contribution is a framework using such objectives to produce a multi-branch encoder in which every exit is a standalone endpoint, evaluated under confidence-threshold early exiting and shallow-exit calibration. As discussed in Section 2.2 and Appendix B (Table 2), none of the prior hierarchical SSL methods evaluates early-exit inference.
Ladder Siamese Networks also attach per-stage projectors, but differ in purpose and mechanism: (i) their intermediate branches are training-only auxiliaries for a better final layer, downstream tasks use final-layer features and no early exit is trained or evaluated, (ii) shallow losses are down-weighted, wi=2i−Nw for N stages, whereas EEL weights all exits equally, (iii) their BYOL-based loss is asymmetric (their Eq. 5):
$ L_i = \|q_i(y^a_i) - y^b_i\|_2^2, y^k_i = p_i(\mathrm{avgpool}(z^k_i))$
where zik is the stage-i feature of view k ∈ a,b, pi a projector and qi a predictor applied to one view only. Following BYOL, view b uses a momentum encoder and stop-gradient. EEL is symmetric, with no predictor or momentum encoder, and a decoupled regularizer (Eq. 3):
Lssli=∥gi(h1i)−gi(h2i)∥22+Lregi
where h1i, h2i are the depth-i features of both views and gi the shared projector. This decoupling lets EEL work with different regularizers (Sections 5.5–5.6).
DAISY keeps a pretrained HuBERT frozen and trains a linear exit branch per layer with a self-supervised loss. In contrast, EEL does not freeze the encoder: it trains the backbone with a separate SSL objective at every exit so that each depth learns a standalone representation, and then fine-tunes the exits with classification heads.
As requested, Section 5.7 and Appendix D.12 compare against final-layer SSL pretraining followed by identical multi-exit fine-tuning. With comparable or better shallow-exit calibration, EEL routes far more samples through Exit 0 (≈59% vs. ≈13% on ResNet34/CIFAR10; ≈45% vs. none on ViT-S/Galaxy10), and final-layer SSL yields a clearly dominated Pareto frontier. So the gains come from hierarchical SSL, not SSL pretraining alone.
Question 2
We thank the reviewer for spotting the incorrect Fig. 29 caption of the initial submission. The calibration benefit does not hold in every setting. Addressing the presentation concern, the new Table 1 summarizes all backbone–dataset pairs (mean ± std over three seeds for our models), including ViT-B/ImageNet100, where Supervised has lower shallow-exit ECE. Section 5.2, the corrected caption (now Fig. 28) and the Conclusion state this explicitly. Even there, our exits route ≈43% of samples through Exit 0 (vs. 9.8%), yielding higher accuracy (64.12%/63.52% vs. 60.22%) at lower latency (≈1.64 vs. 2.37 ms).
Regarding Figs. 2 and 10 of the initial submission, we agree the efficiency trade-off varies across settings, as revised Section 5.2 discusses. Following Kubaty et al. (2025) (line 377, initial submission), we report calibration and efficiency separately rather than assuming they are always correlated.
Regarding near-zero ECE, per-exit ECE is computed only over the samples that exit at that branch to reproduce actual exit behavior. For Supervised ViT-S on Galaxy10, no prediction exceeds τ = 0.9 at Exit 0, so its per-exit ECE is undefined, not low. Table 21 now reports full-test-set ECE without thresholding: this exit matches our accuracy (77.8%) but has an ECE of 0.175 vs. 0.028, indicating the problematic calibration that prevents early exiting.
Question 3
We agree and have softened the third contribution. For a single task, SSL pretraining plus fine-tuning costs more than supervised training (Appendix C.7), as the revised Conclusion states. The benefit arises when an encoder is reused: fine-tuning needs at most 50 epochs vs. 300 for a supervised model trained from scratch. Appendix E (line 2160, initial submission) shows this on three CIFAR10 sub-tasks: 150 epochs or fewer in total for our models vs. 900 for Supervised, with comparable trade-offs.
Question 4
Alignment is an exploratory component (lines 61–62, initial submission) testing whether cross-depth alignment helps. Align gives no consistent gain in calibration or efficiency and is insensitive to λ (Appendix D.15, initial submission). Section 5.7 (lines 488–489) now frames this properly as Standard suffices on its own (see also Table 22).