acceptodds
Under review as a conference paper at ICLR 2027

Optimal Depth of Neural Networks: Resource-Correct Stopping with Bounded Logit Increments

Abstract

When can a network stop without executing the suffix that justifies the stop? For a fixed inspection schedule, finite observed record, confidence-resource utility, declared charges, and coupled bounded logit increments, we derive an information-optimal stopping certificate. One aligned completion attains every future confidence support. A separate sharp guard preserves the full-recall returned label, not unknown-label correctness. We compile both tests into checked integer kernels and prove their equivalence to arbitrary-precision comparisons. Five-seed experiments select schedules separately for scalar and return-conjoined policies, publicly freezing 870 choices before new evaluation. Return protection changes two of 300 main-grid schedule choices and eliminates all returned-label disagreements; precision and temperature are evaluated with newly constructed scores and supports. Actual image-to-return execution on Xeon and Neoverse CPUs makes the compiled multi-exit path faster than full execution for the registered 1,000-image workload, while retaining adverse small-batch results. A finite-family work bound and a separated design procedure distinguish pointwise certification, task risk, and statistical selection. All prior proofs, saturated-training diagnostics, and unfavorable results are preserved behind an active-claim and artifact index.

Primary area:
learning theory
Keywords:
neural networks, optimal stopping, adaptive computation, deep learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HauxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the problem of certified early-stopping in multi-exit neural networks, in the context of recall-based execution where deeper layers can be executed while a controller can still decide and return an earlier exit's prediction, at the cost of redundant computation. In the proposed approach, each stage adds a bounded increment to the input logits (output contact). The paper proves that a single aligned completion for all remaining stages (boosting the top logit and lowering all competitors by a fixed bias) forms a sharp closed-form stopping certificate. Additionally, to guarantee an intermediate sub-model returns the same class prediction as the final output, a separate full-recall return guard is proposed, that further boosts the runner-up probability. Additional remedies for hard-clipping saturation, guard-specific inspection scheduling, logit precision and temperature sensitivity are devised, yielding a complete and effective analytical framework for early-stopping decisions.

Critical strengths
  • The core theorems of the manuscript (specifically 1 and 3) are insightful and self-contained: employing a single adversarial logit trajectory to prove both soundness and information-relative maximality (separating scalar confidence regret from returned-label preservation) is a strong theoretical contribution.
  • The conducted analysis is rigorous and offers comprehensive empirical evidence for both positive and negative results.
  • The analysis on recall dynamics (impact of inspection checks) provides structural insights into sequential stopping with recall, that can guide and inspire future work and best practices in the field.
Critical weaknesses
  • The scale of the empirical results is severely limited to tiny 4-exit CNN-based architecture and toy datasets (e.g. Fashion-MNIST); and still show a limited speed-up (up to 10%) restricted to specific settings (e.g. batch sizes). While negative results are also reported for larger (ResNet-based models) on CiFAR-100 render the practical applicability of the proposed theoretical contribution of this work an open problem.
  • The presentation of the manuscript requires significant improvement. In the current form, the main body of the manuscript does not stand on its own (e.g. fails to provide any information on the adopted models and datasets of the analysis), while the writing style is severely compressed and difficult to follow; and the information in the appendix is overwhelmingly presented.
Questions for authors

An analysis on what causes the slowdown of the proposed selected guarded execution on small batch sizes (e.g. 32) would help understand the effectiveness and limitations of the proposed approach.

Reviewer_MqKyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper derives optimal stopping rules for networks with multiple exists. Several theoretical results for multi-exit networks are presented.

Critical strengths

Implementation results on actual CPU architectures are presented. The paper presents results on both in terms of regret bound and the prediction performance.

Critical weaknesses
  • The paper is not well-written. There are phrases like "architectural contract" which does not make sense unless one is writing a paper on architecture, or rather entirely different than the standard terminology used in this line of research or any theoretical research in fact. We encourage the authors to read the relevant papers and use standard terminology.
  • The reference list is unusually short for a paper of this length and in fact misses many of the prior work that has been done (both past and recent) on conditional computation.
  • The presented theoretical results like Theorem 1 are trivial and I fail to see why they would represent an advance in the field. Again, this may stem from the fact that the previous literature has not been elaborated in detail.
  • The chosen experimental setting like Fashion-MNIST and full connected neural networks is outdated and do not carry much relevance in the current line of research in AI/ML.
Questions for authors

No questions.

Reviewer_RvpbOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the early exit problem in the recall setting, with computation costs charged for every executed layer. Under certain assumptions (bounded additive logit updates), the authors derive an optimal stopping certificate, where optimality is relative to the observed information. They run experiments with five seeds and select inspection schedules on an image classification task (Fashion MNIST), using a small convolutional network with four exits.

Critical strengths

Modeling early exit as a sequential decision problem, specifically as an optimal stopping problem, is an interesting question.

Critical weaknesses

Weaknesses

W1. Clarity of the abstract and introduction. Many central notions are used before they are defined. The second sentence of the abstract stacks five undefined qualifiers (fixed inspection schedule, finite observed record, confidence resource utility, declared charges, coupled bounded logit increments) and then claims an information optimal stopping certificate. A reader cannot tell whether information optimality refers to information theoretic optimality or to something else. The introduction repeats this pattern with recall, coupled witness, information relative maximality, return guard, and aligned completion. The abstract and introduction should state the problem in plain terms and define each notion before using it.

W2. The problem setting is not introduced. The introduction speaks of adaptive depth and cites halting (Graves, 2016), early exit (Teerapittayanon et al., 2016; Huang et al., 2018), and dynamic routing (Wang et al., 2018). It is therefore unclear whether the paper concerns early exit within one network, a cascade of separate models, or layer skipping.

W3. Section 3 is not self contained.

  • The section never states that the network solves a KK class classification task; the returned label first appears in Section 4.2.
  • The score gj∈[0,1]g_j \in [0, 1] is introduced abstractly. That larger is better is only implicit through Bt=max⁡j≤tgjB_t = \max_{j \le t} g_j, and its instantiation as max softmax confidence appears only at Eq. (3).
  • More importantly, no relation between gjg_j and predictive accuracy is assumed, so Theorem 1 certifies regret in confidence minus cost rather than in accuracy. The paper acknowledges this in Section 4.2, but the objective should be stated and motivated at the start of Section 3.

W4. Positioning relative to Qi (2025). Qi (2025) poses depth selection as classical optimal stopping. There, the reward for halting at a layer depends only on that layer's representation and the accumulated cost, so earlier layers cannot be recalled. The submission states that the depth stopping formulation of Qi (2025) motivates the with recall problem, but it does not make this distinction explicit, and a reader may infer that Qi already studies recall.

W5. Completeness of the theory and alignment with practice.

  • (a) The optimality class is undefined. Theorem 1 claims optimality over rules "using only the same observed scores and logits, increment budgets, charges, and comparator", but this class is never formally defined. What is the information set, and are randomized rules included? An optimality theorem should make this quantifier explicit.

  • **(b) The formal assumptions is not stated in the main text.

  • (c) The empirical scope is narrow. All positive evidence comes from one small Fashion MNIST CNN with four exits and ten classes, as the paper itself notes in Section 8. Proposition 2 is proved only for K=10K = 10 and L=3L = 3.

W6. Related work on early exit, especially with recall. Early exit is a dense literature, but the related work section discusses only a handful of early exit papers.

  • Yang et al. (2026b) is summarized in a single clause as studying loss trajectories, which undersells it. That work formalizes early exit and cascaded inference as a multistage decision problem. It shows that recall is necessary and sufficient for provable guarantees: no recall strategies admit no constant factor approximation, while a recall based index policy is optimal and computable in polynomial time. Since the present paper also relies on recall, it should position its worst case certificate against that result.

Minor: terminology. The term compile is used in at least three senses:

  • offline precomputation of an integer threshold (Section 3.1);
  • lowering exact rational comparisons to checked 256 bit integer arithmetic (Proposition 2);
  • native C++ compilation whose cost is timed (Section 7).

The compiled multi exit path mentioned in the abstract is therefore ambiguous. Please disambiguate.

Questions for authors
  • Please formally define the class of rules in the optimality claim of Theorem 1, including the information set and whether randomized rules are allowed. Please also confirm that “information optimal” means maximal sound acceptance relative to this information, rather than an information theoretic statement.
  • Yang et al. (2026b) show that a recall based index policy is optimal in their framework. How does your certificate relate to it?

Yang, Y., Zhang, R., Morgenstern, J., & Xu, H. (2026). T-Tamer: Provably taming trade-offs in ML serving. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=JFY9MZtWTu

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.