acceptodds
Under review as a conference paper at ICLR 2027

TIME-UNIFORM GUARANTEES FOR TRANSFORMER EARLY EXIT VIA MARTINGALE STOPPING

Abstract

Can a transformer exit early and still guarantee that its prediction stays close to the full-depth answer? Existing early exit methods (entropy thresholds, learned confidence scores, patience counters) offer no such guarantee, and in practice collapse unpredictably: CALM loses 14–19 MCC points on two of six GLUE tasks under identical training conditions. We introduce *Martingale Transformers*, which connect early exit to martingale theory. The key observation is that a transformer's layer-wise predictions form a stochastic process amenable to concentration analysis. We first show this process is non-martingale in pretrained BERT (, all 12 layers, six GLUE tasks), then add a zero-parameter regularizer that enforces the martingale property; stopping is simple, exiting when the quadratic variation of the belief sequence drops below a threshold . Our main theoretical result is a time-uniform deviation bound via Ville's inequality: the early exit prediction satisfies simultaneously at all layers , with probability , which we verify empirically at on all datasets. On BERT-large (24 layers, 5 seeds), QV stopping matches or improves baseline MCC on 5 of 6 GLUE tasks while saving 5–9% of FLOPs; on TinyLlama-1.1B it degrades gracefully to full-depth inference (98% MCC retention on MNLI) rather than producing incorrect early exits.

Primary area:
foundation or frontier models, including LLMs
Keywords:
early exit, adaptive inference, transformers, martingale theory, quadratic variation, time-uniform bounds, Ville's inequality, efficient inference, dynamic computation, BERT, deviation bounds, anytime-valid inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Fuo9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies early exit for transformer classifiers with a guarantee that the early prediction stays close to the full-depth prediction. It treats the layer-wise softmax outputs of a shared classifier as a stochastic process, adds a regularizer that penalizes the batch-mean change of these outputs across layers, and exits when the relative quadratic variation of the outputs falls below a threshold. A time-uniform deviation bound via Ville's inequality (Proposition 3) is claimed for the regularized model, together with a non-martingality result (Proposition 1) and an expressivity result (Proposition 2). Validation covers six GLUE tasks with BERT-base and BERT-large against selected early exit baselines, Pareto sweeps, an empirical check of the bound, and an extension to TinyLlama.

Critical strengths
  1. Connecting early exit to martingale theory and anytime-valid inference is an interesting direction.
  2. The method is simple and adds no parameters.
Critical weaknesses
  1. The theory is not established

Proposition 3 has only an invalid proof sketch whose premise the regularizer does not satisfy, and Propositions 1 and 2 are not proved.

  1. The experiments do not support the motivation of the paper

The described stopping rule collapses in the paper's own appendix, and the baselines are out-dated and not run as in their original papers.

Questions for authors
  1. Where is the full proof of Proposition 3? Line 269 and the Reproducibility Statement promise one in the appendix, but Appendix A contains only results. The sketch does not hold: SℓS_\ell contains the terminal BLB^L, so it is not adapted to Fℓ\mathcal{F}_\ell; S0=exp⁡(η∥B0−BL∥2)≥1S_0 = \exp(\eta\|B^0 - B^L\|^2) \ge 1, so E[S0]≤1\mathbb{E}[S_0] \le 1 fails; the supermartingale property is only asserted; and the optimized η\eta depends on the random QVLQV_L, while Ville's inequality needs η\eta fixed in advance. The assumption ∥Bℓ+1−Bℓ∥≤c\|B^{\ell+1} - B^\ell\| \le c always holds with c=2c = \sqrt{2}, and cc never enters the bound.
  2. Why is the regularized sequence a martingale? Eq. 3 only drives the unconditional batch mean of the increments to zero, which Remark 1 calls "precisely the martingale condition", but a martingale needs a zero mean conditional on Fℓ\mathcal{F}_\ell. With Lmart=0\mathcal{L} _ {mart} = 0 exactly, two mirrored groups of inputs that reach one vertex by layer 7 and flip to the other by layer 24 violate Eq. 9 for every input at τ=7\tau = 7. Moreover, B0B^0 is the same vector for every input, so under the paper's probability space a martingale forces a constant classifier.
  3. How are Propositions 1 and 2 proved? The argument for Proposition 1 asserts the drift without derivation. The proof of Proposition 2 uses a path that it admits is not a martingale, and as stated the proposition is false: with Lmart=0\mathcal{L}_{mart} = 0, the batch mean of BLB^L must equal the constant B0B^0 on every batch, which an accurate classifier cannot satisfy on both an all-positive and an all-negative batch.
  4. Is the bound informative where it is checked? By Cauchy-Schwarz, Eq. 9 holds for any sequence whenever L−τ<2log⁡(1/α)L - \tau < 2\log(1/\alpha), that is, for every exit on BERT-base and every exit at layer 15 or later on BERT-large. It also needs QVLQV_L, which is unknown at exit time. Figure 3 and Table 13 check KL rather than the squared L2 of Proposition 3, and the check fails at α=0.05\alpha = 0.05 on BERT-large CoLA (14.8%).
  5. Proposition 1 concerns a model trained with cross-entropy, but Table 1 uses pretrained BERT-base. Which classifier produces the beliefs there, and why does Table 1 report 12/12 significant layers when Table 14 marks three layers as non-significant?
  6. Table 19 reports standard QV at θ=0.02\theta = 0.02 with MCC .461 on BERT-large SST-2, while Table 9 reports .870 for QV at the same θ\theta. Which numbers reflect the stopping rule of Algorithm 1, and which rule produces Tables 2 and 3?
  7. CALM [2] calibrates its threshold to guarantee consistency with the full model, but here it exits at the same layer for every threshold, at about ℓmin\ell_{min} on SST-2 (Appendix A.12). How was CALM implemented and tuned, and how were the thresholds of DeeBERT, PABEE, and Entropy chosen?
  8. The paper states that existing early exit methods offer no guarantee (lines 12 to 13), but CATs [1], CALM [2], and [3] give guarantees with respect to the full model, and [4] and [9] already use martingale arguments across exits or layers. How does the method compare with [1] to [4]?
  9. The newest compared baseline is from 2022. How does QV Exit compare with ConsistentEE, LayerSkip, RAEE, and [5] to [8], which report 1.5 to 2.2 times speedup on GLUE, while QV Exit saves at most 9% of FLOPs?
  10. What is the agreement between the exit and full-depth predictions of the same model? This is the property the paper promises, but only MCC against labels is reported.
  11. In Tables 9 and 10, entropy stopping reaches higher MCC than QV at matched FLOPs (for example .814 vs .732 at about 78% on QNLI), contrary to lines 363 to 365. What does QV stopping add over entropy, and what does it give without Lmart\mathcal{L}_{mart}?
  12. On TinyLlama, SST-2 drops from .874 to .492 at 80% FLOPs (Table 4). How does this fit "degrades gracefully", and why is no baseline run there?
  13. Tables 2 and 3 differ from Tables 5 and 6 for the same five seeds, and the ΔMCC column of Table 15 does not equal Full minus Exit. Which numbers are correct?

[1] Tal Schuster, Adam Fisch, Tommi Jaakkola, and Regina Barzilay. Consistent accelerated inference via confident adaptive transformers. In EMNLP, 2021.

[2] Tal Schuster, Adam Fisch, Jai Gupta, et al. Confident adaptive language modeling. In NeurIPS, 2022.

[3] Metod Jazbec, Alexander Timans, Tin Hadži Veljković, Kaspar Sakmann, Dan Zhang, Christian A. Naesseth, and Eric Nalisnick. Fast yet safe: Early-exiting with risk control. In NeurIPS, 2024.

[4] Metod Jazbec, Patrick Forré, Stephan Mandt, Dan Zhang, and Eric Nalisnick. Early-exit neural networks with nested prediction sets. In UAI, 2024.

[5] D. J. Bajpai and M. K. Hanawal. BEEM: Boosting performance of early exit DNNs using multi-exit classifiers as experts. In ICLR, 2025.

[6] D. J. Bajpai and M. K. Hanawal. Beyond greedy exits: Improved early exit decisions for risk control and reliability. In NeurIPS, 2025.

[7] J. He, Q. Zhang, D. Miao, K. Yi, S. Hao, H. Zhang, and Z. Wei. Improving prediction certainty estimation for reliable early exiting via null space projection. In IJCAI, 2025.

[8] J. He, Q. Zhang, H. Zhang, X. Huang, U. Naseem, and D. Miao. COSEE: Consistency-oriented signal-based early exiting via calibrated sample weighting mechanism. In AAAI, 2025.

[9] Michele Caprio and Sayan Mukherjee. Concentration inequalities and optimal number of layers for stochastic deep neural networks. IEEE Access, 11:38458-38470, 2023.

Reviewer_pPpROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies a natural problem: is it possible for a transformer to stop computing before passing through all layers and have some guarantee on the output being close to the answer that uses all layers? They show empirically that current transformers can not do so, but a minor adjustment to the training loss results in belief values allows us to do exactly this. They provide analytical and experimental guarantees for this new construction.

Critical strengths
  1. The paper is well-written, I found it a clear and breezy read, which is very much appreciated. (The first paragraph in particular, is exceptionally good, it makes everything extremely clear.)
  2. The problem studied is certainly interesting and timely, and the theoretical approach is well-grounded; experimental results complement and help support the theory where it is lacking.
Critical weaknesses

The main weakness according to me is that the paper oversells its theoretical results (but Proposition 3 is correct and as significant as is claimed):

  1. Proposition 1 is not so much a theoretical result as much as it is an empirical observation (that the beliefs do not form a martingale sequence).
  2. Proposition 2 is not too interesting or relevant: it is not surprising that there exists a martingale sequence which matches what we want, the question is whether this martingale sequence can be reached, which this proposition does not answer.
Questions for authors

No questions that might materially change my assessment. I think this is a decent paper, but the overclaiming of theoretical results makes me support it weakly.

Reviewer_RXmVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents an early exit method by introducing martingale transformers that connect early exits to the rich martingale theory. the paper provides theory heavy cnnection between their backbone and the martingale properties, this might help in more relaible exiting.

Critical strengths
  1. The paper connects transformer models to martingales.

  2. The problem statement tackled is important.

Critical weaknesses
  1. A very critical issue with this paper is the timeliness of the work, the field of early exiting has moved a lot in last few years, while all the papers cited by the work are from 2022 and max 2023 which clearly indicates a lack of literature knowledge, a lot of works and guarantees over early exits have been made and the paper neither compres against them nor cites them. [1] [2] [3] [4] [5] and many more.

  2. The paper works only on encoder-only models, and the application to causal language modelling is discussed in a paragraph, limiting the usefulness of the method to non-autoregressive models, where the latency issue is quite less compared to the autoregressive models.

  3. Also, to develop the martingale transformer, the model forces subsequent layers to learn a close to identity mapping, this regularizes the model better but also can lose performance, which is clearly seen in the result tables where the finetuning has lead to lower baseline performance as compared to vanilla finetuning.

  4. The paper wordings are different but eventually it is just checking the normalized MSE between subsequent hidden layers, similar to which there are existing ideas which alreay do a hidden layerwise check using cosine similarity [5].

  5. Also, the paper requires a lot of improvements in writing style, like in the introduction at multiple places, things are directly used without defining them.

Overall, the paper requires significant improvements before reaching a publishable state.

[1] https://proceedings.neurips.cc/paper_files/paper/2025/file/f92ad7a76bb102afab768fd4e675c073-Paper-Conference.pdf.

[2] https://proceedings.neurips.cc/paper_files/paper/2024/file/ea5a63f7ddb82e58623693fd1f4933f7-Paper-Conference.pdf.

[3] https://proceedings.iclr.cc/paper_files/paper/2025/file/9c84feb75eae1ef6389f31b3ef050b6a-Paper-Conference.pdf

[4] https://openreview.net/pdf?id=jX2DT7qDam [ICLR 2024].

[5] https://openaccess.thecvf.com/content/CVPR2023/papers/Tang_You_Need_Multiple_Exiting_Dynamic_Early_Exiting_for_Accelerating_Unified_CVPR_2023_paper.pdf

Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.