acceptodds
Under review as a conference paper at ICLR 2027

Deeper Is Not Better for Quantized CLIP: Early Readout Recovers Accuracy While Cutting Compute

Abstract

Readout depth is a decision variable. CLIP predicts a label from the direction of its final-layer image embedding, on the premise that every transformer block moves that direction closer to the correct class. We show that this premise fails for INT8-quantized CLIP and is already unreliable in full precision. In FP32 ViT-B/32, the cosine agreement between intermediate and final representations rises with depth but regresses at three of the eight candidate layers; under INT8, the deviation introduced by quantization grows from 0.09% of the native inter-layer change at Layer 1 to 75.64% at Layer 11. We model transformer depth as a noisy readout channel in which each block adds a semantic increment and a perturbation, and show that under this model a per-sample readout gated on the top-1/top-2 margin, with depth-dependent thresholds, is expected to beat full-depth readout once perturbation growth outpaces semantic gain. LRA-EE implements this rule on a frozen INT8 backbone: a patch-token average replaces the immature shallow [CLS] token, a learned gate trained without ground-truth labels decides when to stop, and exit layers are selected by an FP32-only maturity criterion applied identically to every backbone. Across four INT8 CLIP backbones on ImageNet-1K, accuracy never falls below the full-depth baseline while FLOPs fall by 7.9–14.5%: +3.27%p on ViT-B/32, +4.27%p on ViT-B/16, +3.16%p on EVA-CLIP, and +0.66%p on ViT-L/14. In other words, the quantized model is more accurate when it computes less. A four-quadrant decomposition attributes the gain to a Rescue Effect—8.1% of samples are correct at an early exit but wrong at full depth, against 4.9% the reverse—and an exact-route control splits it into a precision-independent component (+2.42%p) and a quantization-specific increment (+0.86±0.03%p over three seeds). The evidence is empirical: the backbone is never updated and the exit heads are calibrated on 2.3% of the ImageNet training images, so the result is post-training readout calibration of a compressed model rather than zero-shot deployment.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
early exit, post-training quantization, CLIP, vision–language models, adaptive inference, representation collapse, selective classification, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WQdTOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates whether the final layer is always the optimal readout point for quantized CLIP models. The authors observe that, due to the accumulation of quantization noise and representation degradation, deeper layers in a quantized model do not necessarily yield higher-quality representations. Based on this insight, they propose the "Layer-wise Representation-Aware Early Exit" (LRA-EE) method, which dynamically selects the readout depth for each input sample by leveraging intermediate layer representations, a learned gating mechanism, and depth-dependent thresholds. Experiments conducted on four INT8 CLIP backbones demonstrate that LRA-EE improves ImageNet accuracy while simultaneously reducing computational cost compared to full-depth inference.

Critical strengths
  1. Compelling perspective and research motivation. This paper reveals an interesting and somewhat counter-intuitive phenomenon: for quantized CLIP models, the final layer is not always the optimal readout point. The analysis regarding representation maturity and quantization noise offers valuable insight into why deeper-level inference can sometimes actually degrade performance. I believe this observation holds practical value, particularly for deploying large-scale vision models in low-precision environments.
  2. Robust empirical evaluation and analysis. The experimental evaluation is comprehensive. The authors validated the proposed method across various CLIP backbones, providing not only final accuracy results but also detailed analyses—including ablation studies on the routing mechanism, comparisons with "early-exit" baselines, and a four-quadrant analysis of samples categorized by performance recovery or degradation. These analyses strengthen the empirical findings and help elucidate the reasons behind the method's effectiveness.
  3. Rigorous and meticulous experimental design. The paper carefully accounts for several potential confounding factors. For instance, the authors distinguish the contribution of adaptive routing from the effect of simply adding intermediate readout heads, and they evaluate actual wall-clock time rather than merely reporting theoretical FLOPs. Furthermore, the information regarding reproducibility and implementation details is thorough.
Critical weaknesses
  1. The innovation at the algorithmic level is relatively limited. The paper's primary contribution lies in the finding that, within quantized CLIP models, intermediate-layer representations outperform final-layer representations. However, the proposed LRA-EE framework itself is largely built upon existing concepts such as "early exit," confidence-based routing, and the utilization of intermediate-layer features. While combining these components in the context of quantized CLIP models offers practical value, the methodological innovation appears modest when compared to papers that introduce entirely new architectures or optimization techniques. The authors should more clearly articulate the distinction between the novelty of the observed phenomenon and the novelty of the proposed algorithm.
  2. The generalizability of the proposed method has not been sufficiently validated. Although the paper notes that the framework is not limited to CLIP, all primary experiments were conducted using CLIP image classification models. It remains unclear whether the same observations and performance gains apply to standard supervised ViTs, non-quantized models, or other vision tasks such as retrieval and dense prediction. Given the broad claims made regarding Transformer readout depth, additional validation in scenarios beyond INT8 CLIP classification would strengthen the persuasiveness of the conclusions.
  3. The extent of actual efficiency gains requires further clarification. Although the paper presents LRA-EE as a highly efficient inference method, the actual reduction in computational cost is relatively modest (with FLOPs decreasing by only about 7.9%–14.5%). Furthermore, while the authors measured wall-clock latency, the overhead introduced by exit heads and gating modules offsets some of the speedup; under certain settings, the adaptive inference path may even run slower than the original model. Consequently, it remains unclear exactly how much benefit LRA-EE offers in practical deployment compared to simply using a smaller model or other acceleration techniques.
  4. There is a lack of a strong connection between the theoretical analysis and the proposed algorithm. The concept of a "noisy readout channel" offers an interesting perspective for understanding the benefits of "early exit," yet its role in the design of the LRA-EE algorithm is relatively limited. The final routing strategy relies on empirical features and learned thresholds rather than being directly optimized or derived from the theoretical model. The authors should clarify whether the theoretical analysis serves any purpose beyond explaining the observed behavior.
Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

To Reviewer WQdT

Thank you for the detailed and encouraging review. We have addressed each of your four points in the revision (App. G, blue).

W1 - Phenomenon vs. algorithm. We agree and have labelled the three contributions in Sec. 1 accordingly. Phenomenon: readout depth is a per-sample decision variable (Rescue 813 vs. Loss 489, Table 4), and under INT8 the intermediate [CLS] token is the fragile readout while the patch average is not (Table 19). Algorithm: a depth-dependent Chow rule whose components are individually known; we claim the derivation and combination. Measurement: the four-quadrant and exact-route decompositions.

W2 - Generality. The result is not tied to one precision, protocol or test set. The gain is +3.67 / +2.47 / +2.91%p (B/32, B/16, EVA) in FP32 and positive under all four INT8 protocols on each (up to +6.56%p on EVA, Table 14); it composes with a trained final readout (67.86% FP32, 65.99% INT8 on B/32, Table 16); and it holds without recalibration on ImageNet-V2 (+1.23%p) and -Sketch (+0.61%p) (Table 21). Retrieval and dense prediction are outside the gate's scope (fixed label set, Sec. 8); supervised ViTs are not evaluated and not claimed.

W3 - Efficiency. The adaptive path is now faster than the original model: −2.1% to −6.6% at the paper's operating point and −10.1% to −14.3% with no tuning, both above baseline accuracy (Table 17). Token pruning on the same INT8 backbone gives 58.50% at 13.5% saving vs. our 60.65% at 11.1% (Table 2); it acts on a different axis and can be combined, as can a smaller backbone.

W4 - Theory and algorithm. Each design choice the theory motivates is tested (App. G.4): (1) margin as the gate statistic: 60.47% margin-only vs. 58.21% confidence-only (Table 10); (2) a threshold decreasing with depth: γ>0\gamma>0 on all four backbones, worth 0.49–0.76%p over a constant threshold; (3) the label-free maturity rule (Eq. 12) matches exhaustive label-based selection over 255 exit sets; (4) on ViT-L/14 the best patch-readout layer moves from 23 to 20–22 under INT8 (Table 20).

Presentation. We have added a five-line method summary and a notation table (App. G). If a specific passage is still hard to follow, we would be glad to revise it.

Reviewer_WQdTOpenReview ↗

Official Comment by Reviewer WQdT

Thanks for your response, which has addressed our concerns. We will maintain the score.

AuthorsOpenReview ↗

To Reviewer WQdT

Thank you very much for taking the time to read our response during such a busy period, and for letting us know that your concerns have been resolved. Your comments have made the paper clearer and more rigorous, and we are grateful for your careful engagement throughout the review.

Reviewer_ECBkOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a sample-wise early-exit method for adapting a quantized CLIP vision encoder to a backbone. Experiments are performed on ImageNet1k to prove efficacy. The results show that early-exit may save computation while even having performance gain. Only the early exit heads need lightweight training.

Critical strengths

Distillation is found to be effective and necessary for increasing the performance of the early-exit readout heads.

Critical weaknesses

The problem is of general interest, but the adopted method seems rather mechanical: a sweep through possible hyper parameters. The experiments are limited to ImageNet1k and CIFAR-100, Food-101 which are rather small scale and not so representative of modern vision tasks.

Questions for authors

No particular questions.

4 comments
AuthorsOpenReview ↗

To Reviewer ECBk

Thank you for the review. We have added experiments on both weaknesses: the method works with no tuning at all (+2.03%p at 17.9% fewer FLOPs), and the gain holds under five quantization protocols and on ImageNet-V2 and -Sketch (App. G).

  • "Mechanical sweep." No sweep is needed. A fixed gate probability of 0.5 already beats the baseline, as does every single threshold from 0.44 to 0.99; the calibrated choice is within 0.2%p of the evaluation-split oracle. The exit layers are not searched either: the label-free rule of Eq. 12 matches the best of all 255 exit sets chosen with labels (60.86% vs. 61.00%, not significant) while saving more compute (Table 18).
  • Scale. The gain is not specific to one setting. It holds under five protocols on three backbones (+3.5 to +4.2%p on ViT-B/32, +2.9 to +6.6%p on EVA-CLIP, Table 14) and in wall-clock time (10–14% faster with no tuning, Table 17). Without recalibration, the ImageNet-calibrated policy stays above the baseline on ImageNet-V2 (50.36 → 51.59, +1.23%p) and ImageNet-Sketch (34.64 → 35.25, +0.61%p); on ImageNet-R/-A it needs recalibration, as App. E states for transfer (Table 21).
  • What is new to learn. A quantized CLIP has to be read from patch tokens at intermediate depth: under INT8 the intermediate [CLS] token degrades far more than the patch average on all four backbones (EVA-CLIP Layer 10: gap 20.0%p in FP32, 51.3%p in INT8; Table 19).
  • Presentation. We have added a five-line method summary and a notation table (App. G).

Is there a benchmark you would consider most representative? We would be glad to add it.

Reviewer_ECBkOpenReview ↗

Table 16 data suggests training the head is important

Thank you for adding Table 16. In Table 16, in static quantization and dynamic B/32 cases, [CLS] head has strong performance when compared to LRA-EE (static B/16 [CLS] is on par with LRA-EE). In addition, FP32 and int8 [CLS] head has high scores close to [CLS] head +exits (there are marginal gains and computation savings, which is as expected as we have more trainable parameters). This leads one to question the "deeper is not better" headline, and wonder which factor is more in play here: readout adaptation or accumulation of quantization errors or both.

AuthorsOpenReview ↗

To Reviewer ECBk

(Edited for precision: the reply now answers the “which factor” question directly.)

Thank you for engaging with Table 16. You are right that training the final readout matters. To answer directly: for accuracy, readout adaptation is the larger factor; early readout adds a smaller accuracy gain on top of it while removing compute, and quantization determines how intermediate layers must be read.

Readout adaptation sets the level. A trained head on the final [CLS] token adds +4.94%p over the zero-shot readout on ViT-B/32 (62.32% vs. 57.38%), above LRA-EE with the native exit (60.86%). We do not claim that early exits replace a trained readout. What early exits add on top. With that head as the final exit, adding the exits is more accurate and cheaper in all ten settings (+0.2 to +0.8%p, 3.6–10.6% fewer block FLOPs; Table 16). We agree that part of this accuracy gain can come from having several trained readouts rather than from depth as such; what the exits contribute that additional full-depth heads cannot is the compute reduction. For patch-average heads, no single depth reaches the baseline (best fixed depth 56.59%), and routing over them gives +4.27%p [+3.69, +4.86] (Table 15). Quantization. Two effects are specific to INT8. (a) The intermediate [CLS] token degrades far more than the patch-token average on all four backbones (Table 19), which is why intermediate exits read patch tokens. (b) On ViT-L/14 the best patch-readout layer moves from Layer 23 in FP32 to Layers 20–22 under INT8 (Table 20). Under dynamic INT8 on ViT-B/16, the [CLS] head of Table 16, trained on full-precision features, falls below the baseline, and LRA-EE with the native exit is the best configuration there. The gain itself is also present in FP32 (Table 14), as the abstract states (“already unreliable in full precision”). The headline refers to per-sample readout depth: for either final readout, native or trained, sending a subset of samples to an earlier exit is more accurate and computes less than taking every sample to full depth (Tables 1 and 16). If the reviewers find the current title broader than Table 16 supports, we are glad to make this per-sample reading explicit in the title and abstract of the final version.

We would be glad to clarify further.

AuthorsOpenReview ↗

To Reviewer ECBk

We have edited our reply above to state points 2 and 3 more precisely; the conclusions for Table 16 are unchanged.

Reviewer_GzY7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies early readout for quantized CLIP and proposes LRA-EE: on a frozen INT8 backbone, exit heads that aggregate patch tokens by mean are trained with target-domain labels and distillation from the FP32 model's final embedding, and samples exit per-sample through a learned margin gate with a depth-decaying threshold, with candidate layers screened by an FP32 representation-maturity criterion. It reports ImageNet gains of 0.66–4.27 points on four backbones while removing 7.9–14.5% of transformer block FLOPs.

Critical strengths

S1. The question is worthwhile and the phenomenon is consistent. Accuracy gains together with block-FLOPs reductions appear on all four backbones (Table 1), and the protocol is leakage-safe as described: gates and thresholds are selected on a held-out split only.

S2. The diagnostics are informative and the reporting is candid. The Rescue/Loss decomposition gives an exact account rather than a mean-accuracy delta (Table 4 sums to 10,000), and the paper reports unfavourable results plainly: a measured wall-clock slowdown, +0.43 pt under static W8A8, and failed transfer without recalibration.

Critical weaknesses

W1. The INT8 baseline is weaker than standard INT8, and its recipe is mis-described. The paper says activations stay in floating point, but the code uses quantize_dynamic with reduce_range=True, quantizing activations per tensor over the whole batch, so each prediction depends on the other images in the batch. The B/16 baseline reaches only 30.06 against a published FP32 reference of 68.6, and under the paper's own static W8A8 recipe the gain falls to +0.43 pt; no standard ViT PTQ recipe is compared (§5.1; App. C; Table 1).

W2. The source of the gain and the efficiency claim are not supported. The exit heads are trained with labels and FP32 distillation, whereas the full-depth arm is CLIP's zero-shot readout: hard-label routing alone gives only +0.50 pt, in the FP32 replay +2.42 of the +3.27 gain remains without any quantization, and no equally trained head on the native final [CLS] embedding is compared. §6.1 says the ablation arms share identical routes, but ablation_factorial_rescue_vitb32.py selects thresholds separately for each arm. The deployed path is also slower end-to-end (32.17 vs. 30.65 ms per image) (Table 3; §7.2).

Questions for authors
  1. What is FP32 full-depth accuracy for all four backbones under your protocol, and re-measured at batch size 1 against a standard ViT PTQ recipe with the baseline's final projection and the exit heads at the same precision, how much of the gain remains?
  2. Does Table 3 hold the per-sample exit routes fixed across the two arms? How much of the +3.27 pt survives against the same CE + distillation head on the native final [CLS] embedding, and against the best fixed depth?
  3. §5.2 states that nothing in Eq. 11 penalizes Layers 4 and 6, yet with γ > 0 shallow thresholds are higher (τ4 = 0.81 vs. τ11 = 0.475). What links maturity/NSR to margin drift, and how does the exit set change if Eq. 12 is replaced by held-out accuracy selection?
  4. Is there any configuration in which the deployed policy is faster end-to-end than the unmodified model? If not, would you restate the efficiency contribution as a reduction in block FLOPs?
1 comment
AuthorsOpenReview ↗

To Reviewer GzY7

Thank you for the code-level review; both factual points were correct. We have run all four controls you asked for, and the result holds in each (App. G, Tables 14–18).

W1 - Baseline. You are correct: quantize_dynamic quantizes activations per tensor at run time, shared across the batch. We have corrected the text (Sec. 5.1, App. C) and re-measured (Table 14):

  • Batch size 1: B/32 58.54 → 62.17 (+3.63%p); B/16 36.67 → 37.08.
  • Static W8A8, per-channel weights: B/32 59.06 → 63.27 (+4.21%p); B/16 54.12 → 55.70 (+1.58%p); EVA 66.51 → 71.26 (+4.75%p; batch 1: 63.38 → 67.84, +4.46 [+3.93, +5.00]). LRA-EE is above the baseline in all fifteen settings.
  • B/16: as you inferred, 30.06 reflects the dynamic batch-32 protocol. Table 14 uses one set of full-precision-feature heads for all protocols (+0.89%p there); the INT8-feature heads of Table 1 read that representation better (+4.27%p). On EVA the same recipe change goes the other way (+6.56%p vs. +3.16%p in Table 1).
  • Static +0.43%p (App. F) used 30k-image heads; with the 128k recipe it is +3.80%p.

W2 - Source of the gain. It is not FP32 distillation: hard-label routing gives +3.36%p [+2.78, +3.93], and on identical routes distilled and hard-label heads differ by +0.02%p (Table 15, row iv′; Sec. 6.1 corrected). The +0.50%p you cite came from the 30k-image setting.

  • Same head on the final [CLS]: you anticipated correctly that it helps: 62.32%, above LRA-EE with the native exit (60.86%). Early exits still improve on it: 63.13% at 7.9% fewer FLOPs (all nine seed combinations significant), and in all ten backbone–protocol settings (+0.2 to +0.8%p, Table 16). The claim holds for either final readout.
  • Fixed depth: patch-average heads without routing are below the baseline (final layer −3.62%p; best depth 56.59%); LRA-EE is +4.27%p [+3.69, +4.86] above the best fixed depth.

Q1. FP32 full depth: 60.82 / 65.94 / 73.07 / 71.58 (B/32, B/16, EVA, L/14). Batch 1 and static PTQ: see W1. The native projection is INT8; running it in FP32 changes the baseline by +0.13 / +0.07 / −0.04 / +0.26%p. Exit heads and gates run in FP32 (<1% of one block; now stated in App. C).

Q2. Routes are now fixed: (v) − (iv′) = +0.02%p. Same head on final [CLS]: 62.32%; with exits 63.13% (−7.9% FLOPs); LRA-EE with native exit 60.86% (−12.7%). Best fixed depth: +4.27%p.

Q3. Held-out accuracy selection over all 255 exit subsets gives {4, 9, 10, 11}: 61.00%, −11.6% FLOPs; Eq. 12 gives {4, 6, 8, 10, 11}: 60.86%, −12.7%. Difference +0.14%p [−0.05, +0.33], not significant (Table 18). On thresholds: τ4>τ11\tau_4>\tau_{11} is the schedule itself, τL=τ0max⁡(1−γ(L−1),12)\tau_L=\tau_0\max(1-\gamma(L-1),\tfrac12) with γ>0\gamma>0; "nothing penalizes" meant no term excludes Layers 4 and 6 (reworded, Sec. 5.2). By Eq. 9, τL⋆\tau_L^\star is high where drift is still ahead (small M(L)M(L)) and falls as perturbation accumulates (rising NSR); the depth-dependent schedule adds 0.49–0.76%p over a constant threshold.

Q4. Yes. With an optimized exit path (100% identical predictions): −2.1% / −6.6% (20 / 1 thread) at 61.77% vs. 58.54%; −10.1% / −14.3% with no tuning at 59.65%; −13.3% / −16.9% at matched calibration accuracy (Table 17). As you suggested, we state the efficiency contribution as a block-FLOPs reduction and report wall-clock separately.

We hope this resolves the soundness concerns; we would be glad to run any further control during the discussion.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to All Reviewers and the AC

We thank the reviewers for their careful reviews and the Area Chair for the time devoted to handling our submission. We have run every control requested, and the paper's result holds under each: LRA-EE is more accurate than the full-depth readout while computing less. The method, the main-text tables and the claims are unchanged; all new results are in one new appendix (App. G, Tables 14–22, blue), and the main text stays within nine pages.

AskedAddedResult
Standard INT8 baseline, batch size 1 (GzY7)Five protocols, Table 14Gain in all fifteen settings: +3.5 to +4.2%p (B/32), +0.4 to +2.5%p (B/16), +2.9 to +6.6%p (EVA)
Same head on final [CLS]; fixed routes (GzY7)Tables 15–16Hard-label routing alone: +3.36%p; early exits improve on the [CLS] head in all ten settings
End-to-end speed (GzY7, WQdT)Table 17Faster than the unmodified model: −2.1% to −6.6%; −10% to −14% with no tuning
"Mechanical sweep" (ECBk)Table 18No tuning: +2.03%p; label-free exit rule matches the best of 255 exit sets
Theory link, generality (WQdT, ECBk)Tables 19–21Under INT8, intermediate [CLS] degrades up to +31.4%p more than the patch average; gain holds on ImageNet-V2/-Sketch

Key numbers (ViT-B/32 unless noted; all heads retrained with one 128k-image recipe; 3 head × 3 gate seeds, 95% paired-bootstrap CIs):

  • Protocols (Table 14). Baseline → LRA-EE: FP32 60.82 → 64.49; dynamic INT8 batch 32 57.38 → 60.86; batch 1 58.54 → 62.17; static per-tensor 59.29 → 63.09; static per-channel 59.06 → 63.27, at 12.4–13.2% fewer block FLOPs. EVA-CLIP: +4.46 to +6.56%p under the four INT8 protocols and +2.91%p in FP32, at 16.7–20.8% fewer FLOPs (every CI excludes 0).
  • Source of the gain (Tables 15–16). Hard labels only: +3.36%p [+2.78, +3.93]; distilled vs. hard-label heads on identical routes: +0.02%p. Best fixed depth: 56.59% (LRA-EE +4.27%p [+3.69, +4.86]). The trained final-[CLS] head reaches 62.32%; with LRA-EE's exits, 63.13% at 7.9% fewer FLOPs. For either final readout, routing a subset of samples earlier is more accurate and cheaper than taking every sample to full depth.
  • Latency (Table 17). Optimized exit path, 100% identical predictions: −2.1% / −6.6% (20 / 1 thread) at 61.77% vs. 58.54%; −10.1% / −14.3% with no tuning at 59.65%.
  • Selection (Table 18). Calibrated threshold within 0.2%p of the evaluation-split oracle; Eq. 12's exit set vs. the best of all 255 subsets chosen with labels: 60.86% vs. 61.00% (n.s.), with more FLOPs saved.
  • Readout under INT8 (Tables 19–20). At EVA-CLIP Layer 10 the patch average leads [CLS] by 20.0%p in FP32 and 51.3%p in INT8; on ViT-L/14 the best patch-readout layer moves from 23 to 20–22.

Revisions. (1) New App. G. (2) Three corrections: the quantization description (Sec. 5.1, App. C); "identical routes" and the role of distillation (Sec. 6.1); the exact head-training recipe and head precision (App. C). (3) A few words of clarification in Sec. 1, 5.2, 5.3.

Strengths noted. "The phenomenon is consistent … the protocol is leakage-safe … the reporting is candid" (GzY7); "rigorous and meticulous experimental design" (WQdT); "only the early exit heads need lightweight training" (ECBk).

AuthorsOpenReview ↗

Summary of the discussion

We thank the reviewers and the Area Chair. The state of each concern:

ConcernReviewerEvidence addedStatus
Baseline protocol, batch dependenceGzY7Table 14Gain in all fifteen settings, incl. batch size 1 and static per-channel PTQ; description corrected
End-to-end speedGzY7, WQdTTable 172–17% faster than the unmodified model, identical predictions
Fixed routes; role of distillationGzY7Table 15+3.36%p with hard labels on shared routes; Sec. 6.1 corrected
Same head on the final [CLS]GzY7, ECBkTable 16Early exits improve on that head in all ten settings with 3.6–10.6% fewer block FLOPs; the head itself gives the larger share of accuracy
Hyperparameter sweepECBkTable 18+2.03%p with no tuning; Eq. 12 matches the best of 255 exit sets
Scale, generalityECBk, WQdTTables 14, 21Positive in FP32 and on ImageNet-V2/-Sketch; recalibration needed on ImageNet-R/-A
Theory–algorithm linkWQdT, GzY7Tables 10, 18, 20Each design choice tested
Scope of the titleECBk–We will state the per-sample reading explicitly in the title and abstract

The method, main-text tables and claims are unchanged; all additions are in App. G.

AuthorsOpenReview ↗

Confidential comment to the Area Chair

Thank you for handling our submission. We are grateful to all three reviewers: their questions led directly to the controls below, and the paper is stronger for them. In case it is useful for the reviewer discussion, here is a brief map of where each point is addressed.

What the revision establishes, using the controls the reviewers suggested

  • Accuracy with less compute, under every protocol tested. LRA-EE is above the full-depth readout in all fifteen backbone–protocol settings, including batch size 1 and static per-channel PTQ (Table 14, suggested by Reviewer GzY7).
  • A real speed-up. The deployed policy is now 2–17% faster end-to-end than the unmodified model, with identical predictions (Table 17, following Reviewers GzY7 and WQdT).
  • No tuning required. An untuned gate already gives +2.03%p, and the label-free exit rule matches the best of all 255 exit sets chosen with labels (Table 18, prompted by Reviewer ECBk).
  • A new quantization-specific finding. Under INT8 the intermediate [CLS] token degrades far more than the patch-token average on all four backbones (Table 19).

Where each review is addressed

Reviewer WQdT kindly confirmed that the response "has addressed our concerns." Reviewer ECBk's points on tuning and evaluation scale are covered in Tables 18 and 21, and the reviewer's later observation on Table 16 helped us state the contribution more precisely. Reviewer GzY7's code-level review was especially valuable: Q1–Q4 are answered in Tables 14–18, and we corrected both descriptions the reviewer identified. We understand how demanding this period is for reviewers and simply note these locations in case they help the discussion.

One point of wording

Table 16, added at Reviewer GzY7's suggestion, shows that training the final readout accounts for the larger share of accuracy, a point Reviewer ECBk also raised and with which we agree; early exits improve on that head in all ten settings while removing compute. We have offered to state this per-sample reading explicitly in the title and abstract — a change of a few words, not of results. The method, main-text tables and claims are otherwise unchanged; all additions are in one appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.