- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates whether the final layer is always the optimal readout point for quantized CLIP models. The authors observe that, due to the accumulation of quantization noise and representation degradation, deeper layers in a quantized model do not necessarily yield higher-quality representations. Based on this insight, they propose the "Layer-wise Representation-Aware Early Exit" (LRA-EE) method, which dynamically selects the readout depth for each input sample by leveraging intermediate layer representations, a learned gating mechanism, and depth-dependent thresholds. Experiments conducted on four INT8 CLIP backbones demonstrate that LRA-EE improves ImageNet accuracy while simultaneously reducing computational cost compared to full-depth inference.
Critical strengths
- Compelling perspective and research motivation. This paper reveals an interesting and somewhat counter-intuitive phenomenon: for quantized CLIP models, the final layer is not always the optimal readout point. The analysis regarding representation maturity and quantization noise offers valuable insight into why deeper-level inference can sometimes actually degrade performance. I believe this observation holds practical value, particularly for deploying large-scale vision models in low-precision environments.
- Robust empirical evaluation and analysis. The experimental evaluation is comprehensive. The authors validated the proposed method across various CLIP backbones, providing not only final accuracy results but also detailed analyses—including ablation studies on the routing mechanism, comparisons with "early-exit" baselines, and a four-quadrant analysis of samples categorized by performance recovery or degradation. These analyses strengthen the empirical findings and help elucidate the reasons behind the method's effectiveness.
- Rigorous and meticulous experimental design. The paper carefully accounts for several potential confounding factors. For instance, the authors distinguish the contribution of adaptive routing from the effect of simply adding intermediate readout heads, and they evaluate actual wall-clock time rather than merely reporting theoretical FLOPs. Furthermore, the information regarding reproducibility and implementation details is thorough.
Critical weaknesses
- The innovation at the algorithmic level is relatively limited. The paper's primary contribution lies in the finding that, within quantized CLIP models, intermediate-layer representations outperform final-layer representations. However, the proposed LRA-EE framework itself is largely built upon existing concepts such as "early exit," confidence-based routing, and the utilization of intermediate-layer features. While combining these components in the context of quantized CLIP models offers practical value, the methodological innovation appears modest when compared to papers that introduce entirely new architectures or optimization techniques. The authors should more clearly articulate the distinction between the novelty of the observed phenomenon and the novelty of the proposed algorithm.
- The generalizability of the proposed method has not been sufficiently validated. Although the paper notes that the framework is not limited to CLIP, all primary experiments were conducted using CLIP image classification models. It remains unclear whether the same observations and performance gains apply to standard supervised ViTs, non-quantized models, or other vision tasks such as retrieval and dense prediction. Given the broad claims made regarding Transformer readout depth, additional validation in scenarios beyond INT8 CLIP classification would strengthen the persuasiveness of the conclusions.
- The extent of actual efficiency gains requires further clarification. Although the paper presents LRA-EE as a highly efficient inference method, the actual reduction in computational cost is relatively modest (with FLOPs decreasing by only about 7.9%–14.5%). Furthermore, while the authors measured wall-clock latency, the overhead introduced by exit heads and gating modules offsets some of the speedup; under certain settings, the adaptive inference path may even run slower than the original model. Consequently, it remains unclear exactly how much benefit LRA-EE offers in practical deployment compared to simply using a smaller model or other acceleration techniques.
- There is a lack of a strong connection between the theoretical analysis and the proposed algorithm. The concept of a "noisy readout channel" offers an interesting perspective for understanding the benefits of "early exit," yet its role in the design of the LRA-EE algorithm is relatively limited. The final routing strategy relies on empirical features and learned thresholds rather than being directly optimized or derived from the theoretical model. The authors should clarify whether the theoretical analysis serves any purpose beyond explaining the observed behavior.
Questions for authors
See Weaknesses.
3 comments
Official Comment by Reviewer WQdT
Thanks for your response, which has addressed our concerns. We will maintain the score.
To Reviewer WQdT
Thank you very much for taking the time to read our response during such a busy period, and for letting us know that your concerns have been resolved. Your comments have made the paper clearer and more rigorous, and we are grateful for your careful engagement throughout the review.
To Reviewer WQdT
Thank you for the detailed and encouraging review. We have addressed each of your four points in the revision (App. G, blue).
W1 - Phenomenon vs. algorithm. We agree and have labelled the three contributions in Sec. 1 accordingly. Phenomenon: readout depth is a per-sample decision variable (Rescue 813 vs. Loss 489, Table 4), and under INT8 the intermediate [CLS] token is the fragile readout while the patch average is not (Table 19). Algorithm: a depth-dependent Chow rule whose components are individually known; we claim the derivation and combination. Measurement: the four-quadrant and exact-route decompositions.
W2 - Generality. The result is not tied to one precision, protocol or test set. The gain is +3.67 / +2.47 / +2.91%p (B/32, B/16, EVA) in FP32 and positive under all four INT8 protocols on each (up to +6.56%p on EVA, Table 14); it composes with a trained final readout (67.86% FP32, 65.99% INT8 on B/32, Table 16); and it holds without recalibration on ImageNet-V2 (+1.23%p) and -Sketch (+0.61%p) (Table 21). Retrieval and dense prediction are outside the gate's scope (fixed label set, Sec. 8); supervised ViTs are not evaluated and not claimed.
W3 - Efficiency. The adaptive path is now faster than the original model: −2.1% to −6.6% at the paper's operating point and −10.1% to −14.3% with no tuning, both above baseline accuracy (Table 17). Token pruning on the same INT8 backbone gives 58.50% at 13.5% saving vs. our 60.65% at 11.1% (Table 2); it acts on a different axis and can be combined, as can a smaller backbone.
W4 - Theory and algorithm. Each design choice the theory motivates is tested (App. G.4): (1) margin as the gate statistic: 60.47% margin-only vs. 58.21% confidence-only (Table 10); (2) a threshold decreasing with depth: γ>0 on all four backbones, worth 0.49–0.76%p over a constant threshold; (3) the label-free maturity rule (Eq. 12) matches exhaustive label-based selection over 255 exit sets; (4) on ViT-L/14 the best patch-readout layer moves from 23 to 20–22 under INT8 (Table 20).
Presentation. We have added a five-line method summary and a notation table (App. G). If a specific passage is still hard to follow, we would be glad to revise it.