Key details of configuration selection and dynamic batched inference remain unclear, and the manuscript provides no code link for verification. I would consider raising my score if the following concerns are addressed.
1. Baseline comparability and attribution of gains
Comparisons against combined baselines are missing. CANiT reduces computation in two ways: confident samples exit and skip subsequent backbone layers, while samples that continue have patch tokens removed using the exit head's attention scores. Its overall gains therefore come from combining early exiting with token pruning. Comparisons against methods using either technique alone demonstrate the complete system's performance, but do not establish an advantage over straightforward combinations of existing techniques.
Although the paper includes ablations that disable pruning or compare pruning signals, two further controls would be useful: retaining early exiting in CANiT while using backbone CLS attention for pruning, and adding simple token pruning to a representative baseline such as EnViT. These comparisons should use consistent settings and account for the cost of scoring and filtering tokens.
Configuration search and data separation are unclear. Appendix B.2 jointly sweeps exit thresholds and pruning rates to select configurations for different compute budgets. It does not specify which dataset is used for selection, whether that dataset is separate from the final evaluation set, or whether thresholds are optimized independently for each exit.
The baseline methods also adjust thresholds, but it is unclear whether their search spaces and selection procedures are comparable to CANiT's. Please explain how the configuration for each reported point is selected and whether the baselines receive comparable optimization.
The contribution of the exit-head design is difficult to isolate. CANiT uses a simplified Transformer layer exit head and freezes the backbone during exit-head training, achieves substantially better results than several existing methods. Table 5 establishes computational savings over a full Transformer exit head, but does not directly establish an advantage over existing efficient attention heads, such as LGViT's design based on spatial pooling.
Most baseline results are taken from EnViT, whose backbone is described as ViT-B/16, whereas CANiT uses fine-tuned DeiT3-B backbones in the non-ImageNet experiments. The authors use similar final classification accuracy to justify this choice, but early-exit performance depends on intermediate representations, the different pretrained weights and training recipes may affect these representations even when final accuracy is similar. Furthermore, the published ImageNet accuracies of the ViT-B at 81.78% and the DeiT-B checkpoint at 81.98% are closer to EnViT's 81.63% than DeiT3-B's 83.8% is, leaving the rationale for selecting DeiT3-B unclear.
The current comparisons therefore do not isolate the contribution of the exit-head design from differences in the backbones or training and threshold-search procedures. Please compare CANiT against existing efficient attention or MLP exit heads using the same backbone checkpoint and consistent training and threshold-calibration procedures, with token pruning disabled.
2. Throughput gains with large batches need further explanation
Table 3 reports throughput speedups relatively close to FLOP speedups on an A40 GPU at batch size 256. However, average per-sample FLOPs do not capture a central limitation of batched early exiting: remaining samples traverse deeper layers in progressively smaller batches. The batch finishes only after its deepest remaining computation completes, and later-stage costs need not decrease proportionally with sample counts. This can substantially reduce throughput gains. Token pruning reduces computation but does not itself resolve the loss of GPU utilization from shrinking batches.
Appendix B.3 describes tensor indexing and torch.compile with dynamic shapes. These can reduce overhead and avoid shape-related recompilation, but do not establish how efficiently smaller batches execute. Per-stage sample counts and execution times would help quantify utilization losses and filtering overhead, clarifying the contribution of early exiting relative to pruning.
The measurement protocol adds uncertainty. Appendix B.3 states that each batch has exit proportions similar to the full dataset, controlling stage workloads that would vary under random batching. Whether this affects throughput remains unclear. Please explain how batches are formed, whether exit decisions are determined in advance, and whether this information or preprocessing simplifies timed execution.
Validation should compare controlled and random batches under identical settings and measure sensitivity to initial batch size. Timing ablations should evaluate early exiting and pruning separately and together, using the same compilation settings for all variants and the backbone baseline.
Author Response
We thank the reviewer for their feedback and concerns.
Regarding the evaluation strategy for Pareto frontiers, accuracy numbers were optimized and evaluated on the same set of samples following established practice of prior literature [1-2]. Moreover, the search is coarse, with thresholds in steps of 0.1 and just 9 pruning rates, so overfitting on a sample size of 50K images is limited. However, we agree that providing additional results on a separate evaluation set would enhance the results. We had planned to include those results, plus our own local training of EnViT and LGViT, to produce the Pareto values. Unfortunately, during the response period we lost access to our compute and storage in a security incident. The additional experiments and an explicit statement of the procedure for matching cross paper numbers, will be included in future revisions.
[1] Dong et al. EnViT: enhancing the performance of early-exit vision transformers via exit-aware structured dropout-enabled self-distillation. AAAI ‘26
[2] Wang et al. Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition. NeurIPS ‘21