- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
LADDER converts pretrained vision-language ViTs into efficient hybrid recurrent encoders. It combines AttnCLS for global aggregation with reinforcement learning to select a small number of full-attention layers. With 22 epochs of image-only training on ImageNet-1K, it largely preserves teacher capabilities on TIPSv2 and SigLIP2 while improving high-resolution inference efficiency.
Critical strengths
- Controlled experiments and ablations support the complementary roles of direct CLS aggregation and retained full-attention layers.
- LADDER-L stays within one percentage point of its teacher on the main retrieval metrics and achieves 1.74× throughput over the FlashAttention teacher on Cityscapes inputs.
Critical weaknesses
- Resolutions 224 and 896 are used in both architecture search and the main evaluation. Systematic testing across multiple held-out resolutions is missing.
Questions for authors
- Does the policy outperform random or local search under the same search budget?
Thank you for your positive assessment and your valuable suggestions of our work. We have added experiments on unseen resolutions and a new Section 4.4 comparing attention placement strategies.
W1. Generalization to unseen resolutions
We originally chose 224, 448, and 896 as representative resolutions, with a 4x increase in patch tokens between adjacent sizes. Following your suggestion, we now evaluate 168, 336, 672, and 1344, all held out from student weight training and attention placement search. We have added these results and discussion to Appendix G, Table A11.
The results show that LADDER maintains strong cross-resolution performance at resolutions not used for attention placement search. At 1344, beyond both student training and placement search resolutions, LADDER-L remains within 0.85 percentage points of its teacher on all four retrieval metrics. At 168, it exceeds the teacher on all four. Both LADDER-B and LADDER-L also outperform their teachers on ADE150 at every new resolution, further supporting LADDER's cross-resolution generalization.
Q1. Comparison with alternative placement strategies
Thank you for encouraging a more direct comparison. JetViT [1] and FlashMorph [2] are relevant external methods for search-based and optimization-based layer selection, respectively. As of this rebuttal, neither work has released its official implementation. JetViT targets specific downstream vision tasks, while FlashMorph targets language models. In the new Section 4.4, we therefore reimplemented their selection strategies, beam search and continuous gating, within LADDER using cross-resolution teacher-feature recovery as the objective, and added greedy search as a simple baseline.
All strategies use the same aligned, frozen visual branches and retain the same number of full-attention layers. Greedy search keeps one candidate per stage, beam search keeps four, and continuous gating optimizes branch mixtures before top-K selection. Implementation details are in Appendix D.4.
Interestingly, all four strategies find the same Base placement among 220 configurations, where greedy search is the cheapest. At Large scale, with 134,596 configurations, RL achieves the best feature recovery at lower estimated cost than beam search and continuous gating. Beam search comes close in quality; greedy is cheaper but yields weaker recovery. RL jointly models retained-layer interactions while keeping selection lightweight.
References
[1] Dongyun Zou et al. "JetViT: Efficient High-Resolution Vision Transformer with Post-Training Attention Search." arXiv:2605.26636, 2026.
[2] Disen Lan et al. "Morphing into Hybrid Attention Models." arXiv:2606.30562, 2026.