acceptodds
Under review as a conference paper at ICLR 2027

Linearizing Vision-Language ViTs Beyond the Training Resolution

Abstract

Linear recurrent visual encoders offer increasing computational advantages at high resolutions, yet representative linear CLIP models exhibit substantial performance degradation when the visual sequence length moves beyond their low training resolution. Strong pretrained vision-language ViTs retain high-quality representations beyond their training resolutions, whereas naively linearized recurrent encoders fail to inherit this capability. In this work, we introduce LADDER (**L**ayer **AD**aptive **D**istillation into **E**fficient Hybrid Encode**R**), a lightweight framework that converts pretrained vision-language ViTs into hybrid recurrent encoders while preserving their cross-resolution capabilities through architectural designs. LADDER combines attention and recurrence within blocks and across layers. For global feature aggregation, our proposed Vision-Delta pairs recurrent patch mixing with Attention-based CLS (AttnCLS), decoupling global readout from the block's recurrent propagation. For dense feature interaction, Resolution-Robust Layer-Adaptive Hybridization optimizes full-attention placement under a fixed budget via reinforcement learning guided by cross-resolution teacher patch-feature recovery. We primarily evaluate LADDER on TIPSv2, with additional validation on SigLIP2. With only 22 epochs of image-only training on ImageNet-1K, LADDER closely matches or surpasses its teachers across vision-language and linear probing evaluations, while matching the ViT teacher to within 1% point at Large scale. On high-resolution Cityscapes inputs, LADDER achieves the throughput of the FlashAttention teacher. Code and models will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Foundation Models, Model Linearization, Efficient Foundation Models, Cross Architectue Knowledge Distillation, Sequence-Length Generalization, Hybrid Attention, Linear CLIP, Hybrid Attention Layer Selection

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.50 over 4 reviews, above 99% of the others.
Reviewer_Zau6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LADDER converts pretrained vision-language ViTs into efficient hybrid recurrent encoders. It combines AttnCLS for global aggregation with reinforcement learning to select a small number of full-attention layers. With 22 epochs of image-only training on ImageNet-1K, it largely preserves teacher capabilities on TIPSv2 and SigLIP2 while improving high-resolution inference efficiency.

Critical strengths
  1. Controlled experiments and ablations support the complementary roles of direct CLS aggregation and retained full-attention layers.
  2. LADDER-L stays within one percentage point of its teacher on the main retrieval metrics and achieves 1.74× throughput over the FlashAttention teacher on Cityscapes inputs.
Critical weaknesses
  1. Resolutions 224 and 896 are used in both architecture search and the main evaluation. Systematic testing across multiple held-out resolutions is missing.
Questions for authors
  1. Does the policy outperform random or local search under the same search budget?
1 comment

Thank you for your positive assessment and your valuable suggestions of our work. We have added experiments on unseen resolutions and a new Section 4.4 comparing attention placement strategies.

W1. Generalization to unseen resolutions

We originally chose 224, 448, and 896 as representative resolutions, with a 4x increase in patch tokens between adjacent sizes. Following your suggestion, we now evaluate 168, 336, 672, and 1344, all held out from student weight training and attention placement search. We have added these results and discussion to Appendix G, Table A11.

ResolutionModelCOCO I→TCOCO T→IFlickr30K I→TFlickr30K T→IADE150
168TIPSv2-B57.1442.5880.3065.5814.85
168LADDER-B56.4642.8679.3064.9817.38
336TIPSv2-B68.3052.3591.4078.3216.84
336LADDER-B66.0051.0288.1076.0019.40
672TIPSv2-B69.4653.9891.8079.8617.88
672LADDER-B66.7852.4289.0078.3419.80
1344TIPSv2-B61.9048.5985.0073.5216.29
1344LADDER-B61.0248.5883.0073.0217.85
168TIPSv2-L60.2248.0183.1071.5620.83
168LADDER-L63.5249.5785.1073.3621.67
336TIPSv2-L71.2255.5493.6080.7624.08
336LADDER-L70.8855.3393.0080.5424.46
672TIPSv2-L71.7056.5494.9082.3625.01
672LADDER-L71.3256.3794.4082.0425.51
1344TIPSv2-L69.0054.5691.4079.5624.53
1344LADDER-L68.1654.8490.8080.1425.02

The results show that LADDER maintains strong cross-resolution performance at resolutions not used for attention placement search. At 1344, beyond both student training and placement search resolutions, LADDER-L remains within 0.85 percentage points of its teacher on all four retrieval metrics. At 168, it exceeds the teacher on all four. Both LADDER-B and LADDER-L also outperform their teachers on ADE150 at every new resolution, further supporting LADDER's cross-resolution generalization.

Q1. Comparison with alternative placement strategies

Thank you for encouraging a more direct comparison. JetViT [1] and FlashMorph [2] are relevant external methods for search-based and optimization-based layer selection, respectively. As of this rebuttal, neither work has released its official implementation. JetViT targets specific downstream vision tasks, while FlashMorph targets language models. In the new Section 4.4, we therefore reimplemented their selection strategies, beam search and continuous gating, within LADDER using cross-resolution teacher-feature recovery as the objective, and added greedy search as a simple baseline.

All strategies use the same aligned, frozen visual branches and retain the same number of full-attention layers. Greedy search keeps one candidate per stage, beam search keeps four, and continuous gating optimizes branch mixtures before top-K selection. Implementation details are in Appendix D.4.

ScaleStrategyPatch cosine @224 (%)Patch cosine @896 (%)Est. GPU-hours
BaseContinuous gating74.9672.311.838
BaseBeam search74.9672.310.161
BaseGreedy search74.9672.310.068
BaseRL (ours)74.9672.310.104
LargeContinuous gating93.9188.656.049
LargeBeam search94.1789.352.489
LargeGreedy search94.0688.930.809
LargeRL (ours)94.2889.481.966

Interestingly, all four strategies find the same Base placement among 220 configurations, where greedy search is the cheapest. At Large scale, with 134,596 configurations, RL achieves the best feature recovery at lower estimated cost than beam search and continuous gating. Beam search comes close in quality; greedy is cheaper but yields weaker recovery. RL jointly models retained-layer interactions while keeping selection lightweight.

References

[1] Dongyun Zou et al. "JetViT: Efficient High-Resolution Vision Transformer with Post-Training Attention Search." arXiv:2605.26636, 2026.

[2] Disen Lan et al. "Morphing into Hybrid Attention Models." arXiv:2606.30562, 2026.

Reviewer_pzoQOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to convert pretrained vision-language ViTs into efficient hybrid encoders while preserving their representations across input resolutions. The claimed contributions are: 1.Vision-Delta combines recurrent patch mixing with direct attention-based CLS readout to improve global representation robustness. 2.A cross-resolution RL policy selects a fixed budget of full-attention layers using shared layer and pairwise preferences. The student inherits compatible teacher weights, aligns its replacement mixers, and undergoes image-only feature distillation for 22 conversion-training epochs on ImageNet-1K. Experiments on TIPSv2 and SigLIP2 cover retrieval, zero-shot segmentation, classification, dense probing, and inference throughput. The evidence supports useful high-resolution conversion, with stronger support for AttnCLS than for the necessity of the particular RL selector.

Critical strengths
  1. The controlled experiments isolate an effective architectural remedy for cross-resolution degradation. In Table 2, changing Vision-Delta's readout from RnnCLS to AttnCLS raises Flickr image-to-text R@1 at 896 from 60.8 to 79.1, while performance at 448 remains nearly unchanged (88.8 vs. 88.7). The paper also includes recurrent conversion baselines trained under the same transfer and distillation protocol.
  2. The method preserves broad pretrained capabilities while delivering measured high-resolution acceleration. Table 1 covers TIPSv2 and SigLIP2 across global and dense tasks. LADDER-L remains within one percentage point of TIPSv2-L on all four retrieval metrics after 22 epochs of image-only conversion. Figure 1 reports 1.74x throughput on Cityscapes inputs relative to the FlashAttention teacher; Appendix E.5 specifies the hardware and inference protocol.
Critical weaknesses

The specific benefit of the RL selector and its pairwise preferences is insufficiently established against simpler or existing selection strategies. Tables 3 and A5 mainly compare uniform placement and the authors' policy variants. Layer-only and layer-plus-pair policies select the same Base architecture. At Large scale, adding pairwise scores changes frozen patch cosine similarity at 896 from 89.3 to 89.5, but final downstream results after distillation are not reported for the layer-only alternative. Section 2.2 and Appendix B.3 discuss JetViT and FlashMorph without matched empirical comparisons. The reported five-seed placement consistency supports reproducibility of selection, but does not quantify a final-task advantage over these alternatives.

Questions for authors

(A) Clarification requests that could affect the assessment Q: With the same teacher, aligned mixers, attention budget, and search-time allowance, how does the final distilled model selected by the pairwise policy compare with layer-only selection and a practical alternative such as greedy search or an adapted continuous-gate selector? A reproducible downstream or search-cost advantage would address my main reservation and could justify raising the rating to 4. Comparable final accuracy and cost would leave the RL contribution less established and favor retaining rating 3 on the strength of the remaining contributions. Q: What is the total conversion cost in GPU-hours, including teacher inference, mixer alignment, architecture search, and feature distillation? How much of that cost is due to search, and how does it compare with a simpler selector under the same conversion setup? Appendix E.4 excludes frozen search and teacher passes from the visual-token count. A small measured search cost would strengthen the case for the proposed selector; substantial overhead without a final-task advantage would weaken that case and could change my overall recommendation.

(B) Suggestions for future work, not required for the current rating Evaluate additional resolutions excluded from both weight training and architecture selection, extending the existing Cityscapes evidence. End-to-end multimodal-model latency would also help establish how encoder-level acceleration translates into deployment benefits.

5 comments
AuthorsOpenReview ↗

Response to Reviewer pzoQ (Part 1/3): Weakness 1.1, 1.2 and Question A.1

Thank you for your positive assessment of LADDER's architectural design and high-resolution conversion results. Following your suggestions, we have added alternative placement comparisons, evaluated the final distilled layer-only model, repeated the complete Base conversion pipeline over five seeds, and added experiments at unseen resolutions.

W1.1 Comparison with alternative placement strategies.

Thank you for encouraging a more direct comparison. JetViT [1] and FlashMorph [2] are relevant external methods for search-based and optimization-based layer selection, respectively. JetViT targets specific downstream vision tasks, while FlashMorph targets language models. As of this rebuttal, neither work has released its official implementation. We implemented their layer-selection strategies within LADDER using cross-resolution teacher-feature recovery as the objective.

All strategies use the same teacher and aligned, frozen visual branches, and retain the same number of full-attention layers. Beam search retains the top bb candidate placements at each stage; we use b=4b=4, with greedy search as its b=1b=1 variant. Continuous gating learns continuous gates that mix attention and recurrent branches, then retains full attention in the top-KK layers. The results are shown in the table below.

ScaleStrategyPatch cosine @224 (%)Patch cosine @896 (%)Est. GPU-hours
BaseContinuous gating74.9672.311.838
BaseBeam search, b=474.9672.310.161
BaseGreedy search, b=174.9672.310.068
BaseRL (ours)74.9672.310.104
LargeContinuous gating93.9188.656.049
LargeBeam search, b=494.1789.352.489
LargeGreedy search, b=194.0688.930.809
LargeRL (ours)94.2889.481.966

Interestingly, all four strategies find the same Base placement, {4, 5, 12}, among 220 configurations. This compact search space permits exhaustive evaluation, and greedy search is the most efficient in this setting.

At Large scale, the space grows to 134,596 placements. Our layer-and-pair RL policy achieves the best feature recovery at lower estimated cost than beam search and continuous gating. Beam search comes close in quality, while greedy search is cheaper but yields weaker recovery. These results highlight the effectiveness and efficiency of our RL method in modeling interactions among multiple retained attention layers in a larger search space.

We have incorporated these experiments and analysis into Section 4.4, "Comparison of Attention Placement Strategies," in the main text. Further implementation details are provided in Appendix D.4.

W1.2 & QA.1 Downstream benefit of layer selection.

To directly examine whether the pairwise preferences improve the final distilled model, we now compare Uniform, layer-only RL, and layer-and-pair RL at Large scale, using the same teacher, aligned mixers, attention budget and linearization settings.

PlacementFrozen cosine @224Frozen cosine @896I→T @224I→T @896T→I @224T→I @896
Uniform93.286.788.391.376.580.6
Layer-only RL93.889.389.493.176.981.5
Layer-and-pair RL94.389.590.393.977.381.9

Cosine and Flickr30K R@1 are expressed as percentages; retrieval is evaluated after the full conversion pipeline. At both reward resolutions, the ranking is Uniform < Layer-only < Layer-and-pair for frozen cosine and for both retrieval directions after distillation. At 896, adding pairwise preferences improves final I→T by 0.8 points and T→I by 0.4 points over layer-only selection. The gains identified during frozen evaluation therefore persist through subsequent training, demonstrating the downstream value of modeling interactions among retained attention layers.

AuthorsOpenReview ↗

Response to Reviewer pzoQ (Part 2/3): Weakness 1.3

W1.3 Performance stability across runs.

Our main conversion comparisons use the same training seed, 423, under matched settings. We further repeated the complete Base conversion pipeline over five seeds: 41, 42, 43, 44, and 45, rerunning mixer alignment, placement selection for LADDER, and feature distillation. We hope you understand that this multi-seed evaluation primarily focuses on the Base scale, as repeating the full linearization pipeline at the Large scale is computationally prohibitive within the short rebuttal window. The table below reports the mean ± standard deviation across the five runs.

ResolutionModelFlickr30K I→TFlickr30K T→IADE150
224Uniform Hybrid-B84.58 ± 0.3470.83 ± 0.2716.27 ± 0.18
224LADDER-B85.80 ± 0.3270.89 ± 0.2717.27 ± 0.17
448Uniform Hybrid-B90.12 ± 0.3277.71 ± 0.2519.92 ± 0.22
448LADDER-B89.50 ± 0.3177.85 ± 0.2419.69 ± 0.21
896Uniform Hybrid-B85.20 ± 0.3675.16 ± 0.2917.39 ± 0.20
896LADDER-B86.78 ± 0.3476.31 ± 0.2818.63 ± 0.19

At 896, LADDER improves the means by 1.58 points in I→T, 1.15 in T→I, and 1.24 mIoU. These gains are substantially larger than the observed per-method standard deviations, supporting the stability of the high-resolution improvements across complete conversion runs.

AuthorsOpenReview ↗

Response to Reviewer pzoQ (Part 3/3): Question A.2 and B

QA.2 Total conversion cost and attention placement search overhead

Following your request, we estimate the complete conversion cost, including teacher inference, mixer alignment, architecture search, and feature distillation.

StageBase: Est. GPU-hoursLarge: Est. GPU-hours
Mixer alignment: 448, 1 epoch4.12414.205
Feature distillation: 224, 20 epochs35.306123.998
Final fine-tuning: 448, 1 epoch7.48925.905
Initial reward calibration: 1,024 images0.0160.053
RL selection0.1041.966
Total conversion47.04166.13
Selection + calibration / total0.26%1.22%

Selection accounts for only 0.26% of Base conversion and 1.22% of Large conversion. The linearization conversion cost is dominated by weight training. RL selection evaluates frozen visual encoders through forward passes and updates only a compact policy, keeping this stage lightweight.

The strategy comparison above also shows that, at Large scale, RL reduces estimated selection cost relative to beam search and continuous gating, while achieving stronger feature recovery. Greedy search is cheaper, but fails to identify a superior placement. Continuous gating incurs the highest selection overhead, as evaluating both mixer branches and backpropagating through the frozen student add considerable computational cost. In contrast, RL evaluates discrete encoders via forward passes and updates only a compact policy. Thus, our proposed RL selector adds very little overhead to the full linearization pipeline.

QB. Unseen resolutions and future applications

Thank you for these valuable suggestions. During the rebuttal, we evaluated 168, 336, 672, and 1344, all held out from student weight training and attention placement search. These results and discussion are now included in Appendix G, Table A11. At 1344, beyond both training and search resolutions, LADDER-L remains within 0.85 percentage points of its teacher on all four retrieval metrics. Both LADDER-B and LADDER-L also outperform their teachers on ADE150 at every new resolution, further supporting LADDER's cross-resolution generalization.

ResolutionModelCOCO I→TCOCO T→IFlickr30K I→TFlickr30K T→IADE150
168TIPSv2-B57.1442.5880.3065.5814.85
168LADDER-B56.4642.8679.3064.9817.38
336TIPSv2-B68.3052.3591.4078.3216.84
336LADDER-B66.0051.0288.1076.0019.40
672TIPSv2-B69.4653.9891.8079.8617.88
672LADDER-B66.7852.4289.0078.3419.80
1344TIPSv2-B61.9048.5985.0073.5216.29
1344LADDER-B61.0248.5883.0073.0217.85
168TIPSv2-L60.2248.0183.1071.5620.83
168LADDER-L63.5249.5785.1073.3621.67
336TIPSv2-L71.2255.5493.6080.7624.08
336LADDER-L70.8855.3393.0080.5424.46
672TIPSv2-L71.7056.5494.9082.3625.01
672LADDER-L71.3256.3794.4082.0425.51
1344TIPSv2-L69.0054.5691.4079.5624.53
1344LADDER-L68.1654.8490.8080.1425.02

We will further explore high-resolution applications and end-to-end multimodal-model latency to examine how the encoder-level acceleration translates into practical deployment benefits.

References

[1] Dongyun Zou et al. "JetViT: Efficient High-Resolution Vision Transformer with Post-Training Attention Search." arXiv:2605.26636, 2026.

[2] Disen Lan et al. "Morphing into Hybrid Attention Models." arXiv:2606.30562, 2026.

Reviewer_9ukyOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors come up with an innovative framework (LADDER) to extract the strengths of ViT's performance across a wider resolution scale, to a recurrent visual encoder architecture which has a competitive computational advantage against the former at higher resolution. A simple linearization of a ViT into a recurrent architecture alone does not achieve this. LADDER achieves this by combining attention and recurrence within blocks across layers of a ViT and converting it into a hybrid recurrent architecture. Most layers undergo this transformation and some chosen layers retain full attention under a fixed budget using RL.

The major papers contributions are as follows:

  1. The Vision-Delta block, a Gated DeltaNet-style patch mixer whose CLS tokens attend directly to the patch keys and values instead of going through the recurrent scan. It also scans half the heads in reverse and initializes the state from mean patch statistics.
  2. RL search is used to evaluate candidates (attention layers that will be completely retained) with a reward mechanism to encourages frozen layer placement which recovers teacher patch features across resolutions.
  3. Experimental analysis of their claims. Accuracy and robustness results are strong for the tested claim.

To validate their approach the authors use LADDER on base and large ViT encoders of TIPSv2 and compare the hybrid outputs (LADDER-B and LADDER-L) to CLIMP and RWKV-CLIP. They report additional experiments converting SigLIP2 into LADDER-Sig to demonstrate the framework's generalizability across different teachers.

Critical strengths
  1. I like the idea of generating a hybrid model by dividing layers between attention layers (for maximizing performance) and recurrent ones (for improving efficiency). Testing all candidates to identify the optimal layers for complete attention retention can be costly, and utilizing RL here seems to be an interesting and intuitive approach.
  2. The CLS token design is an effective engineering fix addressing a specific failure mode. For a CLIP-style model, the global embedding comes from the CLS token, and routing it through a recurrent scan compresses it, leading to a clear performance jump.
  3. The paper is well written and easy to follow.
Critical weaknesses
  1. Placement search (selection of layers that retain full attention) is weakly validated. The paper lacks random-search, greedy or beam-search baseline. For LADDER-Base, I assume that all 220 placements could be evaluated with the frozen post-alignment model at lower cost than the RL search.
  2. Competing conversion methods (like JetViT or FlashMorph) have not been included as baselines for comparison, which makes the absolute effect of their framework difficult to evaluate. Comparison with other hybrid methods is a reasonable ablation for this paper.
  3. The learned placement improves over uniform placement mainly at the unseen 896 resolution, and to a lesser extent at 224. At 448, the fine-tuning resolution, it is flat or slightly worse for both Base and Large. Absolute gains are small (about 1 to 3 points) and are reported from single runs, so it is hard to tell whether they exceed run-to-run variance and noise.
Questions for authors
  1. Could the authors report whether the frozen-model reward correlates with performance after distillation? This will help gain better clarity about the impact RL has for their use case.
  2. Related work lists several competing conversion methods (ViT-Linearizer, ICCV 2025; ViT-AdaLA, 2026; KL-guided layer selection, ICLR 2026; JetViT and FlashMorph, 2026), but results in the paper includes only naive Mamba conversions and uniform placement. Will it be possible to add experimental results for the TIPSv2-B teacher converted with ViT-Linearizer’s and ViT-AdaLA’s recipes under the same data and token budget? I understand that JetViT and FlashMorph are very recent publications and beating them will not be a fair ask for this conference's acceptance. Just the former results should suffice.
  3. The AttnCLS idea of a summary token attending directly to patch keys and values has clear precedent (CaiT’s class attention, SigLIP2’s MAP head), which the appendix acknowledges. Can the authors move this positioning into the main text and add a baseline with a single late class-attention/MAP readout on a fully recurrent student, to show that the per-block, pre-scan design is what drives the cross-resolution gains?
  4. Table 2 reports single runs. Could you provide mean and standard deviation over several seeds of the full pipeline? This will help address weakness 3.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 9uky(Part 1/3): Weakness

Thank you for your positive assessment of LADDER's hybrid design. Following your suggestions, we have added alternative placement strategies, evaluated the requested conversion methods, repeated the complete Base conversion pipeline over five seeds, and added a last-block attention-readout control.

W1 and W2. Comparisons with other attention placement search

Thank you for encouraging a more direct comparison. JetViT [1] and FlashMorph [2] are relevant external methods for search-based and optimization-based layer selection, respectively. As of this rebuttal, neither work has released its official implementation. JetViT targets specific downstream vision tasks, while FlashMorph targets language models. We therefore implemented their layer-selection strategies within LADDER using cross-resolution teacher-feature recovery as the objective. Beam search retains the top bb candidate placements at each stage, where bb is the beam width. We also added greedy search as a variant of beam search. Continuous gating learns continuous gates that mix attention and recurrent branches, then retains full attention in the top-KK layers. The results are shown in the table below.

ScaleStrategyPatch cosine @224 (%)Patch cosine @896 (%)Est. GPU-hours
BaseContinuous gating74.9672.311.838
BaseBeam search, b=474.9672.310.161
BaseGreedy search, b=174.9672.310.068
BaseRL (ours)74.9672.310.104
LargeContinuous gating93.9188.656.049
LargeBeam search, b=494.1789.352.489
LargeGreedy search, b=194.0688.930.809
LargeRL (ours)94.2889.481.966

We agree with your observation about Base scale: 220 placements make exhaustive evaluation feasible. Interestingly, all four strategies find the same placement, {4, 5, 12}, and greedy search is the most efficient in this setting.

At Large scale, the space grows to 134,596 placements. Our layer-and-pair RL policy achieves the best feature recovery at lower estimated cost than beam search and continuous gating. Beam search comes close in quality, while greedy search is cheaper but yields weaker recovery. These results highlight the effectiveness and efficiency of our RL method in modeling interactions among multiple retained attention layers in a larger search space.

We have incorporated these experiments and a more comprehensive analysis into Section 4.4, “Comparison of Attention Placement Strategies,” in the main text. Further implementation details are provided in Appendix D.4.

W3 and Q4. Cross-resolution gains and variation across runs

Our main conversion comparisons use the same training seed, 423, under matched settings. To address your concern about seed robustness, we repeated the complete Base conversion pipeline over five seeds: 41, 42, 43, 44, and 45, rerunning mixer alignment, placement selection for LADDER, and feature distillation. We hope you understand that this multi-seed evaluation primarily focuses on the Base scale, as repeating the full linearization pipeline at the Large scale is computationally prohibitive within the short rebuttal window. The table below reports the mean ± standard deviation across the five runs.

ResolutionModelFlickr30K I→TFlickr30K T→IADE150
224Uniform Hybrid-B84.58 ± 0.3470.83 ± 0.2716.27 ± 0.18
224LADDER-B85.80 ± 0.3270.89 ± 0.2717.27 ± 0.17
448Uniform Hybrid-B90.12 ± 0.3277.71 ± 0.2519.92 ± 0.22
448LADDER-B89.50 ± 0.3177.85 ± 0.2419.69 ± 0.21
896Uniform Hybrid-B85.20 ± 0.3675.16 ± 0.2917.39 ± 0.20
896LADDER-B86.78 ± 0.3476.31 ± 0.2818.63 ± 0.19

At 896, LADDER improves the means by 1.58 points in I→T, 1.15 in T→I, and 1.24 mIoU. These gains are substantially larger than the observed per-method standard deviations, supporting the stability of the high-resolution improvements.

The concentration of gains away from 448 reflects our objective: placement is selected for feature recovery at 224 and 896, while 448 is the final fine-tuning resolution. Table 3 also shows that changing the reward to 448 alone raises frozen cosine at 448 to 96.2, compared with 95.8 for Uniform and 95.7 for the cross-resolution policy. Thus, the selection objective accounts for the resolution trade-off.

AuthorsOpenReview ↗

Response to Reviewer 9uky(Part 2/3): Question 1&2

Q1. Relationship between the frozen reward and performance after distillation

Thank you for raising this important question. To directly examine whether the frozen-model reward translates into downstream gains after full training, We now compare Uniform, layer-only RL, and layer-and-pair RL at Large scale, using the same linearization settings. As shown below, we observe a consistent ordering between frozen feature recovery and final cross resolution retrieval.

PlacementFrozen cosine @224Frozen cosine @896I→T @224I→T @896T→I @224T→I @896
Uniform93.286.788.391.376.580.6
Layer-only RL93.889.389.493.176.981.5
Layer-and-pair RL94.389.590.393.977.381.9

Cosine and Flickr30K R@1 are expressed as percentages. At both reward resolutions, the ranking is Uniform < Layer-only < Layer-and-pair for frozen cosine and for both retrieval directions after distillation. The gains identified by the frozen evaluation therefore persist through subsequent training. This supports cross resolution teacher feature recovery as a useful criterion for selecting the final hybrid architecture.

Q2. Comparisons with other linearization methods

In the newly added Section 4.4, we have implemented related hybrid layer-selection strategies within LADDER, including beam search inspired by JetViT [1] and continuous gating inspired by FlashMorph [2]. To further compare against purely linearization methods, we conducted additional analyses and evaluations on ViT-Linearizer [3] and ViT-AdaLA [4].

Q2.1. ViT-Linearizer under the same short conversion budget.

Since ViT-Linearizer has not released official code, we reproduced its core activation matching and masked prediction objectives using the TIPSv2-B/14 teacher and the official Adventurer-B architecture. To ensure a fair comparison with LADDER, the student is initialized with compatible pretrained weights from the teacher.

Training uses ImageNet-1K and the same 22-epoch resolution schedule as LADDER: one epoch at 448, twenty at 224, and one at 448, totaling approximately 9.18B input patch tokens. The results are shown in the table below.

ResolutionModelCOCO I→TCOCO T→IFlickr30K I→TFlickr30K T→IADE150
224TIPSv2-B teacher63.8648.5286.9072.9015.00
224ViT-Linearizer6.6810.3211.1017.022.75
448TIPSv2-B teacher70.2053.6691.9079.6017.40
448ViT-Linearizer8.3215.6016.9025.922.97
896TIPSv2-B teacher67.7252.9790.0078.3017.60
896ViT-Linearizer2.007.614.3012.680.76

The results indicate that ViT-Linearizer fails to preserve usable zero-shot representations under a short conversion budget. Our interpretation is that masked representation learning requires a longer training budget to recover the teacher's vision-language space. With only 25% of the image content visible, its supervision is more indirect than LADDER's dense feature matching on complete images. This is consistent with the original ViT-Linearizer's 300-epoch distillation schedule. Under our short conversion budget, this masked-distillation recipe does not adequately preserve zero-shot image-text alignment, with further degradation at 896.

Q2.2. ViT-AdaLA's official pretrained student.

At the time of our initial main text submission, ViT-AdaLA was only available as an incomplete preprint. Recently, the authors updated their finalized NIPS 2026 version and released official model checkpoints. Here, we evaluate the authors' official CLIP-L checkpoint at resolutions of 256, 512, and 1024 on image-text retrieval tasks. The results are shown in the table below.

ResolutionModelCOCO I→TCOCO T→IFlickr30K I→TFlickr30K T→I
256CLIP-L teacher65.5849.9390.1075.52
256ViT-AdaLA9.7410.2223.5018.14
512CLIP-L teacher63.5247.6788.0074.84
512ViT-AdaLA11.6224.1630.5045.48
1024CLIP-L teacher52.6839.1481.7066.96
1024ViT-AdaLA9.8621.7716.1030.22

The official student falls substantially below its teacher in global image-text retrieval. Meanwhile, its cross-resolution performance also exhibits additional degradation when moving beyond its training resolution. This is because ViT-AdaLA focuses on transferring representations for supervised visual downstream tasks. Its implementation aligns final patch features without a direct global CLS distillation objective, while LADDER explicitly supervises the global image-text representation alongside patch features and focuses on preserving cross-resolution capabilities.

AuthorsOpenReview ↗

Response to Reviewer 9uky(Part 3/3): Question 3

Q3. AttnCLS positioning and a single late attention readout

Thank you for your interest and constructive suggestions regarding AttnCLS. We have added a dedicated discussion in Section 4.5 ("AttnCLS discussion") of the main text to clarify its positioning relative to CaiT's class attention [5] and SigLIP2's multihead attention pooling [6]. Importantly, rather than a direct repurposing of existing pooling mechanisms, AttnCLS is specifically motivated by, and designed to resolve, the cross resolution failure mode of recurrent CLS readout uncovered in our work.

Following your suggestion, Table 5 compares three readout arrangements in fully recurrent Base encoders. The new last-block AttnCLS variant uses only one AttnCLS in block 12. All twelve blocks use recurrent patch mixing, and all three students follow the same 22 epoch conversion schedule and Web CLS evaluation protocol.

ReadoutI→T @224I→T @448I→T @896T→I @224T→I @448T→I @896
TIPSv2-B teacher86.991.990.072.979.678.3
RnnCLS76.088.860.866.277.558.1
Last-block AttnCLS76.488.364.865.477.562.3
Per-block AttnCLS82.188.779.169.477.369.7

While all three readouts perform comparably at the fine-tuning resolution of 448, a single lastblock attention readout fails to prevent severe cross resolution degradation, whereas perblock AttnCLS maintains robust retrieval across both 224 and 896. This confirms that cross resolution robustness stems from our perblock, prescan integration rather than late attention pooling alone: a last block readout still relies on CLS queries and residual representations distorted by preceding recurrent scans, whereas perblock AttnCLS progressively refines the global representation at every depth. Moreover, in hybrid LADDER, retained full attention blocks further allow this global representation to interact with patch features. These results highlight the value of maintaining direct global aggregation throughout depth for cross resolution robustness.

References

[1] Dongyun Zou et al. "JetViT: Efficient High-Resolution Vision Transformer with Post-Training Attention Search." arXiv:2605.26636, 2026.

[2] Disen Lan et al. "Morphing into Hybrid Attention Models." arXiv:2606.30562, 2026.

[3] Guoyizhe Wei and Rama Chellappa. "ViT-Linearizer: Distilling Quadratic Knowledge into Linear-Time Vision Models." ICCV, 2025.

[4] Yifan Li et al. "ViT-AdaLA: Adapting Vision Transformers with Linear Attention." NeurIPS, 2026.

[5] Hugo Touvron et al. "Going Deeper With Image Transformers." ICCV, 2021.

[6] Michael Tschannen et al. "SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features." arXiv:2502.14786, 2025.

Reviewer_bWR2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper takes a strong pretrained vision-language ViT and converts it into a faster hybrid encoder that still works well across image resolutions. Plain Mamba-style conversions match the teacher at the resolution they were tuned at but fall off at other sizes. LADDER fixes this by replacing most attention blocks with a new recurrent mixer, keeping attention for the CLS readout, and using RL to choose which quarter of the layers keep full attention.

Critical strengths

The benefits at much higher resolution are meaningful. The methodology is generally sound.

Critical weaknesses

I believe the main reason to use hybrid approaches is efficiency. Maybe there are other good reasons, but that seems like a big one. Because of that I wish there had been some comparisons against other methods like windowed vision transformers such as Swin or other.

Questions for authors

none

1 comment

Thank you very much for your positive assessment and your interest in our work.

W1. Comparison with window attention

Recent recurrent and hybrid visual backbones, including VMamba [1] and MambaVision [2] have surpassed Swin [3] on standard vision benchmarks at comparable FLOPs. To further examine this choice in LADDER, we compare recurrent and Swin-style shifted-window attention within the same TIPSv2-B backbone.

Swin's hierarchical downsampling is well suited to dense vision tasks, while LADDER converts plain vision-language ViTs. We therefore preserve the teacher's patch grid and replace the nine recurrent patch mixers in Uniform Hybrid-B with window attention. Adjacent window layers alternate between regular and shifted window partitions, following Swin. Both variants retain full attention at layers 4, 8, and 12 and use identical AttnCLS readout. The table below reports Flickr30K zero-shot retrieval R@1 (%), ADE150 zero-shot segmentation mIoU (%), and GFLOPs at 1024×10241024\times1024.

ModelI→T 224I→T 448I→T 896T→I 224T→I 448T→I 896ADE 224ADE 448ADE 896GFLOPs
TIPSv2-B (teacher)86.991.990.072.979.678.315.017.417.62061.7
Uniform Hybrid-linear-B84.590.285.170.977.875.116.220.017.31291.5
Uniform Hybrid-window-B82.189.580.169.777.170.618.520.218.51269.7

The window variant achieves slightly lower FLOPs and stronger zero-shot segmentation at all three resolutions. The recurrent variant performs better on high-resolution image-text retrieval. We find this local-global trade-off interesting. Window attention strengthens local spatial interactions, which benefits dense segmentation. In hierarchical Swin, downsampling also expands the image region covered by deeper windows. In our plain backbone, fixed windows cover a smaller fraction of the image as resolution increases, while recurrence carries context across distant patches through its state. This helps preserve the global representations needed for high-resolution image-text retrieval.

We have added this comparison and analysis to Appendix H, Table A12.

References

[1] Yue Liu et al. "VMamba: Visual State Space Model." NeurIPS, 2024.

[2] Ali Hatamizadeh and Jan Kautz. "MambaVision: A Hybrid Mamba-Transformer Vision Backbone." CVPR, 2025.

[3] Ze Liu et al. "Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows." ICCV, 2021.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.