- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Conventional Vision Transformers (ViT) operate with a fixed depth in the form of stacked blocks. This paper introduces bViT, which recurrently applies a shared transformer block to match the effective depth of a conventional ViT. The authors compare the performance of bViT to ViT on ImageNet-1K performance as well as transfer learning. They find that bViT nears or matches ViT performance at comparable compute budgets while using 1/10th the parameters. They find that widening the recurrent block improves performance whereas increasing depth (number of recurrent steps) does not. They conduct attention analysis as well as spectral weight analysis. Through their analysis they find step-dependent changes in how bViT processes information.
Critical strengths
- Shows that bViT trained from scratch can achieves performance comparable to a conventional deep ViT while using less parameters and extends prior work to standard image classification
- Analyzes attention and spectral weights to give additional insight into the behavior of the architecture
- Investigates trade off between width and recurrent depth
Showing that bViT trained from scratch extends prior work on distilling deep networks to shallow recurrent networks and shows that these networks trained from scratch can get comparable performance to deep ViT. This allows for significant parameter efficiency and doesn't require distillation. The trade off of width and recurrent depth also provides a useful characteristic for future applications of this architecture. The attention analysis allows for a better understanding by the reader about what the model has learned, the dynamics of the architecture and the implications of the architectural changes.
Critical weaknesses
- Lack of experiments on other vision tasks
While the method is evaluated across datasets of different domains in image classification, to further contextualize the authors method in the field of computer vision it would be helpful to know how these dynamics transfer/change with different tasks. This could be rectified with linear probing on other tasks.
Questions for authors
- Can the authors conduct any linear probing or fine-tuning of their models on other vision tasks? Other tasks that may be useful for analysis such as semantic segmentation (ADE20K), depth estimation (NYUv2) and object detection (COCO or Pascal VOC).
3 comments
I thank the authors for their response and additional experiments. The new results on semantic segmentation and depth estimation address my original concern regarding the use of bViT beyond image classification.
I also appreciate the clarification of the paper's research objective in response to Reviewer kouB. In particularly the emphasis on investigating complete parameter sharing and step-dependent computation rather than optimizing computational efficiency.
I consider my original concerns resolved and maintain my current score at this stage.
We thank the reviewer for the positive feedback. We are glad that the additional experiments resolved the concern.
We also appreciate the reviewer's comments on the research objective of our work.
Rebuttal
We thank the reviewer for the positive assessment of our work and for recognizing the value of our experimental results and mechanistic analyses.
We also appreciate the suggestion to extend our transfer learning study from Section 4.3 to additional vision tasks, which helps improve the manuscript. We conducted experiments on ADE20K semantic segmentation and NYUv2 monocular depth estimation, following the experimental setup of the Raptor paper [Jacobs et al., Block-Recurrent Dynamics in Vision Transformers, ICLR 2026] (e.g., training for 10 epochs). We evaluated both ViT-B and bViT-B using linear probing and full fine-tuning, as suggested by the reviewer. In addition, we investigated time-step embedding tuning and rank-8 LoRA applied to the query and value (QV) projections. The results (mean, std, 3 seeds) are summarized below:
ADE20K (segmentation, mIoU %, higher is better)
NYUv2 (depth estimation, RMSE, lower is better)
Table: Trainable parameter counts refer to backbone adaptation, excluding heads (linear probing uses a frozen backbone and serves as the parameter reference).
These new results further demonstrate that bViT representations transfer effectively beyond image classification. Under full fine-tuning, bViT-B outperforms ViT-B on NYUv2 depth estimation (0.5649 vs. 0.5839 RMSE) and achieves competitive semantic segmentation performance on ADE20K (40.65% vs. 42.67% mIoU), despite having approximately 10 times fewer backbone parameters. Rank-8 LoRA remains close to ViT-B (35.14% vs. 36.42% mIoU; 0.6226 vs. 0.6137 RMSE) while using 25K rather than 296K trainable parameters. Time-step embedding tuning provides a lightweight adaptation approach for recurrent models. By training only 9K parameters while keeping the shared transformer block frozen, we improve ADE20K mIoU from 28.28% to 33.45% and reduce NYUv2 RMSE from 0.8512 to 0.6749 compared with linear probing.
We will mention these experiments in Section 4.3 (Transfer learning) and include the complete experimental results/setup in the corresponding Appendix F (Transfer learning setup).