acceptodds
Under review as a conference paper at ICLR 2027

Investigating Single-Block Recurrence in Vision Transformers for Image Recognition

Abstract

Vision Transformers (ViTs) implement depth by stacking independently parameterized blocks, but it remains unclear how much of this parameterization is necessary and how much can be replaced by recurrent reuse. We study this question with bViT, a single-block recurrent ViT that repeatedly applies the same transformer block while preserving the iterative computation of a deep model. On ImageNet-1K, bViT-B reaches 0.779 validation accuracy compared with 0.789 for ViT-B under the same training recipe and computational budget, while using 8.6M rather than 86.6M parameters. This correspondence becomes stronger with model width, while narrow recurrent models exhibit a substantial performance gap. Beyond classification, the single-block formulation provides a controlled testbed for studying how transformer computation evolves with depth, since the same heads, neurons, and weight matrices can be tracked across recurrent steps. Analyses of attention, activation patterns, and step-conditioned spectral pruning reveal temporally organized behavior and step-dependent utilization of the shared parameters. bViT also transfers competitively to downstream tasks while enabling highly parameter-efficient adaptation. Our work shows that much of the performance associated with independently parameterized ViT depth can be recovered through recurrent reuse of a single sufficiently wide transformer block.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
vision transformers, recurrence, parameter sharing, mechanistic analysis, spectral analysis, transfer learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_icgmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Conventional Vision Transformers (ViT) operate with a fixed depth in the form of stacked blocks. This paper introduces bViT, which recurrently applies a shared transformer block to match the effective depth of a conventional ViT. The authors compare the performance of bViT to ViT on ImageNet-1K performance as well as transfer learning. They find that bViT nears or matches ViT performance at comparable compute budgets while using 1/10th the parameters. They find that widening the recurrent block improves performance whereas increasing depth (number of recurrent steps) does not. They conduct attention analysis as well as spectral weight analysis. Through their analysis they find step-dependent changes in how bViT processes information.

Critical strengths
  • Shows that bViT trained from scratch can achieves performance comparable to a conventional deep ViT while using less parameters and extends prior work to standard image classification
  • Analyzes attention and spectral weights to give additional insight into the behavior of the architecture
  • Investigates trade off between width and recurrent depth

Showing that bViT trained from scratch extends prior work on distilling deep networks to shallow recurrent networks and shows that these networks trained from scratch can get comparable performance to deep ViT. This allows for significant parameter efficiency and doesn't require distillation. The trade off of width and recurrent depth also provides a useful characteristic for future applications of this architecture. The attention analysis allows for a better understanding by the reader about what the model has learned, the dynamics of the architecture and the implications of the architectural changes.

Critical weaknesses
  • Lack of experiments on other vision tasks

While the method is evaluated across datasets of different domains in image classification, to further contextualize the authors method in the field of computer vision it would be helpful to know how these dynamics transfer/change with different tasks. This could be rectified with linear probing on other tasks.

Questions for authors
  • Can the authors conduct any linear probing or fine-tuning of their models on other vision tasks? Other tasks that may be useful for analysis such as semantic segmentation (ADE20K), depth estimation (NYUv2) and object detection (COCO or Pascal VOC).
3 comments

We thank the reviewer for the positive assessment of our work and for recognizing the value of our experimental results and mechanistic analyses.

We also appreciate the suggestion to extend our transfer learning study from Section 4.3 to additional vision tasks, which helps improve the manuscript. We conducted experiments on ADE20K semantic segmentation and NYUv2 monocular depth estimation, following the experimental setup of the Raptor paper [Jacobs et al., Block-Recurrent Dynamics in Vision Transformers, ICLR 2026] (e.g., training for 10 epochs). We evaluated both ViT-B and bViT-B using linear probing and full fine-tuning, as suggested by the reviewer. In addition, we investigated time-step embedding tuning and rank-8 LoRA applied to the query and value (QV) projections. The results (mean, std, 3 seeds) are summarized below:

ADE20K (segmentation, mIoU %, higher is better)

MethodViT-BParamsbViT-BParams
Linear probe28.89 ± 0.11—28.28 ± 0.14—
Time embeddings——33.45 ± 0.099K
LoRA (r=8)36.42 ± 0.25296K35.14 ± 0.1225K
Full fine-tuning42.67 ± 0.3385.8M40.65 ± 0.227.8M

NYUv2 (depth estimation, RMSE, lower is better)

MethodViT-BParamsbViT-BParams
Linear probe0.8176 ± 0.0023—0.8512 ± 0.0018—
Time embeddings——0.6749 ± 0.00219K
LoRA (r=8)0.6137 ± 0.0028296K0.6226 ± 0.003325K
Full fine-tuning0.5839 ± 0.002885.8M0.5649 ± 0.00237.8M

Table: Trainable parameter counts refer to backbone adaptation, excluding heads (linear probing uses a frozen backbone and serves as the parameter reference).

These new results further demonstrate that bViT representations transfer effectively beyond image classification. Under full fine-tuning, bViT-B outperforms ViT-B on NYUv2 depth estimation (0.5649 vs. 0.5839 RMSE) and achieves competitive semantic segmentation performance on ADE20K (40.65% vs. 42.67% mIoU), despite having approximately 10 times fewer backbone parameters. Rank-8 LoRA remains close to ViT-B (35.14% vs. 36.42% mIoU; 0.6226 vs. 0.6137 RMSE) while using 25K rather than 296K trainable parameters. Time-step embedding tuning provides a lightweight adaptation approach for recurrent models. By training only 9K parameters while keeping the shared transformer block frozen, we improve ADE20K mIoU from 28.28% to 33.45% and reduce NYUv2 RMSE from 0.8512 to 0.6749 compared with linear probing.

We will mention these experiments in Section 4.3 (Transfer learning) and include the complete experimental results/setup in the corresponding Appendix F (Transfer learning setup).

I thank the authors for their response and additional experiments. The new results on semantic segmentation and depth estimation address my original concern regarding the use of bViT beyond image classification.

I also appreciate the clarification of the paper's research objective in response to Reviewer kouB. In particularly the emphasis on investigating complete parameter sharing and step-dependent computation rather than optimizing computational efficiency.

I consider my original concerns resolved and maintain my current score at this stage.

We thank the reviewer for the positive feedback. We are glad that the additional experiments resolved the concern.

We also appreciate the reviewer's comments on the research objective of our work.

Reviewer_tDV4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies full parameter sharing in ViTs. bViT applies one shared block L times, keeping FLOPs while cutting parameters by roughly 9-21x. On ImageNet-1K, bViT-B/L come within 1.0/0.3 points of ViT-B/L, while bViT-S lags by 10 points. The weight sharing across steps is then used for mechanistic analyses and parameter-efficient transfer.

Critical strengths
  1. The comparison is well controlled, with the same recipe and number of block evaluations for ViT and bViT. The compute- and parameter-matched ablations show when recurrence can replace distinct blocks. For example, at a fixed parameter budget, width beats extra distinct blocks.
  2. Since every step reuses the same weights, step-wise importance and pruning are cleanly defined. The model can also be adapted by tuning only a 9K-parameter step embedding. This beats linear probing on transfer (0.844 vs 0.803).
Critical weaknesses
  1. The main claim is confounded by weak baselines. Without stochastic depth or aggressive dropout, ViT-B is only 0.7 points above ViT-S and well below DeiT-B (78.9 vs 81.8). Also, six distinct blocks applied twice already match twelve distinct blocks. The shrinking gap at larger widths may just reflect overfitting of the unshared models.
  2. Novelty is limited. Universal Transformers and ALBERT already studied full cross-layer sharing in language models. SReT and block-recurrent ViTs are closely related, but published sharing schemes are not compared experimentally. Since FLOPs are unchanged, the benefit is memory and cheaper adaptation, not speed.
  3. The mechanistic analyses lack controls. The early-exit and localization curves have no standard ViT reference, so the findings cannot be tied to recurrence. The persistence test assumes independent steps, but importance profiles are often correlated across steps. The "persistent backbone" is therefore partly expected.
Questions for authors
  1. Does the B/L gap stay this small when ViT is trained with a tuned DeiT recipe and reaches ~81.8 for ViT-B?
  2. What do the localization and early-exit curves look like for a standard ViT-B? Does spectral persistence hold under a null that keeps cross-step correlation?
3 comments
AuthorsOpenReview ↗

Rebuttal 1

We thank the reviewer for the constructive assessment and valuable suggestions. During the rebuttal period, we conducted additional experiments to address the raised issues. Regarding novelty, we believe our main contribution lies in systematically characterizing the limits of complete parameter sharing in ViTs and examining how identical transformer components support different computational functions across recurrent steps.

Question 1

Training recipe and baseline performance. Our main experiments use standard ViT architectures with learned positional embeddings, trained under identical recipes and matched computational budgets. This controlled setting allows us to isolate the effect of replacing independently parameterized blocks with a single shared block, rather than optimizing either architecture for maximum classification accuracy.

To investigate the sensitivity of our findings to the training configuration, we conducted additional ImageNet-1K experiments with stochastic depth and rotary positional embeddings (RoPE), evaluating both ViT-B and bViT-B.

ConfigurationViT-B (%)bViT-B (%)
Original78.977.9
Stochastic depth (10%)80.177.4
RoPE79.579.3
Stochastic depth + RoPE80.3—

Table: The bViT-B experiment combining stochastic depth and RoPE, and the corresponding Large variants, have not been completed within the rebuttal period.

These results show that the accuracy gap depends on the training configuration. Stochastic depth improves ViT-B to 80.1% but slightly reduces bViT-B accuracy, possibly because randomly dropping residual updates interferes with recurrent refinement. In contrast, RoPE improves both architectures, with bViT-B reaching 79.3% compared with 79.5% for ViT-B, a difference of only 0.2 percentage points. We hypothesize that incorporating positional relationships directly into attention at every recurrent step may be particularly beneficial for shared architectures. Combining stochastic depth and RoPE further improves ViT-B to 80.3%.

Our new results highlight that regularization and positional encoding choices can interact differently with shared and independently parameterized architectures. Although we do not reach the 81.8% reported for DeiT-B, the RoPE experiment demonstrates that improving the training configuration does not necessarily increase the performance gap between ViT and bViT.

Importantly, our paper's main results do not depend only on absolute classification accuracy or the precise gap between the architectures. Our experiments establish a dependence of recurrent performance on representation width. Moreover, our mechanistic analyses investigate how identical parameters are utilized across recurrent steps, while interventional spectral pruning demonstrates the functional importance of their temporal organization.

We will retain the original controlled comparisons and mechanistic analyses in the main paper and include the additional stochastic-depth and RoPE experiments in Appendix B. We will also clarify the sensitivity of the classification gap to the training configuration.

AuthorsOpenReview ↗

Rebuttal 2

Question 2

Q2/1. Localization analysis. Following the reviewer's suggestion, we extended the pointing-game analysis from Fig. 2b (to be updated) to the standard ViT-B baseline, using the same evaluation protocol. The results are presented below:

Step / BlockbViT-B MeanViT-B MeanbViT-B Best HeadViT-B Best Head
125.6138.6728.3741.16
226.9537.3237.3837.28
334.6546.1939.5442.33
442.6945.8649.9152.19
543.8246.9455.9160.84
638.0524.5763.8646.61
740.1517.3069.1114.46
843.0816.1270.3016.02
937.2215.7555.7015.06
1026.5316.7539.6218.82
1120.3222.8930.2824.97
1221.7528.6435.6130.18

Table: All values are pointing-game hit ratios (%), with higher values indicating better object localization. For each architecture, the best-head index is selected based on the highest score across all heads and steps or blocks, and its performance is reported across depth. However, attention heads in different ViT-B blocks are independently parameterized and are not necessarily functionally aligned.

Both architectures exhibit good object localization at intermediate depths, but their trajectories reveal different patterns of spatial processing. ViT-B reaches its peak mean localization at block 5, followed by a sharp decline, whereas bViT-B maintains relatively strong localization through later recurrent steps. This suggests that spatial localization is more transient in ViT-B but remains prominent over a broader range of recurrent computation in bViT-B. Notably, the selected bViT-B attention head reaches 70.30% at step 8, compared with 60.84% at block 5 for ViT-B. Importantly, the evolving localization behavior in bViT-B emerges through repeated application of identical attention parameters, demonstrating that the same attention heads can exhibit different functional behavior as representations evolve.

Q2/2. Early-exit classification. We also extended the early-exit analysis from Fig. 2a (to be updated) to the standard ViT-B baseline. The table below reports ImageNet-1K Top-1 accuracy after each recurrent step for bViT-B and each transformer block for ViT-B.

Step / BlockbViT-B (%)ViT-B (%)
10.100.14
20.110.26
30.150.38
40.110.87
50.233.10
60.8710.75
76.4421.48
837.6133.53
961.9645.08
1071.2960.32
1176.1275.91
1277.9578.96

Both architectures progressively improve their classification accuracy, but their trajectories differ. ViT-B improves more gradually across depth, whereas bViT-B remains close to chance accuracy during the initial recurrent steps before improving sharply around steps 8–10. Despite this delayed progression, bViT-B matches or exceeds ViT-B at steps 8–11 and approaches its final accuracy after 12 steps.

Q2.3. Spectral persistence and cross-step correlation. We acknowledge that correlations between importance profiles contribute to the observed persistence of spectral directions under top-k selection. However, these correlations are themselves a property of the learned recurrent computation, as the same transformation processes evolving representations. As shown in Fig. 4, importance profiles remain relatively stable across steps for patch tokens, while CLS profiles exhibit stronger step-dependent changes.

Appendix E (Fig. 15) currently compares the observed persistence with independent random selection. We will extend this analysis with a second limiting case in which the importance profiles are identical across recurrent steps, resulting in the same directions being selected at every step. Our empirical results lie between these two extremes, exhibiting both persistent and stage-specific spectral directions.

Our main evidence for the functional relevance of this temporal organization comes from the interventional pruning experiments (Fig. 6, top row). Step-conditioned pruning outperforms global pruning at the same spectral budget, while temporally permuting the step-conditioned masks reduces classification accuracy despite preserving the same collection of masks. These experiments demonstrate that the assignment of spectral directions to recurrent steps matters for model performance, independently of how frequently individual directions are selected.

We thank the reviewer again for the constructive comments and suggestions. We would like to briefly summarize the additional experiments and analyses conducted in response to the concerns raised in the review.

We investigated alternative training configurations, including stochastic depth and RoPE, and extended our attention localization and early-exit analyses with standard ViT-B baselines. These experiments provide additional evidence about the capabilities and computational behavior of complete parameter sharing. We also clarified the interpretation of spectral persistence and its supporting interventional evidence.

We would also like to inform the reviewer that the bViT-B experiment combining stochastic depth and RoPE, which was still running when we submitted our original response, has now been completed. It achieves 78.3% ImageNet-1K accuracy, compared with 79.3% for RoPE alone, further supporting our observation that stochastic depth does not improve bViT-B performance under the tested configurations.

We have incorporated these results into the revised manuscript. We hope that these additional experiments and clarifications address the reviewer's concerns and would appreciate their consideration in the final assessment of our work.

Reviewer_pvbROpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes recurrently reuse the parameters in ViT layers to reduce the total parameter counts while preserving model performance. This idea is motivated by a research question that whether transformer depth require distinct parameters at different stages. Consequently, a single-block ViT with shared block parameters is constructed, namely bViT. One benefit of adopting bViT is that the single-block architecture facilitates the investigation of ViT computation behaviour change across different layers. With the same training recipe, bViT achieves close or competitive performance as standard ViTs with significantly reduced parameters.

Critical strengths
  1. The exploration of fully shared block ViT is interesting and novel. There are limited prior studies on this topic.

  2. The effectiveness of bViT becomes more prominent as the backbone ViT size increases. On bViT-L, the accurray gap is negligible while the parameter counts reduced to less than 10% of the original ViT-L.

  3. The experiments are extensive and the experimental results are convincing.

Critical weaknesses

None

Questions for authors

None

2 comments

We thank the reviewer for the positive assessment of our work and for recognizing the novelty and value of investigating complete parameter sharing in Vision Transformers.

During the rebuttal period, we further strengthened the manuscript through additional experiments, including alternative training configurations, comparisons with standard ViT-B in our mechanistic analyses, and transfer learning evaluations on semantic segmentation and depth estimation. These new results broaden our experimental evaluation and provide additional support for our conclusions.

We sincerely appreciate the reviewer's support and encouraging evaluation.

I appreciate the authors' response.

Reviewer_kouBOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper explores the topic about which independently parameterized depth in Vision Transformers can be replaced by recurrent application of a single shared Transformer block. The proposed bViT architecture maintains the computational depth of standard ViTs while substantially reducing the number of unique parameters. On ImageNet-1K, bViT-B achieves 77.9% accuracy with 8.6M parameters, compared to 78.9% with 86.6M parameters for ViT-B. The paper further investigates the role of representation width, analyzes the evolution of attention and neuron selectivity across recurrent steps, and introduces step-conditioned spectral sensitivity and pruning analyses. Transfer learning experiments demonstrate competitive downstream performance with substantially fewer trainable adaptation parameters.

Critical strengths
  1. The paper addresses a meaningful question about the relationship between computational depth and parameter depth in Vision Transformers. The single-block architecture provides a relatively clean experimental setting for studying extreme parameter sharing.
  2. The ImageNet results are encouraging, especially for bViT-B and bViT-L, which preserve much of the performance of their independently parameterized counterparts with substantially fewer parameters.
  3. The step-conditioned spectral analysis is an interesting aspect of the work. In particular, the pruning and temporal mask-permutation experiments provide evidence that the functional importance of shared parameter directions varies across recurrent steps.
  4. The transfer learning evaluation covers eight downstream datasets and several adaptation strategies, providing additional evidence that the recurrent representations remain useful beyond ImageNet classification.
Critical weaknesses
  1. The fundamental architecture is a straightforward application of complete cross-layer parameter sharing, an idea closely related to Universal Transformers [6] and earlier recursive Vision Transformers such as SReT [2]. Although the paper acknowledges these connections and positions its contribution as an empirical investigation, the absence of direct comparisons with existing recurrent or partially shared ViT architectures makes it difficult to assess the significance of the proposed formulation relative to prior work.
  2. The current experiments primarily compare bViT against standard ViTs, without sufficient comparisons to existing state-of-the-art parameter-efficient or lightweight vision architectures. Can the authors provide comparisons with representative methods, including weight-sharing approaches such as MiniViT [3] and SReT [2], as well as efficient architectures such as DeiT-Tiny [5] and EfficientFormer [4]? Without such comparisons, it is difficult to assess whether bViT achieves a competitive accuracy–parameter–FLOPs trade-off, particularly under comparable resource budgets.
  3. While the substantial parameter reduction achieved by bViT-B and bViT-L is encouraging, the practical significance of the proposed approach remains insufficiently demonstrated. Parameter efficiency is particularly valuable for memory-constrained edge devices, where model storage and memory bandwidth can be important bottlenecks. However, the experiments primarily focus on relatively large models, while the smallest evaluated configuration, bViT-S, suffers a substantial 10.1 percentage-point accuracy degradation compared with ViT-S. This raises an important question about whether the proposed approach is effective in precisely the resource-constrained regimes where aggressive parameter sharing would be most beneficial. Evaluating smaller architectures, such as ViT-Tiny, would provide a more informative characterization of the method's practical limitations and applicability.
  4. Furthermore, all experiments are restricted to standard ViTs. Extending the evaluation to more efficient or hierarchical architectures, such as Swin Transformer and EfficientFormer, would help establish whether the findings generalize beyond homogeneous ViT blocks.
  5. Real-device latency and memory measurements would be valuable for determining whether the reduced parameter footprint translates into meaningful deployment advantages, particularly given that recurrent sharing does not reduce the number of block evaluations.
  6. The main comparisons preserve FLOPs between ViT and bViT at each model scale, but the substantial parameter reductions come without corresponding reductions in computational depth. For example, bViT-B achieves 77.9% accuracy at 33.7 GFLOPs, whereas ViT-S achieves 78.2% at only 8.5 GFLOPs. Thus, the practical benefit depends heavily on whether parameter storage or computational cost is the primary constraint. Although Appendix A includes parameter-matched comparisons, these configurations are not FLOP-matched. A more comprehensive accuracy–parameters–FLOPs analysis, together with measured latency, throughput, and memory consumption, would substantially strengthen the practical motivation.


[1] Liu, Ze, et al. "Swin transformer: Hierarchical vision transformer using shifted windows." 2021 IEEE/CVF international conference on computer vision (ICCV). Ieee, 2021.


[2] Shen, Zhiqiang, Zechun Liu, and Eric Xing. "Sliced recursive transformer." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.


[3] Zhang, Jinnian, et al. "Minivit: Compressing vision transformers with weight multiplexing." 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2022.


[4] Li, Yanyu, et al. "Efficientformer: Vision transformers at mobilenet speed." Advances in neural information processing systems 35 (2022): 12934-12949.

[5] Touvron, Hugo, et al. "Training data-efficient image transformers & distillation through attention." International conference on machine learning. PMLR, 2021.


[6] Dehghani, Mostafa, et al. "Universal transformers." arXiv preprint arXiv:1807.03819 (2018).

Questions for authors

As shown in weaknesses

4 comments

We thank the reviewer for the detailed assessment and for recognizing the value of our research question, experimental results and spectral analysis.

Research objective and novelty. We believe that several of the listed weaknesses follow from evaluating bViT primarily as an efficiency-oriented architecture. We would like to clarify that our work does not aim to introduce a new computationally efficient ViT architecture that achieves the best accuracy–parameter–FLOPs trade-off. Instead, we investigate the limits of complete parameter sharing and how a single shared block can support stage-dependent computation, which we believe is an important research question in its own right.

Beyond classification performance, we examine how the same attention heads, FFN neurons, and spectral directions are utilized differently across recurrent steps. We acknowledge the relevance of Universal Transformers, SReT, MiniViT, and related models in Related Work. However, our research objective differs. In particular, SReT and MiniViT primarily focus on architectural efficiency, incorporating additional mechanisms or layer-specific components. Comparisons with these architectures would not cleanly isolate the effects investigated in our work. Moreover, our step-conditioned spectral pruning relies on tracking identical parameter directions across recurrent steps, an analysis naturally enabled by the single-block formulation. Hierarchical architectures such as Swin change width and resolution between stages, so a single block cannot be shared across the entire backbone without extra modifications.

Experimental comparisons. Our experimental section begins with comparisons between ViT and bViT trained using identical recipes and matched computational budgets, followed by mechanistic analyses and transfer learning experiments. Appendix A "Width and recurrence analysis" further investigates how representation width, recurrent depth, the number of attention heads, and the allocation of computation between shared and independently parameterized blocks affect performance. In particular, Table 3 compares complete and partial sharing under matched training and computation (e.g., 2 blocks applied 6 times or 6 blocks applied twice). Regarding smaller models, we acknowledge the substantial performance gap of bViT-S, which illustrates a limitation of complete sharing at narrow widths.

Computational efficiency. We do not claim improvements in inference speed or computational efficiency. As stated in the Introduction, the relevant benefit of parameter sharing is the memory cost of storing, deploying, and adapting models. Our transfer learning experiments show that bViT achieves competitive performance using approximately 12 times fewer backbone LoRA parameters than ViT. Since bViT preserves the number of block evaluations and their architecture, its theoretical computational cost remains comparable to that of the corresponding ViT, while the number of stored parameters is reduced.

Soundness assessment. We respectfully ask the reviewer to reconsider the soundness score of 1 and possibly the overall recommendation. We believe that the listed weaknesses primarily concern experimental scope, comparisons with efficiency-oriented architectures, and practical applicability, rather than methodological errors or invalid conclusions in our investigation of single-block recurrence. We would be grateful if the reviewer could clarify which specific methodological issues or conclusions raise concerns about the scientific soundness of our work.

Thank you for the clarification. I understand that the primary goal of this work is not to develop the most computationally efficient ViT architecture, but rather to investigate the limits of complete parameter sharing.

However, I believe the practical motivation and scientific significance of the work still require stronger justification.

The paper explicitly highlights substantial parameter reduction and its relevance to the memory cost of storing, deploying, and adapting models. While I agree that reducing computational FLOPs is not necessarily the objective, the claimed memory-related benefits should still be evaluated in the context of existing parameter-efficient architectures. In particular, if alternative methods can achieve comparable or better accuracy under similar memory budgets, the practical advantage of complete parameter sharing becomes less clear. I am not suggesting that the proposed method must outperform all existing approaches, but rather that appropriate comparisons are important for establishing the significance of the reported results.

If the primary contribution is instead a scientific investigation of recurrent computation, I believe the novelty and significance should be established through mechanistic insights that substantially advance our understanding beyond existing studies of recurrent Transformers and cross-layer parameter sharing. While the step-conditioned spectral analysis provides an interesting perspective on how shared parameters are utilized across recurrent steps, it remains unclear what fundamentally new understanding of Transformer depth these findings establish, or how they advance beyond the already known phenomenon that shared parameters can exhibit different behaviors when applied to evolving hidden states.

Therefore, my main concern is not that the proposed architecture is incorrectly implemented, but that the current evidence does not yet convincingly establish the significance of the contribution from either a practical memory-efficiency perspective or a mechanistic understanding perspective.

We thank the reviewer for acknowledging our research objective, raising the soundness score, and clarifying that the remaining concern relates to the scientific significance of our findings rather than to incorrect implementation.

Our paper offers several contributions beyond the general observation that shared parameters can behave differently across recurrent steps. Several were also recognized among the strengths in the reviewer's original evaluation.

  1. Step-conditioned spectral analysis. Step-conditioned pruning outperforms a single global mask at the same spectral budget (approximately 0.54 vs. 0.23 accuracy after pruning 60% of the value directions), and permuting the same masks across steps reduces accuracy. This shows that one set of weights carries step-specific functions: the specialization across depth usually attributed to separate layers can be realized within shared parameters.

  2. Parameter-efficient adaptation under recurrence. A single LoRA adapter, reused at every step, adapts the entire model with approximately 12 times fewer trainable parameters than per-block adapters in a 12-block ViT, including on the added segmentation and depth estimation tasks. In a standard ViT, sharing an adapter across layers is an additional constraint; in bViT it follows from the architecture, and the gap to full fine-tuning remains comparable to that of per-layer LoRA in ViT-B. A fixed low-rank update can therefore have step-dependent effects, as it acts on an evolving representation.

  3. Functional differentiation and capacity limits. The same attention heads can be tracked across depth, which reveals specialization: one head reaches 70.3% pointing-game accuracy at step 8 and loses this role in later steps. The added ViT-B comparison shows that localization and classification evolve differently in the two architectures. We also demonstrate that complete sharing is limited by width.

To the best of our knowledge, these findings have not been established by previous studies of recurrent Transformers or cross-layer parameter sharing. We respectfully ask the reviewer to reconsider the scientific significance of these contributions and the overall recommendation for our paper.

We thank the reviewer for the continued engagement in the rebuttal and for increasing the overall score.

We appreciate the reviewer's consideration of our clarifications and experimental evidence.

Comments on the paper

2 comments

We thank all four reviewers for their careful assessments and constructive feedback. We appreciate their recognition of the relevance of our research question on single-block recurrence in Vision Transformers and the value of our controlled experimental setting (kouB, pvbR, tDV4), the insights provided by our mechanistic and spectral analyses (kouB, tDV4, icgm), and the breadth of our experimental evaluation, including parameter-efficient transfer learning (kouB, pvbR, tDV4, icgm).

During the rebuttal period, we addressed the main concerns raised by the reviewers and conducted several additional experiments to strengthen the manuscript.

1. Research objective and scope (kouB). We clarified that our primary objective is not to optimize inference efficiency or the accuracy–FLOPs trade-off, but to investigate the capabilities, limitations, and computational behavior of complete parameter sharing. We also discussed the distinctions from related architectures and the scope of our experimental comparisons. We hope this better highlight our contributions.

2. Additional experiments. Following the reviewers' suggestions, we extended our evaluation in several directions.

  • Additional vision tasks (icgm). We extended our transfer learning study to ADE20K semantic segmentation and NYUv2 depth estimation. The results further support the transferability of bViT representations and demonstrate competitive performance with substantially fewer backbone and adaptation parameters.

  • Mechanistic comparisons (tDV4). We extended Fig. 2 with standard ViT-B baselines for attention localization and early-exit classification, revealing similarities and differences in their evolution across depth. We also clarified the spectral persistence analysis and the evidence provided by our interventional pruning experiments.

  • Additional training configurations (tDV4). We extended our ImageNet-1K evaluation with additional ViT-B and bViT-B variants using stochastic depth and RoPE. These experiments provide further insights into how training and architectural choices interact with complete parameter sharing.

3. Manuscript revisions. We will incorporate the additional experimental results and analyses into the main manuscript and existing appendices, as detailed in our individual responses to the reviewers.

We believe these additional experiments and clarifications strengthen the manuscript and address important concerns raised during review. We thank the reviewers and area chair for considering our responses.

Dear Reviewers and Area Chairs,

As the rebuttal period comes to an end, we would like to thank you again for your constructive comments, suggestions and engagement in the discussion. During the rebuttal, we conducted additional experiments and provided clarifications regarding the scientific objectives and contributions of our work.

We particularly thank reviewers pvbR and icgm for their positive assessments. We appreciate that reviewer icgm explicitly acknowledged that our additional experiments resolved their original concerns. We thank reviewer tDV4 for the detailed suggestions, which we addressed with additional experiments and analyses. We also thank reviewer kouB for engaging in the discussion and reconsidering the assessment of our work, increasing both the soundness score and the overall recommendation.

We have now updated the manuscript to incorporate the new experimental results and analyses discussed in our responses. The main modifications are summarized below.

  1. Attention localization and early-exit classification (tDV4). We updated Fig. 2 to include standard ViT-B comparisons, allowing a more comprehensive analysis of how localization and classification capabilities evolve across depth. We discuss these new findings in Sec. 4.2.1.

  2. Additional model variants (tDV4). We extended Appendix B with results for ViT-B and bViT-B variants using RoPE and stochastic depth. We also included the bViT-B configuration combining both modifications, which was still being trained at the time of our original rebuttal response.

  3. Spectral analysis (tDV4). We extended the discussion in Sec. 4.2.3 by clarifying the singular-value-based reference for spectral direction selection and its relationship to step-conditioned utilization.

  4. Transfer learning (icgm). We extended Sec. 4.3 with results on ADE20K semantic segmentation and NYUv2 depth estimation. We also included additional results for LoRA adaptation of FFN layers and expanded the experimental details in Appendix F.

  5. Presentation improvements. We enlarged Fig. 3 to improve its readability.

We believe that these additional experiments and revisions have strengthened the manuscript, broadened its experimental evaluation and provided extra evidence supporting our scientific conclusions.

We thank all reviewers and area chairs for their time, thoughtful feedback and consideration of our revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.