acceptodds
Under review as a conference paper at ICLR 2027

One Block, Multiple Depths: Recurrent Vision Transformers with Depth-Programmed Experts

Abstract

In this work, we show that a single Transformer block, applied recurrently, can match the accuracy of a full-depth vision encoder at comparable inference FLOPs without intermediate feature distillation. reViT restores depth-specific transformations by representing the FFN at each recurrent depth as a convex combination of a small shared expert bank. A continuous normalized-depth coordinate programs this mixture, defining a resampleable trajectory through FFN parameter space. We evaluate this design in two regimes: supervised ImageNet-1k training and distillation from a DINOv2 teacher. Across both regimes, controlled adaptations identify weight-space merging as the strongest tested MoE family at a matching one-FFN budget, ahead of the token-dispatch and output-mixture alternatives. Trained from scratch, reVit-B/16 attains DeiT III accuracy with about 70% fewer stored parameters. An 8-experts model distilled using only the teacher's output features retains nearly all of its DINOv2 teacher's linear-probe accuracy and transfers across classification, segmentation, and depth prediction. Elastic-depth training allows one checkpoint (trained model) to operate at multiple tested depths by resampling the same normalized coordinate interval. For fixed-depth deployment, the recurrent block can be materialized as a conventional dense graph, removing online routing and merging without changing the one-FFN-per-depth compute but expanding deployment storage.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
vision transformers, recurrent vision transformers

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sPL1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes reViT, a recurrent Vision Transformer that reduces model parameters by sharing a Transformer block across multiple depths. The method uses a depth-dependent router to combine several FFN experts into a single FFN at each recurrent step, while sharing the attention parameters. It also supports different inference depths using the same model. Experiments on ImageNet-1k and DINOv2 distillation show that reViT achieves competitive accuracy with significantly fewer parameters.

Critical strengths
  1. The proposed architecture is simple and effective. It achieves substantial parameter reduction while maintaining competitive accuracy. For example, reViT-B/16 uses 73% fewer parameters than DeiT III with slightly better accuracy.
  2. The experiments cover different model sizes and downstream tasks, including classification, segmentation, and depth prediction, demonstrating the potential generalizability of the approach.
  3. The paper provides comparisons with several MoE methods and includes useful analyses of expert utilization and elastic-depth inference.
Critical weaknesses
  1. The methodological novelty is somewhat limited. The approach mainly combines existing ideas from recurrent Transformers, parameter sharing, and expert weight merging. The paper should better explain its technical contributions beyond these existing methods.
  2. Although Sec 4.4 compares depth-only and feature-only routing, the contribution of continuous depth programming could be further clarified. In particular, a comparison with directly learned depth-specific expert mixing weights would help demonstrate the advantage of the proposed design.
  3. Sec 4.3 compares several MoE methods under a common recurrent architecture. However, these methods use different expert configurations, and matching nominal FFN computation does not ensure equal overall computational costs. Additional comparisons with matched parameters and actual inference costs would strengthen the conclusions.
  4. Some improvements over the strongest baselines are relatively small. For example, in Table 4, reViT achieves 83.5% ImageNet accuracy compared with 83.4% for SMEAR. Although results are averaged over three runs, standard deviations are not reported. Statistical evidence is needed to support these improvements.
  5. The practical inference efficiency remains a concern. Table 7 shows that the dynamic reViT-S model has higher inference latency than DeiT III-S despite using fewer parameters. Although the authors suggest precomputing the merged weights, the actual latency of this deployment strategy has not been evaluated.6. The elastic-depth evaluation in Section 4.4 is limited to a small set of inference depths. Further experiments across a wider depth range would help establish the robustness of this capability.
Questions for authors

see weaknesses

Reviewer_4VbVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents reViT, a recurrent vision encoder that shares attention and normalization across depth. The FFN is a soft parameter mixture of an expert bank/pool. A router takes normalized depth as input. The learned parameter trajectory can be resampled at different recurrence depths. Experiments were done on ImageNet training and DINOv2 distillation. The B/16 supervised model reaches 83.0% ImageNet accuracy with 23.6M parameters versus 82.8% and 86.6M for DeiT III. The eight-expert distilled model reaches 83.9% versus the teacher’s 84.5%.

Critical strengths
  • Parameter saving is true under the current post-training or distillation experiment setting.

  • The proposed method is simple and easy to understand.

Critical weaknesses
  • Performance improvement against existing parameter-saving methods is limited (E.g. SMEAR in Table 4)

  • Novelty is highly limited: adding depth as a condition is trivial if the performance/efficiency is not significantly better.

Questions for authors
  • Could the authors further illustrate the novelty of the work?

  • Could this be combined with SMEAR?

Reviewer_XSUYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes reViT, which reduces the number of model parameters through recursive computation using a single block. It further enables depth-adaptive computation by introducing a group of FFN experts and a two-layer MLP to differentiate computations across different depths.

Critical strengths
  1. The direction is worth pursuing. Decoupling the parameters a vision encoder stores from the depth it executes is a real deployment problem, and the paper gives it a positive answer at scale: with one shared attention and four FFN sets, reViT-B/16 reaches 83.0% against 82.8% for DeiT III at matched FLOPs with 23.6M rather than 86.6M parameters, and L/16 reaches 83.8% against 84.1%.
  2. Treating depth as a continuous coordinate for parameter composition is a sensible angle, and it buys properties that per-depth parameterizations cannot.
Critical weaknesses
  1. I am concerned about the new insight of this paper into the problem it targets. On novelty, the mechanism remains an extension of ALBERT-style parameter sharing and parameter reuse rather than a new solution.
  2. The gate distribution is not explained enough. In mechanisms of this kind, where an MLP outputs softmax scores to combine parameters, the gate tends to degenerate toward an oligarchic, near one-hot distribution. Such a distribution would be unhealthy: once it appears, each step effectively uses the parameters of a single FFN, the model degenerates to single-set parameter reuse, accuracy drops severely, and E loses its meaning whatever its value. The paper does not adequately state what distribution these normalized scores should have, what counts as healthy, or what the trained distribution actually is.
  3. Under full layer tying, whether attention or the FFN is the main source of the lost accuracy is a natural and reasonable question. The paper could discuss it to support your design.
Questions for authors
  1. Please add a discussion of the gate score distribution: what is the actual distribution of the softmax scores produced by the normalizing MLP across depths and across values of E? What distribution do the authors consider healthy, and does the actual distribution degenerate toward one-hot?
  2. Please add a discussion or some evidence: under full layer tying, does the lost accuracy come mainly from the shared attention or from the shared FFN? Is it a better option to fully reuse Attention instead of FFN? Hope the author provides the corresponding analysis or ablation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.