- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes reViT, a recurrent Vision Transformer that reduces model parameters by sharing a Transformer block across multiple depths. The method uses a depth-dependent router to combine several FFN experts into a single FFN at each recurrent step, while sharing the attention parameters. It also supports different inference depths using the same model. Experiments on ImageNet-1k and DINOv2 distillation show that reViT achieves competitive accuracy with significantly fewer parameters.
Critical strengths
- The proposed architecture is simple and effective. It achieves substantial parameter reduction while maintaining competitive accuracy. For example, reViT-B/16 uses 73% fewer parameters than DeiT III with slightly better accuracy.
- The experiments cover different model sizes and downstream tasks, including classification, segmentation, and depth prediction, demonstrating the potential generalizability of the approach.
- The paper provides comparisons with several MoE methods and includes useful analyses of expert utilization and elastic-depth inference.
Critical weaknesses
- The methodological novelty is somewhat limited. The approach mainly combines existing ideas from recurrent Transformers, parameter sharing, and expert weight merging. The paper should better explain its technical contributions beyond these existing methods.
- Although Sec 4.4 compares depth-only and feature-only routing, the contribution of continuous depth programming could be further clarified. In particular, a comparison with directly learned depth-specific expert mixing weights would help demonstrate the advantage of the proposed design.
- Sec 4.3 compares several MoE methods under a common recurrent architecture. However, these methods use different expert configurations, and matching nominal FFN computation does not ensure equal overall computational costs. Additional comparisons with matched parameters and actual inference costs would strengthen the conclusions.
- Some improvements over the strongest baselines are relatively small. For example, in Table 4, reViT achieves 83.5% ImageNet accuracy compared with 83.4% for SMEAR. Although results are averaged over three runs, standard deviations are not reported. Statistical evidence is needed to support these improvements.
- The practical inference efficiency remains a concern. Table 7 shows that the dynamic reViT-S model has higher inference latency than DeiT III-S despite using fewer parameters. Although the authors suggest precomputing the merged weights, the actual latency of this deployment strategy has not been evaluated.6. The elastic-depth evaluation in Section 4.4 is limited to a small set of inference depths. Further experiments across a wider depth range would help establish the robustness of this capability.
Questions for authors
see weaknesses