- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors introduce ResFuser which is a more efficient version of the Vision Transformer operating on less tokens, where they utilize the residual structure and token sampling for improved performance over conventional efficient Transformer architectures. They formulate two residual routes inspired from other Transformer architectures, and try four token sampler schemes for training and validation. The experimental results show mixed results on various domains and tasks.
Critical strengths
- Experiments across multiple domains: The authors perform controlled experiments across climate, medical, and natural image domains using the same settings.
- Explicit FLOPs presentation and calculation: The authors present the FLOPs counts for verifiable and reproducible research.
- Paper is written in a straightforward manner and easy to follow.
Critical weaknesses
- The model is tiny (7.4M parameters) and trained for only 5,000 steps, where the authors also admitted that this could not ensure convergence. Thus, the "within 7% performance" claim could be an optimistically selected estimate.
- The encoder, the decoder and the residual route all still run at full resolution, and no wall-clock or memory numbers are reported.
- Other conventional token reduction alternatives are not compared and reported. Alternatives such as U-ViT, HDiT, and Swin/SwinIR would all reduce cost, and these baselines should also be compared under controlled settings. Without these baselines at equal FLOPs, the claim that the proposed method reduces cost while retaining the performance could not be verified.
- Compared to previous works such as SPRINT, ORBIT-2, MAE, AdaFormer, SparseViT and UViT, the proposed work lacks novelty since every technical component already exists, and the paper acknowledges this by citing most of them. Their proposed sequential skip is highly similar to SPRINT with the identical sampler used. Also, lack of comparison with other Transformer variants is a critical weakness.
Questions for authors
Please refer to the weaknesses section.