- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents the method RepScan to address the hinderance of quadratic complexity of vision transformers as well as fixed scan orders in state-space models. RepScan utilizes the proposed "Representation Relation Modeling" (RRM) to create a directed relation between tokens that is input dependent and therefore dynamic/adaptive. RRM for each token to its candidate successors learns a relational affinity between the two tokens and learns a predictive discrepancy. That is, is this successor token compatible to the source token and is this successor token the representation that would be predicted to be compatible with the source token. Using RRM, Adaptive Scan Initialization selects the starting token that has the most promising connectivity. Then, Adaptive Scan Ordering selects the subsequent tokens to traverse from the set of unvisited tokens with a small lookahead. Applying RepScan (RRM, ASI and ASO) to a mamba state space model results in improved performance on visual tasks such as ImageNet-1k classification, MSCOCO Object Detection & Instance Segmentation and ADE20K Semantic Segmentation. However, RepScan introduces a runtime overhead that results in higher latency, lower throughput and more peak memory compared to VMamba and InfoScan.
Critical strengths
- The method is well motivated, formulated and clear to follow
- The experimental results are thorough and relevant to the general field that the method aims to address. the method performs well on the experiments compared to prior works
The method formulation and motivation is clear which makes it easier to adopt for future work and its motivation is a common limitation of prior methods that it directly addresses. The experimental results address multiple common tasks in visual processing and the method outperforms prior methods making it reasonable for future work.
Critical weaknesses
- While motivated by "High-resolution visual representation learning", this domain is not experimented in the main paper results.
This is important because while the drawback of quadratic complexity of ViT is addressed by the use of an SSM and the rigid/fixed token order is addressed by RepScan but while motivated by high resolution, there is not sufficient demonstration of the methods effectiveness for "high-resolution visual representation learning".
The experiments primarily use standard image sizes: 224x224 and 512x512. In the appendix, the authors show scaling to larger image sizes such as 768x768 and 1024x1024 but, do not report performance on any particular task as this scale and the computational efficiency at this scale is generally worse than the standard VMamba method and the prior work InfoScan.
- Experimental results don't currently support: "a favorable accuracy–efficiency trade-off" for RepScan against prior methods. Looking at table 1, RepScan-B performs better than "smaller" models with similar or less parameters; however, uses more FLOPS, GPU memory and higher latency (C.3). see second question.
Questions for authors
-
Based on the visualization in Figure 3 and Figure 4, RepScan only visits a subset of tokens in the image but in the methods section it is stated that "the procedure produces a valid permutation containing all tokens exactly once" which contradicts the visualizations. Do these visualization depict only partial scan orderings? Can you clarify if the full sequence of tokens are processed?
-
Related to weakness 2: could the authors provide accuracy-efficiency comparisons against larger baseline models with matched or more parameters? Ideally a way to visualize the trade off of FLOPs to accuracy for RepScan compared with prior methods. It would strengthen the results to either group results in a plot by size/flops vs accuracy to demonstrate that RepScan achieves better accuracy-efficiency trade-off and also include comparisons to larger model such as the base sizes of prior methods like ConvNeXt, Swin, VMamba