acceptodds
Withdrawn from ICLR 2027

RepScan: Representation-Adaptive Scanning for Visual State Space Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

High-resolution visual representation learning is hindered by the quadratic complexity of Vision Transformers and the rigid scanning strategies used by existing efficient architectures, which employ uniform token processing or fixed scanning orders that fail to adapt to the highly non-uniform distribution of semantic information in real-world images. To address this, we propose RepScan, a representation-adaptive visual scanning framework that dynamically constructs input-dependent serialization orders. RepScan first introduces Representation Relation Modeling to characterize directed token relations by jointly considering representation affinity and predictive discrepancy. Based on these relations, Adaptive Scan Initialization selects a content-dependent starting token with strong outgoing relational support, while Adaptive Scan Ordering determines the subsequent traversal according to the remaining representation structure. A differentiable permutation relaxation further enables end-to-end learning of the discrete scanning process. RepScan preserves the efficient state-space propagation of the underlying visual SSM while introducing a sparse, input-dependent scan-construction process for adaptive serialization. Extensive experiments across classification, detection, and segmentation demonstrate that RepScan achieves a favorable accuracy–efficiency trade-off across diverse visual recognition tasks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Visual State Space Models, Adaptive Scanning, Efficient Vision Models, Dynamic Serialization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_aEVwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents the method RepScan to address the hinderance of quadratic complexity of vision transformers as well as fixed scan orders in state-space models. RepScan utilizes the proposed "Representation Relation Modeling" (RRM) to create a directed relation between tokens that is input dependent and therefore dynamic/adaptive. RRM for each token to its candidate successors learns a relational affinity between the two tokens and learns a predictive discrepancy. That is, is this successor token compatible to the source token and is this successor token the representation that would be predicted to be compatible with the source token. Using RRM, Adaptive Scan Initialization selects the starting token that has the most promising connectivity. Then, Adaptive Scan Ordering selects the subsequent tokens to traverse from the set of unvisited tokens with a small lookahead. Applying RepScan (RRM, ASI and ASO) to a mamba state space model results in improved performance on visual tasks such as ImageNet-1k classification, MSCOCO Object Detection & Instance Segmentation and ADE20K Semantic Segmentation. However, RepScan introduces a runtime overhead that results in higher latency, lower throughput and more peak memory compared to VMamba and InfoScan.

Critical strengths
  • The method is well motivated, formulated and clear to follow
  • The experimental results are thorough and relevant to the general field that the method aims to address. the method performs well on the experiments compared to prior works

The method formulation and motivation is clear which makes it easier to adopt for future work and its motivation is a common limitation of prior methods that it directly addresses. The experimental results address multiple common tasks in visual processing and the method outperforms prior methods making it reasonable for future work.

Critical weaknesses
  • While motivated by "High-resolution visual representation learning", this domain is not experimented in the main paper results.

This is important because while the drawback of quadratic complexity of ViT is addressed by the use of an SSM and the rigid/fixed token order is addressed by RepScan but while motivated by high resolution, there is not sufficient demonstration of the methods effectiveness for "high-resolution visual representation learning".

The experiments primarily use standard image sizes: 224x224 and 512x512. In the appendix, the authors show scaling to larger image sizes such as 768x768 and 1024x1024 but, do not report performance on any particular task as this scale and the computational efficiency at this scale is generally worse than the standard VMamba method and the prior work InfoScan.

  • Experimental results don't currently support: "a favorable accuracy–efficiency trade-off" for RepScan against prior methods. Looking at table 1, RepScan-B performs better than "smaller" models with similar or less parameters; however, uses more FLOPS, GPU memory and higher latency (C.3). see second question.
Questions for authors
  • Based on the visualization in Figure 3 and Figure 4, RepScan only visits a subset of tokens in the image but in the methods section it is stated that "the procedure produces a valid permutation containing all tokens exactly once" which contradicts the visualizations. Do these visualization depict only partial scan orderings? Can you clarify if the full sequence of tokens are processed?

  • Related to weakness 2: could the authors provide accuracy-efficiency comparisons against larger baseline models with matched or more parameters? Ideally a way to visualize the trade off of FLOPs to accuracy for RepScan compared with prior methods. It would strengthen the results to either group results in a plot by size/flops vs accuracy to demonstrate that RepScan achieves better accuracy-efficiency trade-off and also include comparisons to larger model such as the base sizes of prior methods like ConvNeXt, Swin, VMamba

Reviewer_HkRmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RepScan, a visual backbone that dynamically constructs scanning orders according to learned token relationships. Its Representation Relation Modeling (RRM) module combines representation affinity with directional predictive discrepancy to estimate relationships between candidate tokens. Adaptive Scan Initialization (ASI) selects the starting token, while Adaptive Scan Ordering (ASO) determines subsequent traversal using both transition scores and forward connectivity. A local straight-through relaxation enables end-to-end training of the discrete scanning decisions. Extensive experiments demonstrate RepScan's effectiveness and show its potential in visual tasks.

Critical strengths
  1. The proposed method treats visual patch serialization as an input-dependent ordering problem rather than a predefined choice, achieving adaptive visual patch perception and learning.

  2. The proposed method is evaluated on image classification, object detection, instance segmentation, and semantic segmentation, with ablations and scanning-path visualizations to investigate its effectiveness. Strongly demonstrates the effectiveness of RepScan.

Critical weaknesses
  1. In the ImageNet-1k result, the authors claim RepScan outperforms InfoScan in various scales. However, when comparing RepScan-S with InfoScan-B (similar parameter size and FLOPs), InfoScan-B achieves a higher Top-1 Acc with fewer parameters. This may indicate that the visual patch reordering method induces heavy computational or memory overhead. The authors should provide a more detailed complexity analysis, including FLOPs-Acc and Params-Acc diagrams, and a theoretical analysis of complexity (e.g., O(N^2) for computing the directed representation relation?).

  2. The authors compare RepScan with Raster and Hilbert scanning methods, but provide limited discussion of tree-style [1] and deformable [2] scanning methods and do not evaluate them experimentally. Including these approaches in the discussion and experimental comparisons would provide a more comprehensive assessment of RepScan against existing scanning strategies.

[1] MambaTree: Tree Topology is All You Need in State Space Model. Xiao et al. [2] DefMamba: Deformable Visual State Space Model. Liu et al.

To me, the novelty relative to existing adaptive scanning methods needs clearer justification. The experimental comparisons are also incomplete, particularly the omission of tree-style and deformable scanning methods. The authors should provide more discussion and comparison on this. Furthermore, the ImageNet-1k results show that RepScan-S underperforms InfoScan-B by 0.3 percentage points in ImageNet-1K Top-1 accuracy under similar parameter counts and FLOPs.

So I tend to recommend weak rejection, but I would be open to raising my rating if the authors adequately address my concerns.

Questions for authors
  1. Why not use graph-based or attention-based methods to build the representation relation between visual patches, which is more direct and reasonable?

  2. Can this reordering method be performed per stage (consider that there is a downsampling operation between stages, performing it only once in the whole network may not work) instead of per layer to reduce the complexity overhead?

  3. I noticed that Figures 1, 2, and 3 do not appear to be explicitly referenced in the main text. Did I miss the corresponding references? Please check the manuscript carefully. The missing references to figures and tables may confuse readers. Moreover, in Figure 1, VMamba appears to use the same scanning mechanism as Vim, whereas, in fact, VMamba adopts cross-scanning and Vim employs bidirectional scanning.

Reviewer_v7wbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RepScan, a representation-adaptive serialization framework for visual state-space models. RepScan constructs an input-dependent permutation using three components. Representation Relation Modeling (RRM) combines representation affinity and directional predictive discrepancy. Adaptive Scan Initialization (ASI) selects the starting token. Adaptive Scan Ordering (ASO) then traverses unvisited tokens based on immediate relation scores and a connectivity-based look-ahead term.

Critical strengths
  1. The paper addresses a meaningful architectural choice in order-sensitive visual state-space models.
  2. Evaluates classification and dense prediction, and supplements the main comparisons with ablation study.
Critical weaknesses
  1. The reported EFR values are incompatible with the stated cumulative-prefix metric. EFR@20% = 72.6% and EFR@30% = 86.4% are inconsistent with the cumulative prefix metric defined in Eq. (63): even if the foreground coverage of all newly added tokens were 1, EFR@30% could reach at most approximately 81.7%.
  2. Accuracy-efficiency trade-off. Compared with the corresponding InfoScan variants, the current comparison is not parameter-matched. In Table 1(a), RepScan-T/S/B have 28M/47M/51M parameters, whereas InfoScan-T/S/B have 10M/24M/38M. RepScan-S improves performance by only 0.3 percentage points, while increasing FLOPs from 4.8G to 6.7G.
  3. Efficiency requires further clarification. ASO constructs the scan order through input-dependent sequential decisions, so sparse arithmetic complexity alone cannot establish that its runtime overhead is low. Table 8 shows that, at a resolution of 224², RepScan-S has both higher latency and greater memory usage than InfoScan-S: 16.9 vs. 14.2 ms and 2.6 vs. 2.0 GB, respectively.
  4. The writing is overly defensive. Several passages repeatedly qualify what the method or experiments do not claim. These caveats are individually reasonable, but their repetition obscures the main findings.
Questions for authors
  1. Table 6 reports EFR@20% = 72.6% and EFR@30% = 86.4%, which are incompatible with Equation (63). Please clarify this discrepancy.
  2. Why does the 46.9% ADE20K result correspond to RepScan-S in Table 1(c), but RepScan-B in the ablation tables? Were these experiments conducted using the same configuration?
  3. RepScan uses substantially more parameters than the corresponding InfoScan variants. What evidence establishes that the gains come from the proposed scanning mechanism rather than increased model capacity?
  4. Do the reported latency and FLOPs include the entire adaptive scan-construction process? What hardware, batch size, and numerical precision were used?
  5. Was the dataset split by patient before extracting slices in BraTS? Does the reported mIoU include background?
Reviewer_SQX4OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary
  1. Fixed geometric scanning can cause semantically relevant image tokens to be too far apart in a one-dimensional sequence. This paper explores whether scanning start points and subsequent sequences can be selected simultaneously based on the input representation to improve visual state-space models.

  2. RRM combines bilinear affinity and source-conditional prediction error on a sparse candidate map composed of fixed local, expanded neighborhoods, and raster successors. ASI selects start points using a relative benchmark, while ASO generates complete permutations using immediate relation scores and one-step lookahead, along with unvisited sets and backtracking rules. Training uses a local straight-through gradient surrogate, and the SSM output is mapped back to the original spatial location.

  3. Core Potential Contribution: Integrating directed relation scoring, start point selection, and sparse traversal with connectivity considerations into a visual backbone. While the paper reports gains in classification, detection, and segmentation, the "content-based ordering" approach already has precedents; the actual contribution should focus on this specific sparse structure and its verifiable benefits.

Critical strengths
  1. The method definition is more comprehensive than simple significance ranking: §3.2–3.4 explicitly lists the candidate set, deduplication, tying, special cases for the last raster token, and backoff when no available neighbors are available, ensuring that each token is visited only once.

  2. It avoids a full pairwise relation matrix: §3.2 and Appendix B.1 use a fixed-size spatial candidate set, eliminating the need to calculate similarity between all tokens; Appendix A.9 also acknowledges the O(NK²) cost of direct lookahead implementation.

  3. It correctly explains the limitations of gradient estimation: §3.4.2 and A.7–A.8 clearly state that straight-through is a biased local proxy, and candidate eligibility, access set, and index updates are still discrete operations, without claiming complete differentiability in the method details.

  4. The evaluation design covers multiple perspectives: Tables 2, 3, 5, 7, 9, and 10 respectively address relation signals, fixed and random scans, starting point and ranking modules, test-period intervention, candidate topology, and budget. These designs help pinpoint contributions, but data consistency still needs improvement.

  5. The actual additional costs are revealed: Appendices C.3 and D simultaneously provide results on resolution scaling, latency, and memory usage, and acknowledge the increased burden of adaptive sequencing; this is more informative than simply describing the overall approach using the asymptotic complexity of SSM.

Critical weaknesses
  1. EFR Value Violates the Cumulative Average Definition in the Text. rᶠᵍ is between 0 and 1. With a fixed number of tokens N, K₃₀·EFR@30 − K₂₀·EFR@20 ≤ K₃₀−K₂₀. Using a large N approximation, the RepScan values ​​in Table 6 implicitly suggest an average prospect rate of (0.30 × 0.864 − 0.20 × 0.726) / 0.10 = 1.14, or 114%; this cannot be obtained from the cumulative average of the same permutation in Equation 63. For N=49 in Table 4, even after applying floor, the result is (14×0.864−9×0.726)/5=111.24%, which cannot be explained by rounding. This directly undermines key evidence of the semantic priority mechanism; if different q use different samples, different permutations, or different weights, the current definition and expression are incomplete.

  2. Model identity and training procedure cannot be compared. RepScan-T has a parameter value of 28M in the classification column and 11M in the ADE20K column, with S also changing from 47M to 39M; the notes in Table 1 refer to both as backbone parameters. The total detection parameters T/S/B are 37/56/75M, and their relationship with the classification backbone and task head needs to be explained. The ADE20K parameter of 46.9 is labeled as S in Table 1, but as B in multiple ablation tables. Using different architectures for different tasks is not necessarily wrong, but without a complete configuration, it is impossible to determine whether the claims of the same model or of the same scale are valid. Furthermore, the classification does not list epochs, batch size, weight decay, pre-training sources, and complete enhancement recipes, making it difficult to verify the sources of 86.8%.

  3. Efficiency Claims Lack Implementation and Complete Timing for Step-by-Step Sequencing. The next step in ASO depends on the current token and the unvisited set; therefore, even if K is fixed and the workload is linear with respect to N, sequencing still involves N dependent decisions; the parallel scan capability of SSM does not imply that overall sequencing is equally efficient. Table 8 shows that RepScan-S has a lower latency of 1024² than InfoScan, but its memory usage reaches 26.8GB, higher than 20.0GB; at 224², RepScan-S is also slower than InfoScan-S. The manuscript does not list hardware, batch, accuracy, synchronization, warm-up, and kernel costs, therefore it cannot be verified whether the cost covers RRM, ASI, ASO, gather/scatter, and reverse engineering. The VMamba-S throughput in Table 1 and Table 8 also differs, and the baseline conditions need to be explained.

  4. Differences from existing content-adaptive sorting work are not adequately established. This paper cites InfoScan and DefMamba, but related work does not cover Content-Aware Mamba (ICLR 2026), MambaIRv2 (CVPR 2025), and ASM-UNet (2025 preprint). These studies have addressed content- or semantically guided scanning/reordering. Therefore, considering the input-dependent sequence topology itself as the main new perspective is too broad; however, this does not mean that its directed relationships, benchmarks, and prospective constructions have been fully addressed.

  5. Existing ablation methods are still insufficient to support relation learning and semantic mechanisms. The effectiveness of Q1/Q2 and ASI/ASO warrants investigation, but “Local ordering” is undefined, so it is unclear how RRM-only affects forward pass or gradient acquisition. When both Q1 and Q2 are removed, the relation score should be a common constant, the lookahead weight is zero, and the process degenerates according to the tie rule; actual permutations need to be provided for confirmation. The same checkpoint order intervention in Table 7 demonstrates dependence on the learned order, but also alters the training-test distribution, and cannot alone prove superiority over other retrained strategies. The predictor is only trained on task gradients; Gaussian interpretation is not equivalent to calibrated semantics or statistical sufficiency.

  6. BraTS's slice segmentation and metrics are insufficient to determine effectiveness. The manuscript only provides 21K/3K/6K slices, without specifying whether patients/volumes are mutually exclusive. If adjacent slices from the same patient cross sets, leakage may occur, but current information is insufficient to assert that this has already happened. Only T1, 2D UPerNet, and mIoU of 15.3% to 23.4% are custom settings and cannot directly represent the standard multimodal BraTS capability; it is also unclear whether the three semantic categories are original mutually exclusive labels or overlapping areas of WT/TC/ET.

  7. The relationship between local candidate images and global semantic jumps and visualizations is unclear. Preset edges only include horizontal/vertical ±1, ±2, and raster successors, not arbitrary long-range semantic neighbors; long-range jumps mainly likely originate from raster cross-column transitions or backtracking without scoring. While the entire sequence can accumulate global information, this does not mean each step should be described as directly selecting a distant semantic token. This paper stipulates that a complete permutation must visit all tokens, but Figure 5 uses a trajectory support mechanism focused on the foreground; if only prefixes, sampled edges, or smoothed edges are displayed, this must be explained.

  8. Key Advantages: Lack of Replicated Experiments and Resource Matching Comparisons. Small and medium-sized models contribute only 0.3 percentage points to InfoScan's classification gain without providing seeds or errors; T parameters are 28M vs. 10M, S FLOPs are 6.7G vs. 4.8G, and B is 10.3G vs. 8.4G, therefore the same suffix does not equate to the same resources. Current results may represent the combined effect of larger models and policies, and the full gain cannot be attributed to the scan.

Questions for authors
  1. Provide image-by-image N, the sum of the three prefixes, permutation, and masking processing procedures to confirm that the three q share the same permutation and the same sample weights; automatically check and recalculate Table 6 using interval value limits, and also verify the EFR of Tables 7 and 9. Supplement the foreground category set, evaluation layer/resolution, handling of invalid labeled tokens, and confidence intervals. This indicates incompatibility between the definition and the numbers, not an assumption of fraud.

  2. Disclose the stage depth, width, SSM state dimensions, scan count, relationship network, and head for each task (T/S/B); list the breakdown of backbone, additional modules, total parameters, and corresponding checkpoints. Correct S/B tags, indicating whether pre-training transfer occurs within the same category. Complete the training and model selection settings, marking cited figures and self-reproduced figures.

  3. Provide end-to-end implementations or verifiable core programs, programming and SSM segmentation profilers, workload and dependency depth analysis. Fully disclose measurement conditions, reporting latency distribution, training step time, and peak memory usage for the same batch/accuracy/input; FLOPs counts should indicate discrete indexing and traversal costs. Define which scenarios offer advantages using accuracy–latency–memory curves.

  4. Add references [1]–[3] to compare specific differences in candidate graphs, sorting units, starting points, traversal targets, gradient proxies, and costs. At least port representative content sorting and the InfoScan strategy under the same backbone and training budget; explain the independent gains of the new method relative to existing sorting methods.

  5. Supplement with precise ablation pseudocode; add a frozen/random relational network, general distance or cosine fraction, μ=0, remove the ASI gate, fix the offset, and implement a matching full retraining control. Report prediction error, score distribution, and effective gradient proportion; enumerate different path losses using small graphs to examine agent learning direction. Differentiate between order compatibility and semantic causal contribution.

  6. Report the ratio of scored edges, raster edges, and backtracking edges, jump distances, and prefix category coverage layer by layer. Differentiate edge types by color, publicly display the complete permutation, and indicate the prefix length/sampling rules in the figure. Table 9 supplements the discussion of the number of identical candidates for each topology, whether to retain raster successors, and KNN search costs, separating global information propagation and direct long-range edge selection.

  7. In Section 3.1, Eq. (2): π* is not proven through global optimization. It is suggested to change it to predict the permutation πθ, or explicitly define the asterisk as representing only the model output; correct the semantics of describing the function itself as a Π element in the main text.

  8. In Section 3.2, Eq. (4): W symmetry is not equivalent to positive semi-definite, nor does it guarantee semantic affinity; explain how to enforce symmetry, the number of layers, dimensions, normalization, and parameter sharing of ϕ and gψ.

  9. In Section 3.2, Eqs. (5)–(7): Fill in the empty candidate set and the complete branch of N=1 to avoid empty softmax/average; fill in the ε value and the processing of feature scale.

  10. Please compare the following papers: [1] Y. Chen et al., “Content-Aware Mamba for Learned Image Compression,” ICLR, 2026. [2] H. Guo et al., “MambaIRv2: Attentive State Space Restoration,” CVPR, pp. 28124–28133, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.