acceptodds
Withdrawn from ICLR 2027

MorphState: Coupling State Aggregation and Detail Recovery for Fine-Grained Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

State space models represented by Mamba achieve efficient long-range modeling through selective state updates. However, for visual models that aggregate spatial features into a small number of states, deciding which information should be written into the states does not directly reveal which local structures remain insufficiently represented, leaving subsequent detail recovery without an explicit reference to the current aggregation. To address this issue, we propose MorphState, a fine-grained segmentation framework that uses the spatial allocation of state writes as a shared reference for global aggregation and for generating recovery candidates through regularized feature reconstruction. First, high-resolution morphological cues modulate the spatial write weights while keeping the state count fixed. Next, the updated allocation is treated as a spatial reconstruction dictionary to regularize the reconstruction of pre-aggregation features, and the components that cannot be sufficiently explained by this dictionary are extracted as complementary candidates. Finally, because such residual components may also contain irrelevant texture variations, the model combines morphological orientation, residual magnitude, and coarse semantic predictions to modulate their class-specific contributions and refine high-resolution segmentation outputs. On BD3-Seg, MDMCS, and CUBIT-Seg, MorphState achieves a five-class mIoU of 85.77%, and foreground mIoUs of 83.45% and 53.07%, respectively, outperforming the strongest compared methods on the three datasets by 5.00, 4.32, and 4.18 percentage points. Ablations show additional gains from the complete allocation-and-recovery configuration over MGCR alone on BD3-Seg and MDMCS.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual State Space Models;State Representation Learning;Dense Prediction;Semantic Segmentation;State Complementary Recovery;Structural Damage Segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_kQ9WOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MorphState is a fine-grained segmentation model on an EfficientViM backbone. High-resolution morphological cues reweight a fixed set of state weights. That allocation is then used as a dictionary for a ridge reconstruction of the pre-aggregation features, and a decoder turns the residual into class-specific corrections.

Critical strengths

The allocation is used twice, and the two uses are different. It reweights which spatial features are written into a fixed number of states, and the same rows become the dictionary for the residual. The detail head starts at zero, so training begins from the semantic branch.

The method section says which tensor each later module consumes, which path is stopped, and that the reconstruction target is the pre-mixing feature rather than a hidden response. The solve cost is stated to scale with spatial length at a fixed state count.

On the three reported defect test sets, mean overlap improves over the compared entries, and the full three-module stack is the best aggregate configuration in the component study. The estimated cost is well below the heavy transformer baseline in the same cost table.

Critical weaknesses

The decoder carries most of the gain. Refinement alone is already close to the full model, and allocation plus recovery without that decoder is weaker. On one dataset, adding either state module to refinement alone lowers the score, and only the joint configuration recovers it.

The paper does not measure the mechanism it names. There is no test that state mass moves onto thin structures, or that residual magnitude marks the pixels the coarse head gets wrong. There is also no plain high-resolution skip at the default ridge weight.

The comparison is uneven. The model loads a distilled initialization, trains with an exponential moving average, and selects the checkpoint by validation overlap, while epoch count and augmentation for the field are not pinned down. One state-space baseline sits far below the others on the main dataset, with no diagnosis.

Evidence is three infrastructure-damage sets, two of them small, and each headline number is one selected checkpoint. Nothing shows that the allocation rule transfers to a standard multi-class segmentation benchmark.

The task statement is about boundaries and connectivity. The reported metrics are region overlap, accuracy, and Dice. On those extra metrics, other models still win some columns, and the qualitative panels are deliberately selected favorable cases.

The paper lacks comparisons with many previous Mamba-based methods, such as [1] SegMAN: Omni-scale Context Modeling with State Space Models and Local Attention for Semantic Segmentation, CVPR 2025; [2] Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion, ICLR 2025; [3] VSSD: Vision Mamba with Non-Causal State Space Duality, ICCV 2025; [4] SparX: A Sparse Cross-Layer Connection Mechanism for Hierarchical Vision Mamba and Transformer Networks, AAAI 2025; and [5] Multi-scale VMamba: Hierarchy in Hierarchy Visual State Space Model, NeurIPS 2024. These methods can likewise be seamlessly adapted to the task considered in this paper, yet no comprehensive comparison or discussion is provided.

Questions for authors

Please refer to Critical Weaknesses.

Reviewer_Zs4qOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MorphState, a fine-grained segmentation framework built on state space models (specifically extending EfficientViM's HSM-SSD formulation). The central idea is to use the SSD's state-write allocation matrix not only as global feature aggregatior but also as to identify what information is erased and to recover erased details via regularized feature reconstruction. The framework has three components:

  • Morphology-Aligned State Allocation (MASA): Uses high-resolution morphological cues to modulate spatial write weights while keeping the state count fixed.
  • State-Span Complement Recovery (SSCR): Treats the allocation as a reconstruction dictionary and extracts ridge-regularized residuals that the state subspace cannot explain.
  • Morphology-Guided Class Refinement (MGCR): Combines morphological orientation, residual magnitude, and coarse semantic predictions to generate class-specific gates for refining high-resolution outputs.

On BD3-Seg, MDMCS, and CUBIT-Seg, MorphState outperformes the baseline with a large margin.

Critical strengths
  • Conceptually novel idea: The idea of using the state-write allocation matrix not only as a global aggregation operator but also as a dictionary for reconstructing lost information is elegant and well motivated.
Critical weaknesses
  • Limited domain scope undermines generality claims: All three datasets are infrastructure/building-damage segmentation. The paper frames MorphState as a general "fine-grained segmentation framework," but provides no evidence on standard benchmarks (e.g., Cityscapes, ADE20K). It is unclear whether the morphological-cue mechanism generalizes beyond crack/spall/corrosion-type structures.

  • Lack of inference speed evaluation: Although MorphState is relatively lightweight in terms of FLOPs count, SSCR performs an N×NN \times N linear solve for each target block. Even if the FLOPs are low, can this operation be executed efficiently in practice?

Questions for authors
  • Does MorphState generalize to general-purpose segmentation benchmarks, such as ADE20K dataset? If the method is domain-specific to infrastructure damage, the framing and claims should be scoped accordingly.
  • Given that MASA and SSCR individually degrade performance on MDMCS, what is the mechanistic explanation for why they help only jointly with MGCR? Can you isolate whether the gain comes from the allocation-reconstruction coupling or simply from more decoder capacity?
  • What is the inference speed of MorphState?
Reviewer_AdEtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MorphState, a fine-grained segmentation framework built on EfficientViM-M3 that explicitly couples compact state aggregation with local-detail recovery. Specifically, MASA extracts multiscale structure-tensor cues from shallow features and uses them to conservatively reweight the spatial allocation of a fixed number of hidden states. SSCR treats the updated allocation matrix as a ridge-regularized reconstruction dictionary and extracts feature components that are insufficiently represented by its row span. Finally, MGCR combines these residuals with shallow and semantic features, performs morphology-conditioned directional filtering, and applies class-specific gated logit corrections.

Critical strengths

The aggregation–recovery coupling is an interesting idea. The strongest conceptual contribution is to define missing local evidence relative to the allocation actually used to write compact states. SSCR is therefore not simply another generic high-resolution branch: its reconstruction basis is tied to the model’s current aggregation. This gives the recovery mechanism a clear interpretation and could be useful beyond the particular decoder proposed here.

Critical weaknesses

The interpretation of SSCR is stronger than what the formulation guarantees. SSCR computes a ridge residual relative to the row span of the spatial allocation SS, whereas the actual state path additionally contains nonlinear state mixing and a separate readout. A component outside this row span cannot affect the linear state write, but this does not imply that it is task-relevant information “missed” by the complete aggregation module; conversely, information inside the span may still be suppressed downstream. Moreover, because both XX and SS are optimized jointly through the segmentation loss, the network can deliberately route useful features into the residual branch. I therefore find the residual well defined mathematically, but its interpretation as evidence of aggregation deficiency is not fully justified.

The proposed “morphology” prior is more heuristic and domain-specific than the paper’s general framing suggests. It is constructed from a learned scalar projection followed by Sobel-based structure tensors, three manually chosen support types, and a small set of discretized directional filters. These cues respond to any local gradient or texture and do not inherently distinguish meaningful object morphology from material texture or illumination boundaries. This inductive bias is plausible for infrastructure damage, particularly thin cracks, but the paper does not establish why it should generalize to fine-grained segmentation more broadly. The current title and motivation therefore appear to overstate the generality of the method.

Questions for authors

Can the authors compare the proposed shared-allocation SSCR against (i) reconstruction using the unmodified allocation S0S_0, (ii) a stop-gradient allocation, and (iii) an independently learned or parameter-matched reconstruction basis? A matched MGCR variant without allocation-derived residuals would also help establish whether the proposed coupling, rather than decoder capacity, explains the remaining gain.

Reviewer_tbtSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MorphState, a fine-grained segmentation framework built on EfficientViM. The method introduces three components: morphology-aligned state allocation (MASA), state-span complement recovery (SSCR), and morphology-guided class refinement (MGCR). These components aim to connect compact state aggregation with the recovery of local structural details. Experiments on three infrastructure damage segmentation datasets report improvements over the evaluated baselines.

Critical strengths

1.The paper proposes an interesting way to connect state aggregation with local feature recovery. Instead of introducing an independent detail branch, SSCR uses the learned state allocation as a reconstruction dictionary to extract complementary features. This provides a reasonable motivation for the proposed framework and makes the design more coherent than simply combining global and local features. 2.MorphState achieves improvements of 5.00, 4.32, and 4.18 mIoU points over the strongest reported baselines on BD3-Seg, MDMCS, and CUBIT-Seg, respectively. The improvements on thin cracks and irregular regions are particularly relevant to the paper's motivation. The reported parameter count and FLOPs are also competitive, suggesting that the method does not rely on a substantially larger model.

Critical weaknesses

1.The main contribution is the connection between state allocation and complementary recovery. However, the ablations suggest that much of the improvement comes from the MGCR decoder. On BD3-Seg, MGCR alone achieves 83.78% mIoU, compared with 85.77% for the full model. On MDMCS, adding MASA or SSCR individually to MGCR actually reduces performance, while combining both improves it. Although the full configuration performs best, these results do not clearly establish why recovery based on the learned state-allocation subspace is better than other detail-recovery strategies. More direct evidence is needed to support the central mechanism. 2.The paper reports strong results, but the comparison protocols are not sufficiently detailed. MorphState uses pretrained EfficientViM-M3 weights, while the pretraining and optimization settings of the competing methods are not clearly specified. The results are also reported without variation across random seeds, which is particularly relevant for BD3-Seg and MDMCS, whose test sets contain only 100 images each. In addition, the main manuscript refers to supplementary experiments on alternative allocation and recovery designs, but these experiments are not available in the provided PDF. These limitations make it difficult to fully assess the reliability of the reported improvements.

Questions for authors
  1. Can the authors provide a controlled comparison between SSCR using the actual learned state allocation and alternatives such as a fixed allocation, an independently learned reconstruction dictionary, or a parameter-matched conventional detail-recovery module? This would help establish whether the proposed coupling is responsible for the additional gains.
  2. Were all competing methods trained with comparable pretrained weights, data augmentations, training budgets, and hyperparameter tuning? Could the authors report mean and standard deviation over multiple runs, particularly for BD3-Seg and MDMCS?
  3. Since efficiency is an important motivation, could the authors report actual inference latency and peak GPU memory under identical hardware and input settings? FLOPs alone may not capture the overhead of the reconstruction and morphological operations.
  4. In Eq. (1), the state-specific term a appears to be added as a spatially constant bias before the softmax over spatial locations. Such a term would cancel under this normalization. Could the authors clarify whether the equation accurately describes the implementation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.