acceptodds
Withdrawn from ICLR 2027

Tracing Object Evidence Across Depth: Text-Modulated Memory Framework for Weakly Supervised Semantic Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Weakly supervised semantic segmentation (WSSS) learns pixel-level predictions from image-level category labels. While CLIP provides rich semantic priors for localization, recovering complete objects requires integrating complementary evidence across visual layers. Conventional multi-layer aggregation lacks an explicit mechanism for retaining and revising concept-specific evidence as semantic context deepens, leaving local structures weakly tied to category semantics and producing incomplete object coverage and spurious background responses. We propose a text-modulated cross-layer xConvLSTM framework that formulates visual-layer integration as concept-conditioned memory evolution across network depth. Treating selected CLIP layers as an ordered sequence, the framework uses text representations to control how spatial memory incorporates, retains, and suppresses visual evidence, progressively connecting local structures and object parts to category semantics. To adapt this process to each image while preserving CLIP's semantic priors, we introduce anchored adaptive prompting, which augments frozen text embeddings with bounded learnable and image-conditioned residuals. The resulting concept-specific memories calibrate visual tokens, guide decoding, and refine pixel relations, coupling evidence accumulation across depth with response propagation across space. Extensive experiments on WSSS benchmarks demonstrate more complete object localization, higher-quality pseudo-labels, and consistent segmentation gains.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Weakly supervised semantic segmentation, Vision-language models, Cross-layer feature modeling, xLSTM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dUBVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses weakly supervised semantic segmentation by introducing a text-modulated cross-layer xConvLSTM framework to mitigate incomplete object coverage and spurious background responses during multi-layer CLIP feature integration. The proposed approach formulates cross-layer aggregation as concept-conditioned memory updates along network depth, where text embeddings control how spatial memory incorporates, retains, and suppresses visual evidence. Furthermore, anchored adaptive prompting is introduced to adapt text representations to individual images via bounded learnable and image-conditioned residuals while anchoring pre-trained CLIP semantic priors. Extensive experiments on PASCAL VOC 2012 and MS COCO 2014 demonstrate the effectiveness of the proposed model.

Critical strengths
  1. This paper proposes a text-guided adaptive fusion mechanism that enables the complementarity between shallow and deep CLIP features, effectively improving the methods for using CLIP features in the experiments of the weakly supervised segmentation field.
  2. The ablation study validates the contribution of each component separately, providing support for the effectiveness of the proposed design.
Critical weaknesses
  1. The abstract and introduction of the paper emphasize "consistent segmentation gains", but Table 1 only shows a 0.4 mIoU improvement on VOC compared with WeCLIP+, and the scores on COCO are identical. This gap can hardly be regarded as a real gain, which contradicts the claim of "consistent improvement".
  2. The whole paper focuses heavily on "text modulation", with the title and contributions all centered around it. However, in Table 2, removing xConvLSTM leads to a 0.9 mIoU drop, while removing text modulation only causes a 0.2 drop — it is the cross-layer memory itself that actually plays the major role, and the marginal effect of text modulation is actually the smallest.
Questions for authors
  1. The abstract and introduction of the paper emphasize "consistent segmentation gains", but Table 1 only shows a 0.4 mIoU improvement on VOC compared with WeCLIP+, and the scores on COCO are identical. This gap can hardly be regarded as a real gain, which contradicts the claim of "consistent improvement".
  2. The whole paper focuses heavily on "text modulation", with the title and contributions all centered around it. However, in Table 2, removing xConvLSTM leads to a 0.9 mIoU drop, while removing text modulation only causes a 0.2 drop — it is the cross-layer memory itself that actually plays the major role, and the marginal effect of text modulation is actually the smallest.
  3. There are no comparisons of parameters, FLOPs, training time, and inference cost in the main table at all. One of the contributions of WeCLIP+ is its low training cost. After adding new modules on the basis of it, this paper does not explain how the cost changes.
  4. The method overview states that the selected layers are L = {2,5,8,10}, while in the experimental part it is described as "the 3rd, 6th, 9th, and 11th CLIP visual blocks". The two parts are inconsistent. Besides, why 4 layers? Why these specific layers?
Reviewer_T1FJOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a single-stage WSSS method using frozen CLIP-DINO representations. Its main contribution focuses on the aggregation of CLIP features across network depth. Specifically, an xConvLSTM module processes features from selected layers in depth order to accumulate spatial evidence. Class-concept queries condition the visual features and recurrent gates, guiding memory updates. These queries incorporate part-level, attribute, and structural descriptions and are adapted through learnable and image-conditioned residuals anchored to frozen text representations. The paper reports segmentation results on PASCAL VOC and MS COCO, with component ablations conducted on VOC.

Critical strengths

The use of spatial recurrent memory to consolidate evidence across CLIP layers is an interesting direction. Maintaining and selectively updating a state across depth offers a plausible mechanism for combining complementary localization cues.

Critical weaknesses

The central claim that explicitly modeling cross-layer aggregation as an ordered process is beneficial, has not been sufficiently isolated or validated. The text modulation and adaptive prompting designs appear incremental, and their independent contributions remain unclear. Relative to the most relevant baseline, WeCLIP+, the full method provides modest gains on VOC (+0.4 mIoU on both validation and test sets) and no gain on COCO.

Questions for authors

1. What evidence establishes the benefit of ordered cross-layer aggregation?

Table 3 does not isolate the contribution of processing features in depth order. In terms of final segmentation mIoU, xConvLSTM without the delta update matches learned weighted fusion at 77.2, while static averaging achieves 77.5. The full xConvLSTM improves to 78.8, suggesting that the delta update is important within this implementation. Could the authors clarify whether the improvement primarily comes from the proposed cross-layer memory formulation or from this particular update rule?

Moreover, all recurrent variants use the same depth order. Please compare forward, reverse, and shuffled layer orders while keeping the architecture and input features fixed to establish whether the original network depth order is necessary.

2. What distinguishes Contributions 2 and 3 from existing conditional modulation and prompting designs?

Text-conditioned feature modulation, image-conditioned prompting, and enriched textual priors are established ideas. Their integration with the proposed memory module may be useful, but the manuscript should more clearly identify the methodological distinction and demonstrate its benefit. Table 2 provides mixed evidence: adding text modulation from L3 to L4 improves raw CAM quality but reduces final segmentation mIoU from 78.8 to 78.5; removing it from the full model reduces mIoU by only 0.2 under CRF evaluation.

Please provide controlled ablations of Eq. (4), separating the learnable residual, image-conditioned residual, and anchoring constraint. To validate the necessity of these components.

3. Can the component comparisons use a consistent evaluation protocol?

Table 2 includes “xConvLSTM + rich anchors, without text modulation” in the leave-one-out analysis with CRF, but the corresponding result without CRF is missing. Reporting all component configurations under the same post-processing settings would make the effects and interactions easier to interpret.

4. Is the rich concept library necessary?

Adding rich anchors to the baseline improves raw CAM mIoU but leaves final segmentation unchanged at 78.0. Although rich anchors help within the full configuration, this does not establish the necessity of the detailed concept vocabulary. Could simple part names or an equal number of class-level prompts achieve comparable results? Please control the number of queries and report the additional computational cost to distinguish the value of richer semantic priors from that of increased query capacity.

Reviewer_MwbDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work looks at weakly supervised semantic segmentation with only image-level labels. The main thing they propose is a text-modulated cross-layer xConvLSTM that treats CLIP layers as an ordered sequence and accumulates object evidence across network depth, instead of just fusing features all at once. They also add an anchored adaptive prompting design to tune text embeddings per-image without drifting too far from CLIP’s original semantic priors. Experiments run on PASCAL VOC and MS COCO, with full component ablation and head-to-head comparison against different cross-layer aggregation strategies.

Critical strengths
  1. Experimental design is solid and honest. They don’t just throw out SOTA numbers — they explicitly compare xConvLSTM against simple average, weighted fusion and vanilla ConvLSTM, which rules out the lazy explanation that “using more layers just naturally works better”. They also report both weak and strong baselines, and openly show gains shrink on the full WeCLIP+ pipeline. This level of rigor makes the work reliable and easy to reproduce.
  2. The cross-layer memory idea targets a real pain point in CLIP-based WSSS. Everyone in this field knows shallow layers keep local details and deep layers carry strong category semantics, but few works model the aggregation as a stateful, sequential process. And it fits into existing RFM pipelines without big rewrites, so it’s not just a paper trick — people can actually plug it into existing systems.
Critical weaknesses
  1. The overall contribution is pretty incremental. At the end of the day it’s adapting xLSTM to cross-layer feature fusion with text conditioning. On the strong WeCLIP+ baseline the mIoU gain is very small — 0.4% on VOC, basically zero on COCO. For teams already running a strong baseline, the practical upside is quite limited. It’s a clean improvement but not a big leap.
  2. Some key design choices feel empirical and under-explored. The concept term vocabulary and the picked layer indices {2,5,8,10} are not fully ablated; it’s hard to tell how sensitive results are to these choices, and how to port this setup to other CLIP backbones.
Questions for authors

why exactly layers {2,5,8,10}? Is there a simple rule of thumb for picking layers that others can reuse?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.