acceptodds
Withdrawn from ICLR 2027

Sub-Semantic Image Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Semantic segmentation tells us what a region is, but many useful visual distinctions depend on how it looks. Bare and moss-covered rock, peeling paint and exposed wood, or grass growing through sand may share semantic labels while forming clearly distinct appearance regions. Traditionally, segmentation partitions scenes into predefined objects and an amorphous “stuff” category. However, as modern segmentation models increasingly integrate with natural language, this rigid di- chotomy becomes obsolete, and there is no longer a reason to discard rich visual information into a generic “stuff” bucket. Yet, driven by a strong semantic bias, these advanced models degrade when encountering regions that are too abstract for a fixed dictionary noun, but visually distinct enough to elicit rich human descrip- tions. To provide a more fitting representation and explicitly target this vulnerability, we introduce sub-semantic image segmentation, the task of autonomously discov- ering such regions, describing each in free-form language and assigning every description an associated mask, all without receiving region prompts or the number of regions in advance. To make this problem measurable, we contribute three com- plementary task-specific benchmarks spanning annotation-derived natural scenes with variable region counts, independently constructed natural surface transitions and entirely new controlled synthetic compositions, together with validation on an established real-world texture dataset. We propose Detecture, an integrated description-to-partition system that generates structured appearance descriptions, extracts isolated grounding states, predicts description-conditioned masks and resolves them through pixelwise competition. Across all four evaluation routes, Detecture consistently outperforms native GLaMM and current Sa2VA in region covering, and leads the evaluated native language-and-mask model class on both natural benchmarks. Controlled interventions also show that descriptions actively guide region assignment: changing appearance wording alters masks, and mask identities follow appearance edits substantially more often than spatial edits among jointly eligible cases. Together, these results establish language as a practical interface for discovering and addressing visual structure beyond fixed category vocabularies.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
sub-semantic image segmentation, texture discovery, vision-language models, grounded generation, open-vocabulary segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HkQCOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces sub‑semantic image segmentation, a task to discover appearance‑driven visual regions and output disjoint masks paired with free‑form textual descriptions without pre‑defined semantic labels. The authors propose Detecture, a frozen two‑backbone model based on Qwen3‑VL‑8B and SAM3. Key designs include structured autonomous generation with [SEG] markers, block‑diagonal attention for isolated grounding, a lightweight Bridge projection, competitive pixel‑wise assignment with a residual channel, and a Shifted‑Zero loss for training stability. Three new benchmarks plus an external texture dataset are built for evaluation. Results show competitive performance against existing language‑mask models, and controlled interventions validate that text guides mask assignment. Nonetheless, concerns remain regarding dataset annotation quality, missing ablations for critical hyper‑parameters, and limitations of the teacher‑forced intervention setup.

Critical strengths
  1. The task is well motivated and unusually carefully specified: this work defines the novel sub-semantic image segmentation task, focusing on appearance-based region partitioning within the same semantic category, and establishes six clear task design principles.
  2. The authors build three complementary benchmarks. TextureADE (annotation-mined natural scenes, variable cardinality), RWTD-COCO (independently constructed natural two-surface transitions), CSTD (controlled synthetic compositions), plus external RWTD.Controlled intervention experiments are carefully designed to verify that text descriptions actively guide mask generation rather than being post-hoc explanations.
  3. The proposed Detecture introduces block-diagonal attention for isolated grounding, lightweight Bridge projection, competitive pixel assignment and Shifted-Zero language loss, enabling frozen two-backbone (VLM + segmentation foundation model) to output disjoint masks with free-form descriptions.
Critical weaknesses
  1. The ground-truth masks of the proposed benchmarks are largely inherited from legacy semantic segmentation datasets instead of re-annotated specifically for sub-semantic appearance boundaries. There lacks human evaluation to verify the correctness of text-mask pairing.
  2. The method called "Qwen3-VL + SAM 3 semantic head" in Table 22 is significantly better on RWTD and CSTD, statistically unresolved on RWTD-COCO, and worse only on TextureADE. Since this baseline is both directly relevant and strong, it should appear in the main comparison.
  3. The source-data exposure of the compared systems is asymmetric and not fully characterized. The paper correctly notes that source identifiers cannot establish absence from all pretraining or development data. This issue should be discussed alongside the corresponding table rather than only in the appendix(p.40).
Questions for authors
  1. Since TextureADE and RWTD-COCO largely inherit masks from legacy semantic annotations, could the authors provide a dedicated human evaluation.
  2. Could the authors include Qwen3-VL + SAM 3 semantic head in the main comparison table? If the authors believe it is not directly comparable, please clearly explain the task or protocol distinction.
  3. Could the authors more explicitly characterize and discuss source-data exposure for all compared systems alongside the main results?
Reviewer_n5grOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a new task, sub-semantic image segmentation, in which a system must autonomously discover image regions defined by appearance rather than by predefined object categories, describe each region in free-form language, and assign a disjoint binary mask, with no image-specific prompts or category lists given at test time. The authors contribute three benchmarks — TextureADE, RWTD-COCO, CSTD, RWTD. The proposed Detecture builds on Qwen3-VL-8B and frozen SAM 3 with three design choices: (i) block-diagonal attention isolation so each grounded region attends only to its own description token slot; (ii) a LoRA-trained "Bridge" projection into independent SAM-3 mask lanes; and (iii) a Shifted-Zero language loss that suppresses the grounding marker while retaining generation. On the covering metric, Detecture leads all compared open-vocabulary grounded methods on the four datasets.

Critical strengths
  1. The six design goals listed in the introduction make the task operational; framing appearance rather than object parts as the organizing axis addresses a real gap between open-vocabulary referring segmentation and unsupervised partition.
  2. The three benchmarks are complementary in structural guarantees (multi-region natural scenes, deterministic two-source transitions, controllable synthetic composition) and the RWTD external split reduces overfitting-to-benchmark risk.
  3. The Section 4 interventions — deleting descriptions drops covering by 29.7 points, and swapping appearance vs. spatial wording changes which slot a region follows in 58.2% vs. 1.0% of eligible cases — directly support the claim that language is doing work rather than being a post-hoc caption of a vision-only partition, which is the failure mode that usually sinks "describe-and-ground" systems.
Critical weaknesses
  1. Every baseline in Table 2 is a prompt-conditioned open-vocabulary grounded model (GLaMM, Sa2VA-4B, Sa2VA Qwen3-VL/SAM3, SAMTok-3B) that the authors hand-feed appearance-region prompts; the natural autonomous baselines — SAM automatic mask generation followed by a captioner per mask, or appearance-only unsupervised segmentation scored by the same covering metric — are missing, so the autonomous discovery claim is not yet substantiated.
  2. The two named technical contributions (block-diagonal attention isolation, Shifted-Zero loss) are introduced as the key mechanisms in the method section, but their supporting evidence lives in Appendices C and M; there is no main-text ablation isolating the Bridge, the isolation mask, or the loss, so the reader cannot tell which component carries the gain.
  3. The entire Related Work discussion is deferred to "Appendix A" after a few inline citations in the introduction; for a new-task proposal, the positioning against class-agnostic segmentation, unsupervised/zero-shot segmentation (where the covering metric itself originates), and SAM-AMG is the literature the paper lives or dies on and cannot live only in the appendix.
  4. The abstract and experiment section claim Detecture "consistently outperforms" across all four routes, but on external RWTD it is 79.4 vs. Sa2VA-4B 79.1 with overlapping bootstrap intervals, and on RWTD-COCO the gap is only 1.7 points; the gains are in fact concentrated on TextureADE and CSTD and should be qualified to match Table 2.
  5. The test sets are small (212/256/256 images) and TextureADE's Miner scores candidates by region structure and image-language signals before Detecture describes them, raising a circularity risk that the benchmark favors the kind of partition Detecture already produces; the paper should report or argue why the Miner's scoring is independent of Detecture's segmentation prior.
Questions for authors

The task is useful, the benchmarks are a real contribution, and the causal interventions are the kind of evidence most describe-and-ground papers lack. But as submitted, the paper lacks the natural autonomous baselines (SAM-AMG + captioner, appearance-only unsupervised segmentation) needed to substantiate the discovery claim, pushes both key-component ablations and the entire related-work discussion into the appendix. See weakness for more details.

Reviewer_iK3COpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Authors introduce "sub-semantic segmentation" as a new task. Their core argument is that now that regular segmentation (things) has made substantial progress, we should now focus on segmenting textures and other "stuff" in the scene that does not have a clearly prescribed object category. Authors train a new model and introduce a benchmark for this task.

Critical strengths

To the best of my knowledge, this is a "new" task that has not been proposed yet. However, I did not investigate this very deeply, so I would defer to other reviewers.

Critical weaknesses

The writing appears substantially AI edited with little thought about clarity or ease of understanding. This can make it hard to understand at several times throughout the paper. I feel that a paper should be written for human understanding and not just for the sake of describing a method on paper. I feel that this paper is not at all ready for review by the ICLR community and authors should strongly consider rewriting it prior to resubmission.

The task of sub-semantic image segmentation is not well motivated. Why do we care about this task to begin with? What is a potential application of this task assuming it is "solved" at some point in the future?

Questions for authors

What is the point of Fig. 1? It was a bit hard to get what the "main point" is.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.