acceptodds
Withdrawn from ICLR 2027

ReTax-Seg: Visually Grounded Reversible Taxonomy Transport for Image Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing taxonomy-aware segmenters reconcile incompatible label names but leave a fundamental ambiguity: a dataset label can be a policy-dependent union of visual concepts, yet its mapping to a unified vocabulary remains global and one-way. A fixed mapping cannot identify the constituent present in a region. A language-derived mapping imports name semantics without observing that region. A learned latent vocabulary can also collapse despite correct final-label predictions. ReTax-Seg addresses these failures jointly. A visual decoder predicts 39 fine-grained semantic atoms, and an end-to-end transport field compiles them into native labels with a different atom–class distribution at every spatial location. Class names supply only a learned-strength CLIP prior, while supervised visual features and reliability-gated MobileSAM geometry determine the local correction. Reverse transport requires predicted native labels to reconstruct the visual atom distribution, making semantic preservation an explicit training objective. To our knowledge, ReTax-Seg first combines pixel-conditioned class–atom transport, visual-over-language arbitration, and bidirectional taxonomy consistency in one differentiable compiler. Six independent checkpoints average 73.63% test mIoU. One jointly trained six-dataset checkpoint, ReTax-Seg-C, shares the atom representation and reaches 74.05%, outperforming the compared methods on every dataset.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Semantic segmentation, taxonomy alignment, visual grounding, dynamic transport, cycle consistency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PjrAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies taxonomy-aware image segmentation. The authors observe that existing approaches typically map dataset-specific labels to a unified vocabulary in a global and unidirectional manner. They propose ReTax-Seg, which incorporates CLIP priors and SAM-derived geometric information through pixel-conditioned class-to-atom transport, visual-over-language arbitration, and bidirectional taxonomy consistency. Experiments on six datasets demonstrate the effectiveness of the proposed method.

Critical strengths
  • The manuscript provides a detailed presentation of both the methodological formulation and the experimental analysis, although much of this material is included only in the supplementary document.
  • The extensive qualitative results complement the quantitative evaluations and substantially enrich the experimental analysis.
Critical weaknesses
  • The excessive number of equations reduces the readability of the paper. For example, Eqs. (1)–(4) describe relatively straightforward neural network computations and could be presented more intuitively using an architectural diagram.
  • Since this work specifically targets taxonomy-aware image segmentation, the experimental comparison should include other taxonomy-aware segmentation methods rather than focusing primarily on conventional segmentation approaches.
  • Several important hyperparameter choices lack sufficient empirical justification. For example, why are 39 atoms used? Is this choice specific to the datasets considered in the paper, and does it generalize to other datasets and scenarios? The paper should provide practical guidance on how to determine this parameter in new settings. Similar concerns apply to the weighting coefficients in Eq. (13).
  • The Introduction and Related Work are poorly organized and does not clearly establish the motivation for the work. Although the authors pose the question, “Which constituent of a coarse or policy-dependent class is supported by the pixels at this location?”, they do not adequately explain why this issue arises or why it is important. This weakens the overall motivation of the paper.
Questions for authors

What is the key distinction between ReTax-Seg and existing taxonomy-aware image segmentation methods? More specifically, what makes the proposed approach both technically different and practically important?

Reviewer_e3x5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents ReTax-Seg, a semantic segmentation method that is class taxonomy-agnostic. Instead of directly predicting a per-pixel score for all dataset-defined classes like most common segmentation methods, ReTax-Seg first predicts a score for 39 ‘semantic atoms’. These semantic atoms represent supercategories, and these scores are subsequently mapped to scores for the dataset-defined classes, by using CLIP language embeddings, MobileSAM features, and a learnable mapping. This taxonomy-agnostic setup could facilitate multi-dataset training, where multiple datasets use different class taxonomies. In this case, only the mapping from the semantic atoms to the dataset-defined classes would need to vary for different datasets. Through experiments in specific single-dataset settings, ReTax-Seg performs better than various existing methods that it reports. Furthermore, multi-dataset training yields a minor improvement over single-dataset training.

Critical strengths

There are no critical strengths that support the acceptance of this paper.

Critical weaknesses
  1. The experimental evaluation of this paper is insufficient to demonstrate the effectiveness of the method.
    1. The reported scores for existing methods are significantly lower than reported in the original papers. For example, Tab. 1 reports very low mIoU scores of 46.98 for SegFormer-B3 and 46.18 for SegFormer-B4 on Cityscapes, whereas the original SegFormer paper [a] reports mIoU scores of 81.7 and 82.3 on Cityscapes for B3 and B4, respectively. This calls into question the correctness of the results, and makes it unclear whether ReTax-Seg is truly better than these models. The differences could be (partly) explained by the fact that the paper’s experimental setup differs from prior work. For example, it uses its own train/val/test splits (L259) and uses only a low 512x512 resolution for training and inference (L267 & L318). It is not clear whether all the large performance gaps can be attributed to this. To properly demonstrate the effectiveness of the method, the paper should use the originally reported performance of existing methods, use a similar experimental setup for ReTax-Seg, and report the performance in this setup. Until this is done, the effectiveness of ReTax-Seg is not demonstrated properly.
    2. The paper does not experimentally compare ReTax-Seg to existing state-of-the-art methods for multi-dataset segmentation or taxonomy-alignment for segmentation, such as LMSeg [b] and DynAlign [c]. In Sec. 2 (L099-L102), the paper mentions these methods as relevant works, but argues that a disadvantage of these methods is that their alignment remains 'forward-only'. However, the paper does not demonstrate that ReTax-Seg actually outperforms these methods. As such, the value of ReTax-Seg over alternative, existing methods has not been properly demonstrated.
    3. In Tab. 1, ReTax-Seg is compared to several existing single-dataset methods for semantic segmentation, but most of these methods are outdated. The paper does not compare ReTax-Seg to current state-of-the-art transformer-based segmentation models like EoMT [d] or OneFormer [e]. As a result, it is not clear whether ReTax-Seg truly outperforms existing methods, which is what the table suggests. This makes it unclear what its value is.
    4. The benefit of training ReTax-Seg in a multi-dataset setting is very limited. While it appears that the main focus of the paper is to allow a segmentation model to learn effectively from multiple datasets with different class taxonomies, the average mIoU only improves from 73.63 to 74.05 from single-dataset to multi-dataset training. This minor improvement suggests that there is little benefit of multi-dataset training with the proposed framework, limiting its value.
    5. The evaluation is missing very simple baselines that are necessary to assess whether the use of the ‘semantic atoms’ and the dynamic mapping to the dataset-specific class labels is effective. A much simpler alternative would be to take the encoder’s output features, and have a single linear layer per dataset that directly maps the features to per-pixel classification scores for that dataset. ReTax-Seg should at least outperform this baseline to demonstrate its value, but such an experiment is currently not provided. Tab. 3 currently only experiments with different atom-to-class mappings, but does not consider the simpler aforementioned alternative.
  2. The paper’s understandability is poor. Some choices are not sufficiently motivated, and the paper frequently does not provide sufficient context or does not explain aspects of the method in sufficient detail. This makes it difficult to understand the paper’s contributions and it’s value compared to existing works.
    1. The paper does not specifically explain what the task objective is, neither in the introduction nor elsewhere. Is it to do single-dataset segmentation or multi-dataset segmentation? If it is the former, then why is it necessary to make a model compatible with different taxonomies? If it is the latter, then why is the model evaluated mostly in the single-dataset scenario, and why isn’t the model compared to alternative methods for multi-dataset segmentation? This is not clear from the paper.
    2. L050 and L116 specify that 39 ‘semantic atoms’ are used, but the paper does not explain why it uses the specific number of 39, or how these are defined.
    3. L144 explains that a learned visual branch extracts S=8S=8 region factors, but the paper does not explain (i) what ‘region factors’ are and what they represent, or (ii) how the learned visual branch extracts these region factors. Since these region factors are used to obtain region states qq (Eq. 4), it is also not clear what the region state represents. The same applies to the transport logits l\mathcal{l} (Eq. 5), which is a function of these region states. Similarly, the paper mentions that it uses dataset-class factors UdU_ d and atom factors VV (L165-L166), but it does not explain (i) what these factors are and what they represent, or (ii) how they are obtained. Because of all this, it is not clear how the MobileSAM features are used to create the mapping from the per-atom scores to the scores for dataset-specific classes, while this is a key aspect of this method as it is supposed to ensure that the mapping varies locally (L173).
    4. The paper frequently uses terminology that isn’t properly explained or defined, making the paper difficult to understand and follow. As this mostly happens in the introduction, it is difficult to understand what the limitations are of existing works, and why ReTax-Seg addresses these limitations. Some (non-exhaustive) examples: (i) L040: “coarse or policy-dependent class”, (ii) L045: “forward label supervision”, (iii) L046 & L068: “semantically recoverable”, (iv) L144: “dynamic compilation”.
    5. Fig. 2, which provides an overview of ReTax-Seg, is not very informative. The bottom panels show some information about the backbone, CLIP, and SAM models, but then the top panel feeds their outputs into a forward model that is visualized with a robot that looks like it’s thinking, which then produces some scores that are visualized with emojis. With this visualization, it is not clear how the model combines the information from the backbone, CLIP, and SAM models, whereas this is key in understanding the method.

Since it is not clear from the paper (i) what the paper’s objective is, (ii) how the proposed method works, (iii) how the method compares to existing methods, and (iv) whether the method truly performs better than alternatives, I give a ‘clear rejection’ rating,

[a] Xie et al., SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers, NeurIPS 2021.

[b] Zhou et al., LMSeg: Language-guided Multi-dataset Segmentation, ICLR 2023.

[c] Sun et al., DynAlign: Unsupervised Dynamic Taxonomy Alignment for Cross-Domain Segmentation, ICLR 2025.

[d] Kerssies et al., Your ViT is Secretly an Image Segmentation Model, CVPR 2025.

[e] Jain et al., OneFormer: One Transformer to Rule Universal Image Segmentation, CVPR 2023.

Questions for authors

I believe the paper needs substantial changes to be considered for acceptance at ICLR. To resolve the aforementioned critical weaknesses, the paper needs to be rewritten substantially, and many further experiments are required. These changes may be too substantial to be made properly in the brief rebuttal period.

Reviewer_y8w7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes ReTax-Seg for semantic segmentation across datasets with incompatible label taxonomies. A visual decoder predicts 39 semantic atoms, which are mapped into native dataset classes through spatially conditioned transport. The transport combines trainable compatibility relations, offline CLIP affinities, and visual features augmented by gated MobileSAM features. A reverse-transport cycle loss encourages reconstruction of the atom distribution. Experiments evaluate six independently trained models and one jointly trained model, together with cross-dataset transfer, component ablations, semantic diagnostics, and computational measurements.

Critical strengths

The framework addresses a relevant problem with a concrete formulation. The low-rank transport provides an implementable mechanism for combining heterogeneous taxonomies while learning from native masks without requiring atom-level annotations.

The reported segmentation results are competitive across diverse domains. Independent models average 73.63% mIoU, while the joint checkpoint reaches 74.05% and achieves the highest reported mIoU on all six datasets, including the additional comparators in Table 12.

The paper includes targeted diagnostics and computational measurements. Coarse–fine probes, atom-utilization audits, CLIP perturbations, and capacity studies directly examine the proposed mechanisms. Runtime and memory costs are also reported.

Critical weaknesses

The necessity of spatially varying taxonomy transport is insufficiently established. A fixed atom-to-class mapping still allows visual atom probabilities to vary across pixels; it does not inherently impose the fixed constituent mixture suggested in Section 3.5. Static aggregation already supports this distinction in NLL+. Although dynamic transport improves native segmentation in Table 3, Table 8 shows only a 0.07-point fine-probe gain over global learned transport, leaving the claimed fine-semantic advantage weakly supported.

Cycle consistency does not establish the claimed semantic preservation. A uniform atom posterior with identical image-conditioned transport columns can yield zero cycle loss despite indistinguishable atoms. This is an allowed degeneracy, rather than evidence that the reported models reach it. However, Table 9 also shows unchanged active-atom counts and dead-atom ratios, with negligible changes in perplexity and effective rank, providing little support for the anti-collapse interpretation.

The reported component effects are internally inconsistent. Removing cycle consistency gives 59.51 FMB mIoU in Table 3 but 60.34 in Table 9. Removing CLIP gives 78.45 AeroScapes mIoU in Table 3 but 80.30 in Table 10. These experiments are described as using matched settings, yet the discrepancies materially change the apparent contributions of the central components.

The presentation and figure placement hinder readability. Dense sentences and extensive notation make the method difficult to follow, while explanations are dispersed across the main text and appendices. Figure 2 appears after much of the method description, requiring readers to revisit earlier sections to connect the components. A clearer narrative, simpler wording, and figures placed closer to their corresponding explanations would improve readability.

Questions for authors

Why is dynamic transport necessary beyond a fixed mapping applied to spatially varying atom predictions, given the small gain in Table 8?

How is the uniform-atom degeneracy addressed, and how do Table 9’s results support the claimed anti-collapse benefit?

Why do the cycle and CLIP ablation results differ between Table 3 and Tables 9–10, and which values are authoritative?

Could you provide a concise explanation connecting the model components, equations, and Figure 2 to clarify the overall framework?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.