- The experimental evaluation of this paper is insufficient to demonstrate the effectiveness of the method.
- The reported scores for existing methods are significantly lower than reported in the original papers. For example, Tab. 1 reports very low mIoU scores of 46.98 for SegFormer-B3 and 46.18 for SegFormer-B4 on Cityscapes, whereas the original SegFormer paper [a] reports mIoU scores of 81.7 and 82.3 on Cityscapes for B3 and B4, respectively. This calls into question the correctness of the results, and makes it unclear whether ReTax-Seg is truly better than these models. The differences could be (partly) explained by the fact that the paper’s experimental setup differs from prior work. For example, it uses its own train/val/test splits (L259) and uses only a low 512x512 resolution for training and inference (L267 & L318). It is not clear whether all the large performance gaps can be attributed to this. To properly demonstrate the effectiveness of the method, the paper should use the originally reported performance of existing methods, use a similar experimental setup for ReTax-Seg, and report the performance in this setup. Until this is done, the effectiveness of ReTax-Seg is not demonstrated properly.
- The paper does not experimentally compare ReTax-Seg to existing state-of-the-art methods for multi-dataset segmentation or taxonomy-alignment for segmentation, such as LMSeg [b] and DynAlign [c]. In Sec. 2 (L099-L102), the paper mentions these methods as relevant works, but argues that a disadvantage of these methods is that their alignment remains 'forward-only'. However, the paper does not demonstrate that ReTax-Seg actually outperforms these methods. As such, the value of ReTax-Seg over alternative, existing methods has not been properly demonstrated.
- In Tab. 1, ReTax-Seg is compared to several existing single-dataset methods for semantic segmentation, but most of these methods are outdated. The paper does not compare ReTax-Seg to current state-of-the-art transformer-based segmentation models like EoMT [d] or OneFormer [e]. As a result, it is not clear whether ReTax-Seg truly outperforms existing methods, which is what the table suggests. This makes it unclear what its value is.
- The benefit of training ReTax-Seg in a multi-dataset setting is very limited. While it appears that the main focus of the paper is to allow a segmentation model to learn effectively from multiple datasets with different class taxonomies, the average mIoU only improves from 73.63 to 74.05 from single-dataset to multi-dataset training. This minor improvement suggests that there is little benefit of multi-dataset training with the proposed framework, limiting its value.
- The evaluation is missing very simple baselines that are necessary to assess whether the use of the ‘semantic atoms’ and the dynamic mapping to the dataset-specific class labels is effective. A much simpler alternative would be to take the encoder’s output features, and have a single linear layer per dataset that directly maps the features to per-pixel classification scores for that dataset. ReTax-Seg should at least outperform this baseline to demonstrate its value, but such an experiment is currently not provided. Tab. 3 currently only experiments with different atom-to-class mappings, but does not consider the simpler aforementioned alternative.
- The paper’s understandability is poor. Some choices are not sufficiently motivated, and the paper frequently does not provide sufficient context or does not explain aspects of the method in sufficient detail. This makes it difficult to understand the paper’s contributions and it’s value compared to existing works.
- The paper does not specifically explain what the task objective is, neither in the introduction nor elsewhere. Is it to do single-dataset segmentation or multi-dataset segmentation? If it is the former, then why is it necessary to make a model compatible with different taxonomies? If it is the latter, then why is the model evaluated mostly in the single-dataset scenario, and why isn’t the model compared to alternative methods for multi-dataset segmentation? This is not clear from the paper.
- L050 and L116 specify that 39 ‘semantic atoms’ are used, but the paper does not explain why it uses the specific number of 39, or how these are defined.
- L144 explains that a learned visual branch extracts S=8 region factors, but the paper does not explain (i) what ‘region factors’ are and what they represent, or (ii) how the learned visual branch extracts these region factors. Since these region factors are used to obtain region states q (Eq. 4), it is also not clear what the region state represents. The same applies to the transport logits l (Eq. 5), which is a function of these region states. Similarly, the paper mentions that it uses dataset-class factors Ud and atom factors V (L165-L166), but it does not explain (i) what these factors are and what they represent, or (ii) how they are obtained. Because of all this, it is not clear how the MobileSAM features are used to create the mapping from the per-atom scores to the scores for dataset-specific classes, while this is a key aspect of this method as it is supposed to ensure that the mapping varies locally (L173).
- The paper frequently uses terminology that isn’t properly explained or defined, making the paper difficult to understand and follow. As this mostly happens in the introduction, it is difficult to understand what the limitations are of existing works, and why ReTax-Seg addresses these limitations. Some (non-exhaustive) examples: (i) L040: “coarse or policy-dependent class”, (ii) L045: “forward label supervision”, (iii) L046 & L068: “semantically recoverable”, (iv) L144: “dynamic compilation”.
- Fig. 2, which provides an overview of ReTax-Seg, is not very informative. The bottom panels show some information about the backbone, CLIP, and SAM models, but then the top panel feeds their outputs into a forward model that is visualized with a robot that looks like it’s thinking, which then produces some scores that are visualized with emojis. With this visualization, it is not clear how the model combines the information from the backbone, CLIP, and SAM models, whereas this is key in understanding the method.
Since it is not clear from the paper (i) what the paper’s objective is, (ii) how the proposed method works, (iii) how the method compares to existing methods, and (iv) whether the method truly performs better than alternatives, I give a ‘clear rejection’ rating,
[a] Xie et al., SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers, NeurIPS 2021.
[b] Zhou et al., LMSeg: Language-guided Multi-dataset Segmentation, ICLR 2023.
[c] Sun et al., DynAlign: Unsupervised Dynamic Taxonomy Alignment for Cross-Domain Segmentation, ICLR 2025.
[d] Kerssies et al., Your ViT is Secretly an Image Segmentation Model, CVPR 2025.
[e] Jain et al., OneFormer: One Transformer to Rule Universal Image Segmentation, CVPR 2023.