- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes UnfoldCRF, a structured mask refinement framework that formulates refinement as inference in a conditional random field with corrected unary potentials, learned pairwise interactions, and image-conditioned latent regions. By unrolling damped mean-field inference, the method provides an interpretable refinement process. Extensive experiments across multiple datasets and upstream segmentation models demonstrate consistent improvements in segmentation quality, robustness, and generalization over matched baselines. However, my primary concern is that the upstream segmentation models used in the main experiments are relatively outdated. Given the substantial advances in segmentation architectures and large-scale training, it remains unclear whether the reported improvements would persist with modern, more capable segmentation models.
Critical strengths
-
The experimental evaluation is relatively comprehensive, covering multiple datasets, segmentation scenarios, and upstream mask generators. Extensive comparisons, ablation studies, and generalization experiments provide empirical support for the proposed method, although the evaluation mainly focuses on binary segmentation with relatively outdated upstream models.
-
The proposed method is mathematically well-formulated, with a coherent energy-based framework and an interpretable structured inference process. The theoretical derivations are clear and provide a solid foundation for the proposed refinement mechanism.
Critical weaknesses
-
I should first clarify that I am not a researcher specializing in mask refinement. In my (a outsider) view, the upstream segmentation models used in the experiments are relatively outdated. For General Open Refinement, the authors consider models such as DeepLabV3 (2017) and FCN-R50 (2015), while Concealed Refinement relies on SINetV2 (2021) and FEDER (2023). While I do not question the importance of mask refinement, substantial advances in segmentation architectures and large-scale training have significantly improved the quality of initial segmentation masks. Many errors exhibited by these older models may already be mitigated by more recent segmentation models. Although evaluating established benchmarks is useful for comparison with prior work, it remains unclear whether the reported improvements would persist when applied to modern segmentation models, whose prediction quality and error distributions may differ considerably. This raises concerns about the practical relevance of the experimental results and the generalizability of the paper's conclusions to contemporary segmentation pipelines.
-
In terms of methodology, the proposed approach appears to be largely an integration of existing techniques rather than a fundamentally new mechanism. The key components of UnfoldCRF have precedents in prior works, as the authors themselves acknowledge in the Related Work section. While the combination of these components is technically reasonable and may yield empirical improvements, I find the methodological novelty somewhat limited. In particular, it is unclear what fundamentally new insights or capabilities the proposed formulation introduces beyond those already established in the literature.
-
The authors also acknowledge in the Limitations section that their experiments are restricted to binary segmentation, despite the formulation being designed to support multiple classes. I consider this a significant limitation, as multi-class segmentation is a common and practically important setting across a wide range of applications. Without experimental validation in such scenarios, it remains unclear whether the proposed structured refinement mechanism can effectively handle interactions among multiple semantic classes. This substantially limits the demonstrated applicability of the proposed method and weakens its claim as a general-purpose mask refinement framework.
Questions for authors
Could the authors provide a clearer and more intuitive explanation of which types of segmentation errors UnfoldCRF is particularly effective at correcting, which errors it can only partially address, and which remain challenging? Additionally, can UnfoldCRF introduce new errors in regions that were originally segmented correctly by the upstream model? While the paper provides some relevant analyses, a more systematic breakdown of different error types, supported by quantitative results and representative examples, would help clarify the strengths and limitations of the proposed method.
10 comments
Response to Reviewer YUTi [2/3]
W2. What is new, in plain terms
UnfoldCRF corrects an existing mask by letting soft groups of pixels exchange evidence about their labels. Our SLIC-based higher-order CRF control uses fixed superpixels. UnfoldCRF instead learns its groups from both the image and the initial mask, allowing each pixel to contribute to several nearby groups. A soft null state weakens a region's label preference when its label votes disagree, and normalization prevents the maximum group penalty from growing with group size. These ingredients have precedents. Our contribution is their specific combination with learned pixel–region associations and energy-derived updates, together with the controlled evidence for it. Against SPFU, which retains the same candidate connections but learns its updates, UnfoldCRF reduces Harm from 10.4% to 8.5% on COD10K with FEDER masks. In the leave-one-generator-out evaluation, mean held-out is 1.0 percentage point for SPFU and 2.0 percentage points for UnfoldCRF.
We compare an adapted higher-order CRF, propagation-matched free updates, and fixed versus learned associations on COD10K [8] with FEDER [9] masks. All controls use matched training data, resolution, depth, and supervision. Results are averaged over five independent training seeds for Attn-S, SPFU, and UnfoldCRF-S and three for the other controls. Held-out gain is mean (pp) under Fig. 4b's leave-one-generator-out protocol.
| Model | Params | ↑ | Harm (%) ↓ | Held-out gain (pp) ↑ |
|---|---|---|---|---|
| Attn-S (recurrent attention control) | 2.6M | .735 | 11.7 | 0.6 |
| Higher-order CRF (adapted Arnab et al. [6], multi-scale SLIC [7], learned pairwise) | 2.5M | .740 | 10.2 | 0.9 |
| Same-path free updates (SPFU) | 2.9M | .739 | 10.4 | 1.0 |
| Fixed soft associations, energy-derived updates | 2.5M | .738 | 10.8 | – |
| UnfoldCRF-S | 2.6M | .745 | 8.5 | 2.0 |
Compared with the adapted higher-order CRF and SPFU, UnfoldCRF-S improves by 0.5 and 0.6 pp and reduces Harm by 1.7 and 1.9 pp, respectively. Removing mass normalization or the null state reduces to .740/.737 and increases Harm to 10.1% in both cases.
W3. Multi-class refinement
The main submitted comparisons follow established foreground–background mask refinement protocols, allowing evaluation on the same coarse masks and metrics. Our formulation already supports labels. We now evaluate joint 19-class refinement using the same energy and mean-field update equations.
For Cityscapes [10], we set , with 19-channel upstream probability maps and unary corrections, a learned symmetric label-compatibility matrix, and 20 region states including null. All classes are updated jointly at each stage, and the final pixel label is selected by argmax. Only class-dependent network dimensions and the multi-class CE + Dice implementation are adapted.
We train one UnfoldCRF-S and one Attn-S checkpoint on Cityscapes train using Mask2Former-R50 [2] predictions. Both refiners operate jointly over 19 classes and are evaluated on the 500 validation images. We also apply the trained UnfoldCRF-S frozen to SegFormer-B0 [11] predictions. Before and after metrics use the released upstream checkpoints' single-scale predictions under the same evaluation protocol. Dataset mIoU uses the aggregate confusion matrix, Boundary F a 3-pixel tolerance, and Harm the percentage of images with decreased per-image mIoU over GT-present classes.
| Cityscapes val, 19 classes | mIoU ↑ | Boundary F ↑ | Harm (%) ↓ |
|---|---|---|---|
| Mask2Former-R50 (released checkpoint) | 79.4 | 62.3 | – |
| + Attn-S, joint | 79.7 | 63.4 | 14.6 |
| + UnfoldCRF-S, joint | 80.2 | 65.1 | 10.8 |
| SegFormer-B0 (released checkpoint), unseen upstream | 76.2 | 56.9 | – |
| + UnfoldCRF-S, frozen | 77.3 | 60.1 | 9.8 |
On Mask2Former-R50, UnfoldCRF-S improves mIoU by 0.8 pp and Boundary F by 2.8 points, with lower Harm than Attn-S. On unseen SegFormer-B0 outputs, it adds 1.1 mIoU pp and 3.2 Boundary F points without retraining. Per-class IoU gains include pole +2.1 pp, traffic light +1.8 pp, and traffic sign +1.5 pp, while road, sky, and building change by at most 0.1 pp.
Response to Reviewer YUTi [3/3]
Q. Which errors are corrected, which remain, and does refinement create new errors?
On COD10K with FEDER masks, UnfoldCRF corrects 40% of boundary errors and 33%/37% of non-boundary FN/FP errors, leaving 60%/67%/63% uncorrected. It harms 8.5% of images, with a median loss of 0.7 IoU pp among harmed images, while 0.9% lose more than 5 pp. Confidently wrong regions, missing object parts, and thin structures remain difficult (Fig. 6). Fig. 7 shows a harmful correction.
On COD10K [8] with FEDER [9] masks, correction, message-intervention, damage, and Harm rates are averaged over five training seeds. Loss summaries are computed among harmed images within each seed and then averaged. Severe-loss conditions use the single Fig. 5 checkpoint, with 18/2,026 images losing >5 IoU pp. Attn-S is the submitted recurrent attention control. Harm means any per-image IoU decrease.
Correction and message contributions. Wrong pixels are grouped into boundary, non-boundary false-negative (FN), and non-boundary false-positive (FP) errors (Fig. 4c). Correction rates are relative to initially wrong pixels in each group. Each message branch is zeroed at every inference stage without retraining, with paired correction-rate changes averaged over five seeds.
| Error type | Share of wrong pixels | Corrected by UnfoldCRF-S / Attn-S (%) | Change when messages are zeroed |
|---|---|---|---|
| Boundary | 44% | 40 / 31 | −12 pp without pairwise, −3 pp without region |
| Non-boundary FN | 33% | 33 / 24 | −11 pp without region, −2 pp without pairwise |
| Non-boundary FP | 23% | 37 / 31 | −10 pp without region, −2 pp without pairwise |
Pairwise messages contribute more to boundary correction, while region messages contribute more to non-boundary FN/FP correction.
Conditions of severe loss. The single-checkpoint post-hoc conditions use predictions and GT and may overlap. Rates are losses >5 IoU pp inside/outside each condition.
| Condition | Images | Severe rate in / out (%) |
|---|---|---|
| Mean upstream predicted-class confidence on wrong pixels ≥.9 | 417 | 2.9 / 0.4 |
| GT component entirely absent from initial mask | 236 | 2.5 / 0.7 |
| Local object width ≤4 px at >30% of skeleton sites | 163 | 2.5 / 0.8 |
| Object area <1% of image | 303 | 1.7 / 0.8 |
| Initial IoU <.5 | 412 | 2.2 / 0.6 |
Severe losses are more frequent with confidently wrong predictions, absent GT components, or thin structures in this checkpoint.
Damage to correct pixels. TP→FN and TN→FP rates use initially correct foreground and background pixels, respectively, within or outside the boundary band.
| Measure | UnfoldCRF-S | Attn-S |
|---|---|---|
| Boundary TP→FN / TN→FP (%) | 5.9 / 4.7 | 8.4 / 6.8 |
| Non-boundary TP→FN / TN→FP (%) | 0.2 / 0.1 | 0.4 / 0.2 |
| Images with any IoU loss / loss >5 pp (%) | 8.5 / 0.9 | 11.7 / 2.0 |
| Median / mean loss among harmed images (IoU pp) | 0.7 / 2.0 | 1.2 / 2.8 |
| Harm by initial IoU <.5 / .5–.7 / .7–.9 / ≥.9 (%) | 3.0 / 5.5 / 10.0 / 18.0 | 4.6 / 8.2 / 13.6 / 23.5 |
Any-decrease Harm rises with initial-mask quality. In the single-checkpoint analysis, severe loss occurs for 2.2% of images with initial IoU <.5 versus 0.6% otherwise. The quantitative analysis complements the successful refinements in Fig. 3 and the difficult cases in Fig. 6.
Multi-class error transitions. On Cityscapes [10] val, we compare the single W3 refiners on the same Mask2Former-R50 [2] predictions. We show the three classes highlighted in W3 and three high-IoU, large-area classes. Rates pool valid pixels across 500 images. For each GT class, Wrong→Correct is normalized by initially wrong pixels, and Correct→Wrong by initially correct pixels. A correction requires the final label to match GT.
| GT class (initial IoU, %) | Wrong→Correct, UnfoldCRF-S / Attn-S (%) | Correct→Wrong, UnfoldCRF-S / Attn-S (%) |
|---|---|---|
| Pole (66.4) | 24.3 / 15.8 | 3.1 / 4.0 |
| Traffic light (72.6) | 20.4 / 14.6 | 2.2 / 3.6 |
| Traffic sign (80.3) | 22.9 / 13.7 | 2.0 / 2.6 |
| Building (92.7) | 7.9 / 6.1 | 0.3 / 0.4 |
| Road (98.3) | 5.2 / 4.4 | 0.1 / 0.2 |
| Sky (95.2) | 6.6 / 4.1 | 0.1 / 0.1 |
The first three classes have higher correction and damage rates than the last three. Across the other 13 classes, UnfoldCRF-S correction rates are 5.9–23.4% and damage rates 0.2–2.4%. In this evaluation, UnfoldCRF-S has higher correction rates in all 19 classes and lower damage rates in 18. Both methods report 0.1% damage for sky at the shown precision.
References: [1] PromptMoE (Price et al., CVPR 2026). [2] Mask2Former (Cheng et al., CVPR 2022). [3] SAM3 (Carion et al., ICLR 2026). [4] DSS (Yang et al., CVPR 2026). [5] SAM3-Adapter (Chen et al., 2025). [6] Higher-order CRF (Arnab et al., ECCV 2016). [7] SLIC (Achanta et al., TPAMI 2012). [8] COD10K (Fan et al., CVPR 2020). [9] FEDER (He et al., CVPR 2023). [10] Cityscapes (Cordts et al., CVPR 2016). [11] SegFormer (Xie et al., NeurIPS 2021).
Response to Reviewer YUTi [1/3]
Manuscript revisions. Sec. 4.2 adds comparisons with recent mask generators and prompted SAM, Sec. 4.4 adds multi-class refinement and error-type analysis, and Secs. 1–2 clarify the contribution. Changes are marked in blue.
W1. Do the gains persist with modern segmentation models?
Yes. The submitted refiners improve predictions from recent generators without retraining on these upstreams. UnfoldCRF-S and Attn-S results are averaged over five independent training seeds per domain. Each checkpoint is applied frozen to additional mask sources excluded from refiner training. These tests complement Fig. 4b's leave-one-generator-out retraining, where the mean held-out ΔFβω is 2.0 pp for UnfoldCRF-S versus 0.6 pp for Attn-S.
General refinement. The five-benchmark suite follows PromptMoE [1] (CVPR 2026). The submitted results already include RMBG-1.4/2.0 and BiRefNet masks. We additionally evaluate Mask2Former [2]. SAM3 [3] uses automatic prompts derived from the image and initial mask without test GT (App. B). Harm is any IoU decrease relative to the input of each row. For SAM3, the input is the prompted SAM3 mask.
Input IoU and gains in the first two rows are from Table 1. Mask2Former provides a stronger closed-set segmentation upstream on VOC, where UnfoldCRF-S adds 2.1 IoU pp versus 0.9 pp for Attn-S. After SAM3 prompting, UnfoldCRF-S adds 1.2 pp versus 0.4 pp for Attn-S, bringing the total gain over the original masks to 5.3 pp.
Refinement of strong COD upstreams. On COD10K, the frozen COD-trained refiners are applied to prompted SAM3, DSS [4] (CVPR 2026), and COD-trained SAM3-Adapter [5] outputs from released implementations. Harm is measured against each upstream mask.
UnfoldCRF-S improves DSS and SAM3-Adapter predictions by 0.9 and 1.4 pp in Fβω, respectively, and achieves higher accuracy and lower Harm than Attn-S on every tested upstream. On SAM3-Adapter masks, boundary errors account for 63% of initially wrong pixels versus 44% on FEDER, and UnfoldCRF-S corrects 36% of these boundary errors.