acceptodds
Under review as a conference paper at ICLR 2027

UnfoldCRF: Structured Mask Refinement with Image-Conditioned Latent Regions

Abstract

Learned mask refiners improve segmentation accuracy, but it is hard to tell how much of the improvement comes from explicit structure rather than from extra capacity, and whether it holds up when the mask generator or its error distribution changes. UnfoldCRF treats refinement as inference in a conditional random field over pixel labels and latent region variables. Its energy has a corrected unary term, learned local pairwise interactions, and image-conditioned latent-region consistency, with a null state that lets a region with weak label agreement withdraw from the consistency term; inference unrolls damped mean-field updates on this one energy. To isolate the effect of structure, we compare against recurrent black-box refiners that read the same inputs and receive the same parameter budget, stage count, and supervision. On COD10K, UnfoldCRF beats the strongest matched control by 1.0 point, improves all four COD metrics, and lowers the fraction of images made worse from 11.7% to 8.5%. Under a train-once protocol over five datasets and several mask sources, the 2.6M-parameter variant gains 4.2 mean IoU against 2.0 for its control, and a variant built on frozen DINOv2 features matches the strongest foundation-model refiner with about a seventh of its resident parameters while staying ahead of its own control. On mask generators never seen in training, the gain is 2.0 points against 0.6 for the control. Zeroing individual messages shows where the corrections come from: the pairwise messages mostly fix boundaries, the region messages mostly fix non-boundary errors. Code and supporting materials will be publicly released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Segmentation Refinement, Conditional Random Fields, Deep Unfolding, Mean-Field Inference, Structured Prediction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YUTiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes UnfoldCRF, a structured mask refinement framework that formulates refinement as inference in a conditional random field with corrected unary potentials, learned pairwise interactions, and image-conditioned latent regions. By unrolling damped mean-field inference, the method provides an interpretable refinement process. Extensive experiments across multiple datasets and upstream segmentation models demonstrate consistent improvements in segmentation quality, robustness, and generalization over matched baselines. However, my primary concern is that the upstream segmentation models used in the main experiments are relatively outdated. Given the substantial advances in segmentation architectures and large-scale training, it remains unclear whether the reported improvements would persist with modern, more capable segmentation models.

Critical strengths
  1. The experimental evaluation is relatively comprehensive, covering multiple datasets, segmentation scenarios, and upstream mask generators. Extensive comparisons, ablation studies, and generalization experiments provide empirical support for the proposed method, although the evaluation mainly focuses on binary segmentation with relatively outdated upstream models.

  2. The proposed method is mathematically well-formulated, with a coherent energy-based framework and an interpretable structured inference process. The theoretical derivations are clear and provide a solid foundation for the proposed refinement mechanism.

Critical weaknesses
  1. I should first clarify that I am not a researcher specializing in mask refinement. In my (a outsider) view, the upstream segmentation models used in the experiments are relatively outdated. For General Open Refinement, the authors consider models such as DeepLabV3 (2017) and FCN-R50 (2015), while Concealed Refinement relies on SINetV2 (2021) and FEDER (2023). While I do not question the importance of mask refinement, substantial advances in segmentation architectures and large-scale training have significantly improved the quality of initial segmentation masks. Many errors exhibited by these older models may already be mitigated by more recent segmentation models. Although evaluating established benchmarks is useful for comparison with prior work, it remains unclear whether the reported improvements would persist when applied to modern segmentation models, whose prediction quality and error distributions may differ considerably. This raises concerns about the practical relevance of the experimental results and the generalizability of the paper's conclusions to contemporary segmentation pipelines.

  2. In terms of methodology, the proposed approach appears to be largely an integration of existing techniques rather than a fundamentally new mechanism. The key components of UnfoldCRF have precedents in prior works, as the authors themselves acknowledge in the Related Work section. While the combination of these components is technically reasonable and may yield empirical improvements, I find the methodological novelty somewhat limited. In particular, it is unclear what fundamentally new insights or capabilities the proposed formulation introduces beyond those already established in the literature.

  3. The authors also acknowledge in the Limitations section that their experiments are restricted to binary segmentation, despite the formulation being designed to support multiple classes. I consider this a significant limitation, as multi-class segmentation is a common and practically important setting across a wide range of applications. Without experimental validation in such scenarios, it remains unclear whether the proposed structured refinement mechanism can effectively handle interactions among multiple semantic classes. This substantially limits the demonstrated applicability of the proposed method and weakens its claim as a general-purpose mask refinement framework.

Questions for authors

Could the authors provide a clearer and more intuitive explanation of which types of segmentation errors UnfoldCRF is particularly effective at correcting, which errors it can only partially address, and which remain challenging? Additionally, can UnfoldCRF introduce new errors in regions that were originally segmented correctly by the upstream model? While the paper provides some relevant analyses, a more systematic breakdown of different error types, supported by quantitative results and representative examples, would help clarify the strengths and limitations of the proposed method.

10 comments
AuthorsOpenReview ↗

Response to Reviewer YUTi [1/3]

Manuscript revisions. Sec. 4.2 adds comparisons with recent mask generators and prompted SAM, Sec. 4.4 adds multi-class refinement and error-type analysis, and Secs. 1–2 clarify the contribution. Changes are marked in blue.

W1. Do the gains persist with modern segmentation models?

Yes. The submitted refiners improve predictions from recent generators without retraining on these upstreams. UnfoldCRF-S and Attn-S results are averaged over five independent training seeds per domain. Each checkpoint is applied frozen to additional mask sources excluded from refiner training. These tests complement Fig. 4b's leave-one-generator-out retraining, where the mean held-out ΔFβω\Delta F_\beta^\omega is 2.0 pp for UnfoldCRF-S versus 0.6 pp for Attn-S.

General refinement. The five-benchmark suite follows PromptMoE [1] (CVPR 2026). The submitted results already include RMBG-1.4/2.0 and BiRefNet masks. We additionally evaluate Mask2Former [2]. SAM3 [3] uses automatic prompts derived from the image and initial mask without test GT (App. B). Harm is any IoU decrease relative to the input of each row. For SAM3, the input is the prompted SAM3 mask.

Benchmark, generatorInput IoU (%)ΔIoU UnfoldCRF-S / Attn-S (pp)Harm UnfoldCRF-S / Attn-S (%)
ECSSD, RMBG-1.4/2.0 and BiRefNet (2024)81.4+4.3 / +2.38.7 / 12.7
MSRA-B, RMBG-1.4/2.0 and BiRefNet (2024)75.2+4.0 / +2.18.3 / 11.9
VOC, Mask2Former Swin-L (2022, COCO→VOC mapping), new84.9+2.1 / +0.910.2 / 14.8
Five-benchmark suite, SAM3 prompted (2025), new80.4+1.2 / +0.47.1 / 11.2

Input IoU and gains in the first two rows are from Table 1. Mask2Former provides a stronger closed-set segmentation upstream on VOC, where UnfoldCRF-S adds 2.1 IoU pp versus 0.9 pp for Attn-S. After SAM3 prompting, UnfoldCRF-S adds 1.2 pp versus 0.4 pp for Attn-S, bringing the total gain over the original masks to 5.3 pp.

Refinement of strong COD upstreams. On COD10K, the frozen COD-trained refiners are applied to prompted SAM3, DSS [4] (CVPR 2026), and COD-trained SAM3-Adapter [5] outputs from released implementations. Harm is measured against each upstream mask.

COD10K inputInput FβωF_\beta^\omegaAfter UnfoldCRF-S / Attn-SHarm UnfoldCRF-S / Attn-S (%)
FEDER (reference).713.745 / .7358.5 / 11.7
SAM3 prompted from FEDER, new.728.746 / .7389.1 / 13.6
DSS, zero-shot SAM pipeline, new.849.858 / .85210.3 / 15.4
SAM3-Adapter, COD-trained, new.880.894 / .88512.6 / 18.3

UnfoldCRF-S improves DSS and SAM3-Adapter predictions by 0.9 and 1.4 pp in FβωF_\beta^\omega, respectively, and achieves higher accuracy and lower Harm than Attn-S on every tested upstream. On SAM3-Adapter masks, boundary errors account for 63% of initially wrong pixels versus 44% on FEDER, and UnfoldCRF-S corrects 36% of these boundary errors.

AuthorsOpenReview ↗

Response to Reviewer YUTi [2/3]

W2. What is new, in plain terms

UnfoldCRF corrects an existing mask by letting soft groups of pixels exchange evidence about their labels. Our SLIC-based higher-order CRF control uses fixed superpixels. UnfoldCRF instead learns its groups from both the image and the initial mask, allowing each pixel to contribute to several nearby groups. A soft null state weakens a region's label preference when its label votes disagree, and normalization prevents the maximum group penalty from growing with group size. These ingredients have precedents. Our contribution is their specific combination with learned pixel–region associations and energy-derived updates, together with the controlled evidence for it. Against SPFU, which retains the same candidate connections but learns its updates, UnfoldCRF reduces Harm from 10.4% to 8.5% on COD10K with FEDER masks. In the leave-one-generator-out evaluation, mean held-out ΔFβω\Delta F_\beta^\omega is 1.0 percentage point for SPFU and 2.0 percentage points for UnfoldCRF.

We compare an adapted higher-order CRF, propagation-matched free updates, and fixed versus learned associations on COD10K [8] with FEDER [9] masks. All controls use matched training data, resolution, depth, and supervision. Results are averaged over five independent training seeds for Attn-S, SPFU, and UnfoldCRF-S and three for the other controls. Held-out gain is mean ΔFβω\Delta F_\beta^\omega (pp) under Fig. 4b's leave-one-generator-out protocol.

ModelParamsFβωF_\beta^\omega ↑Harm (%) ↓Held-out gain (pp) ↑
Attn-S (recurrent attention control)2.6M.73511.70.6
Higher-order CRF (adapted Arnab et al. [6], multi-scale SLIC [7], learned pairwise)2.5M.74010.20.9
Same-path free updates (SPFU)2.9M.73910.41.0
Fixed soft associations, energy-derived updates2.5M.73810.8–
UnfoldCRF-S2.6M.7458.52.0

Compared with the adapted higher-order CRF and SPFU, UnfoldCRF-S improves FβωF_\beta^\omega by 0.5 and 0.6 pp and reduces Harm by 1.7 and 1.9 pp, respectively. Removing mass normalization or the null state reduces FβωF_\beta^\omega to .740/.737 and increases Harm to 10.1% in both cases.

W3. Multi-class refinement

The main submitted comparisons follow established foreground–background mask refinement protocols, allowing evaluation on the same coarse masks and metrics. Our formulation already supports LL labels. We now evaluate joint 19-class refinement using the same energy and mean-field update equations.

For Cityscapes [10], we set L=19L=19, with 19-channel upstream probability maps and unary corrections, a learned symmetric 19×1919\times19 label-compatibility matrix, and 20 region states including null. All classes are updated jointly at each stage, and the final pixel label is selected by argmax. Only class-dependent network dimensions and the multi-class CE + Dice implementation are adapted.

We train one UnfoldCRF-S and one Attn-S checkpoint on Cityscapes train using Mask2Former-R50 [2] predictions. Both refiners operate jointly over 19 classes and are evaluated on the 500 validation images. We also apply the trained UnfoldCRF-S frozen to SegFormer-B0 [11] predictions. Before and after metrics use the released upstream checkpoints' single-scale predictions under the same evaluation protocol. Dataset mIoU uses the aggregate confusion matrix, Boundary F a 3-pixel tolerance, and Harm the percentage of images with decreased per-image mIoU over GT-present classes.

Cityscapes val, 19 classesmIoU ↑Boundary F ↑Harm (%) ↓
Mask2Former-R50 (released checkpoint)79.462.3–
+ Attn-S, joint79.763.414.6
+ UnfoldCRF-S, joint80.265.110.8
SegFormer-B0 (released checkpoint), unseen upstream76.256.9–
+ UnfoldCRF-S, frozen77.360.19.8

On Mask2Former-R50, UnfoldCRF-S improves mIoU by 0.8 pp and Boundary F by 2.8 points, with lower Harm than Attn-S. On unseen SegFormer-B0 outputs, it adds 1.1 mIoU pp and 3.2 Boundary F points without retraining. Per-class IoU gains include pole +2.1 pp, traffic light +1.8 pp, and traffic sign +1.5 pp, while road, sky, and building change by at most 0.1 pp.

AuthorsOpenReview ↗

Response to Reviewer YUTi [3/3]

Q. Which errors are corrected, which remain, and does refinement create new errors?

On COD10K with FEDER masks, UnfoldCRF corrects 40% of boundary errors and 33%/37% of non-boundary FN/FP errors, leaving 60%/67%/63% uncorrected. It harms 8.5% of images, with a median loss of 0.7 IoU pp among harmed images, while 0.9% lose more than 5 pp. Confidently wrong regions, missing object parts, and thin structures remain difficult (Fig. 6). Fig. 7 shows a harmful correction.

On COD10K [8] with FEDER [9] masks, correction, message-intervention, damage, and Harm rates are averaged over five training seeds. Loss summaries are computed among harmed images within each seed and then averaged. Severe-loss conditions use the single Fig. 5 checkpoint, with 18/2,026 images losing >5 IoU pp. Attn-S is the submitted recurrent attention control. Harm means any per-image IoU decrease.

Correction and message contributions. Wrong pixels are grouped into boundary, non-boundary false-negative (FN), and non-boundary false-positive (FP) errors (Fig. 4c). Correction rates are relative to initially wrong pixels in each group. Each message branch is zeroed at every inference stage without retraining, with paired correction-rate changes averaged over five seeds.

Error typeShare of wrong pixelsCorrected by UnfoldCRF-S / Attn-S (%)Change when messages are zeroed
Boundary44%40 / 31−12 pp without pairwise, −3 pp without region
Non-boundary FN33%33 / 24−11 pp without region, −2 pp without pairwise
Non-boundary FP23%37 / 31−10 pp without region, −2 pp without pairwise

Pairwise messages contribute more to boundary correction, while region messages contribute more to non-boundary FN/FP correction.

Conditions of severe loss. The single-checkpoint post-hoc conditions use predictions and GT and may overlap. Rates are losses >5 IoU pp inside/outside each condition.

ConditionImagesSevere rate in / out (%)
Mean upstream predicted-class confidence on wrong pixels ≥.94172.9 / 0.4
GT component entirely absent from initial mask2362.5 / 0.7
Local object width ≤4 px at >30% of skeleton sites1632.5 / 0.8
Object area <1% of image3031.7 / 0.8
Initial IoU <.54122.2 / 0.6

Severe losses are more frequent with confidently wrong predictions, absent GT components, or thin structures in this checkpoint.

Damage to correct pixels. TP→FN and TN→FP rates use initially correct foreground and background pixels, respectively, within or outside the boundary band.

MeasureUnfoldCRF-SAttn-S
Boundary TP→FN / TN→FP (%)5.9 / 4.78.4 / 6.8
Non-boundary TP→FN / TN→FP (%)0.2 / 0.10.4 / 0.2
Images with any IoU loss / loss >5 pp (%)8.5 / 0.911.7 / 2.0
Median / mean loss among harmed images (IoU pp)0.7 / 2.01.2 / 2.8
Harm by initial IoU <.5 / .5–.7 / .7–.9 / ≥.9 (%)3.0 / 5.5 / 10.0 / 18.04.6 / 8.2 / 13.6 / 23.5

Any-decrease Harm rises with initial-mask quality. In the single-checkpoint analysis, severe loss occurs for 2.2% of images with initial IoU <.5 versus 0.6% otherwise. The quantitative analysis complements the successful refinements in Fig. 3 and the difficult cases in Fig. 6.

Multi-class error transitions. On Cityscapes [10] val, we compare the single W3 refiners on the same Mask2Former-R50 [2] predictions. We show the three classes highlighted in W3 and three high-IoU, large-area classes. Rates pool valid pixels across 500 images. For each GT class, Wrong→Correct is normalized by initially wrong pixels, and Correct→Wrong by initially correct pixels. A correction requires the final label to match GT.

GT class (initial IoU, %)Wrong→Correct, UnfoldCRF-S / Attn-S (%)Correct→Wrong, UnfoldCRF-S / Attn-S (%)
Pole (66.4)24.3 / 15.83.1 / 4.0
Traffic light (72.6)20.4 / 14.62.2 / 3.6
Traffic sign (80.3)22.9 / 13.72.0 / 2.6
Building (92.7)7.9 / 6.10.3 / 0.4
Road (98.3)5.2 / 4.40.1 / 0.2
Sky (95.2)6.6 / 4.10.1 / 0.1

The first three classes have higher correction and damage rates than the last three. Across the other 13 classes, UnfoldCRF-S correction rates are 5.9–23.4% and damage rates 0.2–2.4%. In this evaluation, UnfoldCRF-S has higher correction rates in all 19 classes and lower damage rates in 18. Both methods report 0.1% damage for sky at the shown precision.

References: [1] PromptMoE (Price et al., CVPR 2026). [2] Mask2Former (Cheng et al., CVPR 2022). [3] SAM3 (Carion et al., ICLR 2026). [4] DSS (Yang et al., CVPR 2026). [5] SAM3-Adapter (Chen et al., 2025). [6] Higher-order CRF (Arnab et al., ECCV 2016). [7] SLIC (Achanta et al., TPAMI 2012). [8] COD10K (Fan et al., CVPR 2020). [9] FEDER (He et al., CVPR 2023). [10] Cityscapes (Cordts et al., CVPR 2016). [11] SegFormer (Xie et al., NeurIPS 2021).

Reviewer_mkGGOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes UnfoldCRF, which refines an initial mask using the RGB image and mean-field updates on a learned CRF energy. The energy combines unary corrections, local pixel interactions, and consistency within learned regions. UnfoldCRF-S has 2.6M parameters, while UnfoldCRF-L uses frozen DINOv2 features with approximately 92M resident and 6M trainable parameters. Experiments for binary segmentation cover general object refinement with five benchmarks and separately trained concealed object refinement with four benchmarks.

Critical strengths
  • Despite advances in state-of-the-art interactive segmentation foundation models (e.g., SAM3), UnfoldCRF-S offers a compact option for domain-specific mask refinement. Specifically, it uses 2.6M parameters, requires only the RGB image and initial mask at inference time, and achieves results competitive with state-of-the-art refinement models on the general/camouflage benchmarks (Tables 1/2).

  • The controlled comparisons match the encoder architecture, parameter budget, refinement stages, and supervision. The main comparisons also report five training seeds and the percentage of masks whose IoU decreases after refinement (see Table 3).

Critical weaknesses
  • [W1] Outdated Task. The practical need for a separately trained mask refiner remains unclear when an initial mask can provide point/box/mask prompts to recent interactive segmentation models (e.g., SAM2/3 [A, E]). Table 1 already includes SAMRefiner/PromptMoE, but please demonstrate a clear advantage or a task-specific benefit over existing interactive or domain-adapted alternatives using the same initial masks (e.g., MedSAM [B] for medical applications). For concealed-object refinement, similar alternatives include click-based test-time adaptation (DC-TTA [F], evaluated on CAMO/COD10K), sketch-guided interaction [G], and training-free SAM pipelines (e.g., [H]).

  • [W2] Weak Novelty. Learning CRF potentials, unrolling inference, and enforcing higher-order region consistency are already established ideas and cited in the paper (e.g., CRF-as-RNN). Therefore, the current contribution needs to go beyond applying learnable CRF inference to mask refinement. Please clarify why the soft pixel-region assignments, mass normalization, null state, and shared-energy updates represent a conceptual advance over existing trainable higher-order CRFs, and validate this through matched comparisons. In Table 3, fixed-affinity/SLIC and recurrent controls support the design choices but do not demonstrate the advance over these learned structured alternatives.

  • [W3] Unclear Training/Inference Costs. Please report latency, peak memory, training cost, and the accuracy-efficiency trade-off across refinement stages. Additionally, the matched DINOv2 control isolates a structural gain, but it remains unclear whether this benefit holds with similar representations (e.g., DINOv3 [C], EUPE [D], or SAM image features [A, E]) at comparable encoder sizes.

References:

[A] SAM 3: Segment Anything with Concepts, ICLR 2026.

[B] Segment Anything in Medical Images, Nature Communications 2024.

[C] DINOv3, arXiv 2025.

[D] Efficient Universal Perception Encoder, arXiv 2026.

[E] SAM 2: Segment Anything in Images and Videos, ICLR 2025.

[F] DC-TTA: Divide-and-Conquer Framework for Test-Time Adaptation of Interactive Segmentation, ICCV 2025.

[G] Let Human Sketches Help: Empowering the Challenging Image Segmentation Task With Freehand Sketches, IEEE Transactions on Multimedia 2026.

[H] Discover, Segment, and Select: A Progressive Mechanism for Zero-shot Camouflaged Object Segmentation, CVPR 2026.

Questions for authors

Please refer to the Weaknesses section for details.

  • Q1. [W1] What evidence demonstrates the practical necessity of a separately trained mask refiner over prompting modern interactive or domain-adapted models (SAM2/3 [A, E] & MedSAM [B]) with the initial mask? For concealed-object refinement, how does UnfoldCRF compare against interactive test-time adaptation, sketch-guided, and training-free SAM alternatives such as DC-TTA [F], sketch-guided interaction [G], and progressive SAM pipelines [H]?

  • Q2. [W2] Given established formulations of learnable CRF unrolling and higher-order consistency (e.g., CRF-as-RNN), what conceptual advances do soft pixel-region assignments, mass normalization, null states, and shared-energy updates provide over existing trainable higher-order CRFs? Under matched conditions, how does the proposed method quantitatively compare against these learned structured alternatives beyond the fixed-affinity and SLIC baselines in Table 3?

  • Q3. [W3] What are the exact latency, peak memory, training costs, and stage-wise accuracy-efficiency trade-offs of UnfoldCRF? Furthermore, does the structural gain observed with DINOv2 hold when evaluating alternative image encoders such as DINOv3 [C], EUPE [D], or SAM image features [A, E] at comparable encoder sizes?

13 comments
AuthorsOpenReview ↗

Response to Reviewer mkGG [1/4]

Manuscript revisions. Secs. 1–2 and 4.1–4.4 clarify the contribution and add SAM comparisons, structured controls, downstream tests, and measured costs. Changes are marked in blue.

W1 / Q1. Value alongside foundation models (1/2)

A separately trained refiner adds value in three measured ways. On the same initial masks, UnfoldCRF-S is comparable to automatic SAM3 prompting in mean ΔIoU\Delta\mathrm{IoU} at one-thirteenth of the latency and with lower Harm. On concealed objects it is more accurate than prompted SAM3 and harms fewer images. It also improves the outputs of prompted SAM3, DSS, and SAM3-Adapter. Part 2 examines complementarity and downstream supervision.

Automatic prompting. We follow PromptMoE's [1] five-benchmark evaluation protocol and compare SAM2 [E], SAM3 [A], and UnfoldCRF on the same initial masks. For SAM2/3, we use a fixed automatic prompting rule with box, positive/negative point, and mask prompts derived from (I,Pinit)(I,P^{\mathrm{init}}). Prompt settings were selected once using VOC training labels, the same labels used to train the refiners, and kept fixed across benchmarks. Multimask outputs are selected by the model's predicted IoU, and no test labels are used (App. B). The SAM2/3 results are our own evaluations under this prompting rule.

Five-benchmark suiteResidentMedian msΔIoU/ΔBIoU (pp)↑Harm (%)↓
SAM2 prompted224M185+3.9/+6.215.1
SAM3 prompted0.85B260+4.1/+6.513.6
UnfoldCRF-S2.6M20+4.2/+6.28.9
UnfoldCRF-L (frozen DINOv2)92M50+6.3/+9.17.4
SAM2 → UnfoldCRF-S224M + 2.6M–+4.6/+7.19.6
SAM3 → UnfoldCRF-S0.85B + 2.6M–+5.3/+8.08.8

Harm is any IoU decrease against the initial mask. Latencies exclude upstream mask generation. The results support low-cost independent correction and further refinement after SAM prompting.

For reference, the published SAMRefiner and PromptMoE results in Table 1 are +5.4/+8.3 and +6.2/+9.0 with standard SAM ViT-H at 641M or more resident parameters. UnfoldCRF-S is below these, and UnfoldCRF-L (+6.3/+9.1, 4482448^2 input) is comparable to standard PromptMoE, while the SAM-HQ configurations in PromptMoE's supplement (+6.83/+9.25 for SAMRefiner-HQ and +7.20/+9.74 for PromptMoE-HQ) remain higher.

Concealed-object refinement. COD10K [6] metrics are listed in the order MAE (MM)↓ / FβωF_\beta^\omega↑ / EϕE_\phi↑ / SαS_\alpha↑. Harm is relative to FEDER [5] for the first group and to each listed upstream for the others.

COD10KMM / FβωF_\beta^\omega / EϕE_\phi / SαS_\alphaHarm (%)↓
FEDER initial mask.032/.713/.899/.822–
SAM3 prompted from FEDER.037/.728/.895/.82122.0
UnfoldCRF-S on FEDER (five seeds).029/.745/.910/.8348.5
SAM3 → UnfoldCRF-S.029/.746/.910/.8349.3 (9.1 vs SAM3)
DSS [H], zero-shot upstream.022/.849/.942/.887–
DSS → UnfoldCRF-S.021/.858/.950/.89210.3
SAM3-Adapter [3], COD-trained upstream.015/.880/.964/.926–
SAM3-Adapter → UnfoldCRF-S.014/.894/.971/.93112.6

The same frozen COD-trained UnfoldCRF-S refines outputs from the released DSS [H] and SAM3-Adapter implementations. Paired comparisons show improvements across all four metrics, including on strong upstream predictions. On COD10K, the SAM3 cascade improves the prompted masks from .728 to .746 but shows no clear advantage over direct UnfoldCRF-S (.745), with higher Harm relative to FEDER (9.3% vs. 8.5%). On the five-benchmark suite, the cascade improves mean ΔIoU\Delta\mathrm{IoU} by 1.1 pp over direct UnfoldCRF-S.

DC-TTA [F] uses interactive correction clicks, while sketch-guided segmentation [G] requires human contours. Their test-time input requirements differ from our automatic refinement setting and are discussed in Sec. 2. MedSAM [B] is evaluated in Part 2.

AuthorsOpenReview ↗

Response to Reviewer mkGG [2/4]

W1 / Q1. Complementarity and downstream utility (2/2)

Correction differences. On the same 2,026 COD10K images with FEDER masks, we compare fixed SAM3 predictions with five independently trained UnfoldCRF-S models. Improvement means ΔIoU>0.5\Delta\mathrm{IoU}>0.5 pp, while Harm counts any decrease. Statistics are averaged across UnfoldCRF seeds.

StatisticSAM3UnfoldCRF-S
Images improved / Harm (%)52 / 22.071 / 8.5
Improved only by this method (both: 40%)1231

Pseudo-label training. We follow SEE's [4] 1/16 labeled-data split on CAMO [7] + COD10K [6]. A teacher trained on the labeled subset generates P0P_0, and UnfoldCRF learns from out-of-fold predictions on labeled images only. The refiner is initialized from the VOC train-once checkpoint, and the 1/16 budget refers to target-domain labels. SAM3 is used without COD adaptation, and SEE pseudo-labels are generated by rerunning its released pipeline under the same split. All pseudo-label variants train the same FEDER [5] student architecture with matched training settings and no unlabeled GT. Results are evaluated on COD10K test, with one student training run per configuration. The SEE rows compare exported pseudo-labels under the common FEDER student, not the full SEE system. Full-label FEDER is a separate reference.

Student supervisionStudent MM / FβωF_\beta^\omega / EϕE_\phi / SαS_\alpha
1/16 labels only.064/.608/.845/.768
Original P0P_0.060/.622/.851/.773
UnfoldCRF R(P0)R(P_0).056/.637/.858/.778
SAM3 G(P0)G(P_0).048/.675/.866/.786
SAM3 → UnfoldCRF.044/.690/.876/.798
SEE pseudo-labels.047/.681/.870/.793
SEE → UnfoldCRF.045/.687/.878/.799
Full-label FEDER (reference).032/.713/.899/.822

UnfoldCRF improves student training from raw pseudo-labels and provides further gains when applied to SAM3- and SEE-generated labels. Corrections are performed offline without increasing student inference cost. Data split and student-training settings are in App. C.8.

Medical refinement. We use the released Kvasir-SEG [8] 880/120 train/validation split, reserving 120 images for evaluation. A single UnfoldCRF-S, initialized from the VOC train-once checkpoint, is trained using out-of-fold U-Net [9] predictions from the 880 training images. It is then applied frozen to U-Net, MedSAM [B], and SAM2/3-Adapter [2, 3] outputs. MedSAM uses boxes derived from U-Net masks, while both Adapters are fine-tuned on the same training split without using evaluation images or labels.

Kvasir-SEG inputmDice / mIoU before → afterHarm (%)
U-Net.832/.755 → .851/.7789.2
MedSAM (U-Net-derived box).846/.771 → .852/.77911.7
SAM2-Adapter.871/.803 → .879/.81210.8
SAM3-Adapter.904/.839 → .909/.84512.5

Harm is any Dice decrease relative to each upstream mask. On the same U-Net initial masks, UnfoldCRF-S (.851/.778, 9.2% Harm) is above MedSAM prompted with U-Net-derived boxes (.846/.771, 19.2% Harm relative to U-Net). Refining the MedSAM output lowers that Harm to 15.0%. The same task-trained refiner raises mDice and mIoU on all four upstream predictions without further training.

AuthorsOpenReview ↗

Response to Reviewer mkGG [3/4]

W2 / Q2. Relation to trainable higher-order CRFs

Learned CRF potentials, unrolled mean-field inference, auxiliary region states, and higher-order consistency have precedents (CRF-as-RNN [11], Arnab et al., 2016 [10]). UnfoldCRF combines image-and-mask-conditioned soft pixel–region associations with a mass-normalized consistency energy over semantic and null region states. Associations determine where evidence is exchanged, and the energy-derived updates determine how it enters the mask. The null state provides a soft label-neutral option (revised Secs. 1–2).

We evaluate trainable CRFs and propagation-matched free-update controls. Accuracy and Harm use COD10K [6] with FEDER [5] inputs. Held-out gain is mean ΔFβω\Delta F_\beta^\omega (pp) under leave-one-generator-out retraining. Controls share the corrected-unary formulation, input resolution, depth, data, and supervision, but are trained independently. Results are seed means (five for submitted models and the learned-association SPFU/FRAP/AMFU, three for the other new controls). Additional results and summaries of seed variability are in App. C.1. Paired confidence intervals for selected comparisons are in Table S5.

ModelParamsFβωF_\beta^\omega ↑Harm (%) ↓Held-out gain (pp) ↑
Attn-S (submitted attention control)2.6M.73511.70.6
Dense Gaussian CRF [12] (learned kernel weights and compatibility)2.4M.73312.40.4
Higher-order CRF A (fixed RGB–XY pairwise, multi-scale SLIC [13])2.4M.73910.50.8
Higher-order CRF B (learned pairwise, multi-scale SLIC)2.5M.74010.20.9
SPFU (free region and pixel updates)2.9M.73910.41.0
FRAP (free region update)2.8M.7419.91.2
AMFU (free pixel update)2.7M.7429.41.5
UnfoldCRF-S2.6M.7458.52.0

A/B adapt Arnab et al.'s trainable superpixel potential and mean-field update to three SLIC granularities, omitting detector-dependent potentials. B retains our learned pairwise term. SPFU preserves UnfoldCRF's encoder and candidate propagation paths but learns both updates. FRAP/AMFU replace one update each, without reducing encoder capacity or adding spatial mixing. These are our adapted runs, not published scores. The FβωF_\beta^\omega difference against SPFU is 0.6 pp (95% CI [0.3, 0.9]). Implementation details are in App. B.

Component ablations. Entries are FβωF_\beta^\omega / Harm (%). Fixed-association variants omit the association head but retain the encoder, so parameter counts differ.

Update ruleFixed soft associationsLearned soft associations
Free (SPFU).736 / 11.2 (2.8M).739 / 10.4 (2.9M)
Energy-derived.738 / 10.8 (2.5M).745 / 8.5 (2.6M)
IngredientVariantFβωF_\beta^\omega / Harm
Association learning and softnessfixed-hard / fixed-soft / learned-hard (straight-through) / learned-soft.736/11.4 · .738/10.8 · .741/9.6 · .745/8.5
Mask conditioningimage-only A(I)A(I), mask-conditioned unary/pairwise.741/9.7
Mass normalizationremoved, updates rederived and retrained (cap βrmkκr\beta_rm_k\kappa_r, vote threshold 1−κr1-\kappa_r unchanged).740/10.1
Null stateremoved.737/10.1
Dampinglearned per stage / fixed / none (free-energy increases in 4/7/19% of images during five trained stages, seed means).745/8.5 · .743/9.0 · .742/9.5

The main comparisons appear in revised Sec. 4.3 and Table 4.

Reviewer_gg1WOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes UnfoldCRF, a structured mask refinement framework formulated as conditional random field (CRF) inference over pixel labels and latent region labels. Its energy function combines unary terms incorporating upstream predictions and learned residuals, symmetric local pairwise interactions, and image-conditioned soft region consistency with an optional null state. The model performs multi-step damped mean-field inference over a shared energy function.

The method is evaluated on five benchmarks, including camouflaged object refinement and leave-one-generator-out testing. Comparisons with parameter-matched convolutional and attention-based recurrent refiners, together with message ablations, are used to assess the benefits of explicit structured interactions.

Critical strengths
  1. The paper addresses whether explicit label interactions provide benefits beyond conventional recurrent refinement under approximately matched architectures, parameter budgets, and supervision. The shared encoders and unary prediction components reduce important confounding factors, making the comparison more informative than a simple performance comparison between unrelated architectures.
  2. The null state provides a principled mechanism for relaxing region consistency when a region overlaps conflicting semantic labels, potentially preventing incorrect corrections near object boundaries. The correspondence between the CRF energy and mean-field updates is clearly formulated. Furthermore, the message ablations in Figure 4c provide useful evidence that pairwise and region-level interactions play complementary roles.
Critical weaknesses
  1. Although the baselines are approximately parameter-matched, they are not matched in spatial information propagation. UnfoldCRF employs multi-scale dilated pairwise interactions and multi-granularity region aggregation, whereas the attention baseline uses local 8×8 windows and the convolutional baseline uses three 3×3 convolutional layers. Consequently, the observed improvements may arise from broader contextual aggregation rather than energy-based structured inference itself. The message-zeroing ablations do not resolve this confounding factor, substantially weakening the paper's central empirical claim.
  2. Shared-energy unrolled inference and differentiable higher-order CRF potentials have established precedents, including CRF-as-RNN and earlier higher-order CRF models (Higher Order CRFs, ECCV 2016, https://arxiv.org/pdf/1511.08119.pdf). The potentially novel contributions lie primarily in the learned soft region associations and specific region energy formulation. However, the experiments do not adequately isolate the effects of soft associations, mass normalization, the null state, and damping. The fixed-SLIC comparison simultaneously changes association learnability and assignment softness. Consequently, the evidence is insufficient to establish which specific methodological innovations account for the reported improvements.
Questions for authors
  1. Do the convolutional and attention-based baselines have access to spatial context and multi-scale label aggregation comparable to UnfoldCRF? If not, how can the reported improvements be attributed specifically to energy-based structured inference rather than differences in information propagation?
  2. What precisely distinguishes the proposed formulation from existing differentiable higher-order CRFs with shared-parameter mean-field inference? Which controlled experiments independently establish the contributions of learned soft associations, mass normalization, the null state, and damping?
  3. Figure 5 suggests that additional inference steps can improve convergence toward an energy fixed point while degrading segmentation performance. How do the authors reconcile this observation with the proposed energy-based interpretation? Does the evidence primarily support supervised finite-step refinement rather than the reliability of the learned energy itself?
  4. Can the authors report end-to-end inference latency and peak GPU memory under identical hardware and input resolutions, particularly against the matched attention baseline? How does the claimed parameter efficiency translate into practical inference efficiency?
  5. For the images classified as Harm in Table 3, what is the distribution of IoU degradation? Are severe failures concentrated in particular conditions, such as confidently incorrect initial masks, missing objects, or fine structures? This information is important for assessing whether the refinement process introduces unacceptable failure cases.
5 comments
AuthorsOpenReview ↗

Response to Reviewer gg1W [1/3]

Manuscript revisions. Secs. 1–2 clarify the contribution, Secs. 4.1–4.3 add structured controls, and Secs. 4.4–5 update inference, cost, and reliability analyses. Changes are marked in blue.

W1 / Q1. Information Propagation versus Energy-Derived Updates

The submitted attention control (Attn-S) matches encoder architecture, parameter budget, depth, and supervision, but uses 8×88\times8 windows rather than UnfoldCRF's dilated interactions and multi-scale regions. We add stronger attention and propagation-matched controls.

Controls. We add unary reinjection and damping to attention, and test region-aware attention at standard and doubled budgets. Same-Path Free Update (SPFU) keeps UnfoldCRF's neighborhoods, region grids, candidate pixel–region paths, mass-normalized aggregation, and backprojection, but replaces both analytic updates with learned functions. Free Region, Analytic Pixel (FRAP) replaces the region-state update, while Analytic Messages, Free Pixel Update (AMFU) replaces the pixel update.

SPFU, FRAP, AMFU, and UnfoldCRF share the full front-end architecture, feature dimensions, message interfaces (including null), initialization, depth, supervision, and damping. Free functions act per region or pixel without extra spatial mixing and share weights across stages. Input-conditioned quantities remain fixed during inference. Free-update models have 2.7–2.9M parameters rather than a reduced encoder. Independent training may change learned affinities and active regions, so same-path refers to candidate propagation, not identical realized messages (App. B).

Results. Accuracy and Harm use COD10K [6] with FEDER [5] upstream. Held-out gain is mean ΔFβω\Delta F_\beta^\omega (pp) from Fig. 4b's leave-one-generator-out protocol, with each model retrained without the tested generator.

ModelParamsFβωF_\beta^\omega ↑Harm (%) ↓Held-out gain (pp) ↑
Attn-S (submitted)2.6M.735 ± .00311.7 ± .5.6
Attn-S + unary + damping2.6M.736 ± .00311.3 ± .5.7
Region-aware attention2.6M.737 ± .00310.9 ± .5.8
Region-aware attention, 2× params5.2M.738 ± .00310.7 ± .5.8
SPFU2.9M.739 ± .00310.4 ± .51.0
FRAP2.8M.741 ± .0039.9 ± .51.2
AMFU2.7M.742 ± .0029.4 ± .41.5
UnfoldCRF-S2.6M.745 ± .0028.5 ± .42.0

± denotes training-seed SD, with five seeds for submitted models and SPFU/FRAP/AMFU, and three for new attention controls.

Read against Attn-S, SPFU closes less than half of the gap to UnfoldCRF-S in FβωF_\beta^\omega, Harm, and held-out gain. Relative to SPFU, UnfoldCRF gains 0.6 pp in FβωF_\beta^\omega (95% CI [0.3, 0.9]), lowers Harm by 1.9 pp (difference CI [−2.6, −1.2]), and improves held-out gain by 1.0 pp (CI [0.5, 1.5]). The 0.3 pp accuracy difference versus AMFU has CI [0.0, 0.6]. A frozen-front-end SPFU diagnostic obtains .740 FβωF_\beta^\omega / 9.9% Harm (App. C.1).

On the five-benchmark train-once suite [8], mean ΔIoU\Delta\mathrm{IoU} is +2.0/+3.3/+3.7/+4.2 pp for Attn-S/SPFU/AMFU/UnfoldCRF-S. On the same frozen DINOv2 [7] backbone, SPFU-L gains +5.4 versus +6.3 pp for UnfoldCRF-L. Together, these comparisons support energy-derived updates beyond matched candidate propagation paths.

AuthorsOpenReview ↗

Response to Reviewer gg1W [2/3]

W2 / Q2. Existing CRFs and Component Contributions

Unrolled mean-field inference, auxiliary region states, and trainable higher-order CRFs have precedents [1, 2]. UnfoldCRF combines image-and-mask-conditioned soft pixel–region associations with mass-normalized consistency over semantic and null region states. Its updates turn normalized votes into region posteriors under an input-conditioned energy fixed across stages (revised Secs. 1–2).

All results use COD10K [6] with FEDER [5] inputs. Controls share the corrected-unary formulation and architecture, input resolution, depth, supervision, and training data, but are trained independently. Entries are seed means. Submitted models and the learned-association SPFU/FRAP/AMFU controls use five training seeds, and the other new controls use three. Additional results and summaries of seed variability are in App. C.1. Paired confidence intervals for selected comparisons are in Table S5.

Trainable CRF controls.

ModelFβωF_\beta^\omega ↑Harm (%) ↓
Dense Gaussian CRF [3] (learned kernel weights and compatibility).73312.4
Higher-order CRF A (fixed pairwise, multi-scale SLIC [4]).73910.5
Higher-order CRF B (learned pairwise, multi-scale SLIC).74010.2
UnfoldCRF-S.7458.5

A/B adapt the superpixel potential and mean-field update of Arnab et al. [1] using SLIC [4] at the three region-grid granularities, each with a trainable potential. Detector-dependent terms are omitted because the protocol supplies no detector output. B retains our learned pairwise branch, comparing regional modeling strategies. All scores are our adapted runs, not published results (App. B).

Association and update controls. Fixed association variants use RGB–XY scores and omit the association head while retaining the encoder. Learned hard uses straight-through estimation. The following cross association choice with update formulation (entries: FβωF_\beta^\omega / Harm, parameter count):

UpdateFixed soft associationsLearned soft associations
Free (SPFU).736 / 11.2% (2.8M).739 / 10.4% (2.9M)
Energy-derived.738 / 10.8% (2.5M).745 / 8.5% (2.6M)

Both association learning and energy-derived updates improve the observed scores in either setting. The parameter counts are shown rather than forced to equality.

Other component ablations. Entries are FβωF_\beta^\omega / Harm (%).

ComponentVariantResult
Association softness and learningfixed hard / fixed soft / learned hard / learned soft.736/11.4 · .738/10.8 · .741/9.6 · .745/8.5
Mask conditioningimage-only associations, with mask-conditioned unary/pairwise.741/9.7
Mass normalizationremoved, energy and updates rederived, retrained.740/10.1
Null stateremoved / removed with normalization.737/10.1 · .733/11.4
Dampinglearned per stage / fixed / none.745/8.5 · .743/9.0 · .742/9.5

Without mass normalization, the truncation cap changes from βrκr\beta_r\kappa_r to βrmkκr\beta_r m_k\kappa_r and the semantic/null vote threshold remains 1−κr1-\kappa_r. Learned/fixed/no damping produce free-energy increases on 4/7/19% of images during the five trained stages. Revised Sec. 4.3 reports the main comparisons.

AuthorsOpenReview ↗

Response to Reviewer gg1W [3/3]

Q3. Free Energy versus Task Accuracy

The energy defines the region and pixel updates, while Eq. (13) trains finite-step predictions. Proposition 1 describes the region penalty, and Lemma 1 shows that damping preserves interior fixed points. Neither implies convergence, monotone free-energy decrease, or better segmentation at a fixed point.

For the single Fig. 5 checkpoint trained at T=5T=5, FβωF_\beta^\omega is .744/.741/.739 at t=5/10/20t=5/10/20, reusing the final damping coefficient beyond TT (Table 4's five-seed mean at t=5t=5 is .745).

Between t=5t=5 and 20 over 2,026 images, free energy ↓ / IoU ↑ occurs in 36%, both ↓ in 45%, both ↑ in 5%, and energy ↑ / IoU ↓ in 8%. In the remaining 6%, at least one change is below tolerance (IoU 0.05 pp, free energy 10−410^{-4}, normalization in App. C.6). For one independently trained checkpoint at each depth T=3/5/10T=3/5/10, the observed peaks occur at stages 3/5/9 with FβωF_\beta^\omega .739/.744/.746 (seed means at the trained depth are in Table S10). Extending Attn-S/SPFU/UnfoldCRF-S from trained depth to step 20 lowers accuracy by 1.1/0.7/0.5 pp under their continuation rules (App. C.6). These diagnostics support finite-step supervised inference rather than guaranteed accuracy improvement from continued energy reduction.

Q4. Latency, Memory, and Training Cost

All models are profiled on one RTX PRO 6000 (FP16, batch 1), with image and mask on GPU. Synchronized timing after 20 warm-ups includes refinement and resizing to original resolution, excluding loading and host transfers. Memory is peak allocated GPU memory. The profiled S and L configurations use 40k updates per training run, with batch sizes 16 and 8, respectively.

ModelResident/trainableInputMedian msPeak GBGPU-h/run
Attn-S2.6M/2.6M3522352^217.54
SPFU/FRAP/AMFU-S2.7–2.9M (all trainable)3522352^219–21.65
UnfoldCRF-S2.6M/2.6M3522352^220.65
Attn-L92M/6M4482448^2462.011
UnfoldCRF-L92M/6M4482448^2502.212

At the same parameter count and input resolution, the accuracy and Harm improvement of UnfoldCRF-S over Attn-S costs an additional 3 ms and 0.1 GB of peak memory. For scale, automatic SAM2/SAM3 prompting on the same hardware takes 185/260 ms per image at the default input resolutions, about 9 to 13 times the latency of UnfoldCRF-S.

Q5. Severity and Conditions of Harmful Corrections

Define Di=100max⁡0,IoUiinit−IoUirefD_i=100\max\\{0,\mathrm{IoU}_i^{\mathrm{init}}-\mathrm{IoU}_i^{\mathrm{ref}}\\} and Hτ=Pr⁡(Di>τ)H_\tau=\Pr(D_i>\tau). Seed-aggregated rates use all 2,026 COD10K [6] test images, while loss summaries are computed among harmed images within each seed and then averaged.

MeasureUnfoldCRF-SAttn-SSPFU
H0/H1/H5/H10H_0/H_1/H_5/H_{10} (%)8.5/3.9/0.9/0.211.7/6.2/2.0/0.610.4/5.1/1.5/0.4
Loss among harmed: median/mean/95th pct (IoU pp)0.7/2.0/7.51.2/2.8/10.21.0/2.4/8.9

Harm increases with initial-mask quality, but stays below Attn-S in each initial-IoU bin. For initial IoU <.5/.5–.7/.7–.9/≥.9, UnfoldCRF's mean IoU gains are +9.8/+5.1/+2.6/+0.6 pp and Harm rates are 3.0/5.5/10.0/18.0% (App. C.5).

Severe-failure conditions. The rates below use the single Fig. 5 checkpoint with 18 severe images, rather than pooling images across seeds. Conditions overlap. Entries compare Pr⁡(Di>5∣C)\Pr(D_i>5\mid C) with Pr⁡(Di>5∣¬C)\Pr(D_i>5\mid\neg C).

ConditionImages with CSevere with CSevere rate C / not C
Mean wrong-pixel predicted-class confidence ≥.9417122.9% / .4%
GT component absent from initial mask23662.5% / .7%
Local object width ≤4 px at >30% of skeleton sites16342.5% / .8%
Object area <1% of image30351.7% / .8%
Initial IoU <.541292.2% / .6%

FEDER [5] confidence comes from probability maps. With 18 severe cases, these conditional rates describe associations rather than causes.

Corrected and newly introduced pixel errors. Each triplet is UnfoldCRF-S/Attn-S/SPFU (%). Correction and damage use different denominators.

RegionFN→TPFP→TNTP→FNTN→FP
Boundary41/31/3638/31/355.9/8.4/7.04.7/6.8/5.6
Non-boundary33/24/2937/31/34.2/.4/.3.1/.2/.2

References (all three comments): [1] Higher-order CRF (Arnab et al., ECCV 2016). [2] CRF-as-RNN (Zheng et al., ICCV 2015). [3] Dense CRF (Krähenbühl and Koltun, NeurIPS 2011). [4] SLIC (Achanta et al., TPAMI 2012). [5] FEDER (He et al., CVPR 2023). [6] COD10K (Fan et al., CVPR 2020). [7] DINOv2 (Oquab et al., 2023). [8] PromptMoE (Price et al., CVPR 2026).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers (gg1W, mkGG, and YUTi) for their thoughtful and constructive feedback, which has helped us strengthen our work. We are encouraged that the reviewers recognize the controlled comparisons under matched architectures, parameter budgets, and supervision (gg1W, mkGG), the well-formulated energy-based framework and interpretable structured inference (gg1W, YUTi), the principled null-state mechanism and the complementary roles of pairwise and region messages (gg1W), the compact design with competitive refinement results (mkGG), and the comprehensive evaluation across datasets and upstream mask generators (YUTi).

In response to the reviewers' valuable comments, we have carefully addressed each concern in our individual responses and summarize the main revisions below:

  • We added propagation-matched free-update controls (SPFU, FRAP, and AMFU) that keep UnfoldCRF's encoder and candidate pixel–region paths but learn the updates, together with stronger attention controls and trainable dense and higher-order CRF controls. Compared with SPFU, UnfoldCRF-S improves FβωF_\beta^\omega by 0.6 pp (95% CI [0.3, 0.9]), reduces Harm from 10.4% to 8.5%, and raises the held-out-generator gain from 1.0 to 2.0 pp. We also clarified the contribution as image-and-mask-conditioned soft pixel–region associations combined with a mass-normalized consistency energy over semantic and null region states. These results support the benefit of energy-derived updates beyond matched candidate propagation and adapted trainable CRFs.
  • We expanded the component ablations to association learning and softness, mask conditioning, mass normalization, the null state, and damping, and report seed statistics with paired confidence intervals for the principal comparisons. These ablations support the roles of the learned soft associations and the normalized region energy.
  • We compared automatic SAM2/SAM3 prompting on the same initial masks, applied frozen refiners to Mask2Former, prompted SAM3, DSS, and SAM3-Adapter outputs, and evaluated downstream pseudo-label training and medical refinement with MedSAM and SAM2/3-Adapter upstreams. We further added joint 19-class refinement on Cityscapes and comparisons across four frozen backbones. These results support the practical value of a compact trained refiner alongside foundation models and its applicability across upstream models, label spaces, and representations.
  • We provided analyses of corrected and newly introduced errors, the frequency, severity, and conditions of harmful corrections, and stage-wise inference dynamics, together with measured latency, memory, and training cost and the accuracy–cost trade-off across trained depths. These analyses clarify the effectiveness and operating limits of finite-step structured refinement.

We thank the reviewers again for their valuable suggestions. We have revised the manuscript accordingly, updating the main text, experimental results, and appendix to incorporate these improvements, with the main changes marked in blue.

We appreciate the reviewers' time and consideration in reviewing our responses. We hope that the additional experiments and clarifications have addressed the concerns raised, and we welcome any further questions or discussion.

Best Regards,

All #4130 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.