acceptodds
Under review as a conference paper at ICLR 2027

Diffuse2Seg: Diffusion Models Can Segment Anything Without Supervision

Abstract

Open-world entity segmentation aims to predict masks for arbitrary visual entities across domains and at multiple granularities, from parts to whole objects. In this setting, SAM sets a strong standard: trained on SA-1B, comprising 11M images and over 1B carefully annotated masks, it achieves remarkable zero-shot performance. Collecting such labels is expensive and time-consuming, however, which limits how far this recipe can scale. Text-to-image diffusion models offer a way around this. Their intermediate features transfer well across perception tasks, and since object structure emerges as the model denoises a noise sample into an image conditioned on a text prompt, that structure is already encoded in these representations. It should therefore be possible to exploit them for open-world entity segmentation without retraining or supervision. To realize this, we present Diffuse2Seg, which repurposes generative diffusion models for automatic mask generation. This is achieved by propagating a grid of point prompts through their self-attention representations in an edge-preserving manner. Diffuse2Seg produces multi-granular instance masks and outperforms prior state-of-the-art label generators by 4.3-7.1 p.p. in across five domains. Training an instance segmentation model on these generated masks advances detector-free open-world segmentation by 7.4 and 7.7 p.p. on "things" and "stuff+things" datasets and surpasses the detector-based UnSAM on "stuff+things" by 2.1 p.p. in . Finally, we show that a model trained on Diffuse2Seg labels provides a strong initialization for semi-supervised learning, matching its fully supervised counterpart with only 5k labeled images and outperforming it with 10k. Code will be made publicly available.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Unsupervised Segmentation, Segment Anything, Diffusion Features

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_wRKQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Diffuse2Seg turns a frozen Stable Diffusion 2 UNet into an automatic, class-agnostic, multi-granular mask generator. Self-attention from the two highest-resolution decoder layers at a single denoising step (t=150, following M2N2) is aggregated into an affinity graph. A grid of one-hot point prompts is propagated over this graph with edge-preserving p-Laplacian regularization (p=1.6); the soft maps are clustered by symmetric KL with average linkage, and cutting the dendrogram at six heights yields masks at several granularities, deduplicated by NMS and refined with CascadePSP. A Mask2Former (DINO ResNet-50) is trained on these labels in two rounds, the second adding crop-based child masks; optionally CutLER masks are mixed in. Reported results: pseudo-label AR1000 is +4.3 to +7.1 above a re-implemented UnSAM pipeline across five datasets; the detector-free student beats SOHES by +7 to +10 AR; with CutLER mixing it beats UnSAM by +2 to +3 AR on stuff+things and ties on things; decoder-only fine-tuning matches a 100k-image supervised reference with 5k labels.

Critical strengths
  1. Strong, broad empirical results. In the detector-free regime the student beats SOHES by +7.4, +7.7, and +10.0 AR1000 (things, stuff+things, parts) with half the training images; with CutLER mixing it is consistently above UnSAM on stuff+things at all three budgets. Pseudo-labels are evaluated on five domains with all generator hyperparameters frozen on a SA-1B holdout, and a LAION-excluded EntitySeg check addresses the obvious leakage concern.

  2. The label generator is carefully ablated. Propagation exponent, timestep, number of hierarchy levels, and NMS threshold are all swept (Figs. 5 and 6, Table 4), so the design choices are justified rather than asserted.

Critical weaknesses
  1. The title claim, that diffusion representations are what enable this, is never isolated. Every component other than the affinity source (grid prompts, p-Laplacian propagation, KL and average-linkage merging, multi-height cuts, two-round training with crop enrichment) is agnostic to where the affinity comes from, yet no experiment keeps the pipeline fixed and swaps SD2 self-attention for DINO ViT-B/8 or DINOv2/v3 affinities. Without it, the +4 to +7 AR cannot be attributed to diffusion features rather than to the new propagation and merging algorithm, which is itself a combination of M2N2 (same layers and timestep), DiffSeg (KL merging), UnSAM (two-round training, crop refinement), and seeded graph propagation (random walker, Power Watershed). The "much smaller encoder" and data-efficiency claims apply only to the student: the labeler is a roughly 0.9B-parameter, LAION-language-supervised model, and CascadePSP is trained on human masks.

  2. Reporting inconsistencies bear directly on the headline comparison with UnSAM. (a) The UnSAM 0.4M row in Table 2 (ADE 39.1, SA-1B 44.8) differs from UnSAM's published values (37.5, 44.5) without being marked as re-evaluated. (b) CutLER has different AR1000 on the same datasets in Table 1 (ADE 24.8, Entity 22.1, PACO 10.7) and Table 2 (26.3, 23.1, 8.9). (c) The re-implemented UnSAM pseudo-labels score below CutLER alone on ADE20K (16.0 vs. 24.8) and UVO (22.8 vs. 32.3), although UnSAM's label pool contains the divide-stage CutLER masks by construction; this suggests the baseline is under-tuned while Diffuse2Seg's own generator is swept over about ten hyperparameters. (d) PartImageNet is dropped because of ImageNet pretraining, a rationale that applies equally to the paper's own ImageNet-pretrained DINO R50 student; it is also UnSAM's strongest benchmark (52.7 vs. 28.3 for SAM).

Questions for authors
  1. Can you run the affinity-swap ablation with an identical pipeline: SD2 self-attention vs. DINO ViT-B/8 vs. DINOv2/v3 affinities at comparable token resolution?

  2. Which Table 2 rows are quoted and which are re-evaluated? Why does CutLER differ between Tables 1 and 2, and why does the UnSAM pseudo-label pool have lower recall than CutLER alone?

  3. What are the PartImageNet numbers for Table 2?

  4. The raw pseudo-labels have mAP of about 1.7% (Table 4) and are generated at six levels with a permissive IoU 0.9 NMS. What is the mean number of masks per image for every method in Table 1 before refinement, and what is the AP of the trained students on all six benchmarks?

4 comments
AuthorsOpenReview ↗

Response to wRKQ (Part 1)

Dear Reviewer wRKQ, thank you for your detailed and constructive review. We address your points below.

W1 & Q1: Affinity-swap ablation with identical pipeline

Thank you for this useful suggestion. We agree that comparing SD2 to affinities of ViT-B models from DINOv1, DINOv2 (with registers), and DINOv3, while keeping the other components of our pipeline fixed, helps to isolate the impact of diffusion features on pseudo-label generation. We ran this ablation on our SA-1B validation split, keeping token resolution, prompt density, affinity normalization, and the number of clustering thresholds the same. Following previous work (CutLER, SOHES, UnSAM), we extract DINO attention-layer features from the last layer. We compute a Q–K affinity, analogous to SD2, and the K–K affinity, which is widely used in unsupervised segmentation and in our baseline UnSAM. To further improve fairness, we tuned the clustering thresholds and temperature on our 1k tuning holdout to match the average mask count of our SD2 model. With the pipeline fixed, SD2 affinities give the highest recall, +9.4 p.p. over the best DINO variant. DINOv1 K–K is better on small objects (3.3 vs. 2.6).

ModelAffinityARS\mathrm{AR}_SARM\mathrm{AR}_MARL\mathrm{AR}_LAR1000\mathrm{AR} _{1000}Masks/image
SD2 (ours)Q–K2.615.338.920.7632
DINOv1 (ViT-B/8)Q–K0.61.24.12.0683
K–K3.37.521.811.3662
DINOv2 (ViT-B/14)Q–K1.02.912.05.4608
K–K1.42.614.76.1652
DINOv3 (ViT-B/16)Q–K1.22.821.48.3659
K–K2.44.923.110.2643

SA-1B validation split (1k images). Bold: best per column.

W1: Data-efficiency claims of the student

We agree that these claims apply to the student only and will scope them accordingly in the revision: the labeler is a roughly 0.9B-parameter SD2 model pretrained on LAION image–text pairs, and CascadePSP is trained on human-annotated masks. SD2 is used once to annotate the training set and is not needed at inference. CascadePSP only refines the training labels, as in the SOHES and UnSAM baselines, so the comparison with them is unaffected.

W2a & Q2: Reported UnSAM numbers

Thank you for catching this. For Table 2, we re-evaluated the publicly released UnSAM 0.4M model with our evaluation pipeline, which yields a slightly higher ADE20K AR than the published value (39.1 vs. 37.5). For SA-1B, SOHES and UnSAM evaluate on randomly sampled validation splits that were not released. We therefore evaluate the released UnSAM 0.4M model on our own randomly sampled split, which gives a marginally different result (44.8 vs. 44.5). In both cases, our numbers for UnSAM are slightly higher than the published ones, so our evaluation favors the baseline. In the revision, we will mark these results as re-evaluated.

W2b & Q2: CutLER in Table 1 and Table 2

Thank you for pointing this out. Table 1 reports our own reproduction of CutLER, to evaluate consistently across our wider range of benchmarks. Table 2 instead reports the values published in SOHES and UnSAM, for consistency with prior work. In the revision, we will mark the Table 1 numbers as reproduced and state this distinction explicitly.

W2c & Q2: CutLER vs. UnSAM pseudo-labels

Thank you for this careful observation. UnSAM's label pool indeed contains CutLER masks, but only those with a confidence above 0.1. In contrast, the CutLER row in Table 1 applies no threshold. For a consistent comparison, we added a table below, where both are compared at the same threshold. At 0.1, UnSAM consistently outperforms CutLER (+3.4 to +8.1 AR1000\mathrm{AR} _{1000}), confirming that it adds useful masks to the CutLER pool. The higher recall of unthresholded CutLER comes from its many low-confidence predictions (below 0.1) that still match ground-truth masks. Regarding tuning, we use the hyperparameters of UnSAM's official implementation, except for the confidence threshold, which we lower from 0.3 (official) to 0.1 to improve recall. We will add this table and discussion to the revision.

MethodConfidenceSA-1BADE20KEntitySegUVOPACO
CutLER0.017.024.822.132.310.7
CutLER0.18.39.215.916.73.3
UnSAM0.116.416.019.322.89.3
Δ at 0.1 (UnSAM − CutLER)+8.1+6.8+3.4+6.1+6.0

AR1000\mathrm{AR} _{1000} of the pseudo-labels. The confidence threshold is the CutLER detection threshold; for UnSAM, it gates the divide-stage parents.

AuthorsOpenReview ↗

Response to wRKQ (Part 2)

W2d & Q3: PartImageNet

You are right: the ImageNet rationale also applies to our DINO-initialized student. We will add the PartImageNet results to the revised paper. The numbers are in the table below, together with ADE20K-Part [1], evaluated following PartGLEE [2], as a second part benchmark without ImageNet overlap.

ModelCutLERImagesPartImageNetADE20K-Part (parts)ADE20K-Part (obj.+parts)PACO†
Diffuse2Seg (ours)✗0.1M43.225.628.927.1
UnSAM (released checkpoint)✓0.4M51.726.230.729.7
Diffuse2Seg (ours)✓0.1M46.326.330.828.9
Diffuse2Seg (ours)✓0.2M46.126.730.829.1
Diffuse2Seg (ours)✓0.4M46.427.031.529.8
Δ at 0.4M (ours − UnSAM)−5.3+0.8+0.8+0.1

AR1000\mathrm{AR} _{1000}. †From Table 2, for reference.

UnSAM is clearly ahead on PartImageNet (−5.3 for ours at 0.4M; UnSAM re-evaluated: 51.7, published: 52.7). On PACO and ADE20K-Part, both models are on par (+0.1 and +0.8 for ours). Our detector-free model reaches 43.2; adding CutLER masks helps most on PartImageNet (+3.1, vs. +1.8 on PACO and +0.7 on ADE20K-Part parts).

A likely reason for the gap is ImageNet overlap: PartImageNet images come from ImageNet, on which UnSAM's label generators (CutLER and DINO) are pretrained, whereas our pseudo labels come from SD2 and are only mildly augmented with CutLER masks during training. For our own student, we tested the initialization directly: replacing ImageNet-pretrained DINO with MoCo-v2 pretrained on PASS [3] in round 1 changes PartImageNet by −1.4 p.p. and all other benchmarks by at most −0.8 p.p. (table below). PartImageNet is thus the benchmark most affected by ImageNet initialization, although this effect is small compared to the gap to UnSAM.

InitializationCOCOLVISADE20KEntitySegSA-1BPACOPartImageNet
DINO, ImageNet-1k38.736.237.940.140.926.543.5
MoCo-v2, PASS38.336.037.139.840.526.542.1
Δ (PASS − ImageNet)−0.4−0.2−0.8−0.3−0.40.0−1.4

AR1000\mathrm{AR} _{1000} after round 1 of detector-augmented Diffuse2Seg training (0.1M images) with different ResNet-50 initializations.

[1] Wei et al., OV-PARTS: Towards Open-Vocabulary Part Segmentation, NeurIPS Datasets and Benchmarks Track, 2023.

[2] Li et al., PartGLEE: A Foundation Model for Recognizing and Parsing Any Objects, ECCV 2024.

[3] Asano et al., PASS: PASS: An ImageNet replacement for self-supervised pretraining without humans, NeurIPS Datasets and Benchmarks Track, 2021.

AuthorsOpenReview ↗

Response to wRKQ (Part 3)

Q4: Masks per image and student AP

Thank you for your suggestion to also report AP and mask count.

Masks per image. The table below reports the mean number of masks per image before refinement for every method in Table 1 (SA-1B validation split). Our pseudo-labels contain about 5% more masks than UnSAM’s (632 vs. 599) while improving AR1000\mathrm{AR}_{1000} by 26% relative (16.4 → 20.7, +4.3 p.p.), so the recall gain does not come from emitting more masks. DiffSeg emits the most masks (1000) at lower recall; SOHES and M2N2 emit far fewer, at substantially lower recall. CutLER is capped at 100 detections per image by default. Regarding the low mAP: our generator has no mask-ranking mechanism, so AP only becomes meaningful once a student learns to rank masks.

MethodARS\mathrm{AR}_SARM\mathrm{AR}_MARL\mathrm{AR}_LAR1000\mathrm{AR} _{1000}Masks/image
DiffSeg0.312.429.916.0999.6
Diffuse2Seg (ours)2.615.338.920.7631.5
UnSAM5.713.926.016.4599.4
M2N20.02.826.59.7162.2
SOHES2.810.615.711.0144.8
CutLER5.414.028.017.099.9

Student AP. The next table reports class-agnostic mask AP of our and UnSAM's detector-augmented 0.4M students on all six benchmarks. When interpreting AP for unsupervised segmentation, it is worth keeping in mind that correct masks of objects or regions that a dataset does not annotate count as false positives, so AP depends strongly on each dataset’s labeling rules. Indeed, ordering the benchmarks by annotation density reveals a clear trend. On densely annotated stuff+things datasets, our student achieves substantially higher AP than UnSAM (1.97× on EntitySeg, 1.32× on SA-1B). On the more sparsely annotated datasets, it trails UnSAM while retaining 80–97% of its AP. This highlights our model’s strength in segmenting both stuff and things.

BenchmarkPixels annotatedGT masks / imageOurs APUnSAM APOurs / UnSAM
EntitySeg99.2%17.88.644.381.97
SA-1B74.9%102.714.2910.811.32
COCO29.3%7.33.423.560.96
ADE20K22.8%11.33.873.970.97
LVIS20.2%10.52.603.270.80
PACO9.5%13.21.081.230.88

Class-agnostic mask AP at 1,000 proposals per image, ordered by the mean share of image pixels covered by ground-truth masks.

To further confirm this, we split the SA-1B validation images into quartiles by annotation density (table below): the AP ratio grows from 1.21 to 1.52 with density, while the AR1000\mathrm{AR} _{1000} ratio stays at 1.06–1.08. This is consistent with the AP gap on sparse benchmarks stemming largely from masks on unannotated regions.

QuartilePixels annotatedAP (Ours)AP (UnSAM)AP ratioAR$_{1000}$ (Ours)AR$_{1000}$ (UnSAM)AR$_{1000}$ ratio
12–63% (mean 45%)11.419.441.2144.842.21.06
263–79% (mean 72%)13.5010.671.2745.843.11.06
379–92% (mean 86%)14.9411.241.3348.645.21.08
492–100% (mean 97%)18.7312.301.5254.250.51.07

SA-1B, 1,000 validation images split into quartiles by the share of pixels covered by ground-truth masks.

Thank you for the detailed explanation and additional experiments. Most of my concerns are addressed well. I will maintain my positive score. Good luck with your manuscript!

Reviewer_uwMQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Diffuse2Seg automatically generates class-agnostic, multi-granularity instance masks from self-attention in a frozen diffusion model. A grid of point prompts is propagated over the self-attention affinity graph using nonlinear p-Laplacian regularization to produce soft object maps. Hierarchical clustering, map merging, and post-processing convert these maps into a pseudo-label pool. The labels are then used to train a Mask2Former segmentor through two rounds of training with crop-based pseudo-label enrichment. Experiments evaluate both pseudo-label quality and zero-shot cross-dataset segmentation performance, with particularly strong results on stuff-and-things benchmarks. The paper also demonstrates benefits for label-efficient semi-supervised fine-tuning.

Critical strengths

The paper presents an effective diffusion-based framework for dense, multi-granularity pseudo-label generation and validates its utility both directly and through downstream model training.

Critical weaknesses
  1. The experiments do not fully disentangle the contributions of the representation source and the mask-generation pipeline. The paper includes extensive hyperparameter analyses, a linear-versus-nonlinear propagation comparison, and training-stage ablations. Comparisons with SD2-based methods also provide evidence for the effectiveness of the proposed pipeline. However, controlled comparisons that swap the affinity source while retaining the same pipeline, or replace propagation while retaining the same merging procedure, are missing. This limits confidence in attributing the gains to diffusion representations versus the proposed processing steps.

  2. Small-object coverage remains a limitation of the pseudo-label generator. Although Table 6 shows that this deficit does not persist in the evaluated detector-augmented student model, the limitation remains relevant to the standalone generator’s goal of densely annotating visual entities across scales.

Questions for authors

My main concern is that the current experiments do not sufficiently disentangle the contributions of the diffusion-derived affinities and the mask-construction pipeline, making it difficult to assess the paper’s central technical contribution. I am open to revising my rating upward if the authors’ response to the questions below addresses this concern through clarification of existing results or additional supporting evidence.

  1. Controlled comparison of affinity sources. How does pseudo-label quality change when the diffusion-derived affinities are replaced with affinities constructed from self-supervised features, such as DINO [1, 2], while retaining the same propagation, merging, and post-processing pipeline? A comparison with comparable spatial resolution, prompt density, and affinity normalization would help assess the contribution of the representation source. Please report overall and size-stratified recall, together with matched qualitative examples illustrating representative differences or failure cases.

  2. Ablation of prompt propagation. How does nonlinear propagation compare with simpler alternatives under the same downstream processing? In particular, please consider both the direct affinity map of each seed token and a locally averaged affinity map: mj(k)=1∣Pk∣∑i∈PkAij,m_j^{(k)}=\frac{1}{|P_k|}\sum_{i\in P_k}A_{ij}, where PkP_k is a 6×66\times6 neighborhood of latent tokens associated with prompt kk, and AA is the same preprocessed affinity matrix used for propagation. Keeping map normalization, merging, and post-processing unchanged, how do these alternatives affect pseudo-label quality and runtime? Quantitative results and representative qualitative failure cases would help establish the benefit of propagation.

  3. Clarification of small-object limitations. What evidence distinguishes the effects of affinity-grid resolution, prompt spacing, and subsequent merging or filtering on small-object recall? Figure 6 provides aggregate parameter sweeps, but size-stratified results or examples showing where small objects are lost in the pipeline would help assess the source and severity of this limitation.

[1] Oquab et al., DINOv2: Learning Robust Visual Features without Supervision. TMLR 2024.

[2] Siméoni et al., DINOv3. arXiv:2508.10104.

2 comments
AuthorsOpenReview ↗

Response to Reviewer uwMQ (Part 1)

Dear Reviewer uwMQ, thank you for your insightful comments. We provide detailed answers to your questions below.

Q1: Controlled comparison of affinity sources

Thank you for this useful suggestion. We ran the requested comparison on our SA-1B validation split: the affinity is swapped, while propagation, merging, and post-processing stay fixed. All models use the same token grid (140×140), prompt grid, and normalization. Only the temperature and cut heights are re-tuned per model on the tuning holdout: we select the configuration with the best AR1000\mathrm{AR}_{1000} at a mask count matching ours. For DINOv1/v2 (with registers)/v3 (last layer, ViT-B), we report the Q–K affinity, analogous to SD2, and the K–K affinity, the standard choice in DINO-based segmentation (CutLER, UnSAM, SOHES). With the pipeline fixed, SD2 affinities give the highest recall, +9.4 p.p. over the best DINO variant (DINOv1 K–K), with gains on medium (+7.8) and large objects (+15.8.) over the best DINO results. DINOv1 K–K is better on small objects (3.3 vs. 2.6). We will add qualitative examples to the appendix of the revised paper.

ModelAffinityARS\mathrm{AR}_SARM\mathrm{AR}_MARL\mathrm{AR}_LAR1000\mathrm{AR} _{1000}Masks/image
SD2 (ours)Q–K2.615.338.920.7632
DINOv1 (ViT-B/8)Q–K0.61.24.12.0683
K–K3.37.521.811.3662
DINOv2 (ViT-B/14)Q–K1.02.912.05.4608
K–K1.42.614.76.1652
DINOv3 (ViT-B/16)Q–K1.22.821.48.3659
K–K2.44.923.110.2643

SA-1B validation split (1k images). Bold: best per column.

Q2: Ablation of prompt propagation

We evaluated both proposed alternatives, the direct affinity map of each seed token and the 6×66 \times 6 locally averaged affinity map, with map normalization, merging, and post-processing unchanged. We also measured the runtime of all three variants on an A100 GPU. Nonlinear propagation has a clear advantage that justifies its higher cost. The local-average variant adds cost to the clustering stage (0.47 s vs. 0.02 s), as it produces a high number of clusters before post-processing due to a strong local window pattern in its maps.

Maps fromAR1000\mathrm{AR} _{1000}ARS\mathrm{AR}_SARM\mathrm{AR}_MARL\mathrm{AR}_LMasks/imageTotal (s)Map constr. (s)Clust. + post (s)
Nonlinear propagation (ours)20.72.615.338.96322.442.420.02
Direct affinity row12.51.48.025.86690.440.410.03
6×66 \times 6 local average6.70.21.618.96621.010.540.47

Recall in %, SA-1B validation split (1k images), no refinement. Times are seconds per image: total, map construction, clustering and post-processing.

AuthorsOpenReview ↗

Response to Reviewer uwMQ (Part 2)

Q3: Clarification of small-object limitations

Thank you for this question. The table below gives the size-stratified version of the Figure 6 sweeps for the three factors you name: affinity-grid resolution, prompt spacing, and subsequent merging and filtering. Each row changes one parameter from the reference configuration of this Appendix A.1 analysis (first row in the table below).

FactorChangePromptsARS\mathrm{AR}_SARM\mathrm{AR}_MARL\mathrm{AR}_LAR1000\mathrm{AR} _{1000}
Reference–2892.512.434.617.7
Affinity-grid resolution rr140→128140 \to 1282561.610.133.216.0
140→160140 \to 1604003.916.037.220.7
140→192140 \to 1925766.921.139.024.5
Prompt spacing ss (tokens)8→128 \to 121442.010.531.915.8
8→58 \to 57842.614.636.619.6
Merging: lowest cut height0.186→0.050.186 \to 0.052893.014.936.719.8
Merging: number of cut levels5→85 \to 82892.713.035.418.4
Filtering: deduplication IoU0.8→0.90.8 \to 0.92892.512.936.118.5
Filtering: minimum mask area100→25100 \to 25 px2892.512.534.617.8

Recall in %, SA-1B holdout split, pseudo-labels before refinement. Prompts: number of point prompts per image. For merging and filtering, we show the more permissive setting, i.e., the one that keeps more small masks.

Resolution vs. prompt spacing. Resolution dominates: r=192r = 192 raises ARS\mathrm{AR}_S from 2.5 to 6.9. As spacing is defined in tokens, this also raises the prompt count to 576. However, increasing the number of prompts at fixed r=140r = 140 (spacing 5, 784 prompts) moves ARS\mathrm{AR}_S only to 2.6. Within the tested range, small-object recall is thus limited by the grid resolution, not by prompt density.

Merging and filtering. Even the more permissive settings improve ARS\mathrm{AR}_S at most by +0.5 p.p. (lowest cut height). Relaxed deduplication and a smaller minimum area have no significant impact on small-object recall. Small objects are therefore not merged away or filtered out, but are already missing from the soft object maps.

Resolution as a primary source. The primary source is the affinity grid: at r=140r = 140, one token covers 8×88 \times 8 px of the input, so a small object (area below 32232^2 px) spans fewer than 4×44 \times 4 tokens. This remains a limitation at our default resolution, which we keep because the affinity matrix grows with r4r^4 (about 3.5×3.5\times larger at r=192r = 192). We will add this table and examples of missed small objects to the appendix.

Reviewer_Ejo7OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Diffuse2Seg take a frozen Stable Diffusion 2, read self-attention off one denoising step, and treat it as a graph. Point prompts scatter over it, each spreads until it hits an edge, and cutting at several thresholds gives masks at different granularities. Those train a Mask2Former R50. Labels beat UnSAM on five domains, the student beats SOHES and edges UnSAM, and decoder-only fine-tuning on 10k labels passes a 100k supervised model.

Critical strengths
  1. The recall gain isn't just more proposals. Tab 5 has 29.2 vs 23.8 with fewer masks per image, 366 vs 387, and UnSAM ran with its threshold loosened in its own favour (sec C.1). That's the first thing I check on an AR-only table. Beating trained CutLER on PACO with no training at all is what sold me.

  2. They police their own contamination. PartImageNet dropped because ImageNet pretrains DINO and CutLER (sec 4.1); Tab 7 redoes EntitySeg without LAION. Both cost them, neither is standard.

Critical weaknesses
  1. I can't separate the labels from the training recipe, and their own ablation is why. The abstract claims +2.1 over UnSAM on stuff+things at 0.4M. Tab 3b says round-2 crop enrichment alone is worth +2.2 (40.3 to 42.5), and per sec D.4 UnSAM is the public checkpoint, so it never saw that recipe. At 0.1M it's margin +3.4, recipe +2.2. Push the public UnSAM through your crop enrichment and report stuff+things; if the margin holds I'll raise. The SOHES comparison is worse, it swaps ViT-B for R50 too.

  2. Nothing isolates the diffusion affinity. Fig 5, Tab 4 and Fig 6 all turn knobs inside the pipeline. No run swaps A in eq 1 for a non-diffusion affinity, so the title is unsupported.

Questions for authors
  1. Is the 1k SA-1B you tune on the same 1k you evaluate on? Sec B.1 picks every generator parameter on "a held-out SA-1B 1k split", sec 4.1 holds out "1000 further images", Tab 5 reports "the SA-1B validation split (1000 images)". If it's the same split, Tab 1's SA-1B column and all of Tab 5 are in-sample for the whole sweep in Fig 6, while DiffSeg got ten calibration images (sec C.1). Transfer columns are fine either way. Separately, Fig 1 calls the +1.4 to +3.0 student gain "zero-shot", and one of those three is SA-1B, which you train on.

  2. CascadePSP runs before student training (sec 3.3) and ships trained on human masks. Do SOHES and UnSAM get it on the same terms? Tab 1 is unrefined so it's clean, Tab 2 isn't.

  3. Swap A in eq 1 for a DINO ViT-B/8 affinity, hold propagation, clustering and NMS fixed, rerun the 1k holdout.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Ejo7 (Part 1)

Dear Reviewer Ejo7, we appreciate your thoughtful comments and insights. We address your points below.

W1: Crop enrichment paired with UnSAM

We fully agree that disentangling the impact of labels and training pipeline on our student models deserves more attention. We therefore ran an additional experiment in which we keep our Diffuse2Seg training pipeline fixed and swap our Diffuse2Seg labels for UnSAM labels. Specifically, we trained a model on the same 100k images used for our models, labeled with UnSAM, for two rounds with our training settings and our crop enrichment. We favor this over fine-tuning the publicly released checkpoint to ensure matching fine-tuning budgets, as the latter already received two-round training including label enrichment.

ThingsStuff+ThingsParts
Training method + label enrichment +COCOLVISAvg.ADEEntitySA-1BAvg.PACO
UnSAM pseudo-labels39.338.338.836.636.239.137.328.2
Diffuse2Seg pseudo-labels (ours)41.038.739.939.842.245.542.528.9
Δ (Diffuse2Seg − UnSAM labels)+1.7+0.4+1.1+3.2+6.0+6.4+5.2+0.7

AR1000\mathrm{AR} _{1000} of students trained with our training pipeline and crop enrichment; only the pseudo-labels differ.

With the training recipe held fixed, Diffuse2Seg labels remain ahead on every benchmark, by +5.2 p.p. on stuff+things. In general, label enrichment is meant to mitigate label-specific shortcomings: our pseudo-labels are particularly dense, so we add detail with crop enrichment, whereas UnSAM masks lack coverage, which their enrichment stage seeks to improve.

W1: Comparison to SOHES

Thank you for your remark. SOHES does indeed use a stronger backbone for its self-training model (ViT-B vs. our ResNet-50). We believe this further supports our detector-free results, as we achieve significantly better performance with half the training images and a smaller backbone. Using ResNet-50 moreover ensures comparability to our strongest baseline, UnSAM.

W2 & Q3: Ablation of the affinity source

We agree that swapping our diffusion-based affinity for non-diffusion-based DINO feature affinities greatly improves understanding the impact of different representations on the label generation pipeline. We thus conducted additional experiments with fixed label-generation pipeline without refinement to compare our SD2-based affinities with various DINO-based ones. Specifically, we consider last-layer Q-K and K-K affinities of DINOv1, Dinov2 (with registers), and DINOv3 ViT-B, which are used in UnSAM and many other DINO-based unsupervised segmentation works. For a fair comparison, we keep the same feature resolution (140 × 140) for DINO. We adjust its clustering thresholds and temperature on our tuning holdout so that the average mask count matches that of our diffusion-based labeling pipeline. Results are reported on the 1k validation holdout set. Compared to these DINO-based affinities, we perform significantly better in mAR (see the table below). Only the DINOv1 K-K configuration is tuned more toward small-object performance, at the expense of other object scales.

ModelAffinityARS\mathrm{AR}_SARM\mathrm{AR}_MARL\mathrm{AR}_LAR1000\mathrm{AR} _{1000}Masks/image
SD2 (ours)Q–K2.615.338.920.7632
DINOv1 (ViT-B/8)Q–K0.61.24.12.0683
K–K3.37.521.811.3662
DINOv2 (ViT-B/14)Q–K1.02.912.05.4608
K–K1.42.614.76.1652
DINOv3 (ViT-B/16)Q–K1.22.821.48.3659
K–K2.44.923.110.2643

SA-1B validation split (1k images). Bold: best per column.

Q1: Re-tuning DiffSeg for a fair comparison Thank you for pointing out that DiffSeg got fewer calibration images than our method. We re-ran its calibration on the full 1k tuning holdout, which improves DiffSeg on the SA-1B validation split. It still underperforms our method and emits more proposals (996 vs. 632 per image). We will include the cross-domain results with the re-tuned DiffSeg thresholds in the revised version.

MethodARS\mathrm{AR}_SARM\mathrm{AR}_MARL\mathrm{AR}_LAR1000\mathrm{AR} _{1000}Masks/image
DiffSeg, paper recipe0.312.429.916.01000
DiffSeg, re-tuned0.914.631.417.8996
Diffuse2Seg (ours)2.615.338.920.7632
AuthorsOpenReview ↗

Response to Reviewer Ejo7 (Part 2)

Q1: Tuning split vs. validation split

Thanks for pointing this out. We use two different, randomly sampled, disjoint 1k holdout splits: one to tune the pseudo-labels and one to validate the pseudo-labels and trained models. We will clarify this in the revision.

Q1: Student gain on SA-1B labeled as zero-shot

Thanks for bringing this up. In the revision, we will make clear which gains are held-out in-domain (SA-1B) and which are cross-domain transfer.

Q2: CascadePSP refinement

CascadePSP was applied consistently across methods (Diffuse2Seg, UnSAM, and SOHES) for the self-training experiments in Table 2. We will state this more clearly in the main paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.