- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Diffuse2Seg turns a frozen Stable Diffusion 2 UNet into an automatic, class-agnostic, multi-granular mask generator. Self-attention from the two highest-resolution decoder layers at a single denoising step (t=150, following M2N2) is aggregated into an affinity graph. A grid of one-hot point prompts is propagated over this graph with edge-preserving p-Laplacian regularization (p=1.6); the soft maps are clustered by symmetric KL with average linkage, and cutting the dendrogram at six heights yields masks at several granularities, deduplicated by NMS and refined with CascadePSP. A Mask2Former (DINO ResNet-50) is trained on these labels in two rounds, the second adding crop-based child masks; optionally CutLER masks are mixed in. Reported results: pseudo-label AR1000 is +4.3 to +7.1 above a re-implemented UnSAM pipeline across five datasets; the detector-free student beats SOHES by +7 to +10 AR; with CutLER mixing it beats UnSAM by +2 to +3 AR on stuff+things and ties on things; decoder-only fine-tuning matches a 100k-image supervised reference with 5k labels.
Critical strengths
-
Strong, broad empirical results. In the detector-free regime the student beats SOHES by +7.4, +7.7, and +10.0 AR1000 (things, stuff+things, parts) with half the training images; with CutLER mixing it is consistently above UnSAM on stuff+things at all three budgets. Pseudo-labels are evaluated on five domains with all generator hyperparameters frozen on a SA-1B holdout, and a LAION-excluded EntitySeg check addresses the obvious leakage concern.
-
The label generator is carefully ablated. Propagation exponent, timestep, number of hierarchy levels, and NMS threshold are all swept (Figs. 5 and 6, Table 4), so the design choices are justified rather than asserted.
Critical weaknesses
-
The title claim, that diffusion representations are what enable this, is never isolated. Every component other than the affinity source (grid prompts, p-Laplacian propagation, KL and average-linkage merging, multi-height cuts, two-round training with crop enrichment) is agnostic to where the affinity comes from, yet no experiment keeps the pipeline fixed and swaps SD2 self-attention for DINO ViT-B/8 or DINOv2/v3 affinities. Without it, the +4 to +7 AR cannot be attributed to diffusion features rather than to the new propagation and merging algorithm, which is itself a combination of M2N2 (same layers and timestep), DiffSeg (KL merging), UnSAM (two-round training, crop refinement), and seeded graph propagation (random walker, Power Watershed). The "much smaller encoder" and data-efficiency claims apply only to the student: the labeler is a roughly 0.9B-parameter, LAION-language-supervised model, and CascadePSP is trained on human masks.
-
Reporting inconsistencies bear directly on the headline comparison with UnSAM. (a) The UnSAM 0.4M row in Table 2 (ADE 39.1, SA-1B 44.8) differs from UnSAM's published values (37.5, 44.5) without being marked as re-evaluated. (b) CutLER has different AR1000 on the same datasets in Table 1 (ADE 24.8, Entity 22.1, PACO 10.7) and Table 2 (26.3, 23.1, 8.9). (c) The re-implemented UnSAM pseudo-labels score below CutLER alone on ADE20K (16.0 vs. 24.8) and UVO (22.8 vs. 32.3), although UnSAM's label pool contains the divide-stage CutLER masks by construction; this suggests the baseline is under-tuned while Diffuse2Seg's own generator is swept over about ten hyperparameters. (d) PartImageNet is dropped because of ImageNet pretraining, a rationale that applies equally to the paper's own ImageNet-pretrained DINO R50 student; it is also UnSAM's strongest benchmark (52.7 vs. 28.3 for SAM).
Questions for authors
-
Can you run the affinity-swap ablation with an identical pipeline: SD2 self-attention vs. DINO ViT-B/8 vs. DINOv2/v3 affinities at comparable token resolution?
-
Which Table 2 rows are quoted and which are re-evaluated? Why does CutLER differ between Tables 1 and 2, and why does the UnSAM pseudo-label pool have lower recall than CutLER alone?
-
What are the PartImageNet numbers for Table 2?
-
The raw pseudo-labels have mAP of about 1.7% (Table 4) and are generated at six levels with a permissive IoU 0.9 NMS. What is the mean number of masks per image for every method in Table 1 before refinement, and what is the AP of the trained students on all six benchmarks?
4 comments
Response to wRKQ (Part 2)
W2d & Q3: PartImageNet
You are right: the ImageNet rationale also applies to our DINO-initialized student. We will add the PartImageNet results to the revised paper. The numbers are in the table below, together with ADE20K-Part [1], evaluated following PartGLEE [2], as a second part benchmark without ImageNet overlap.
| Model | CutLER | Images | PartImageNet | ADE20K-Part (parts) | ADE20K-Part (obj.+parts) | PACO† |
|---|---|---|---|---|---|---|
| Diffuse2Seg (ours) | ✗ | 0.1M | 43.2 | 25.6 | 28.9 | 27.1 |
| UnSAM (released checkpoint) | ✓ | 0.4M | 51.7 | 26.2 | 30.7 | 29.7 |
| Diffuse2Seg (ours) | ✓ | 0.1M | 46.3 | 26.3 | 30.8 | 28.9 |
| Diffuse2Seg (ours) | ✓ | 0.2M | 46.1 | 26.7 | 30.8 | 29.1 |
| Diffuse2Seg (ours) | ✓ | 0.4M | 46.4 | 27.0 | 31.5 | 29.8 |
| Δ at 0.4M (ours − UnSAM) | −5.3 | +0.8 | +0.8 | +0.1 |
. †From Table 2, for reference.
UnSAM is clearly ahead on PartImageNet (−5.3 for ours at 0.4M; UnSAM re-evaluated: 51.7, published: 52.7). On PACO and ADE20K-Part, both models are on par (+0.1 and +0.8 for ours). Our detector-free model reaches 43.2; adding CutLER masks helps most on PartImageNet (+3.1, vs. +1.8 on PACO and +0.7 on ADE20K-Part parts).
A likely reason for the gap is ImageNet overlap: PartImageNet images come from ImageNet, on which UnSAM's label generators (CutLER and DINO) are pretrained, whereas our pseudo labels come from SD2 and are only mildly augmented with CutLER masks during training. For our own student, we tested the initialization directly: replacing ImageNet-pretrained DINO with MoCo-v2 pretrained on PASS [3] in round 1 changes PartImageNet by −1.4 p.p. and all other benchmarks by at most −0.8 p.p. (table below). PartImageNet is thus the benchmark most affected by ImageNet initialization, although this effect is small compared to the gap to UnSAM.
| Initialization | COCO | LVIS | ADE20K | EntitySeg | SA-1B | PACO | PartImageNet |
|---|---|---|---|---|---|---|---|
| DINO, ImageNet-1k | 38.7 | 36.2 | 37.9 | 40.1 | 40.9 | 26.5 | 43.5 |
| MoCo-v2, PASS | 38.3 | 36.0 | 37.1 | 39.8 | 40.5 | 26.5 | 42.1 |
| Δ (PASS − ImageNet) | −0.4 | −0.2 | −0.8 | −0.3 | −0.4 | 0.0 | −1.4 |
after round 1 of detector-augmented Diffuse2Seg training (0.1M images) with different ResNet-50 initializations.
[1] Wei et al., OV-PARTS: Towards Open-Vocabulary Part Segmentation, NeurIPS Datasets and Benchmarks Track, 2023.
[2] Li et al., PartGLEE: A Foundation Model for Recognizing and Parsing Any Objects, ECCV 2024.
[3] Asano et al., PASS: PASS: An ImageNet replacement for self-supervised pretraining without humans, NeurIPS Datasets and Benchmarks Track, 2021.
Response to wRKQ (Part 3)
Q4: Masks per image and student AP
Thank you for your suggestion to also report AP and mask count.
Masks per image. The table below reports the mean number of masks per image before refinement for every method in Table 1 (SA-1B validation split). Our pseudo-labels contain about 5% more masks than UnSAM’s (632 vs. 599) while improving by 26% relative (16.4 → 20.7, +4.3 p.p.), so the recall gain does not come from emitting more masks. DiffSeg emits the most masks (1000) at lower recall; SOHES and M2N2 emit far fewer, at substantially lower recall. CutLER is capped at 100 detections per image by default. Regarding the low mAP: our generator has no mask-ranking mechanism, so AP only becomes meaningful once a student learns to rank masks.
| Method | Masks/image | ||||
|---|---|---|---|---|---|
| DiffSeg | 0.3 | 12.4 | 29.9 | 16.0 | 999.6 |
| Diffuse2Seg (ours) | 2.6 | 15.3 | 38.9 | 20.7 | 631.5 |
| UnSAM | 5.7 | 13.9 | 26.0 | 16.4 | 599.4 |
| M2N2 | 0.0 | 2.8 | 26.5 | 9.7 | 162.2 |
| SOHES | 2.8 | 10.6 | 15.7 | 11.0 | 144.8 |
| CutLER | 5.4 | 14.0 | 28.0 | 17.0 | 99.9 |
Student AP. The next table reports class-agnostic mask AP of our and UnSAM's detector-augmented 0.4M students on all six benchmarks. When interpreting AP for unsupervised segmentation, it is worth keeping in mind that correct masks of objects or regions that a dataset does not annotate count as false positives, so AP depends strongly on each dataset’s labeling rules. Indeed, ordering the benchmarks by annotation density reveals a clear trend. On densely annotated stuff+things datasets, our student achieves substantially higher AP than UnSAM (1.97× on EntitySeg, 1.32× on SA-1B). On the more sparsely annotated datasets, it trails UnSAM while retaining 80–97% of its AP. This highlights our model’s strength in segmenting both stuff and things.
| Benchmark | Pixels annotated | GT masks / image | Ours AP | UnSAM AP | Ours / UnSAM |
|---|---|---|---|---|---|
| EntitySeg | 99.2% | 17.8 | 8.64 | 4.38 | 1.97 |
| SA-1B | 74.9% | 102.7 | 14.29 | 10.81 | 1.32 |
| COCO | 29.3% | 7.3 | 3.42 | 3.56 | 0.96 |
| ADE20K | 22.8% | 11.3 | 3.87 | 3.97 | 0.97 |
| LVIS | 20.2% | 10.5 | 2.60 | 3.27 | 0.80 |
| PACO | 9.5% | 13.2 | 1.08 | 1.23 | 0.88 |
Class-agnostic mask AP at 1,000 proposals per image, ordered by the mean share of image pixels covered by ground-truth masks.
To further confirm this, we split the SA-1B validation images into quartiles by annotation density (table below): the AP ratio grows from 1.21 to 1.52 with density, while the ratio stays at 1.06–1.08. This is consistent with the AP gap on sparse benchmarks stemming largely from masks on unannotated regions.
| Quartile | Pixels annotated | AP (Ours) | AP (UnSAM) | AP ratio | AR$_{1000}$ (Ours) | AR$_{1000}$ (UnSAM) | AR$_{1000}$ ratio |
|---|---|---|---|---|---|---|---|
| 1 | 2–63% (mean 45%) | 11.41 | 9.44 | 1.21 | 44.8 | 42.2 | 1.06 |
| 2 | 63–79% (mean 72%) | 13.50 | 10.67 | 1.27 | 45.8 | 43.1 | 1.06 |
| 3 | 79–92% (mean 86%) | 14.94 | 11.24 | 1.33 | 48.6 | 45.2 | 1.08 |
| 4 | 92–100% (mean 97%) | 18.73 | 12.30 | 1.52 | 54.2 | 50.5 | 1.07 |
SA-1B, 1,000 validation images split into quartiles by the share of pixels covered by ground-truth masks.
Thank you for the detailed explanation and additional experiments. Most of my concerns are addressed well. I will maintain my positive score. Good luck with your manuscript!
Response to wRKQ (Part 1)
Dear Reviewer wRKQ, thank you for your detailed and constructive review. We address your points below.
W1 & Q1: Affinity-swap ablation with identical pipeline
Thank you for this useful suggestion. We agree that comparing SD2 to affinities of ViT-B models from DINOv1, DINOv2 (with registers), and DINOv3, while keeping the other components of our pipeline fixed, helps to isolate the impact of diffusion features on pseudo-label generation. We ran this ablation on our SA-1B validation split, keeping token resolution, prompt density, affinity normalization, and the number of clustering thresholds the same. Following previous work (CutLER, SOHES, UnSAM), we extract DINO attention-layer features from the last layer. We compute a Q–K affinity, analogous to SD2, and the K–K affinity, which is widely used in unsupervised segmentation and in our baseline UnSAM. To further improve fairness, we tuned the clustering thresholds and temperature on our 1k tuning holdout to match the average mask count of our SD2 model. With the pipeline fixed, SD2 affinities give the highest recall, +9.4 p.p. over the best DINO variant. DINOv1 K–K is better on small objects (3.3 vs. 2.6).
SA-1B validation split (1k images). Bold: best per column.
W1: Data-efficiency claims of the student
We agree that these claims apply to the student only and will scope them accordingly in the revision: the labeler is a roughly 0.9B-parameter SD2 model pretrained on LAION image–text pairs, and CascadePSP is trained on human-annotated masks. SD2 is used once to annotate the training set and is not needed at inference. CascadePSP only refines the training labels, as in the SOHES and UnSAM baselines, so the comparison with them is unaffected.
W2a & Q2: Reported UnSAM numbers
Thank you for catching this. For Table 2, we re-evaluated the publicly released UnSAM 0.4M model with our evaluation pipeline, which yields a slightly higher ADE20K AR than the published value (39.1 vs. 37.5). For SA-1B, SOHES and UnSAM evaluate on randomly sampled validation splits that were not released. We therefore evaluate the released UnSAM 0.4M model on our own randomly sampled split, which gives a marginally different result (44.8 vs. 44.5). In both cases, our numbers for UnSAM are slightly higher than the published ones, so our evaluation favors the baseline. In the revision, we will mark these results as re-evaluated.
W2b & Q2: CutLER in Table 1 and Table 2
Thank you for pointing this out. Table 1 reports our own reproduction of CutLER, to evaluate consistently across our wider range of benchmarks. Table 2 instead reports the values published in SOHES and UnSAM, for consistency with prior work. In the revision, we will mark the Table 1 numbers as reproduced and state this distinction explicitly.
W2c & Q2: CutLER vs. UnSAM pseudo-labels
Thank you for this careful observation. UnSAM's label pool indeed contains CutLER masks, but only those with a confidence above 0.1. In contrast, the CutLER row in Table 1 applies no threshold. For a consistent comparison, we added a table below, where both are compared at the same threshold. At 0.1, UnSAM consistently outperforms CutLER (+3.4 to +8.1 AR1000), confirming that it adds useful masks to the CutLER pool. The higher recall of unthresholded CutLER comes from its many low-confidence predictions (below 0.1) that still match ground-truth masks. Regarding tuning, we use the hyperparameters of UnSAM's official implementation, except for the confidence threshold, which we lower from 0.3 (official) to 0.1 to improve recall. We will add this table and discussion to the revision.
AR1000 of the pseudo-labels. The confidence threshold is the CutLER detection threshold; for UnSAM, it gates the divide-stage parents.