- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
see weakness.
Critical strengths
see weakness.
Critical weaknesses
The paper addresses an interesting problem, namely concept leakage in training-free semantic grounding with MM-DiT models, but the literature coverage, technical novelty, empirical depth, and presentation are insufficient for ICLR.
-
The literature review is incomplete and shows insufficient familiarity with the visual grounding literature.
The paper discusses diffusion-based attention grounding and a few interpretability baselines, but it omits many relevant visual grounding and open-vocabulary segmentation works, including region-based grounding, prompt-driven segmentation, and training-free localization methods. This weakens the claimed gap and makes it difficult to judge whether anchor propagation over attention graphs is genuinely new. -
The technical novelty is limited and the core idea is common.
Selecting a high-confidence attention anchor and propagating it over a self-attention affinity graph is a natural combination of attention rollout, seed propagation, and graph-based refinement, and the paper does not provide a strong argument for why this design is fundamentally different from existing training-free grounding pipelines. The output-space cosine affinity and attention gate are engineering choices rather than a new formulation, so the contribution reads as incremental. -
The empirical workload is not solid and feels rushed.
The evaluation focuses on two standard segmentation benchmarks and a newly constructed Multi-Concept Confusion Dataset, but the new dataset is largely synthetic and relies on Qwen2.5-VL and SAM2 for localization and masks with only manual inspection. The paper reports no multiple seed variance, no significance tests, and no comparison with several strong training-free grounding or open-vocabulary segmentation baselines, so the gains are not fully supported. -
The method analysis and ablation are not sufficiently rigorous.
The ablations vary layers, propagation steps, and affinity retention, but they do not isolate the effect of the anchor selection quality, the attention gate, and the cosine weighting under matched parameter and compute budgets. The paper also acknowledges that a single anchor can miss spatially separated instances, yet the main experiments do not systematically evaluate multi-instance coverage or anchor failure cases. -
The presentation is poor and key details are difficult to verify.
The manuscript contains long line number blocks, inconsistent figure and table placement, and several dense tables that are hard to parse. Important implementation details such as timestep selection, layer choice, thresholding, and dataset construction are scattered across the main text and appendix, which makes the paper feel hastily prepared and reduces confidence in reproducibility.
Questions for authors
see weakness.
5 comments
Response to Weakness2
We thank the reviewer for this comment. While prior work has explored seed propagation and graph-based refinement, AnchorDiff addresses a distinct challenge: concept leakage among visually confusable objects in MM-DiTs. Our key distinction lies in suppressing leakage at both semantic initialization and graph propagation, rather than simply combining existing techniques. Our original submission already includes comparisons with representative training-free grounding methods (Tables 3 and 4). Following the reviewer's suggestion, we further extend these comparisons to DiffSeg, DWC, M2N2, SeeDiff, and iSeg to clarify the technical differences and demonstrate the effectiveness of our design.
1. Fundamental Differences
Sparse anchor vs. dense initialization (iSeg, SeeDiff). iSeg starts from a dense CA heatmap, while SeeDiff uses thresholded multi-pixel seeds. In visually confusable scenes, these initializations may already activate similar non-target objects. AnchorDiff instead retains a single high-confidence CA anchor, reducing leakage at initialization.
Structurally gated propagation vs. raw SA (M2N2). M2N2 propagates a one-hot seed over processed raw self-attention. However, raw SA in MM-DiTs is highly local (Fig. 3), and repeated propagation can spread responses across object boundaries. AnchorDiff instead uses row-wise attention similarity to restrict propagation to structurally consistent connections, suppressing cross-object leakage.
Target-specific propagation vs. class-agnostic partitioning (DiffSeg, DWC). DiffSeg merges attention maps using KL divergence, while DWC uses self-attention-derived affinities for graph partitioning. Both partition the image without a target-specific seed. In contrast, AnchorDiff uses row-wise similarity as a sparse binary edge selector to restrict propagation from a concept-specific anchor and suppress connections across visually similar objects.
2. Controlled Comparisons
We ran the official implementations of DiffSeg, SeeDiff, M2N2, and iSeg on their original backbones (Table A), then adapted all five methods to SD3.5-Large using the same timestep and layers with our method (Table B). M2N2 uses our anchor instead of a user click, while DiffSeg and DWC select the segment containing that anchor. Since DWC has no public code, we implemented it following the paper.
Table A: Original backbones
| Method | mIoU_fg ↑ | mAP_fg ↑ | NAR ↓ | Acc_fg ↑ |
|---|---|---|---|---|
| DiffSeg | 31.16 | 46.02 | 52.58 | 50.33 |
| SeeDiff | 38.50 | 49.19 | 51.14 | 52.05 |
| M2N2 | 47.24 | 82.21 | 48.45 | 68.76 |
| iSeg | 66.44 | 85.07 | 43.21 | 85.02 |
Table B: Shared SD3.5 features
| Method | mIoU_fg ↑ | mAP_fg ↑ | NAR ↓ | Acc_fg ↑ |
|---|---|---|---|---|
| DiffSeg | 43.50 | 74.33 | 19.22 | 76.41 |
| M2N2 | 44.09 | 83.29 | 44.80 | 72.41 |
| SeeDiff | 50.68 | 80.04 | 11.42 | 75.87 |
| DWC | 61.38 | 77.61 | 29.65 | 82.64 |
| iSeg | 66.52 | 88.46 | 34.84 | 85.78 |
| AnchorDiff | 68.89 | 89.38 | 12.20 | 90.02 |
With the same anchor, M2N2 achieves only 44.09 mIoU_fg and 44.80 NAR, highlighting the importance of our graph construction. iSeg achieves competitive mIoU_fg (66.52) but nearly three times our NAR (34.84 vs. 12.20). SeeDiff obtains slightly lower NAR (11.42) but substantially lower mIoU_fg (50.68), while DWC and DiffSeg also trail in target coverage. These results demonstrate that AnchorDiff achieves a better balance between target coverage and concept leakage.
For ease of review, we provide qualitative comparisons between our AnchorDiff and the additional 5 methods in (https://anonymous.4open.science/r/supplementary-figure-7-0D57/figure7.pdf).
3. Motivation for the Hybrid Graph
The combination of and is motivated by two distinct properties of MM-DiT self-attention (Section 3). Raw is highly local (Fig. 3), while output-space affinity provides long-range within-object similarities but also assigns high affinity to visually confusable objects (79.40% for same-object pairs vs. 67.27% for confusable cross-object pairs, Table 1). In contrast, row-wise attention similarity better distinguishes object boundaries (21.83% vs. 4.65%). We therefore use to retain structurally consistent connections and to weight these connections for effective within-object propagation. This design addresses the limitations of both signals rather than simply combining them. The component ablations in Table 14 support this motivation. On Multi-Concept, alone achieves only 11.89 mIoU_fg, while alone reaches 66.71 and their combination achieves 68.89. This confirms that the structural gate provides the primary improvement, while output-space weighting further refines within-object propagation and improves propagation efficiency, as shown in Table 15.
Response to Weakness3
We thank the reviewer for these suggestions. Below, we provide additional evidence on dataset reliability, multi-seed stability, and comparisons with recent training-free methods.
1. Dataset Design. Public segmentation benchmarks rarely contain two or three visually confusable concepts co-occurring in a single image with separate per-concept masks, and collecting such examples at scale from web images is challenging. To build a sufficiently large and diverse dataset, we collected 51 real images from the web and supplemented them with 1,221 synthetic images: 340 generated by FLUX.1-dev, 583 by Seedream 3.0, and 298 by Seedream 4.0. The resulting dataset contains 1,272 images in total. These generators are distinct from our evaluated SD3 and SD3.5 backbones, enabling systematic evaluation of concept confusion while reducing reliance on any single generator and avoiding direct self-evaluation. Every mask generated via Qwen2.5-VL and SAM2 was manually inspected to check annotation quality. Moreover, our results are also supported by evaluations on real images with human annotations: ImageNet-Segmentation (4,276 images), PascalVOC Single-Class (928 images), and PascalVOC Multi-Class (518 images). AnchorDiff achieves competitive performance across metrics (Tables 3 and 11).
2. Multi-Seed Stability AnchorDiff is training-free, with randomness arising from Gaussian noise sampled at the feature-extraction timestep. We reran SD3.5 using five independent random seeds across all three benchmarks:
- ImageNet-Segmentation: mIoU (paper: )
- PascalVOC Single-Class: mIoU (paper: )
- Multi-Concept Confusion: mIoU_fg and NAR (paper: and )
These results show small variation across random seeds, with the five-seed mean mIoU remaining higher than that of the strongest baseline on each dataset, including the newly added methods. We have included these multi-seed statistics in Appendix A.11 (Table 19).
3. Additional training-free baselines. Our original submission already compares AnchorDiff with 17 representative training-free methods (Table 3), covering CLIP/ViT interpretability, self-supervised ViT attention, CLIP-based segmentation, and diffusion grounding. Following the reviewer's suggestion, we further evaluated five recent methods: DiffSeg, DWC, M2N2, SeeDiff, and iSeg (see Tables A and B in our response to W2). We tested four methods on their original backbones using official implementations and adapted all five to shared SD3.5 features for controlled comparison. AnchorDiff achieves the highest mIoU_fg (68.89), mAP_fg (89.38), and Acc_fg (90.02) among these adapted baselines. Even the strongest new baseline, iSeg (66.44% mIoU_fg on its native backbone and 66.52% on shared SD3.5 features), exhibits substantially higher concept leakage (43.21% and 34.84% NAR, respectively), with the latter nearly ours (12.20%). It also trails AnchorDiff by 4.24 points in Acc_fg (85.78% vs. 90.02%). These additional comparisons further support the effectiveness of AnchorDiff in separating visually confusable concepts. Finally, we welcome any additional baselines the reviewer would like us to include.
Response to Weakness1
We thank the reviewer for this suggestion. We agree that our original related work focused too narrowly on diffusion-based grounding. We have expanded Section 2 to cover the three research directions highlighted by the reviewer and clarify how AnchorDiff differs from existing approaches.
Region-based grounding and prompt-driven segmentation. MDETR, GLIP, and Grounding DINO learn phrase-to-region alignment using large-scale detection and grounding supervision. SAM/SAM 2 support prompt-based segmentation with points and boxes, while SEEM and LISA extend segmentation to more flexible visual or language instructions. These methods rely on separately trained grounding or segmentation components. In contrast, AnchorDiff extracts concept-specific masks directly from the attention of a frozen text-to-image MM-DiT, without task-specific grounding supervision or a trained segmentation decoder. We have discussed these representative works to better position our method within the broader visual grounding literature.
CLIP/ViT-based interpretability, localization, and segmentation. We have expanded Section 2 to discuss methods that exploit pretrained visual representations through attention-based interpretation, feature decomposition, or spatial refinement, including Rollout, TransInterp, TextSpan, and CLIPasRNN. We also discuss DINO, DINOv2, and iBOT, which learn patch-level visual representations that can be used for object localization and segmentation. These methods are included among the baselines in Table 3. On the Multi-Concept Confusion Dataset, CLIPasRNN obtains a NAR of 47.20, compared with 12.20 for AnchorDiff SD3.5 (Table 4), highlighting the remaining challenge of separating visually similar concepts.
Training-free diffusion segmentation and propagation. We now additionally discuss DiffSeg, Diffusing-Walking-Cutting (DWC), M2N2, iSeg, and SeeDiff in Section 2. DiffSeg merges attention maps using KL divergence, while DWC constructs self-attention-derived affinities for class-agnostic partitioning with Normalized Cuts. M2N2 constructs Markov maps through propagation over self-attention and uses them for point-prompted interactive segmentation. iSeg iteratively refines cross-attention responses using entropy-reduced self-attention, while SeeDiff spreads cross-attention-derived seeds through multi-scale self-attention. In contrast, AnchorDiff propagates from a single concept-specific anchor over a structurally gated graph in MM-DiTs. A sparse row-wise attention gate suppresses connections across visually similar objects, while output-space affinity reweights the retained edges to support within-object propagation and reduce concept leakage. We also adapted all five methods to shared SD3.5 backbone for controlled comparison, as detailed in Table B of our response to W2. With the same anchor, M2N2 achieves 44.09 mIoU_fg and 44.80 NAR, compared with our 68.89 and 12.20. Although iSeg achieves competitive mIoU_fg (66.52), its NAR remains substantially higher (34.84), while SeeDiff obtains lower NAR (11.42) at the cost of considerably lower mIoU_fg (50.68). These results support the effectiveness of our graph construction in balancing target coverage and concept leakage.
We have expanded the related work section to include these studies and incorporate the additional comparisons into the revision.