acceptodds
Under review as a conference paper at ICLR 2027

AnchorDiff: Training-Free Concept Grounding for MM-DiTs via Anchor-Based Graph Propagation

Abstract

Multi-Modal Diffusion Transformers (MM-DiTs) encode rich representations for training-free concept grounding, but existing attention-based methods often produce overlapping activations on visually confusable concepts—a failure mode we call concept leakage—where target responses spill over to non-target objects. To address this issue, we propose AnchorDiff, a training-free grounding method that decouples semantic localization from structural refinement. AnchorDiff selects a high-confidence anchor from concept-to-image attention map and propagates it as a one-hot seed over a hybrid graph derived from image-to-image self-attention. The graph uses output-space similarity for dense within-object propagation and a row-wise attention gate to suppress cross-object connections. Additionally, we introduce the Multi-Concept Confusion Dataset, which contains images with multiple visually similar concepts and separate masks, enabling explicit evaluation of concept leakage. Experiments show that AnchorDiff achieves strong grounding performance on ImageNet-Segmentation and PascalVOC, while substantially reducing concept leakage on our Multi-Concept Confusion Dataset.

Primary area:
generative models
Keywords:
Diffusion Model, Open-Vocabulary Semantic Segmentation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zqrnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

see weakness.

Critical strengths

see weakness.

Critical weaknesses

The paper addresses an interesting problem, namely concept leakage in training-free semantic grounding with MM-DiT models, but the literature coverage, technical novelty, empirical depth, and presentation are insufficient for ICLR.

  1. The literature review is incomplete and shows insufficient familiarity with the visual grounding literature.
    The paper discusses diffusion-based attention grounding and a few interpretability baselines, but it omits many relevant visual grounding and open-vocabulary segmentation works, including region-based grounding, prompt-driven segmentation, and training-free localization methods. This weakens the claimed gap and makes it difficult to judge whether anchor propagation over attention graphs is genuinely new.

  2. The technical novelty is limited and the core idea is common.
    Selecting a high-confidence attention anchor and propagating it over a self-attention affinity graph is a natural combination of attention rollout, seed propagation, and graph-based refinement, and the paper does not provide a strong argument for why this design is fundamentally different from existing training-free grounding pipelines. The output-space cosine affinity and attention gate are engineering choices rather than a new formulation, so the contribution reads as incremental.

  3. The empirical workload is not solid and feels rushed.
    The evaluation focuses on two standard segmentation benchmarks and a newly constructed Multi-Concept Confusion Dataset, but the new dataset is largely synthetic and relies on Qwen2.5-VL and SAM2 for localization and masks with only manual inspection. The paper reports no multiple seed variance, no significance tests, and no comparison with several strong training-free grounding or open-vocabulary segmentation baselines, so the gains are not fully supported.

  4. The method analysis and ablation are not sufficiently rigorous.
    The ablations vary layers, propagation steps, and affinity retention, but they do not isolate the effect of the anchor selection quality, the attention gate, and the cosine weighting under matched parameter and compute budgets. The paper also acknowledges that a single anchor can miss spatially separated instances, yet the main experiments do not systematically evaluate multi-instance coverage or anchor failure cases.

  5. The presentation is poor and key details are difficult to verify.
    The manuscript contains long line number blocks, inconsistent figure and table placement, and several dense tables that are hard to parse. Important implementation details such as timestep selection, layer choice, thresholding, and dataset construction are scattered across the main text and appendix, which makes the paper feel hastily prepared and reduces confidence in reproducibility.

Questions for authors

see weakness.

5 comments
AuthorsOpenReview ↗

Response to Weakness1

We thank the reviewer for this suggestion. We agree that our original related work focused too narrowly on diffusion-based grounding. We have expanded Section 2 to cover the three research directions highlighted by the reviewer and clarify how AnchorDiff differs from existing approaches.

Region-based grounding and prompt-driven segmentation. MDETR, GLIP, and Grounding DINO learn phrase-to-region alignment using large-scale detection and grounding supervision. SAM/SAM 2 support prompt-based segmentation with points and boxes, while SEEM and LISA extend segmentation to more flexible visual or language instructions. These methods rely on separately trained grounding or segmentation components. In contrast, AnchorDiff extracts concept-specific masks directly from the attention of a frozen text-to-image MM-DiT, without task-specific grounding supervision or a trained segmentation decoder. We have discussed these representative works to better position our method within the broader visual grounding literature.

CLIP/ViT-based interpretability, localization, and segmentation. We have expanded Section 2 to discuss methods that exploit pretrained visual representations through attention-based interpretation, feature decomposition, or spatial refinement, including Rollout, TransInterp, TextSpan, and CLIPasRNN. We also discuss DINO, DINOv2, and iBOT, which learn patch-level visual representations that can be used for object localization and segmentation. These methods are included among the baselines in Table 3. On the Multi-Concept Confusion Dataset, CLIPasRNN obtains a NAR of 47.20, compared with 12.20 for AnchorDiff SD3.5 (Table 4), highlighting the remaining challenge of separating visually similar concepts.

Training-free diffusion segmentation and propagation. We now additionally discuss DiffSeg, Diffusing-Walking-Cutting (DWC), M2N2, iSeg, and SeeDiff in Section 2. DiffSeg merges attention maps using KL divergence, while DWC constructs self-attention-derived affinities for class-agnostic partitioning with Normalized Cuts. M2N2 constructs Markov maps through propagation over self-attention and uses them for point-prompted interactive segmentation. iSeg iteratively refines cross-attention responses using entropy-reduced self-attention, while SeeDiff spreads cross-attention-derived seeds through multi-scale self-attention. In contrast, AnchorDiff propagates from a single concept-specific anchor over a structurally gated graph in MM-DiTs. A sparse row-wise attention gate suppresses connections across visually similar objects, while output-space affinity reweights the retained edges to support within-object propagation and reduce concept leakage. We also adapted all five methods to shared SD3.5 backbone for controlled comparison, as detailed in Table B of our response to W2. With the same anchor, M2N2 achieves 44.09 mIoU_fg and 44.80 NAR, compared with our 68.89 and 12.20. Although iSeg achieves competitive mIoU_fg (66.52), its NAR remains substantially higher (34.84), while SeeDiff obtains lower NAR (11.42) at the cost of considerably lower mIoU_fg (50.68). These results support the effectiveness of our graph construction in balancing target coverage and concept leakage.

We have expanded the related work section to include these studies and incorporate the additional comparisons into the revision.

AuthorsOpenReview ↗

Response to Weakness2

We thank the reviewer for this comment. While prior work has explored seed propagation and graph-based refinement, AnchorDiff addresses a distinct challenge: concept leakage among visually confusable objects in MM-DiTs. Our key distinction lies in suppressing leakage at both semantic initialization and graph propagation, rather than simply combining existing techniques. Our original submission already includes comparisons with representative training-free grounding methods (Tables 3 and 4). Following the reviewer's suggestion, we further extend these comparisons to DiffSeg, DWC, M2N2, SeeDiff, and iSeg to clarify the technical differences and demonstrate the effectiveness of our design.

1. Fundamental Differences

Sparse anchor vs. dense initialization (iSeg, SeeDiff). iSeg starts from a dense CA heatmap, while SeeDiff uses thresholded multi-pixel seeds. In visually confusable scenes, these initializations may already activate similar non-target objects. AnchorDiff instead retains a single high-confidence CA anchor, reducing leakage at initialization.

Structurally gated propagation vs. raw SA (M2N2). M2N2 propagates a one-hot seed over processed raw self-attention. However, raw SA in MM-DiTs is highly local (Fig. 3), and repeated propagation can spread responses across object boundaries. AnchorDiff instead uses row-wise attention similarity to restrict propagation to structurally consistent connections, suppressing cross-object leakage.

Target-specific propagation vs. class-agnostic partitioning (DiffSeg, DWC). DiffSeg merges attention maps using KL divergence, while DWC uses self-attention-derived affinities for graph partitioning. Both partition the image without a target-specific seed. In contrast, AnchorDiff uses row-wise similarity as a sparse binary edge selector to restrict propagation from a concept-specific anchor and suppress connections across visually similar objects.

2. Controlled Comparisons

We ran the official implementations of DiffSeg, SeeDiff, M2N2, and iSeg on their original backbones (Table A), then adapted all five methods to SD3.5-Large using the same timestep and layers with our method (Table B). M2N2 uses our anchor instead of a user click, while DiffSeg and DWC select the segment containing that anchor. Since DWC has no public code, we implemented it following the paper.

Table A: Original backbones

MethodmIoU_fg ↑mAP_fg ↑NAR ↓Acc_fg ↑
DiffSeg31.1646.0252.5850.33
SeeDiff38.5049.1951.1452.05
M2N247.2482.2148.4568.76
iSeg66.4485.0743.2185.02

Table B: Shared SD3.5 features

MethodmIoU_fg ↑mAP_fg ↑NAR ↓Acc_fg ↑
DiffSeg43.5074.3319.2276.41
M2N244.0983.2944.8072.41
SeeDiff50.6880.0411.4275.87
DWC61.3877.6129.6582.64
iSeg66.5288.4634.8485.78
AnchorDiff68.8989.3812.2090.02

With the same anchor, M2N2 achieves only 44.09 mIoU_fg and 44.80 NAR, highlighting the importance of our graph construction. iSeg achieves competitive mIoU_fg (66.52) but nearly three times our NAR (34.84 vs. 12.20). SeeDiff obtains slightly lower NAR (11.42) but substantially lower mIoU_fg (50.68), while DWC and DiffSeg also trail in target coverage. These results demonstrate that AnchorDiff achieves a better balance between target coverage and concept leakage.

For ease of review, we provide qualitative comparisons between our AnchorDiff and the additional 5 methods in (https://anonymous.4open.science/r/supplementary-figure-7-0D57/figure7.pdf).

3. Motivation for the Hybrid Graph

The combination of WcosW_{\mathrm{cos}} and GattnG_{\mathrm{attn}} is motivated by two distinct properties of MM-DiT self-attention (Section 3). Raw AI→IA_{I\to I} is highly local (Fig. 3), while output-space affinity WcosW_{\mathrm{cos}} provides long-range within-object similarities but also assigns high affinity to visually confusable objects (79.40% for same-object pairs vs. 67.27% for confusable cross-object pairs, Table 1). In contrast, row-wise attention similarity better distinguishes object boundaries (21.83% vs. 4.65%). We therefore use GattnG_{\mathrm{attn}} to retain structurally consistent connections and WcosW_{\mathrm{cos}} to weight these connections for effective within-object propagation. This design addresses the limitations of both signals rather than simply combining them. The component ablations in Table 14 support this motivation. On Multi-Concept, WcosW_{\mathrm{cos}} alone achieves only 11.89 mIoU_fg, while GattnG_{\mathrm{attn}} alone reaches 66.71 and their combination achieves 68.89. This confirms that the structural gate provides the primary improvement, while output-space weighting further refines within-object propagation and improves propagation efficiency, as shown in Table 15.

AuthorsOpenReview ↗

Response to Weakness3

We thank the reviewer for these suggestions. Below, we provide additional evidence on dataset reliability, multi-seed stability, and comparisons with recent training-free methods.

1. Dataset Design. Public segmentation benchmarks rarely contain two or three visually confusable concepts co-occurring in a single image with separate per-concept masks, and collecting such examples at scale from web images is challenging. To build a sufficiently large and diverse dataset, we collected 51 real images from the web and supplemented them with 1,221 synthetic images: 340 generated by FLUX.1-dev, 583 by Seedream 3.0, and 298 by Seedream 4.0. The resulting dataset contains 1,272 images in total. These generators are distinct from our evaluated SD3 and SD3.5 backbones, enabling systematic evaluation of concept confusion while reducing reliance on any single generator and avoiding direct self-evaluation. Every mask generated via Qwen2.5-VL and SAM2 was manually inspected to check annotation quality. Moreover, our results are also supported by evaluations on real images with human annotations: ImageNet-Segmentation (4,276 images), PascalVOC Single-Class (928 images), and PascalVOC Multi-Class (518 images). AnchorDiff achieves competitive performance across metrics (Tables 3 and 11).

2. Multi-Seed Stability AnchorDiff is training-free, with randomness arising from Gaussian noise sampled at the feature-extraction timestep. We reran SD3.5 using five independent random seeds across all three benchmarks:

  • ImageNet-Segmentation: 77.28±0.13%77.28 \pm 0.13\% mIoU (paper: 77.29%77.29\%)
  • PascalVOC Single-Class: 81.59±0.05%81.59 \pm 0.05\% mIoU (paper: 81.53%81.53\%)
  • Multi-Concept Confusion: 68.76±0.19%68.76 \pm 0.19\% mIoU_fg and 12.23±0.23%12.23 \pm 0.23\% NAR (paper: 68.89%68.89\% and 12.20%12.20\%)

These results show small variation across random seeds, with the five-seed mean mIoU remaining higher than that of the strongest baseline on each dataset, including the newly added methods. We have included these multi-seed statistics in Appendix A.11 (Table 19).

3. Additional training-free baselines. Our original submission already compares AnchorDiff with 17 representative training-free methods (Table 3), covering CLIP/ViT interpretability, self-supervised ViT attention, CLIP-based segmentation, and diffusion grounding. Following the reviewer's suggestion, we further evaluated five recent methods: DiffSeg, DWC, M2N2, SeeDiff, and iSeg (see Tables A and B in our response to W2). We tested four methods on their original backbones using official implementations and adapted all five to shared SD3.5 features for controlled comparison. AnchorDiff achieves the highest mIoU_fg (68.89), mAP_fg (89.38), and Acc_fg (90.02) among these adapted baselines. Even the strongest new baseline, iSeg (66.44% mIoU_fg on its native backbone and 66.52% on shared SD3.5 features), exhibits substantially higher concept leakage (43.21% and 34.84% NAR, respectively), with the latter nearly 3×3\times ours (12.20%). It also trails AnchorDiff by 4.24 points in Acc_fg (85.78% vs. 90.02%). These additional comparisons further support the effectiveness of AnchorDiff in separating visually confusable concepts. Finally, we welcome any additional baselines the reviewer would like us to include.

Reviewer_1MVHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AnchorDiff, a training-free method to localize a text concept in MMDits. The authots say that existing dense concept maps (e.g. row cross-attention, Seg4Diff, ConceptAttention) also activate on other objects that look similar. They call this "concept leakage". Their method has two steps: take the location of the maximum of the concept-to-image cross-attention map. Start a one-hot vector at the anchor and propagate it for multiple steps on a graph built from image self-attention. The authors also introduce the Multi-concept confusion dataset, automatically labeled and manually inspected images with multiple similar concepts.

Critical strengths

The method is simple and doesn't need training. The ablations are informative: without the gate performance collapses, and Table 6 addresses the dependence of ConceptAttention on auxiliary background concepts.

The improvement on the confusable-object setting is large, and PascalVOC mIoU gain is substantial.

A multi-concept benchmark with per-concept masks is useful.

Appendix honestly reports a limitation that the lowest same-category instance recall, 67% vs 75–88% for the others.

Critical weaknesses

The paper does not cite the line of work that uses diffusion self-attention for segmentation:

  • DiffSeg (CVPR’24) groups patches by comparing rows of the self-attention matrix with KL divergence. "Diffusing, Walking and Cutting" builds the graph from the cosine or dot product of attention rows, which has the same form as Eq. 1.
  • M2N2 (CVPR'25) starts from a one-hot vector at a pixel and propagates it with SD self-attention as a Markov transition matrix. This is the "one-hot seed + N steps" mechanism.
  • iSeg and SeeDiff use cross-attention to find the object and self-attention to refine or spread it. SeeDiff does it with seeds, like AnchorDiff. Without these, I cannot tell what is new beyond applying known ingredients to MM-DiT. The paper should state the difference clearly.

Overlap of attention between similar subjects has been studied in generation since 2023: CONFORM, Bounded Attention, DeLeaker. None is cited.

The claim of "complementary cues" is stronger than the ablation. In Table 5, removing W_cos reduces mIoU_fg by only 1.5-2.2 points, about 5% of the total gain over the A_{C→I}-only baseline. Appendix Table 14 shows that the gate alone gives 77.25 / 81.34 mIoU on ImageNet-Seg / PascalVOC against 77.29 / 81.53 for the full graph. W_cos mainly makes propagation faster (Table 15) and helps only on the authors' own dataset. The gate is close to the graphs in DiffSeg and "Diffusing, Walking and Cutting". This should be stated in the main text.

The layers were chosen as the best of 28 two-layer pairs on the Multi-Concept dataset (Table 7), and the step count (160) was also chosen on it. There is no separate validation set, so the Table 4 numbers may be optimistic. However, using the same settings on ImageNet-Seg and PascalVOC is reassuring.

In Table 17, a fixed 160-step budget is far from optimal for large objects. For targets above 25% of the image, mIoU_fg is 71.20 at 160 steps and 83.84 at 400 steps. Small targets peak at 120 steps. The step count depends on the object size, and the paper does not explain how to set it per image.

Questions for authors
  1. What is the technical difference between AnchorDiff and M2N2, "Diffusing, Walking and Cutting", DiffSeg and SeeDiff? Please cite them and compare, or adapt one of them to MM-DiT as a baseline.

  2. How does concept leakage differ from the attention overlap studied in CONFORM and Bounded Attention?

  3. Layers, steps and retention ratio were chosen on the Multi-Concept dataset. Do the results stay the same if you choose them on a separate validation split?

  4. Table 14 shows that W_cos adds only +0.04 and +0.20 mIoU on ImageNet-Seg and PascalVOC. Why do you describe the gate and W_cos as complementary? What does W_cos add besides fewer propagation steps?

  5. The anchor is wrong in about 10% of the cases (Table 2). What is the mIoU for anchor-correct and anchor-wrong cases? Is there a confidence score or a fallback?

  6. In A.5, the gain over Seg4Diff is only 0.32 mIoU, but it is 22 points on your dataset. Why does the gain nearly vanish in the multi-class setting? Can you report results on Pascal-Context and COCO-Object?

7 comments
AuthorsOpenReview ↗

Response to Q1 & Weakness 1

We thank the reviewer for pointing us to DiffSeg, Diffusing-Walking-Cutting (DWC), M2N2, SeeDiff and iSeg. We will cite and discuss all five methods. These works share some components with ours, but they target different problems, and our controlled comparisons below show that none of them resolves concept leakage among visually confusable objects. The key difference is not which ingredients are used, but where leakage is stopped: AnchorDiff blocks it at both the seed and the graph, whereas each prior method leaves at least one of these open.

1. Fundamental Differences

Sparse anchor vs. dense or multi-seed initialization (iSeg, SeeDiff). iSeg starts from a dense CA heatmap, and SeeDiff from all pixels with CAc[x]≥αCA_c[x]\ge\alpha. In confusable scenes, CA already fires on similar non-targets, so leakage is present before any refinement: iSeg's Ent-Self can sharpen SA but cannot remove activation already seeded on the wrong object, while SeeDiff's background subtraction suppresses it only by eroding the target. AnchorDiff keeps a single argmax anchor pk∗p_k^*, so propagation starts from the target alone.

Structurally gated propagation vs. raw-SA propagation (M2N2). M2N2 introduced one-hot propagation for click-based interactive segmentation, but it walks over processed raw SA. In MM-DiT, raw AI→IA_{I\to I} is highly local (Fig. 3), so multi-step propagation over it can cover the target while also spreading responses to background and visually confusable objects. GattnG_{attn} keeps only the top-2% structurally consistent pairs, reducing confusable cross-object affinity from 67.27% to 0.04% (Table 1), so this improvement is substantial. Additionally, using the same anchor and 160 steps, raw AI→IA_{I\to I} yields 19.73 mIoU_fg, whereas our graph yields 68.89 (Table 14). It is thus evident that the key to resolving the issue of response leakage is not "one-hot propagation" itself, but rather the graph structure we have constructed.

Target-seeded propagation vs. class-agnostic partitioning (DiffSeg, DWC). DiffSeg merges attention maps via KL divergence, and DWC computes row cosine similarity in the same form as our Eq. 1, but both use it as a dense affinity for partitioning the whole image with no notion of a target. Partitioning tends to merge similar objects into one segment or fragment a single object. We use row similarity in a different role, as a binary edge selector that determines which edges a concept-specific seed may traverse. Sparsity is essential to this role, as NAR rises monotonically from 9.38 to 12.20, 15.23, and 21.56 as ρ\rho increases from 1% to 2%, 3%, and 5%, respectively (Table 16). This trend is consistent with the more severe leakage onto similar objects observed with DWC's dense row-cosine affinity.

2. Comparisons on the Multi-Concept Confusion Dataset

We ran the official code of DiffSeg, SeeDiff, M2N2 and iSeg on their original backbones (Table A), and adapted all five methods to SD3.5-Large using the same timestep and layers as AnchorDiff (Table B). For M2N2, the user click is replaced by our anchor. For the class-agnostic DiffSeg and DWC, we take the segment containing the same anchor. DWC has no public code, so we implemented it following the paper.

Table A: Original backbones

MethodmIoU_fg ↑mAP_fg ↑NAR ↓Acc_fg ↑
DiffSeg31.1646.0252.5850.33
SeeDiff38.5049.1951.1452.05
M2N247.2482.2148.4568.76
iSeg66.4485.0743.2185.02

Table B: Shared SD3.5 features

MethodmIoU_fg ↑mAP_fg ↑NAR ↓Acc_fg ↑
DiffSeg43.5074.3319.2276.41
M2N244.0983.2944.8072.41
SeeDiff50.6880.0411.4275.87
DWC61.3877.6129.6582.64
iSeg66.5288.4634.8485.78
AnchorDiff68.8989.3812.2090.02

3. Analysis

The results confirm each difference above. M2N2, given exactly our anchor, reaches only 44.09 mIoU_fg with 44.80 NAR, 3.7x our leakage, which isolates the graph as the decisive component. iSeg, the strongest baseline, still has nearly 3× our NAR (34.84 vs. 12.20), indicating that its Ent-Self refinement cannot fully remove leakage introduced by dense initialization. SeeDiff attains low NAR (11.42) only by severely under-segmenting the target (-18.21 mIoU_fg, -14.15 Acc_fg), and the updated Fig. 7 shows its responses confined to small facial regions. DWC, despite using the same row-cosine form, leaks onto similar objects (29.65 NAR) and trails us by 7.51 mIoU_fg, while DiffSeg fragments the target (-25.39 mIoU_fg).

AnchorDiff is the only method that combines full target coverage with low leakage, and it achieves the best mIoU_fg, mAP_fg and Acc_fg. We have expanded Section 2 and added the comparisons to Table 4 and Figure 7 in the revision. For ease of review, we provide qualitative comparisons between our AnchorDiff and the additional 5 methods in (https://anonymous.4open.science/r/supplementary-figure-7-0D57/figure7.pdf).

AuthorsOpenReview ↗

Response to Q2

We thank the reviewer for pointing us to CONFORM, Bounded Attention, and DeLeaker. We have cited these works in the Introduction and clarified the distinction between attention leakage during image generation and concept leakage during visual grounding.

CONFORM, Bounded Attention, and DeLeaker address leakage during image generation, where attention overlap between subjects can lead to missing objects, incorrect attributes, or blended appearances. In contrast, our concept leakage concerns grounding in a fixed image. Even when objects are visually distinct and correctly depicted, the response map for a queried concept may still activate similar non-target objects. On our Multi-Concept Confusion Dataset, where each concept corresponds to a distinct annotated object, existing grounding methods still place 19.08–52.58% of their response on non-target objects (NAR, Table 4). Unlike these generation methods that modify latents or attention during denoising, AnchorDiff extracts concept-specific masks from a frozen MM-DiT without intervening in generation. Additionally, our analysis is also consistent with and complementary to these works. As Bounded Attention observes, features of similar subjects can become mixed: in MM-DiT, self-attention output features assign 67.27% affinity to confusable different-object pairs (Table 1). However, we further find that the row-wise structure of the same self-attention still preserves object-discriminative information, and using it as a gate reduces cross-object affinity to 0.04%.

We will clarify in the Introduction that "concept leakage" refers to response spill-over onto non-target objects during grounding, rather than semantic leakage in generated images.

AuthorsOpenReview ↗

Response to Q3

We thank the reviewer for this question. Our additional experiments confirm that the main results remain robust under validation-only hyperparameter selection.

We create three image-level splits of the Multi-Concept Confusion Dataset using fixed random seeds (42, 43, and 44). Each split assigns 20% of the images to validation and 80% to testing, while preserving the proportions of two- and three-concept images. For each split, we use only the validation set to select the eight best-performing single layers and search over their 28 possible pairs, 12 propagation step counts, and four retention ratios, yielding 1,344 configurations in total. We then evaluate the selected configuration once on the test set.

SeedConfigSteps / ρmIoU_fg ↑mAP_fg ↑NAR ↓Acc_fg ↑
42Original160 / 2%69.0689.5712.1390.12
42Val-selected120 / 3%69.4190.5413.6789.90
43Original160 / 2%68.4989.0512.4989.60
43Val-selected40 / 5%69.0689.2913.5988.42
44Original160 / 2%68.7289.1912.1990.04
44Val-selected80 / 3%68.9589.2012.1789.57
MeanOriginal160 / 2%68.76 ± 0.2989.2712.2789.92
MeanVal-selected—69.14 ± 0.2489.6813.1589.30

Three results stand out. First, validation-only selection recovers the original layer pair, L23+L31, in all three splits, indicating that the layer choice is stable across these splits. Second, the reported numbers are not inflated. On test images, the original configuration reaches 68.76 mIoU_fg, essentially identical to the 68.89 reported on the full dataset. Third, re-tuning gives no meaningful gain. Only +0.38 mIoU_fg on average, at the cost of higher leakage (NAR +0.87) and lower Acc_fg (−0.62). All of these differences are far below our margins over baselines, and we therefore keep the original setting.

Although the selected step counts and retention ratios vary, they reflect a similar trade-off: denser graphs generally require fewer propagation steps. Our 160 / 2% setting also lies on this flat ridge, within 0.6 mIoU_fg of the re-selected one on every split. Together with its unchanged transfer to ImageNet-Segmentation and PascalVOC, this shows that AnchorDiff does not depend on tuning to the evaluation set.

Reviewer_8SCpOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies about concept leakage in training-free semantic grounding with multi-modal diffusion transformers, where concept-response maps may activate multiple visually similar objects. Proposes AnchorDiff, which selects a high-confidence semantic anchor from concept-to-image attention and propagates a one-hot activation through an image-attention-derived affinity graph. The graph combines output-space feature similarity with a structural gate obtained from row-wise self-attention similarity. Another main contribution from the paper is that it introduces a Multi-Concept Confusion Dataset, that has images with multiple visually similar concepts and per-concept segmentation masks. Main experiments are with SD3 Medium and SD3.5 Large evaluate the method on ImageNet-Segmentation, PascalVOC, and the proposed dataset.

Critical strengths

An interesting finding about MM-DiT attention: The main strength of this paper is showing that pretrained diffusion transformers already contain useful structural information to distinguish visually similar objects, even though they were not specifically trained for this kind of behaviour. The paper use this with a simple anchor-based graph propagation method, and got strong improvements without training.

Along with this finding, this also introduced a new dataset: Multi-Concept Confusion Dataset to explicitely evaluate the concept leakage. The results across different datasets and backbones, along with many ablations experiments further support this finding.

Critical weaknesses
  1. [IMP] The method always selects an anchor and produces a mask, even when the queried concept may not be present in the image (Equations 7-13). How the method handles absent concepts is unclear.
  2. An incorrect anchor can be selected, and then the graph propagation has no further semantic information to correct it. it would be interesting to understand how the method performs specifically on these cases.
  3. Propagation length or steps, large-object mIoU improves when the steps are increased from 160 to 400. longer propagation hurting smaller area objects, so, 160 is taken as fixed, but, what other measures to be taken so that, the propagation lenght can be left as an undeciding factor.
Questions for authors
  1. How does AnchorDiff handle cases where the queried concept is not present in the image?
  2. How does the method perform when an incorrect semantic anchor is selected? Can it recover from such errors?
  3. Can the number of propagation steps be determined adaptively based on the object, rather than using a fixed 160 steps?
3 comments
AuthorsOpenReview ↗

Response to Q1 & Weakness 1

We thank the reviewer for raising this question. Like existing training-free grounding methods such as ConceptAttention and Seg4Diff, AnchorDiff follows the standard localization setting that assumes the queried concept is present. Its goal is to determine where a concept is, rather than whether it exists. ConceptAttention notes that it may activate the closest matching concept when the queried concept is absent, while Seg4Diff evaluates only classes known to be present in each image.

However, absent concepts can be identified before returning a mask using a lightweight, training-free verification step without modifying AnchorDiff. Following the query rejection approach of CLIPasRNN, we evaluated a post-propagation check on 90 PascalVOC images. Each image was queried with its ground-truth present class, a semantically related absent class, and an unrelated absent class. For example, we queried "motorcycle" on an image containing a bicycle. Even for mismatched queries, AnchorDiff's hybrid graph Wcos⊙GattnW_{\mathrm{cos}}\odot G_{\mathrm{attn}} helps isolate coherent object regions rather than scattered noise. We then score the predicted masks using CLIP based on a fixed COCO-80 reference vocabulary. This distinguishes present from absent concepts with an AUROC of 0.95 overall and 0.93 for semantically related absent concepts, suggesting that absent queries can be effectively identified before returning a mask.

AuthorsOpenReview ↗

Response to Q2 & Weakness 2

We sincerely appreciate this question from the reviewer. On the Multi-Concept Confusion Dataset (SD3.5), queries with incorrect anchors (9.75%) achieve 25.23 mIoU_fg, compared with 73.61 for correct anchors. The ability to recover depends on where the incorrect anchor falls:

Incorrect anchor locationCasesAnchorDiff (mIoU_fg ↑)+ Query competition (mIoU_fg ↑)
Background near the target8947.848.5
Another queried object18814.527.5

Recovery from incorrect anchors. When the anchor falls on the background near the target, typically only one patch away, propagation can still recover much of the target (47.8 mIoU_fg), with 53% of these queries reaching IoU above 0.5. However, when the anchor falls on another visually similar object, propagation tends to segment that object instead. Such errors originate from the semantic response AC→IA_{C\to I} rather than our constructed graph.

Anchor correction via query competition. Concept leakage across objects often cause different queries to produce overlapping masks, providing a GT-free signal for detecting anchor errors (AUROC 0.78). Based on this observation, we introduce a query-competition fallback. When more than half of a query's mask overlaps another query's mask, the two queries try alternative anchors outside each other's masks. We retain the anchor assignment with the higher total concept response. Among the 188 cross-object errors, the fallback correctly reassigns 98 cases, improving mIoU_fg on this 98-case subset from 19.1 to 44.0 (+24.9). Across all incorrect-anchor queries, mIoU_fg improves from 25.23 to 34.25, while correct-anchor queries remain essentially unaffected (73.61 vs. 73.57).

These results show that AnchorDiff can recover from small anchor localization errors through propagation, while the optional fallback partially corrects cross-object anchor errors. We have included this analysis and the optional query-competition fallback in Appendix A.13 (Tables 22 and 23).

AuthorsOpenReview ↗

Response to Q3 & Weakness 3

We thank the reviewer for this point. We have carefully considered and adopted the reviewer's suggestions. In additional experiments, we have evaluated a adaptive stopping rule in place of the fixed step budget. It needs no ground-truth target sizes at inference, no training and no per-dataset step-count tuning.

Adaptive stopping. We only add a check after each propagation step. Let sts_t be the heatmap at step tt. We retain the positive differences between its spatial mean and heatmap values, and normalize the resulting map:

q_t = \frac{b_t}{\sum_i b_t(i)}.$$ We then measure how much this normalized distribution changes in one step: $$r_t = \mathrm{Hellinger}(q_t,q_{t-1}).$$ We stop at the first $t$ satisfying $r_t\le c\bar g$, where $c$ is a shared stopping tolerance and $\bar g$ is the image-specific diffusion scale. While this distribution is still changing appreciably, propagation continues. Once it stabilizes, further propagation may increase leakage, so we stop. **Image-specific scale.** A fixed threshold on the unnormalized distribution change $r_t$ may not generalize across graphs or edge-retention ratios $\rho$, since graph structure and sparsity affect propagation speed. We therefore normalize the distribution change $r_t$ by each image's propagation scale $\bar g$, allowing a shared stopping tolerance $c$ across different graphs. To estimate $\bar g$, we use the real eigenvalues $\lambda$ of the propagation matrix $\hat W=D^{-1}W$, where $W$ is the symmetric graph constructed from attention and output features averaged over the two selected layers. Eigenvalues closer to 1 indicate slower decay during propagation. We compute $\bar g$ by averaging $1-\lambda$ over the $K$ largest non-trivial eigenvalues of $\hat W$. This scale is computed once per image and shared by all queried concepts. We use two constants, $c=0.005$ and $K=80$, selected on a subset of the Multi-Concept Confusion Dataset and kept fixed across all three datasets. **Results (SD3.5).** The rule allocates more steps to larger targets without being told their size: | Target area | n | Fixed 160 step | Best fixed step | **Adaptive** step | Avg. step | |:---:|:---:|:---:|:---:|:---:|:---:| | < 10% | 1,970 | 65.96 | 66.45 (120) | **66.56** | 137 | | [10%, 25%) | 800 | 75.91 | 78.15 (320) | **78.18** | 144 | | ≥ 25% | 72 | 71.20 | 83.84 (400) | **78.89** | 170 | The best fixed step uses ground-truth target sizes to select the best evaluated step count for each size group. Without knowing the target size, our adaptive rule achieves comparable results on small and medium targets and improves large-target mIoU_fg from 71.20 to 78.89, approaching the best fixed-step result of 83.84. | Method | ImageNet-Seg mIoU | PascalVOC mIoU | Multi-Concept mIoU_fg | NAR ↓ | |:---|:---:|:---:|:---:|:---:| | Fixed 160 step | 77.29 | 81.53 | 68.89 | 12.20 | | Adaptive step | 77.26 | 81.12 | **70.14** | **11.52** | On Multi-Concept, mIoU_fg improves by +1.25. On ImageNet-Seg and PascalVOC, the adaptive rule achieves comparable performance without dataset-specific step-count tuning. Computing $\bar g$ adds only ≈0.4 s per image, with the result shared by all queried concepts. We have included the stopping rule and these results in Appendix A.12 (Tables 20 and 21). We thank the reviewer again for this valuable suggestion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.