acceptodds
Withdrawn from ICLR 2027

From Spatial Evidence to Target Understanding: Aligning Diffusion and Grounding Models for Reliable Visual Grounding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual grounding maps an image and a referring query to a bounding box. Currently, text-to-image diffusion models appear well suited to augment this ability because of their dense visual features. However, we find that widely adopted diffusion augmentation can be harmful for visual grounding, which may perform even worse than noisy inputs. To diagnose this problem, we examine spatial evidence in model features, and find that naively generated images lead the model to disperse focus over the whole image, whereas noisy inputs coincidentally align with the grounding model’s target-centered focus, yielding higher performance. This suggests that only aligned information from diffusion models benefits visual grounding. Motivated by this finding, we propose DiReCT, a diffusion-supervised framework that strengthens vision-grounding alignment from both the diffusion and grounding sides. It employs a frozen diffusion model as a teacher with a residual-feature mechanism to extract better-aligned features, and a grounding model as a student that uses paired valid and counterfactual queries to filter out misaligned diffusion-augmented features. The student then learns from the frozen teacher to capture detailed visual information, while only the VLM student is retained at inference. Extensive experiments on GroundingME, FineCops, gRefCOCO, LISA, and RefCOCOg show that DiReCT consistently improves diffusion-augmented visual grounding and outperforms state-of-the-art methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
open-world visual grounding, multimodal large language models, diffusion priors, representation distillation, spatial reasoning, rejection-aware learning, referring expression comprehension

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6ULAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how diffusion models can support visual grounding, including both bounding-box localization and rejection of unsupported referring expressions. It first shows that supplying a query-generated reference image can degrade grounding performance relative to single-image, duplicate-image, and pixel-noise controls. It then proposes DiReCT, which transfers regional spatial relations from a frozen, image-anchored diffusion teacher to a vision-language student. The method combines conditional and residual teacher features, query-gated student features, Gram-matrix alignment, and paired positive/counterfactual objectives for target-existence calibration. The teacher is removed at inference.

Experiments primarily use Qwen3.5-9B and SD3.5 with 5,000 training examples. On GroundingME, full DiReCT improves over supervised fine-tuning by 1.92 points in mIoU and 2.29 points in overall [email protected]. The localization-only variant achieves larger localization gains, while the full method improves rejection accuracy. Additional experiments examine FineCops transfer and report smaller improvements on LISA and RefCOCOg.

Critical strengths

Useful localization gains with a training-only teacher. DiReCT-Loc improves GroundingME mIoU from 39.51 to 41.84 and overall [email protected] from 41.89 to 44.58 relative to SFT. The shuffled-teacher control performs substantially worse than the correctly aligned teacher, supporting the importance of corresponding teacher information. Removing the teacher at inference makes this a practically attractive direction.

Explicit evaluation of localization–rejection trade-offs. The paper distinguishes forced-box localization from standard bbox-or-null evaluation and separately ablates feature and output pairing. This exposes trade-offs that aggregate metrics could conceal. In particular, Table 5 transparently shows that feature pairing alone does not improve rejection, whereas output pairing increases rejection at a localization cost.

Critical weaknesses
  1. The proposed explanation of diffusion-augmentation failure is not adequately established. Table 1 measures the behavior of a VLM receiving a second image, whereas the GT-lift analysis measures activation magnitudes inside a diffusion model. Higher diffusion-feature concentration within a ground-truth box does not by itself demonstrate that the VLM uses that information or explain why its predictions change. Moreover, a pixel-noise reference image and a diffusion latent at the pure-noise endpoint are different controls, but the narrative treats them as closely interchangeable. Figure 2’s aggregate pure-noise response is approximately uniform, and Table 9 reports correct GT lifts of 0.998 and 0.984 for pure-noise conditional and residual features. These results do not substantiate a general target-centered advantage of noise. The initial performance observation is interesting, but the central mechanistic interpretation is stronger than the evidence supports.
  2. The empirical evidence does not yet justify the full method or the breadth of the performance claims. Full DiReCT underperforms DiReCT-Loc on GroundingME overall mIoU and [email protected]. Feature pairing alone reduces mIoU without improving rejection, and adding it to output pairing recovers only part of the localization loss. Full-method transfer gains over SFT are also small: FineCops forced-box mIoU increases from 74.06 to 74.12, RefCOCOg from 82.50 to 82.56, and FineCops balanced accuracy from 71.21 to 71.92. Confidence intervals for these differences and variation across training seeds are not reported. Furthermore, the supplied manuscript does not present the stated gRefCOCO no-target evaluation results, and Table 2 includes a published baseline with higher overall accuracy than either proposed variant. Thus, improvements over the controlled SFT baseline are supported, but broad state-of-the-art claims and the necessity of the additional objectives are insufficiently demonstrated.
Questions for authors
  1. Diagnostic validity: What evidence connects diffusion-feature GT lift to the VLM’s localization errors in Table 1? Please clarify the distinction between the pixel-noise image control and the pure-noise latent control, and reconcile the target-centered-noise explanation with Figure 2 and Table 9.
  2. Statistical reliability and model selection: Can you provide paired confidence intervals for the principal improvements and variability across training seeds? Which held-out data were used to select the SD-GRAM weight, checkpoint, and output-pair stopping point? Appendix A.3 appears to motivate the selected weight using GroundingME results, so clarification is important for interpreting GroundingME as an out-of-domain evaluation.
  3. Query-gate mechanism: Equation 8 rescales individual token vectors, while Equation 10 normalizes pooled spatial vectors. Under what crop-and-pool conditions does the query-dependent scaling survive normalization? Please specify the relevant spatial resolutions, pooling operation, and gate parameters so that the contribution of this mechanism can be assessed.
  4. Complete evaluation and comparison: Can you provide the stated gRefCOCO no-target results, the standard bbox-or-null results for LISA and RefCOCOg, and the precise balanced-accuracy definition? Which comparisons support the state-of-the-art claim, and is there a matched comparison with an existing diffusion-feature distillation approach?
Reviewer_AQnUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

DIRECT takes an image and a referring expression and predicts a bounding box or null, evaluated through localization and rejection metrics. Its motivation is that diffusion-generated references can match the query semantics while disrupting the scene geometry needed for grounding. Training uses 5K positive and negative examples from FineCops and gRefCOCO. First, a frozen SD3.5 teacher extracts query-conditioned and query-free features from a noisy latent of the original image. Conditional and residual features are cropped around annotated target regions and cached as relational supervision. Next, Qwen3.5-9B constructs query-gated visual features and matches their regional Gram matrices to the teacher's. Answer-token supervision and paired valid/counterfactual objectives train localization and target-existence calibration, with output pairing applied early in training. At inference, the VLM alone generates the box or null. Overall, the reported evidence shows a GroundingME improvement with limited positive-localization transfer gains.

Critical strengths
  1. The paper explicitly distinguishes localization from target-existence decisions. Reporting both forced-box and box-or-null results helps reveal when improved rejection comes at the expense of localization.

  2. Same-backbone SFT, shuffled-teacher, and component controls provide useful evidence about the proposed training design. The reported GroundingME gain is positive, although its generality and practical significance remain uncertain.

Critical weaknesses
  1. I find the methodological novelty and practical promise insufficient. Diffusion supervision and relation-matrix distillation are established ideas; the contribution is primarily a task-specific combination with regional targets and existence calibration. Even if this combination is effective, the current comparisons do not establish a substantial conceptual or empirical advance. Full DIRECT improves forced-box mIoU over SFT by only 0.06 on FineCops (74.06 to 74.12), 0.70 on LISA (65.09 to 65.79), and 0.06 on RefCOCOg (82.50 to 82.56). These limited transfer gains provide weak support for diffusion supervision as a promising grounding direction. The paper also omits teacher-extraction GPU-hours, feature-storage requirements, and total training overhead. Retaining only the VLM at inference does not establish that this additional training machinery is worthwhile relative to simpler alternatives.

  2. The empirical scope is too restricted for the broad grounding and state-of-the-art claims. Training uses only 5K examples, approximately one-sixteenth of the 80,512 training expressions in the RefCOCOg UMD split, without source or data-size controls showing that the benefit persists at a conventional larger grounding scale. FineCops negatives are relevant, but the specific selection and mixture remain insufficiently justified. RefCOCO/RefCOCO+ results are absent, and the announced gRefCOCO no-target results are not reported in a verifiable table. External mIoU/mAcc are unavailable, leaving those comparisons confined to one backbone; overall accuracy is also below the listed Qwen3-VL-A22B result. Final-model success/failure comparisons are missing despite poor small-object retention (Base 40.67 versus DIRECT 11.33). Figure 1's small labels and unclear Ours protocol hinder interpretation, while the brief limitations section does not adequately integrate these observed weaknesses.

Questions for authors
  1. What are the GPU-hours, peak memory, cached-feature size, and total training cost, including teacher extraction? Does the advantage persist against a strong non-diffusion grounding or distillation alternative under comparable data/compute budgets, and are the small forced-box gains statistically reliable?

  2. Why were precisely 5K examples and these source proportions selected? Please specify sampling and valid/counterfactual pairing and provide a data-size or source comparison establishing whether the gains persist beyond this small mixture, toward the roughly 80K-expression training scale.

  3. Why was frozen SD3.5 Medium selected as the teacher? What property makes its features particularly suitable for grounding, and how sensitive is the result to another suitable teacher?

  4. Please report RefCOCO/RefCOCO+ and the announced gRefCOCO no-target results, with representative strong grounding baselines on shared protocols. A box-based ORES evaluation would also help test richer referring expressions: convert target masks to boxes and explicitly define the evaluated subset and handling of visual references and multiple targets.

  5. Can representative open-source baselines be evaluated with the same mIoU/mAcc and coordinate parser? Please also identify the model, training state, and subset behind Figure 1's Ours bars and provide final SFT-versus-DIRECT predictions covering false rejection, unsupported-query acceptance, and small-object failures.

Reviewer_XMvrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates why query-generated diffusion references can reduce visual grounding performance despite their semantic relevance. It introduces DiReCT, a diffusion-supervised framework that transfers regional relations from a frozen, image-anchored teacher to a vision-language student. Conditional and residual teacher features supervise a contextual query-gated spatial representation through crop-and-pool Gram alignment.

Matched positive and counterfactual queries additionally support feature-space separation and early bbox-null output calibration. The framework distinguishes DiReCT-Loc, which emphasizes localization, from full DiReCT, which incorporates both paired objectives. Evaluation considers localization and rejection under complementary protocols, while inference retains only the trained student.

Critical strengths

The duplicate-image and pixel-noise controls make the generated-reference comparison more informative than a single-image baseline alone. By matching the additional image-token budget, the experiment in Table 1 supports the observation that semantically relevant references can still be detrimental under the tested input configuration.

Aligning regional spatial relations provides a plausible interface between heterogeneous teacher and student representations. The crop-and-pool Gram alignment in Equation (10) avoids requiring equal channel dimensions and focuses supervision on relational structure rather than dimension-wise feature correspondence.

Separating localization from rejection helps expose the operating trade-offs introduced by the paired objectives. Table 5 reports both the adverse standalone effect of feature pairing and its partial recovery of localization when combined with output pairing, providing a comparatively transparent account of component behavior.

Critical weaknesses

The proposed explanation for the noise advantage appears stronger than the diagnostic evidence establishes. Section 3.3 measures response concentration in diffusion hidden states, whereas the motivating augmentation experiment concerns predictions from a grounding model receiving an additional image. The link between teacher-feature concentration and the grounding model’s use of that input remains indirect, including the attribution of the noise advantage to target-centered spatial alignment.

The contribution of feature pairing to target-existence calibration appears conditional on its interaction with the output objective. In Table 5, feature pairing alone leaves rejection unchanged and reduces overall mIoU, while its addition to output pairing recovers some localization without further increasing rejection. The manuscript acknowledges this pattern, but the evidence supports a narrower role than an independently effective existence-calibration mechanism.

Evidence for transfer to LISA and RefCOCOg is more limited than the broader reliable-grounding claim may suggest. Section 5.4 reports forced-box improvements on these target-present datasets but does not provide corresponding standard bbox-or-null results or uncertainty estimates. These observations support a restricted localization claim, while the effect on abstention and the stability of the small gains remain uncertain.

The general claim of outperforming state-of-the-art methods appears broader than the reported comparison supports. Table 2 includes a published model with higher overall [email protected] than either proposed variant, although the proposed variants have stronger rejection accuracy. The evidence therefore supports improvements over the controlled SFT baseline and a different localization-rejection balance more directly than uniform superiority over the listed methods.

The manuscript’s description of the full training configuration appears insufficient for independent reproduction without the supplementary implementation. Although Appendix A.3 specifies several teacher-feature settings, the student layer, query-gate parameters, paired-loss weights and margins, and output-pair stopping update remain unspecified in the supplied PDF. The construction of matched positive-counterfactual pairs, including the treatment of no-target training samples, is also not fully explicit.

Questions for authors

See the Weaknesses Above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.