- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates how diffusion models can support visual grounding, including both bounding-box localization and rejection of unsupported referring expressions. It first shows that supplying a query-generated reference image can degrade grounding performance relative to single-image, duplicate-image, and pixel-noise controls. It then proposes DiReCT, which transfers regional spatial relations from a frozen, image-anchored diffusion teacher to a vision-language student. The method combines conditional and residual teacher features, query-gated student features, Gram-matrix alignment, and paired positive/counterfactual objectives for target-existence calibration. The teacher is removed at inference.
Experiments primarily use Qwen3.5-9B and SD3.5 with 5,000 training examples. On GroundingME, full DiReCT improves over supervised fine-tuning by 1.92 points in mIoU and 2.29 points in overall [email protected]. The localization-only variant achieves larger localization gains, while the full method improves rejection accuracy. Additional experiments examine FineCops transfer and report smaller improvements on LISA and RefCOCOg.
Critical strengths
Useful localization gains with a training-only teacher. DiReCT-Loc improves GroundingME mIoU from 39.51 to 41.84 and overall [email protected] from 41.89 to 44.58 relative to SFT. The shuffled-teacher control performs substantially worse than the correctly aligned teacher, supporting the importance of corresponding teacher information. Removing the teacher at inference makes this a practically attractive direction.
Explicit evaluation of localization–rejection trade-offs. The paper distinguishes forced-box localization from standard bbox-or-null evaluation and separately ablates feature and output pairing. This exposes trade-offs that aggregate metrics could conceal. In particular, Table 5 transparently shows that feature pairing alone does not improve rejection, whereas output pairing increases rejection at a localization cost.
Critical weaknesses
- The proposed explanation of diffusion-augmentation failure is not adequately established. Table 1 measures the behavior of a VLM receiving a second image, whereas the GT-lift analysis measures activation magnitudes inside a diffusion model. Higher diffusion-feature concentration within a ground-truth box does not by itself demonstrate that the VLM uses that information or explain why its predictions change. Moreover, a pixel-noise reference image and a diffusion latent at the pure-noise endpoint are different controls, but the narrative treats them as closely interchangeable. Figure 2’s aggregate pure-noise response is approximately uniform, and Table 9 reports correct GT lifts of 0.998 and 0.984 for pure-noise conditional and residual features. These results do not substantiate a general target-centered advantage of noise. The initial performance observation is interesting, but the central mechanistic interpretation is stronger than the evidence supports.
- The empirical evidence does not yet justify the full method or the breadth of the performance claims. Full DiReCT underperforms DiReCT-Loc on GroundingME overall mIoU and [email protected]. Feature pairing alone reduces mIoU without improving rejection, and adding it to output pairing recovers only part of the localization loss. Full-method transfer gains over SFT are also small: FineCops forced-box mIoU increases from 74.06 to 74.12, RefCOCOg from 82.50 to 82.56, and FineCops balanced accuracy from 71.21 to 71.92. Confidence intervals for these differences and variation across training seeds are not reported. Furthermore, the supplied manuscript does not present the stated gRefCOCO no-target evaluation results, and Table 2 includes a published baseline with higher overall accuracy than either proposed variant. Thus, improvements over the controlled SFT baseline are supported, but broad state-of-the-art claims and the necessity of the additional objectives are insufficiently demonstrated.
Questions for authors
- Diagnostic validity: What evidence connects diffusion-feature GT lift to the VLM’s localization errors in Table 1? Please clarify the distinction between the pixel-noise image control and the pure-noise latent control, and reconcile the target-centered-noise explanation with Figure 2 and Table 9.
- Statistical reliability and model selection: Can you provide paired confidence intervals for the principal improvements and variability across training seeds? Which held-out data were used to select the SD-GRAM weight, checkpoint, and output-pair stopping point? Appendix A.3 appears to motivate the selected weight using GroundingME results, so clarification is important for interpreting GroundingME as an out-of-domain evaluation.
- Query-gate mechanism: Equation 8 rescales individual token vectors, while Equation 10 normalizes pooled spatial vectors. Under what crop-and-pool conditions does the query-dependent scaling survive normalization? Please specify the relevant spatial resolutions, pooling operation, and gate parameters so that the contribution of this mechanism can be assessed.
- Complete evaluation and comparison: Can you provide the stated gRefCOCO no-target results, the standard bbox-or-null results for LISA and RefCOCOg, and the precise balanced-accuracy definition? Which comparisons support the state-of-the-art claim, and is there a matched comparison with an existing diffusion-feature distillation approach?