- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GroundAnything, a blockwise diffusion model designed to reduce the inference latency associated with sequential autoregressive (AR) generation in visual grounding. The model uses bidirectional context to predict multiple tokens in parallel within each output block while retaining sequential dependencies across blocks. Training combines AR prediction and masked-token reconstruction objectives and includes supervised fine-tuning and GRPO-based post-training. The model uses a shared output format for object detection, referring expression grounding, point prediction, OCR, GUI grounding, and document layout recognition, and is evaluated on 30 benchmarks. The paper also distinguishes pure diffusion generation from an optional inference mode that verifies diffusion-generated drafts autoregressively, and analyzes the speed–performance trade-offs of different decoding strategies on COCO.
Critical strengths
-
The paper integrates diverse tasks, including object detection, point prediction, OCR, GUI grounding, and document layout recognition, into a single model with a shared output format. Evaluation spans 30 benchmarks, with competitive results in dense object detection, referring expression grounding, and GUI grounding using pure diffusion generation without AR verification. This is significant because it demonstrates that diffusion-based generation can handle multiple types of spatial outputs rather than being restricted to a particular task.
-
The same model weights support both pure diffusion generation and an inference mode that uses AR verification of diffusion-generated drafts. On COCO, the AR-verified configuration increases output generation speed while keeping the performance loss relative to the AR baseline small. The model supports a choice of inference modes without requiring a separate draft model, and the analysis of how block size and decoding strategy affect speed and performance provides useful evidence for selecting an inference configuration suited to the intended application.
Critical weaknesses
-
The main speed–performance analysis is concentrated on COCO, making it difficult to assess the acceleration benefits for OCR and document layout tasks. The AR and pure diffusion models achieve F1mIoU scores of 71.15 and 43.64 on SROIE, 85.78 and 68.38 on DocLayNet, and 74.76 and 56.69 on M6Doc, respectively, indicating substantial performance gaps. The reductions in processing time associated with these performance losses are not sufficiently quantified, and it remains unclear whether speed gains persist under decoding configurations that reduce the performance losses. Consequently, it is difficult to determine how much acceleration can be achieved across tasks with an acceptable loss in performance.
-
In the draft-quality comparison with bidirectional block-MTP, block-MTP predicts the draft in a single pass, whereas the diffusion model uses two steps: an initial prediction followed by remasking and re-predicting low-confidence positions. Since this comparison changes both the training objective and computational cost, the observed differences alone do not isolate the additional benefits of diffusion. Evidence from comparisons against bidirectional block-MTP at matched wall-clock time is limited, leaving it unclear how much diffusion training and iterative refinement contribute beyond the use of bidirectional context. This limits the evidence supporting the choice of diffusion for parallel visual grounding.
Questions for authors
On the RefCOCOg validation set, the AR and diffusion models achieve F1mIoU scores of 84.37 and 91.61, respectively, for box prediction, and F1@Point scores of 90.44 and 85.75, respectively, for point prediction. Please explain the main reasons why the direction of the performance change relative to AR differs between these two tasks. In particular, please clarify whether the decline in point-prediction performance primarily results from selecting the wrong target or from failing to place the point within the target object's region after correctly identifying it. This analysis is important for assessing which types of spatial prediction benefit from diffusion and where its limitations lie.
5 comments
2. W2: Why Diffusion for Parallel Grounding
Thank you for this incisive observation. The central question is whether diffusion produces more useful grounding predictions per unit time, beyond bidirectional context alone. We have strengthened the evidence through one-pass comparisons, fixed-time measurements, and direct evaluation of iterative refinement (Appendix I.4).
(1) One-pass prediction without additional refinement.
We compare one-pass bidirectional block-MTP with one-step DLM, using the strengthened MTP control described below. The controls share model initialization, grounding data, token interface, and a frozen causal verifier, with matched adaptation examples, supervised-token exposure, and optimization settings.
At K=16, one-step DLM increases acceptance from 20.20% to 21.76% and reduces full-round cost per accepted draft token from 6.10 to 5.38 ms. At K=32, the cost decreases from 8.70 to 6.88 ms (Table S39). The advantage is therefore already observable before adding a second diffusion step.
(2) Useful progress at matched wall-clock time.
Within 200 ms, DLM commits 47.9 tokens, compared with 45.0 for bidirectional block-MTP; within 400 ms, it commits 97.1 versus 89.2 tokens (Table S40). These comparisons directly measure effective progress within the same time budget.
(3) Whether iterative refinement repays its computation.
Diffusion can revisit uncertain proposals before verification. In the direct one-step/two-step comparison at K=16, a second refinement step reduces full-round cost per accepted draft token from 5.38 to 4.70 ms. The common-quality end-to-end results further show:
| Task | Common quality | Bidirectional block-MTP, 1 pass | DLM, 1 step | DLM, 2 steps |
|---|---|---|---|---|
| Dense200 | 74.80 | 2.50 s | 2.30 s | 2.00 s |
| RefCOCOg val | 84.00 | 0.56 s | 0.59 s | 0.70 s |
| SROIE | 70.20 | 2.80 s | 2.90 s | 3.10 s |
| DocLayNet | 85.10 | 3.00 s | 2.60 s | 2.30 s |
| M6Doc | 74.40 | 3.20 s | 3.10 s | 2.80 s |
These are AR-verified drafting configurations; quality is F1mIoU, with loose matching for SROIE. Measurements use H100 80GB, batch size 1, BF16, and a shared backend, including preprocessing, visual encoding, prefill, decoding, and parsing (Figure S30 and Table S42).
On Dense200, DocLayNet, and M6Doc, the second diffusion step further reduces total latency. Two-step DLM lowers latency relative to bidirectional block-MTP by 20.0%, 23.3%, and 12.5%, respectively. Improved verification and commitment can therefore repay the extra denoising computation, while cached context and efficient execution help control its overhead.
To further strengthen the comparison, the MTP results above use Box-aligned bidirectional block-MTP, following LocateAnything’s geometric organization: complete box tuples remain within prediction units, with bidirectional attention inside each unit. This adds an explicit structural prior to the baseline.
Together, the evidence supports diffusion for parallel visual grounding through joint prediction and iterative refinement that can improve useful output per unit time, including against this more structured alternative. We appreciate your suggestion, which helped us substantiate this choice with stronger timing-based evidence.
3. Q1: Box versus Point Prediction
Thank you for this particularly perceptive question. Distinguishing correct target selection from precise point placement is central to understanding these results, and prompted us to inspect nearly 1,000 failure cases more closely.
(1) Box localization and point-in-mask evaluation require different geometric precision.
On RefCOCOg validation, diffusion improves box F1mIoU from 84.37 to 91.61, while point F1@Point decreases from 90.44 to 85.75. A point can lie inside the correct bounding box but outside the foreground mask, especially for thin or irregular objects. Estimating the object’s overall extent therefore does not automatically resolve a valid foreground point.
Our qualitative interpretation highlights fine point placement relative to the mask as an important contributor: the model may localize the approximate target region while still struggling to place the point precisely on its foreground.
(2) Iterative prediction reveals unresolved localization decisions.
We observed that some uncommitted coordinate positions are predicted repeatedly while their entropy remains close to, but slightly above, the commitment threshold, particularly in pointing examples. We hypothesize that the model is still adjusting fine placement—for example, resolving the difference between a point inside the bounding box and one inside the foreground mask.
This connects directly to our motivation for iterative denoising: uncertain spatial decisions can be reconsidered as within-block context develops, before commitment. It also explains how improved box localization can coexist with lower point-in-mask performance: estimating spatial extent and resolving foreground membership are different geometric requirements.
(3) Implications for suitable tasks.
Compact spatial outputs offer opportunities for joint prediction and iterative refinement, especially when a few uncertain coordinates must satisfy coupled constraints. Pointing additionally requires precise foreground membership and is sensitive to small coordinate differences. Longer outputs place greater demands on language modeling, instance organization, and text–geometry consistency across the sequence.
Thus, output length, geometric precision, and output-organization complexity jointly influence diffusion’s behavior. We are especially grateful that your question helped us distinguish coarse target localization from precise foreground placement and sharpen our understanding of iterative grounding.
More broadly, your constructive suggestions have led to stronger experiments and useful directions for model improvement. We apologize that the main-paper page limit has led us to collect the additional results toward the end of the appendix; if accepted, we will consolidate them into a clearer camera-ready presentation.
We welcome further discussion. If additional analyses or experiments that we have not considered would strengthen the paper, please let us know; we will make every effort to complete them and improve the manuscript.
1. W1: OCR/Layout Speed and Accuracy
Thank you for highlighting this important gap in the original evaluation. We have expanded the measurements beyond COCO to include Dense200, RefCOCOg, ICDAR2015, DocLayNet, SROIE, and M6Doc (Appendix I.3, Figures S24–S27 and Tables S32–S37).
(1) Acceleration at the original and higher-quality operating points.
For the original pure-diffusion configurations, end-to-end latency decreases from 7.40 to 3.10 s on SROIE, 5.90 to 2.80 s on DocLayNet, and 6.80 to 3.20 s on M6Doc, yielding 2.39×, 2.11×, and 2.13× acceleration, respectively.
Importantly, acceleration also persists when reducing the accuracy loss:
Quality is F1mIoU, with loose matching for OCR; TPS measures decoding throughput. Thus, pure diffusion achieves 4.24× throughput with a 0.20 pp decrease on ICDAR2015, while verified decoding achieves 6.61×/7.02× throughput with 0.50/0.98 pp decreases on ICDAR2015/DocLayNet. Common-quality end-to-end comparisons are additionally provided in Table S42.
(2) Diagnosing the OCR gap and improving the model.
Your comment also motivated qualitative error analysis. We found cases where the text was largely correct but word/line granularity differed from the annotations—for example, several words were grouped into one line—reducing text–box matching scores.
Approximately 80% of our pretraining mixture consists of box-structured supervision, emphasizing coordinate ordering, geometric consistency, and label–box associations. We hypothesize that denoising becomes particularly adapted to these coupled constraints, while transcription tokens may become confident before their word/line organization is sufficiently consistent.
We therefore increased OCR supervision during pretraining and GRPO and adjusted rewards for word/line granularity and text–geometry agreement, using GRPO for targeted adaptation with the aim of preserving box-grounding capabilities. Appendix I.1, Table S30 reports:
All four OCR benchmarks improve. ICDAR2015 and TotalText exceed the original AR counterpart, while the layout gaps to AR narrow to 1.84 pp on DocLayNet and 2.93 pp on M6Doc.
We are especially grateful for your suggestion, which led to both broader efficiency evaluation and concrete model improvements.