acceptodds
Desk rejected at ICLR 2027

GroundAnything: Reconciling Parallel Decoding with Precise Visual Grounding at Flash Speed

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Autoregressive (AR) grounding models serialize spatial predictions, introducing sequential latency and imposing a causal order on output tokens. We view grounding as visual evidence extraction: objects, locations, and spatial relations are jointly constrained by the image and query, yet their dependencies do not imply an intrinsic left-to-right generation order. This distinction makes bidirectional diffusion a natural fit, allowing spatial hypotheses to emerge in parallel and be jointly refined through iterative denoising. We introduce GroundAnything, a 4B-parameter grounding foundation model that reconciles fast parallel decoding with precise localization through blockwise denoising. Training combines grounding pretraining from public datasets and dedicated data engines, direct AR-to-diffusion conversion with joint AR and diffusion objectives, supervised fine-tuning, and GRPO-based reinforcement post-training. Across 30 grounding benchmarks, our autoregressive variant, GroundAnything-VLM, establishes a new overall state of the art among similarly sized models at 72.42%, remaining competitive with GPT-6 Astra (71.35%). With entropy-guided decoding, GroundAnything also surpasses the prior state of the art at this scale, averaging 61.75% versus 53.32% for the fast MTP-based LocateAnything model. We further explore decoding strategies, showing that an optional self-speculative mode achieves a 4.51 speedup over the AR counterpart with a 0.74 percentage-point drop in COCO F1mIoU. Infrastructure experiments show that progressive inference optimizations translate parallel decoding into practical speedups. These gains support efficient visual grounding in latency-sensitive real-world systems.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Foundation Models, Diffusion Language Models, Vision-Language Models, Parallel Decoding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pYPoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GroundAnything, a blockwise diffusion model designed to reduce the inference latency associated with sequential autoregressive (AR) generation in visual grounding. The model uses bidirectional context to predict multiple tokens in parallel within each output block while retaining sequential dependencies across blocks. Training combines AR prediction and masked-token reconstruction objectives and includes supervised fine-tuning and GRPO-based post-training. The model uses a shared output format for object detection, referring expression grounding, point prediction, OCR, GUI grounding, and document layout recognition, and is evaluated on 30 benchmarks. The paper also distinguishes pure diffusion generation from an optional inference mode that verifies diffusion-generated drafts autoregressively, and analyzes the speed–performance trade-offs of different decoding strategies on COCO.

Critical strengths
  1. The paper integrates diverse tasks, including object detection, point prediction, OCR, GUI grounding, and document layout recognition, into a single model with a shared output format. Evaluation spans 30 benchmarks, with competitive results in dense object detection, referring expression grounding, and GUI grounding using pure diffusion generation without AR verification. This is significant because it demonstrates that diffusion-based generation can handle multiple types of spatial outputs rather than being restricted to a particular task.

  2. The same model weights support both pure diffusion generation and an inference mode that uses AR verification of diffusion-generated drafts. On COCO, the AR-verified configuration increases output generation speed while keeping the performance loss relative to the AR baseline small. The model supports a choice of inference modes without requiring a separate draft model, and the analysis of how block size and decoding strategy affect speed and performance provides useful evidence for selecting an inference configuration suited to the intended application.

Critical weaknesses
  1. The main speed–performance analysis is concentrated on COCO, making it difficult to assess the acceleration benefits for OCR and document layout tasks. The AR and pure diffusion models achieve F1mIoU scores of 71.15 and 43.64 on SROIE, 85.78 and 68.38 on DocLayNet, and 74.76 and 56.69 on M6Doc, respectively, indicating substantial performance gaps. The reductions in processing time associated with these performance losses are not sufficiently quantified, and it remains unclear whether speed gains persist under decoding configurations that reduce the performance losses. Consequently, it is difficult to determine how much acceleration can be achieved across tasks with an acceptable loss in performance.

  2. In the draft-quality comparison with bidirectional block-MTP, block-MTP predicts the draft in a single pass, whereas the diffusion model uses two steps: an initial prediction followed by remasking and re-predicting low-confidence positions. Since this comparison changes both the training objective and computational cost, the observed differences alone do not isolate the additional benefits of diffusion. Evidence from comparisons against bidirectional block-MTP at matched wall-clock time is limited, leaving it unclear how much diffusion training and iterative refinement contribute beyond the use of bidirectional context. This limits the evidence supporting the choice of diffusion for parallel visual grounding.

Questions for authors

On the RefCOCOg validation set, the AR and diffusion models achieve F1mIoU scores of 84.37 and 91.61, respectively, for box prediction, and F1@Point scores of 90.44 and 85.75, respectively, for point prediction. Please explain the main reasons why the direction of the performance change relative to AR differs between these two tasks. In particular, please clarify whether the decline in point-prediction performance primarily results from selecting the wrong target or from failing to place the point within the target object's region after correctly identifying it. This analysis is important for assessing which types of spatial prediction benefit from diffusion and where its limitations lie.

5 comments
AuthorsOpenReview ↗

1. W1: OCR/Layout Speed and Accuracy

Thank you for highlighting this important gap in the original evaluation. We have expanded the measurements beyond COCO to include Dense200, RefCOCOg, ICDAR2015, DocLayNet, SROIE, and M6Doc (Appendix I.3, Figures S24–S27 and Tables S32–S37).

(1) Acceleration at the original and higher-quality operating points.

For the original pure-diffusion configurations, end-to-end latency decreases from 7.40 to 3.10 s on SROIE, 5.90 to 2.80 s on DocLayNet, and 6.80 to 3.20 s on M6Doc, yielding 2.39×, 2.11×, and 2.13× acceleration, respectively.

Importantly, acceleration also persists when reducing the accuracy loss:

TaskConfigurationAR → configuration qualityAR → configuration TPSSpeedup
ICDAR2015Pure diffusion, B=16, τ=0.842.50 → 42.3041 → 1744.24×
ICDAR2015Linear self-speculation, B=3242.50 → 42.0041 → 2716.61×
DocLayNetLinear self-speculation, B=3285.78 → 84.8044 → 3097.02×

Quality is F1mIoU, with loose matching for OCR; TPS measures decoding throughput. Thus, pure diffusion achieves 4.24× throughput with a 0.20 pp decrease on ICDAR2015, while verified decoding achieves 6.61×/7.02× throughput with 0.50/0.98 pp decreases on ICDAR2015/DocLayNet. Common-quality end-to-end comparisons are additionally provided in Table S42.

(2) Diagnosing the OCR gap and improving the model.

Your comment also motivated qualitative error analysis. We found cases where the text was largely correct but word/line granularity differed from the annotations—for example, several words were grouped into one line—reducing text–box matching scores.

Approximately 80% of our pretraining mixture consists of box-structured supervision, emphasizing coordinate ordering, geometric consistency, and label–box associations. We hypothesize that denoising becomes particularly adapted to these coupled constraints, while transcription tokens may become confident before their word/line organization is sufficiently consistent.

We therefore increased OCR supervision during pretraining and GRPO and adjusted rewards for word/line granularity and text–geometry agreement, using GRPO for targeted adaptation with the aim of preserving box-grounding capabilities. Appendix I.1, Table S30 reports:

BenchmarkOriginal diffusionRevised diffusionImprovement
HierText33.3140.18+6.87 pp
ICDAR201541.8149.27+7.46 pp
TotalText43.0849.84+6.76 pp
SROIE43.6446.73+3.09 pp
DocLayNet68.3883.94+15.56 pp
M6Doc56.6971.83+15.14 pp

All four OCR benchmarks improve. ICDAR2015 and TotalText exceed the original AR counterpart, while the layout gaps to AR narrow to 1.84 pp on DocLayNet and 2.93 pp on M6Doc.

We are especially grateful for your suggestion, which led to both broader efficiency evaluation and concrete model improvements.

AuthorsOpenReview ↗

2. W2: Why Diffusion for Parallel Grounding

Thank you for this incisive observation. The central question is whether diffusion produces more useful grounding predictions per unit time, beyond bidirectional context alone. We have strengthened the evidence through one-pass comparisons, fixed-time measurements, and direct evaluation of iterative refinement (Appendix I.4).

(1) One-pass prediction without additional refinement.

We compare one-pass bidirectional block-MTP with one-step DLM, using the strengthened MTP control described below. The controls share model initialization, grounding data, token interface, and a frozen causal verifier, with matched adaptation examples, supervised-token exposure, and optimization settings.

At K=16, one-step DLM increases acceptance from 20.20% to 21.76% and reduces full-round cost per accepted draft token from 6.10 to 5.38 ms. At K=32, the cost decreases from 8.70 to 6.88 ms (Table S39). The advantage is therefore already observable before adding a second diffusion step.

(2) Useful progress at matched wall-clock time.

Within 200 ms, DLM commits 47.9 tokens, compared with 45.0 for bidirectional block-MTP; within 400 ms, it commits 97.1 versus 89.2 tokens (Table S40). These comparisons directly measure effective progress within the same time budget.

(3) Whether iterative refinement repays its computation.

Diffusion can revisit uncertain proposals before verification. In the direct one-step/two-step comparison at K=16, a second refinement step reduces full-round cost per accepted draft token from 5.38 to 4.70 ms. The common-quality end-to-end results further show:

TaskCommon qualityBidirectional block-MTP, 1 passDLM, 1 stepDLM, 2 steps
Dense20074.802.50 s2.30 s2.00 s
RefCOCOg val84.000.56 s0.59 s0.70 s
SROIE70.202.80 s2.90 s3.10 s
DocLayNet85.103.00 s2.60 s2.30 s
M6Doc74.403.20 s3.10 s2.80 s

These are AR-verified drafting configurations; quality is F1mIoU, with loose matching for SROIE. Measurements use H100 80GB, batch size 1, BF16, and a shared backend, including preprocessing, visual encoding, prefill, decoding, and parsing (Figure S30 and Table S42).

On Dense200, DocLayNet, and M6Doc, the second diffusion step further reduces total latency. Two-step DLM lowers latency relative to bidirectional block-MTP by 20.0%, 23.3%, and 12.5%, respectively. Improved verification and commitment can therefore repay the extra denoising computation, while cached context and efficient execution help control its overhead.

To further strengthen the comparison, the MTP results above use Box-aligned bidirectional block-MTP, following LocateAnything’s geometric organization: complete box tuples remain within prediction units, with bidirectional attention inside each unit. This adds an explicit structural prior to the baseline.

Together, the evidence supports diffusion for parallel visual grounding through joint prediction and iterative refinement that can improve useful output per unit time, including against this more structured alternative. We appreciate your suggestion, which helped us substantiate this choice with stronger timing-based evidence.

AuthorsOpenReview ↗

3. Q1: Box versus Point Prediction

Thank you for this particularly perceptive question. Distinguishing correct target selection from precise point placement is central to understanding these results, and prompted us to inspect nearly 1,000 failure cases more closely.

(1) Box localization and point-in-mask evaluation require different geometric precision.

On RefCOCOg validation, diffusion improves box F1mIoU from 84.37 to 91.61, while point F1@Point decreases from 90.44 to 85.75. A point can lie inside the correct bounding box but outside the foreground mask, especially for thin or irregular objects. Estimating the object’s overall extent therefore does not automatically resolve a valid foreground point.

Our qualitative interpretation highlights fine point placement relative to the mask as an important contributor: the model may localize the approximate target region while still struggling to place the point precisely on its foreground.

(2) Iterative prediction reveals unresolved localization decisions.

We observed that some uncommitted coordinate positions are predicted repeatedly while their entropy remains close to, but slightly above, the commitment threshold, particularly in pointing examples. We hypothesize that the model is still adjusting fine placement—for example, resolving the difference between a point inside the bounding box and one inside the foreground mask.

This connects directly to our motivation for iterative denoising: uncertain spatial decisions can be reconsidered as within-block context develops, before commitment. It also explains how improved box localization can coexist with lower point-in-mask performance: estimating spatial extent and resolving foreground membership are different geometric requirements.

(3) Implications for suitable tasks.

Compact spatial outputs offer opportunities for joint prediction and iterative refinement, especially when a few uncertain coordinates must satisfy coupled constraints. Pointing additionally requires precise foreground membership and is sensitive to small coordinate differences. Longer outputs place greater demands on language modeling, instance organization, and text–geometry consistency across the sequence.

Thus, output length, geometric precision, and output-organization complexity jointly influence diffusion’s behavior. We are especially grateful that your question helped us distinguish coarse target localization from precise foreground placement and sharpen our understanding of iterative grounding.

More broadly, your constructive suggestions have led to stronger experiments and useful directions for model improvement. We apologize that the main-paper page limit has led us to collect the additional results toward the end of the appendix; if accepted, we will consolidate them into a clearer camera-ready presentation.

We welcome further discussion. If additional analyses or experiments that we have not considered would strengthen the paper, please let us know; we will make every effort to complete them and improve the manuscript.

Reviewer_SE9kOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits whether visual grounding in vision-language models inherently requires sequential autoregressive decoding. The authors propose GroundAnything, a framework that integrates autoregressive and blockwise diffusion decoding within a shared model. Extensive experiments across diverse grounding benchmarks demonstrate that the proposed approach can improve decoding efficiency at the cost of some overall grounding accuracy.

Critical strengths

The proposed framework effectively integrates autoregressive and diffusion decoding within a shared model, demonstrating the feasibility of combining these two generation paradigms for visual grounding. The extensive experiments also demonstrate tangible improvements in decoding efficiency, validating the practical value of this design.

Critical weaknesses
  1. GroundAnything exhibits substantial performance degradation on several OCR and document layout benchmarks (e.g., −27.51 pp on SROIE, −18.07 pp on M6Doc, and −17.40 pp on DocLayNet; Table 3). The reasons remain unclear, raising concerns about whether diffusion decoding has inherent limitations in complex, fine-grained structured localization tasks. The authors should provide further analysis and explanations.

  2. The authors perform GRPO using only causal AR rollouts and likelihoods, while directly reusing the updated parameters for diffusion inference (Section 3.2). However, the paper does not clearly demonstrate or explain how this AR-based optimization benefits diffusion grounding. The authors should provide a direct ablation and clarify the mechanism behind this cross-mode improvement.

  3. The reported speedups primarily concern decoding throughput rather than end-to-end inference latency (Section 4.2 and Appendix C.2). Since visual encoding and prompt prefill are excluded, it remains unclear how much acceleration the complete system achieves. The authors should provide end-to-end latency comparisons to demonstrate the practical efficiency gains.

  4. I am unsure whether the reported speedups are worth the substantial loss in grounding accuracy. Since this is not my primary research area, I would appreciate the authors' explanation of whether such a trade-off is acceptable and necessary in practical applications. I am also open to the perspectives of other reviewers on this issue.

Questions for authors

The authors are expected to provide point-by-point responses to the concerns raised in the Weaknesses section.

Additionally, I would like the authors to clarify what exactly Stage III refers to in Figure 10, including its specific training procedure and whether it corresponds to the GRPO post-training stage.

8 comments
AuthorsOpenReview ↗

1. W1: OCR/Layout Analysis and Improvements

We sincerely thank the reviewer for this insightful comment. We acknowledge the original OCR/layout gaps and have added cross-task timing experiments, qualitative analysis, and targeted training improvements (Appendices I.1 and I.3). Your suggestion helped us identify two connected issues.

(1) Structural constraints and training distribution. Box generation must jointly satisfy several constraints: four coordinates in a prescribed order, geometrically consistent corners, valid delimiters, and correct label–box associations. A locally plausible token can therefore produce an invalid spatial prediction when combined with the remaining fields. This is the additional structural complexity we refer to beyond generating a textual span.

Approximately 80% of our pretraining mixture consists of box-structured supervision. We hypothesize that the model’s denoising behavior becomes particularly adapted to resolving these coupled constraints, whereas transcription tokens may reach the commitment threshold earlier—before word/line organization and text–box associations are sufficiently consistent. This offers a possible explanation for the fast OCR decoding observed in our extended experiments together with its lower matching accuracy.

(2) Inconsistent word/line granularity. Our qualitative comparison with AR revealed cases where the recognized text was largely correct, but the diffusion model grouped several words into a line when separate word instances were expected, or split a line differently from the annotations. The associated boxes then describe different units, reducing text–geometry matching scores despite correct textual content. This identifies output-granularity stability as a concrete training issue.

Following this diagnosis, we increased OCR supervision during pretraining and GRPO and adjusted the reward treatment of word/line granularity and text–geometry agreement. We used GRPO for additional targeted adaptation with the aim of preserving box-grounding capabilities. Appendix I.1, Table S30 reports:

BenchmarkOriginal AROriginal diffusionRevised diffusionImprovement
HierText41.3333.3140.18+6.87 pp
ICDAR201542.5041.8149.27+7.46 pp
TotalText49.0343.0849.84+6.76 pp
SROIE71.1543.6446.73+3.09 pp
DocLayNet85.7868.3883.94+15.56 pp
M6Doc74.7656.6971.83+15.14 pp

OCR uses loose-match F1mIoU; layout uses F1mIoU. Improvements are relative to original diffusion.

All four OCR benchmarks improve, with ICDAR2015 and TotalText exceeding the original AR counterpart. For layout, which requires region-category and box prediction, the gaps to AR narrow to 1.84 pp on DocLayNet and 2.93 pp on M6Doc.

These results demonstrate that targeted training improvements can substantially strengthen diffusion grounding on these tasks. We are especially grateful that your comment helped us turn the observed gaps into actionable model improvements.

AuthorsOpenReview ↗

2. W2 + Additional Question: Causal GRPO and Stage III

Thank you for requesting a direct ablation and clearer training explanations.

W2: How causal GRPO benefits diffusion grounding

We added a with/without causal-GRPO comparison evaluated using pure diffusion inference (Appendix I.2, Table S31). OCR improves by 1.71 pp on HierText, 1.91 pp on ICDAR2015, 2.38 pp on TotalText, and 3.84 pp on SROIE. We explain GRPO’s role in two parts.

(1) Improving output-format stability. Our qualitative inspection of many ablation outputs identifies format stability as an important benefit of RL. In grounding, this means maintaining structural boundaries, organizing coordinate fields into complete instances, and consistently associating textual identifiers with geometry. For OCR, it also includes stable word/line units: correct characters can still receive a lower score if their organization into output instances is inconsistent.

Causal and diffusion modes share the backbone and output head. GRPO thus updates the same parameters used during diffusion inference to predict structural tokens, text, and coordinates. Reward-driven improvements to these shared output conventions can transfer to diffusion generation, providing a concrete mechanism for the observed OCR gains.

(2) Directly optimizing the self-speculative verifier. Causal GRPO also directly optimizes the policy that verifies diffusion-generated drafts. During self-speculative decoding, diffusion proposes tokens in parallel, while the causal policy determines acceptance and the final output. The AR-based objective therefore directly serves this deployment mode.

Our main benchmark tables use pure diffusion to evaluate diffusion generation itself; self-speculative decoding is reported separately as a deployment option. We appreciate your suggestion, which helped us distinguish these two roles and provide direct experimental evidence.

Additional question: Stage III

We apologize for the ambiguous terminology. Stage III denotes the general visual/video understanding stage of base VLM pretraining, including general VQA supervision—not GRPO post-training. Specifically, Figure 10 tests whether general visual/VQA data from Stage III are included during ViT alignment. The underlying base stage uses causal language-model supervision and updates the vision encoder, projector, and language model (Appendix E.3, Table S6).

When specializing an already pretrained AR model, a natural strategy is to emphasize task-specific grounding data and treat further general visual supervision as secondary. We included this ablation to examine that assumption in our AR-to-diffusion conversion setting. Our hypothesis is that substantial general visual/VQA supervision remains valuable for developing and maintaining multimodal language-modeling capabilities during diffusion adaptation. The diffusion model’s pronounced sensitivity to this data choice supports that motivation.

We have clarified this interpretation in Sections 3.2 and 4.3, the Figure 10 caption, and Appendices E.3 and G.3. Thank you for helping us make both the procedure and its motivation clearer.

AuthorsOpenReview ↗

3. W3–W4: End-to-End Efficiency and Practical Value

Thank you for these important questions about practical efficiency. We have added end-to-end measurements and explain below when the available operating points are useful.

W3: End-to-end latency

For the original AR and pure-diffusion configurations, Appendix I.3, Figure S27 and Table S32 report:

TaskAR qualityDiffusion qualityAR E2E latencyDiffusion E2E latencySpeedup
Dense20075.5570.044.80 s2.60 s1.85×
RefCOCOg val84.3791.610.62 s0.51 s1.22×
SROIE71.1543.647.40 s3.10 s2.39×
DocLayNet85.7868.385.90 s2.80 s2.11×
M6Doc74.7656.696.80 s3.20 s2.13×

Quality is F1mIoU, with loose OCR matching for SROIE. These results demonstrate 1.22–2.39× end-to-end acceleration across representative grounding, OCR, and layout tasks.

W4: When is the trade-off worthwhile?

We agree that the answer depends on the application.

For large-scale offline annotation, faster end-to-end inference allows more candidate annotations to be generated within the same time or compute budget. This is valuable when the pipeline already includes confidence filtering, teacher agreement, or human review, allowing increased generation capacity to expand data collection.

Conversely, for a standalone detection or OCR tool with stringent accuracy requirements, we agree that the original pure-diffusion trade-off may not be worthwhile. If every prediction is directly consumed and accuracy is the overriding priority, we would favor the more accurate AR configuration.

For applications seeking acceleration while retaining high output quality, AR-verified self-speculative decoding provides an additional option. We added a controlled comparison using an H100 80GB GPU, batch size 1, BF16, and the same inference backend, timing preprocessing, visual encoding, prompt prefill, decoding, and output parsing (Appendix I.4.2, Figure S30 and Table S42).

At the reported common quality within each task, two-step diffusion drafting with AR verification achieves:

TaskCommon qualityCausal-only E2EVerified diffusion E2ESpeedup
Dense20074.804.90 s2.00 s2.45×
DocLayNet85.106.00 s2.30 s2.61×
M6Doc74.406.90 s2.80 s2.46×

The practical value is therefore the ability to select an inference mode for the workload: pure diffusion for throughput-oriented usage, AR for accuracy-first usage, and verified decoding for another useful speed–quality operating point.

We sincerely appreciate the reviewer’s careful and constructive suggestions, which helped us improve the experiments and clarify the model’s practical value. We also apologize that the main-paper page limit has led us to collect the additional results toward the end of the appendix. If accepted, we will consolidate these findings into a clearer camera-ready presentation.

We welcome further discussion. If additional analyses or experiments that we have not considered would strengthen the paper, please let us know; we will make every effort to complete them and improve the manuscript.

Reviewer_AHVmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GroundAnything, a visual grounding model designed to reduce the sequential decoding cost of autoregressive (AR) coordinate generation. Starting from an AR grounding model with a 4B language backbone, the authors introduce blockwise masked diffusion: tokens within a block are predicted with bidirectional attention, while completed blocks provide causal context for subsequent blocks. Training combines grounding pretraining, AR-to-diffusion conversion with joint AR and diffusion objectives, supervised fine-tuning, and reinforcement post-training. The model is evaluated on 30 benchmarks spanning detection, referring expressions, pointing, OCR, document layout, and GUI grounding. The pure-diffusion model averages 61.75% across these benchmarks, compared with 72.42% for its AR counterpart. The paper also studies decoding trade-offs; an optional diffusion-drafting mode with AR verification achieves a 4.51× throughput improvement over the AR counterpart on COCO, with a 0.74-point decrease in F1mIoU.

Critical strengths
  1. A substantive extension of diffusion-based grounding. The paper moves beyond task-specific demonstrations by building a single diffusion model for detection, referring expressions, pointing, OCR, document layout, and GUI grounding. Its pure-diffusion variant achieves a 61.75% average across 30 benchmarks, demonstrating that the approach can support a broad range of structured spatial outputs. This breadth, together with the strong results on several individual tasks, is the main reason I view the contribution as more than an incremental application.

  2. A useful, carefully differentiated speed–accuracy analysis. The paper evaluates pure diffusion separately from diffusion drafting with AR verification, rather than treating their results as interchangeable. On COCO, pure diffusion achieves 2.56× the AR model’s decoding throughput with a 2.98-point F1mIoU decrease; the optional verified mode reaches 4.51× throughput with a 0.74-point decrease. These operating points make the proposed approach practically relevant despite its remaining accuracy gaps.

Critical weaknesses
  1. The speed–accuracy advantage is not yet established across the full task range. Pure GroundAnything scores 61.75% across 30 benchmarks, versus 72.42% for its AR counterpart, with particularly large gaps on OCR and document layout tasks. The most favorable speed–accuracy result—4.51× throughput with only a 0.74-point F1mIoU decrease—uses AR verification and is reported on COCO, rather than across the broader benchmark suite. This matters because the paper’s central claim concerns fast and precise grounding across diverse tasks; it remains unclear how often that trade-off holds outside the reported COCO setting.

  2. The diffusion-specific benefit over strong parallel alternatives is not fully isolated. The controlled drafting comparison primarily uses an in-house causal MTP baseline, whereas LocateAnything uses bidirectional, box-aligned blocks. The appendix also shows that bidirectional block-MTP improves proposal quality, but does not provide a matched final accuracy–latency comparison against it. Consequently, the current evidence does not clearly separate the contribution of diffusion from the benefits of bidirectional block prediction and the broader training recipe. This uncertainty materially affects my assessment of the method’s distinct contribution.

Questions for authors
  1. Clarification of the speed–accuracy claim: Could the authors report matched end-to-end latency and accuracy for pure diffusion, self-speculative decoding, and the AR counterpart on representative non-COCO tasks, particularly OCR, layout, and dense grounding? This would help determine whether the favorable COCO trade-off extends to tasks where pure diffusion currently shows larger accuracy losses.

  2. Clarification of the diffusion-specific contribution: Under matched training data, inference budget, and evaluation protocols, how does GroundAnything compare with a bidirectional block-MTP baseline? The appendix shows that bidirectionality also improves block-MTP proposal quality, so a comparison of final accuracy and latency would materially affect my assessment of what diffusion contributes beyond parallel bidirectional prediction.

4 comments
AuthorsOpenReview ↗

1. W1/Q1: Cross-Task Speed and Accuracy

Thank you for recognizing the breadth of our framework and for emphasizing the need for stronger evidence beyond COCO. Your comment motivated expanded efficiency measurements and targeted improvements to OCR and layout performance.

(1) Cross-task accuracy–latency comparisons.

We added decoding sweeps on Dense200, RefCOCOg, ICDAR2015, and DocLayNet, alongside end-to-end measurements covering OCR, layout, and dense grounding (Appendix I.3, Figures S24–S27 and Tables S32–S37).

For the original GroundAnything-VLM and pure-diffusion configurations, Table S32 reports:

TaskAR qualityPure-diffusion qualityAR E2E latencyPure-diffusion E2E latencySpeedup
Dense20075.5570.044.80 s2.60 s1.85×
RefCOCOg val84.3791.610.62 s0.51 s1.22×
SROIE71.1543.647.40 s3.10 s2.39×
DocLayNet85.7868.385.90 s2.80 s2.11×
M6Doc74.7656.696.80 s3.20 s2.13×

Quality is F1mIoU, with loose OCR matching for SROIE.

We additionally compare causal-only decoding and diffusion drafting with AR verification at common final quality (Figure S30 and Table S42). Verified decoding achieves 2.45×, 2.62×, 2.61×, and 2.46× end-to-end acceleration on Dense200, SROIE, DocLayNet, and M6Doc, respectively, using one-step drafting for SROIE and two-step drafting for the other three tasks.

This controlled verified-decoding experiment uses H100 80GB, batch size 1, BF16, and the same backend, timing preprocessing, visual encoding, prompt prefill, decoding, and output parsing.

The decoding sweeps also identify higher-quality configurations: pure diffusion on ICDAR2015 achieves 4.24× decoding throughput with a 0.20 pp decrease, while linear self-speculation achieves 6.61×/7.02× throughput with 0.50/0.98 pp decreases on ICDAR2015/DocLayNet. These results establish useful operating points beyond the COCO example.

(2) Diagnosing and improving OCR/layout performance.

We acknowledge the original accuracy gaps and appreciate your attention to them. Qualitative inspection revealed cases where the text was largely correct but word/line granularity and text–box organization differed from the annotations.

Approximately 80% of our pretraining mixture consists of box-structured supervision, emphasizing coordinate ordering, geometric consistency, and label–box associations. We hypothesize that denoising becomes particularly adapted to these coupled constraints, while transcription tokens may become confident before their word/line organization is sufficiently consistent.

We therefore increased OCR supervision during pretraining and GRPO and adjusted rewards for word/line granularity and text–geometry agreement, using GRPO for targeted adaptation with the aim of preserving box-grounding capabilities. Appendix I.1, Table S30 reports:

BenchmarkOriginal diffusionRevised diffusionImprovement
HierText33.3140.18+6.87 pp
ICDAR201541.8149.27+7.46 pp
TotalText43.0849.84+6.76 pp
SROIE43.6446.73+3.09 pp
DocLayNet68.3883.94+15.56 pp
M6Doc56.6971.83+15.14 pp

All four OCR benchmarks improve. ICDAR2015 and TotalText exceed the original AR counterpart, while the layout gaps to AR narrow to 1.84 pp on DocLayNet and 2.93 pp on M6Doc.

We are especially grateful that your suggestion led to both broader practical evaluation and measurable model improvements.

AuthorsOpenReview ↗

2. W2/Q2: Isolating Diffusion’s Contribution

Thank you for identifying this central attribution question. We understand that separating diffusion’s contribution from bidirectionality and the broader training recipe is essential to assessing the method’s distinct contribution. We have added stronger controls that examine proposal generation, iterative refinement, and their practical time costs separately.

(1) Controlling the training recipe and strengthening MTP.

Following your suggestion, we implemented Box-aligned bidirectional block-MTP, adopting LocateAnything’s geometric organization. We reprocessed and cleaned the training targets into complete box-aligned units, with bidirectional attention within each unit and causal conditioning across units.

The controls share model initialization, grounding data, token interface, and a frozen causal verifier, with matched adaptation examples, supervised-token exposure, optimizer updates, and effective batch size. MTP therefore receives both bidirectional context and an explicit geometric prior, while the common training ingredients are controlled (Appendix I.4.1 and Table S38).

(2) Comparing proposals without additional refinement.

The stronger baseline indeed improves proposal quality: at K=16, acceptance rises from 16.81% for causal MTP to 20.20% for box-aligned bidirectional block-MTP. One-step DLM further reaches 21.76% and reduces full-round cost per accepted draft token from 6.10 to 5.38 ms relative to the box-aligned baseline. At K=32, the cost decreases from 8.70 to 6.88 ms (Table S39).

Both methods make one proposal pass. These gains are therefore measured beyond structured bidirectional prediction and do not depend on giving DLM an extra refinement step.

(3) Testing refinement with the trained model held fixed.

We then compare one-step and two-step decoding of the same DLM, keeping its parameters and verifier fixed. At K=16, the second step reduces full-round cost per accepted draft token from 5.38 to 4.70 ms.

This directly tests the inference-time value of revisiting uncertain proposals, without changing the training data or recipe. Under a fixed 400 ms budget, DLM also commits 97.1 tokens, versus 89.2 for box-aligned block-MTP (Table S40).

(4) Comparing final quality and end-to-end latency.

Under the shared causal verifier, Figure S30 and Table S42 report:

TaskCommon qualityCausal-onlyBox-aligned block-MTP, 1 passDLM, 1 stepDLM, 2 steps
Dense20074.804.90 s2.50 s2.30 s2.00 s
RefCOCOg val84.000.64 s0.56 s0.59 s0.70 s
SROIE70.207.60 s2.80 s2.90 s3.10 s
DocLayNet85.106.00 s3.00 s2.60 s2.30 s
M6Doc74.406.90 s3.20 s3.10 s2.80 s

Quality is F1mIoU, with loose matching for SROIE. Measurements use H100 80GB, batch size 1, BF16, and a shared backend, including preprocessing through output parsing.

On Dense200, DocLayNet, and M6Doc, the second diffusion step further reduces total latency. Two-step DLM reduces latency relative to box-aligned block-MTP by 20.0%, 23.3%, and 12.5%, respectively. Better proposals can repay the refinement computation through more effective verification and commitment, with cached context and efficient execution helping control overhead.

Together, these comparisons clarify what diffusion adds: the one-pass comparison evaluates diffusion-trained proposals beyond structured bidirectional prediction; the fixed-model comparison establishes the additional value of iteration without changing the training recipe. The matched-time and common-quality results show how these capabilities translate into practical efficiency.

We sincerely appreciate your suggestion and hope these stronger controls strengthen confidence in the method’s distinct contribution. We also apologize that the main-paper page limit has led us to collect the additional results toward the end of the appendix; if accepted, we will consolidate them into a clearer camera-ready presentation.

We welcome further discussion. If additional analyses or experiments that we have not considered would strengthen the paper, please let us know; we will make every effort to complete them and improve the manuscript.

Reviewer_AHVmOpenReview ↗

Post-Rebuttal Assessment: Maintain Weak Acceptance

Thank you for the additional experiments. The cross-task end-to-end measurements and matched box-aligned bidirectional block-MTP comparisons substantially address my main concerns. The controlled one-step comparison and fixed-model refinement experiment provide stronger evidence for diffusion’s contribution beyond bidirectional parallel prediction.

The benefits are nevertheless task-dependent: DLM improves latency over block-MTP on Dense200 and the layout benchmarks, but not on RefCOCOg or SROIE. Please make this scope explicit and clearly distinguish pure diffusion, AR-verified decoding, and the revised-training results.

Overall, the rebuttal strengthens the empirical support for the paper. I maintain my rating of 3 (Weak acceptance).

AuthorsOpenReview ↗

Clarifying Task-Dependent Gains and Decoding Settings

We sincerely thank you for your careful and conscientious assessment, your encouraging recognition of the additional evidence, and your continued support. We especially appreciate your sharp insight in identifying the need to specify exactly where the gains hold and which experimental setting supports each conclusion. This perceptive observation reflects the rigor and care with which you have examined our work, and has helped us present its contribution more precisely. We fully agree that the latency advantage over box-aligned block-MTP is task-dependent, and will make this scope explicit while clearly distinguishing the following three settings.

(1) Pure diffusion, without AR verification. In Table S32, RefCOCOg improves from 0.62 s (AR) to 0.51 s (DLM), a modest 1.22× speedup. This is a single-target task: AR needs only a few output tokens, whereas our DLM processes at least one 32-token canvas per denoising pass. Even with blockwise generation, this minimum canvas introduces overhead that limits gains on very short outputs and can make diffusion slower in some configurations—a structural limitation of fixed-block diffusion decoding.

Short OCR outputs can incur the same overhead. For SROIE, our qualitative analysis additionally identified word/line grouping mismatches. We hypothesize that resolving this granularity ambiguity requires additional refinement, reducing the efficiency advantage. This reflects an interaction between OCR-specific output organization and diffusion decoding. The original pure-DLM configuration is faster than AR (3.10 vs. 7.40 s), but has lower quality.

(2) Diffusion drafting with AR verification. The block-MTP comparison in Table S42 uses a shared causal verifier and a common final quality within each task. End-to-end latencies are:

TaskBox-aligned block-MTPDLM, 1 stepDLM, 2 steps
Dense2002.50 s2.30 s2.00 s
RefCOCOg val0.56 s0.59 s0.70 s
SROIE2.80 s2.90 s3.10 s
DocLayNet3.00 s2.60 s2.30 s
M6Doc3.20 s3.10 s2.80 s

Here, the mechanisms discussed above concern draft generation, while AR verification determines token acceptance and final outputs. Refinement improves overall latency only when better drafts offset its computational cost. This occurs on Dense200 and the layout tasks, but not on RefCOCOg or SROIE. SROIE still achieves 2.62× acceleration over causal-only decoding with one-step DLM (7.60 vs. 2.90 s), while remaining slightly slower than block-MTP.

(3) Revised training, evaluated with pure diffusion. These are separate checkpoint results. With the same data-mixture and GRPO adjustments applied to both AR and DLM (Table S43), the DocLayNet/M6Doc gaps shrink from 17.40/18.07 pp to 2.33/2.80 pp; SROIE retains a substantial 25.73 pp gap. These results demonstrate gap reduction under matched revised training; they should not be combined with the earlier latency measurements as a single operating point.

Thank you again for the considerable time and thought you have invested in this review. Your incisive questions have led to stronger experimental controls and a clearer understanding of where diffusion provides practical value. We deeply appreciate both your constructive guidance and your recognition of these improvements, and will take particular care to make the scope and distinctions you highlighted clear in the final presentation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Main Strengths and New Evidence

We sincerely thank the PC and AC and reviewers for recognizing the paper’s strengths:

  • Broad grounding coverage (AHVm, pYPo): one 4B diffusion model supports 30 benchmarks. On RefCOCOg validation, pure diffusion achieves 91.61 F1mIoU versus 84.37 for AR (+7.24 pp).
  • A unified AR/diffusion framework (SE9k, pYPo): shared parameters support parallel generation and causal verification, without a separate draft model.
  • Practical acceleration (SE9k, AHVm, pYPo): self-speculative decoding achieves 4.51× decoding throughput with only a 0.74 pp decrease on COCO, with pure diffusion and verified decoding reported separately.

Our revision builds on these strengths to address concerns that the “speed–accuracy advantage is not yet established across the full task range” and the evidence “does not clearly separate the contribution of diffusion” (AHVm; related concerns from pYPo). We also investigated SE9k’s concern about “inherent limitations” in OCR/layout.

1. Acceleration extends beyond COCO, including at high output quality. (SE9k-W3/W4; pYPo-W1; AHVm-W1/Q1)

New decoding sweeps and end-to-end measurements span dense grounding, referring grounding, OCR, and layout (Appendix I.3). Original pure diffusion achieves 1.22–2.39× end-to-end acceleration across five tasks (Table S32). On ICDAR2015, pure diffusion with B=16 and τ=0.8 achieves 4.24× decoding throughput with only a 0.20 pp decrease versus AR (Table S35).

At common final quality, AR-verified DLM decoding also reduces end-to-end latency:

TaskCommon F1mIoUCausal-only → DLM + verification
Dense20074.804.90 → 2.00 s
SROIE70.207.60 → 2.90 s
DocLayNet85.106.00 → 2.30 s
M6Doc74.406.90 → 2.80 s

These comparisons use H100 80GB, batch 1, BF16, and a shared backend, timing preprocessing, visual encoding, prefill, decoding, and parsing; DLM uses two steps except one on SROIE (Table S42).

2. Targeted training substantially improves OCR and layout. (SE9k-W1; pYPo-W1; AHVm-W1)

Qualitative OCR inspection identified mismatched word/line granularity and text–box organization. We increased OCR supervision during pretraining and GRPO and revised the text–geometry rewards, using targeted GRPO adaptation with the aim of preserving box grounding.

In this revised-training experiment, all four OCR benchmarks improve by 3.09–7.46 pp; ICDAR2015 and TotalText exceed the original AR counterpart. DocLayNet and M6Doc improve by 15.56/15.14 pp, narrowing their AR gaps to 1.84/2.93 pp (Table S30). These results show that targeted training substantially reduces the original gaps. We thank the reviewers for helping identify this direction.

3. Stronger controls establish benefits beyond bidirectional prediction. (pYPo-W2; AHVm-W2/Q2)

We implemented Box-aligned bidirectional block-MTP, adopting LocateAnything’s geometric organization and reprocessing targets into complete box units. Comparisons share initialization, grounding supervision, token interface, and a frozen verifier, with matched adaptation exposure and optimization settings.

  • One pass versus one step: at K=16, DLM reduces full-round cost per accepted draft token from 6.10 to 5.38 ms versus box-aligned block-MTP (Table S39).
  • Refinement with the same model: a second DLM step further reduces this cost to 4.70 ms, without changing trained parameters or the training recipe.
  • Matched wall-clock budget: within 400 ms, the DLM-based verified pipeline commits 97.1 versus 89.2 tokens (Table S40).
  • Matched final quality: two-step DLM reduces end-to-end latency by 12.5–23.3% versus box-aligned block-MTP on Dense200, DocLayNet, and M6Doc (Table S42).

Together, these controls strengthen attribution to learned diffusion proposals and iterative refinement, directly addressing uncertainty about the method’s distinct contribution.

4. Additional analyses clarify training and task behavior.

The causal-GRPO ablation evaluated with pure diffusion shows OCR gains of 1.71–3.84 pp, with qualitative evidence of more stable output organization (SE9k-W2; Table S31). Inspection of nearly 1,000 failure cases clarifies the distinction between locating the correct box region and placing a point inside the foreground mask (pYPo-Q1). We clarified that the Stage III ablation concerns general visual/VQA data during ViT alignment, not GRPO.

We greatly appreciate the reviewers’ insightful guidance. Appendix I collects the evidence; each reviewer has a complete, independent response. We apologize that page limits placed much of this material at the appendix’s end; if accepted, we will consolidate it in the camera-ready. We welcome further discussion and suggestions for overlooked experiments, and will make every effort to strengthen the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.