acceptodds
Under review as a conference paper at ICLR 2027

Grounding Beyond Tokens: Continuous Geometry from Visual Evidence

Abstract

Vision-language models increasingly formulate visual grounding as autoregressive generation of discrete coordinate tokens, unifying language understanding and spatial prediction within a single framework. However, standard token-level supervision treats coordinates as categorical labels without explicitly modeling their spatial ordering or geometric dependencies. Language-modeling objectives also supervise object-relevant visual evidence indirectly, leaving the correspondence among referring expressions, visual referents, and spatial predictions insufficiently constrained. In this work, we propose Locus, a generative framework for object detection and visual grounding that combines explicit visual evidence supervision with continuous coordinate prediction. Locus supervises the spatial distribution of phrase-conditioned visual attention, encouraging the alignment between linguistic representations and corresponding visual referents. It also interprets coordinate-token probabilities as distributions over an ordered spatial domain, enabling continuous coordinate estimation, distance-aware supervision, and box-level geometric optimization. Notably, Locus preserves the autoregressive generation paradigm and requires no additional detection modules or architectural changes. Experiments across detection and referring expression benchmarks demonstrate consistent gains in localization accuracy and state-of-the-art performance, supporting the effectiveness of combining visual evidence supervision with continuous geometric modeling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, visual grounding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Dp4rOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper proposes Locus, a fine-tuning recipe for autoregressive VLM-based detection and visual grounding, built on Qwen3-VL-4B, that keeps the coordinate-token interface unchanged. It adds two groups of auxiliary objectives. Visual Evidence Supervision (VES) combines an attention concentration loss, which maximizes phrase-conditioned attention mass on the visual patches overlapping the ground-truth box, with a coverage regularizer that penalizes attention collapsing onto few patches. Continuous Coordinate Prediction (CCP) guides the model how to predict coordinate tokens. The paper reports gains on COCO, LVIS, Dense200, VisDrone, HumanRef and RefCOCOg, together with ablations of each component.

Critical strengths
  1. The method requires no architectural changes and preserves the autoregressive generation interface.
  2. The evaluation covers a broad set of detection, dense-detection, and referring benchmarks, with component ablations on three of them.
Critical weaknesses

1.The motivation is unclear and not justified. The introduction states that token-level supervision does not constrain the correspondence between the expression and its supporting visual evidence, but this is asserted rather than demonstrated. A well-trained VLM should learn to condition each coordinate token on the relevant visual context, as language models do for next-token prediction, and it is unclear why the existing mechanisms are insufficient. Likewise, if the predicted coordinates are correct, it is unclear why the supporting evidence requires separate supervision. The only evidence offered consists of two examples in Fig. 1, whose caption does not explain the box colors, and two attention maps in Fig. 5. The claimed limitation should be justified experimentally. Another point that was unclear to me is that attention concentration may conflict with relational grounding. Localizing a referent among similar objects (e.g., "the second boy from the left" in Fig. 2) plausibly requires attending to distractors and context, not only to the target region. In addition the absence of a dedicated module is presented as an advantage without justification. Decoder-based approaches such as OMG-LLaVA (and OMG-Seg-style decoders) typically add small modules and often yield solid gains. The paper neither shows that the end-to-end design is better in a quality/complexity trade-off nor compares against such approaches. The claim that quantized coordinates "reduce prediction complexity" is also not supported by a citation or an experiment.

  1. The "zero-shot" claims and comparisons are unclear. Locus is trained with the box-generation objective on a large detection and referring corpus, and "zero-shot" is never defined. It is unclear whether the training data overlaps with COCO or LVIS (the referring datasets in Table 5 are built on COCO images). Table 1 compares models with different training data and budgets, so the state-of-the-art claim cannot be attributed to the method alone. The only controlled comparison is the autoregressive baseline in the ablations.
Questions for authors
  1. What experiment shows that standard token-level training leaves the visual evidence insufficiently constrained?
  2. Would that make sense to compare L_coord + L_W with a simple L1/regression loss on the coordinates to simplify the objective?
  3. Would that make sense to compare Locus to an autoregressive baseline that uses post-hoc box sorting instead of L_valid?
  4. How is "zero-shot" defined, and does the training data overlap with COCO or LVIS images?
  5. How does Locus compare with a dedicated-module approach (such as OMG-LLaVA) in terms of quality and complexity?
8 comments

Thank you for the detailed questions about visual evidence, coordinate supervision, and evaluation. We provide additional experiments on target-region affinity, L1 regression, post-hoc sorting, and a dedicated-module approach, and clarify our data protocol. Together, these results support Locus's central contribution: jointly supervising visual evidence and continuous box geometry through the autoregressive decoder.

1. Motivation for visual evidence supervision

Precise localization requires the model to identify the target and translate its visual evidence into box boundaries. Standard token-level supervision directly optimizes the generated coordinates, while the correspondence between the expression and its supporting visual regions is learned indirectly. Previous work has shown that VLMs may encode relevant visual evidence yet fail to use it effectively when producing an answer (Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs, ICLR 2026).

This issue is particularly relevant when multiple objects are generated sequentially. As the output grows, later predictions are conditioned on an increasingly long generated prefix and may rely more heavily on preceding text. Zhao et al. document language bias in VLMs (Looking Beyond Text: Reducing Language Bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance, EMNLP 2025), and Sun et al. observe declining reliance on image inputs during long multimodal reasoning (Mitigating Visual Forgetting via Take-along Visual Conditioning for Multi-modal Long CoT Reasoning, ACL 2025). These findings motivate testing whether explicit visual supervision helps when generating multiple boxes.

We argue that each object and its boundaries require evidence from the image. Related work also shows that adding region-based attention supervision to next-token training improves visual localization tasks (Direct Visual Grounding by Directing Attention of Visual Tokens, arXiv:2511.12738). This supports an explicit training signal linking referring phrases to their corresponding visual regions. VES provides this signal by supervising phrase-conditioned visual affinities at selected response positions in the final six decoder layers. The decoder's native attention retains access to the full image, including at the supervised layers. This allows contextual cues to support referent identification while encouraging attention to the target region for localization. The coverage term further encourages spatially distributed support within that region, providing evidence for precise boundary prediction.

We test VES directly by adding it to the CCP-equipped model. On COCO, target-region affinity increases from 24.3% to 67.8%, alongside a Mean F1 increase from 52.1% to 57.5%. On RefCOCOg test, the corresponding changes are 21.9% to 65.4% and 79.8% to 82.4%. Thus, explicit evidence supervision improves localization even when coordinate and box geometry are already supervised.

Table R3a. VES ablation on COCO.

ModelTarget affinity (%)[email protected][email protected]Mean F1
Locus (CCP)24.370.318.752.1
Locus (CCP + VES)67.874.322.557.5

Table R3b. VES ablation on RefCOCOg test.

ModelTarget affinity (%)[email protected][email protected]Mean F1
Locus (CCP)21.990.539.779.8
Locus (CCP + VES)65.492.843.582.4

In Tables R3a and R3b, target affinity is the mass assigned to box-overlapping patches by the phrase-conditioned, image-normalized affinity in Eqs. 2–3. F1 scores are percentages. The appendix attention maps illustrate the corresponding spatial changes in two examples.

We will also revise Figure 1's caption to explain the box colors.

2. Why use quantized coordinate tokens?

Atomic coordinate tokens provide a compact autoregressive interface: each boundary occupies one coordinate token, rather than a potentially longer digit sequence. This representation is used by Rex-Omni (arXiv:2510.12798) and LocateAnything. In Locus, the discrete bins also provide an ordered support for learning a coordinate distribution. The discrete interface and continuous output serve complementary purposes: discrete tokens drive autoregressive generation, while expectation readout recovers fractional coordinates from their probabilities. Consequently, the reported boundary need not coincide with an integer bin.

3. Comparison with L1 regression

We compare our coordinate objective with L1 regression on expected coordinates. Both settings retain coordinate-token type supervision, validity supervision, and GIoU, so that both learn to generate coordinate tokens and optimize box geometry.

The key distinction is what the coordinate objective supervises. L1 evaluates the expected location; different distributions can share that expectation. For example, equal mass at two distant bins can have the correct mean despite assigning no mass near the target. Two-bin supervision instead preserves the fractional target location, and Wasserstein supervision penalizes displacement of probability mass along the ordered bins. This provides a learning signal beyond the accuracy of the mean alone.

Table R3c. Coordinate-supervision comparison on COCO. Both settings retain coordinate-token type supervision, validity, and GIoU. Scores are percentages.

Coordinate supervision[email protected][email protected]Mean F1
L1 regression71.117.554.2
Lcoord+λWLW\mathcal{L}_{\mathrm{coord}} + \lambda_{\mathrm{W}} \mathcal{L}_{\mathrm{W}}74.322.557.5

Distributional supervision improves Mean F1 by 3.3 points and [email protected] by 5.0 points, supporting its contribution to precise localization beyond expected-coordinate regression.

4. Validity supervision versus post-hoc sorting

These approaches intervene at different stages. Post-hoc sorting corrects boundary order after generation. The validity objective supplies gradients during training that encourage the coordinate distributions to respect x1≤x2x_1\leq x_2 and y1≤y2y_1\leq y_2. This is especially useful alongside GIoU: GIoU evaluates the assembled box after its expected boundaries are ordered, whereas validity supervision directly encourages the predicted boundary roles to remain consistent.

In manuscript Table 4(b), adding validity to Wasserstein supervision improves COCO [email protected] from 69.9% to 70.5% and [email protected] from 17.8% to 18.6%. We additionally compare the complete model with a variant trained without validity supervision that applies sorting at inference.

Table R3d. Validity supervision and post-hoc sorting on COCO. Scores are percentages.

Method[email protected][email protected]Mean F1
Without Lvalid\mathcal{L}_{{valid}} + inference-time sorting73.121.355.2
Locus with Lvalid\mathcal{L}_{{valid}}74.322.557.5

The gains of 2.3 Mean-F1 points and 1.2 [email protected] points support learning consistent boundary ordering during training, beyond repairing the final boundary order.

5. Definition of zero-shot evaluation and data overlap

Following the evaluation usage in Rex-Omni and LocateAnything, “zero-shot” in our COCO and LVIS comparisons denotes evaluation without training on those evaluation images. We checked the fine-tuning corpus for image-level overlap with both evaluation splits and excluded overlapping images from training.

This check applies across data sources, including the COCO-derived referring datasets listed in Table 5. The relevant separation is therefore defined at the image level across the combined training corpus, rather than by dataset name alone. Locus learns grounding from the remaining detection and referring annotations and is evaluated on the excluded benchmark images. We will make this definition explicit alongside Table 1. The benchmark results and matched ablations provide complementary evidence for Locus. Table 1 establishes its performance relative to existing detection and grounding systems; under the same training setting, manuscript Table 4(c) shows that Locus improves COCO Mean F1 from 46.5% for the autoregressive baseline to 57.5%, an 11.0-point gain. The paired cross-model experiments further show consistent gains when applying VES and CCP to the same initialization and fine-tuning data. These comparisons connect the benchmark performance to the proposed training objectives.

6. Comparison with a dedicated-module approach

We evaluate OMG-LLaVA, which produces segmentation masks through a dedicated visual decoder, and convert its masks to boxes for RefCOCOg test evaluation. Locus-4B reaches 82.4% Mean F1 and 43.5% [email protected], with a measured throughput of 2.1 boxes per second.

Table R3e. RefCOCOg test comparison. F1 scores are percentages; BPS is measured boxes per second.

Model[email protected][email protected]Mean F1BPS
OMG-LLaVA-7B80.031.565.70.9
Qwen3-VL-8B (manuscript Table 3 checkpoint)88.633.875.21.5
Locus-4B92.843.582.42.1

Qwen3-VL-8B, whose nominal size is closer to OMG-LLaVA-7B, also achieves higher Mean F1 (75.2% versus 65.7%) and measured throughput (1.5 versus 0.9 BPS). This row uses the Qwen3-VL-8B checkpoint reported in manuscript Table 3; Tables R2a and R2b report the additional paired fine-tuning experiment on Objects365-based data.

The designs offer different localization interfaces. OMG-LLaVA generates a [SEG] representation that conditions a visual decoder; its mask is then resized, thresholded, and converted into a box. Locus predicts boundaries through the autoregressive decoder and reads continuous coordinates from its output distributions without an additional localization model pass. VES and CCP strengthen this direct path by jointly supervising phrase-to-region correspondence and box geometry. The results demonstrate a favorable accuracy–throughput trade-off for Locus on box localization, while OMG-LLaVA's decoder additionally supports pixel-level mask prediction.

Reviewer_S1oSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Locus, a framework to improve object localization in vision-language models (VLMs) by fine-tuning Qwen3-VL-4B using two main components: Visual Evidence Supervision (VES) and Continuous Coordinate Prediction (CCP). VES encourages the model to attend to the correct visual regions while preventing attention from concentrating on only a few image patches. CCP improves coordinate prediction through soft coordinate supervision, distance-aware loss, box validity constraints, and GIoU optimization.

The authors evaluate Locus on multiple object detection and visual grounding benchmarks, including COCO, LVIS, and RefCOCOg. The results show promising improvements over existing VLM-based methods. Ablation studies demonstrate the contributions of the proposed loss components, while qualitative results illustrate the model's localization performance.

Critical strengths

1.The paper addresses an important limitation of VLMs in object localization, particularly the lack of explicit visual attention supervision and the limitations of discrete coordinate prediction.

2.The proposed VES and CCP losses improve visual grounding and coordinate precision without requiring additional detection modules or architectural changes.

3.Locus achieves promising results across multiple object detection and visual grounding benchmarks, outperforming existing VLM-based methods on several metrics.

4.The authors evaluate the contributions of the proposed loss components, showing how VES and CCP individually and jointly improve localization performance.

5.The paper is well-organized, and the methodology is easy to understand and follow.

Critical weaknesses

1.All experiments are conducted using Qwen3-VL-4B. Although the authors present Locus as a general framework, it remains unclear whether the proposed method generalizes to other VLM architectures or model sizes.

2.The qualitative results in Figure 3 only show predictions from Locus without comparing them with the original Qwen3-VL-4B model. Including the baseline predictions on the same images would help demonstrate how the proposed method improves localization.

3.The authors introduce several hyperparameters, such as the minimum coverage ratio rcov=0.4r_{\mathrm{cov}}=0.4, but do not provide sufficient analysis to justify these choices or show how sensitive the performance is to different values.

4.Although Locus does not introduce additional detection modules, the paper does not provide a detailed comparison of inference time or computational cost against existing VLM-based methods. This makes it difficult to assess its practical efficiency.

Questions for authors
  1. Since VES encourages attention to focus on the target bounding box, how does it handle referring expressions that require contextual information outside the target region? Fro example “The person standing next to the bus”

2.In Target Coverage Regularization, the coverage ratio rcovr_{\mathrm{cov}} is introduced in Equation (7), but its role is not clearly explained in Section 3.2. Although the authors set it to 0.4 in Section 4.1, this choice is not justified. How sensitive is the model's performance to this parameter? Could the authors provide a sensitivity analysis using different values, such as 0.2, 0.4, 0.6, and 0.8?

3.The optimal coverage ratio may depend on object size, shape, and the amount of background included in the bounding box. Since GIoU is computed using the expected continuous coordinates during training, while autoregressive decoding remains conditioned on discrete coordinate tokens, how does this difference affect geometric consistency during inference?

4.Although Locus achieves strong zero-shot performance among VLM-based methods, DINO-Swin-L still outperforms it on COCO. Could the authors further discuss the trade-offs between Locus and task-specialized detectors, particularly in terms of localization accuracy and inference efficiency?

5.How well does Locus generalize to other VLM architectures, such as LLaVA or InternVL? Could the authors provide additional experiments or discuss the architectural requirements of the proposed method?

Minor Issue: Table 1 states that bold and underlined values indicate the best and second-best results, respectively. However, the underlining appears to be applied only to the LVIS columns and not consistently to the COCO columns. The authors should ensure that the formatting is consistent across both datasets.

3 comments

Thank you for your questions about relational grounding, coverage regularization, geometric consistency, and generalization. We clarify how VES preserves access to contextual information, explain the coverage threshold and continuous coordinate readout, and report paired fine-tuning results on three additional VLM checkpoints. We also discuss the accuracy–efficiency trade-off and the presentation improvements we will make.

1. Relational expressions and visual evidence supervision

VES supervises a phrase-conditioned, image-only affinity distribution at selected response positions in the final six decoder layers. This auxiliary distribution is computed from the decoder's native query and key projections, while native self-attention continues to access all image tokens. For “the person standing next to the bus,” the bus provides contextual information for identifying the intended person, and the person's patches provide spatial evidence for localizing that person's boundaries. Contextual information therefore remains available throughout autoregressive decoding, including in the supervised layers.

The annotated box provides VES with an explicit link between the referring phrase and its visual target. The coverage term then acts on affinities renormalized within that box, encouraging spatially distributed support for boundary localization. On the RefCOCOg test split, adding VES alone increases mean F1 from 74.6% to 78.1%, a gain of 3.5 percentage points (manuscript Table 6(a)). This controlled improvement supports the effectiveness of explicit visual evidence supervision for referring-expression localization.

2. Meaning of the coverage ratio and sensitivity analysis

In manuscript Eq. (7), rcovr_{\mathrm{cov}} specifies the minimum effective coverage as a fraction of the patches overlapping the target box. For a phrase–instance pair gg, let IgI_g denote these patches and NgN_g their number. Let aˉg,jℓ\bar{a}_{g,j}^{\ell} denote the affinity at layer ℓ\ell, normalized within the target box for each head and then averaged across heads. The effective patch count is

= \frac{1}{\displaystyle\sum_{j\in I_g}\left(\bar{a}_{g,j}^{\ell}\right)^2}.$$ A distribution concentrated on one patch has an effective count of 1, while a uniform distribution over $m$ patches has an effective count of $m$. The regularizer therefore measures how broadly the visual evidence is distributed within the target box. For boxes with $N_g\geq m_{\mathrm{cover}}=4$, the coverage penalty is active when $N_{\mathrm{eff},g}^{\ell}<r_{\mathrm{cov}}N_g$. For example, if a box overlaps 10 patches and $r_{\mathrm{cov}}=0.4$, the penalty becomes zero once the effective patch count reaches 4. Boxes overlapping fewer than $m_{\mathrm{cover}}=4$ patches contribute zero coverage loss. This makes the threshold interpretable in relation to the spatial support available for each target. We evaluated $r_{\mathrm{cov}}\in\{0.2,0.4,0.6,0.8,1.0\}$ on HumanRef and Dense200 (Appendix, Fig. 4). Mean F1 is highest at $r_{\mathrm{cov}}=0.4$ on both benchmarks and decreases at larger tested values. The consistent trend across referring-expression and dense-detection benchmarks supports the choice of $r_{\mathrm{cov}}=0.4$ in our main experiments. ### 3. Coverage across objects and geometric consistency at inference The coverage target $r_{\mathrm{cov}}N_g$ scales with the number of patches overlapping each box, while the $m_{\mathrm{cover}}=4$ rule handles targets with very limited patch support. Object shape and box tightness determine how foreground and background are distributed among these patches. The choice $r_{\mathrm{cov}}=0.4$ encourages broader evidence while allowing nonuniform affinity within the box: the regularizer reaches zero when the effective coverage target is met. The HumanRef and Dense200 sweep in Fig. 4 provides empirical support for this setting across the two evaluated tasks. Training and inference use the same expectation-based coordinate readout. During training, the decoder predicts coordinate-token distributions under teacher-forced discrete prefixes, and GIoU supervises the box assembled from their expected coordinates. During inference, each generated discrete token is fed back into the autoregressive decoder; the recorded coordinate-token distributions are then converted into continuous coordinates by the same expectation operation. The final reported box therefore uses continuous coordinates, while the generated discrete sequence supplies the context for subsequent decoding steps.

Manuscript Table 4(b) directly evaluates whether this training objective improves free-running localization. With Wasserstein and validity losses enabled and the same free-running decoding and expectation readout, adding GIoU increases COCO [email protected] from 18.6% to 22.5%, a gain of 3.9 percentage points. This demonstrates that optimizing expected boxes during training improves precise localization when the model generates its own discrete prefixes at inference.

4. Localization accuracy and inference efficiency

DINO-Swin-L achieves 62.1% mean F1 on COCO, compared with 57.5% for Locus, demonstrating its strength on this detection benchmark. Locus provides a unified language-conditioned interface for object detection, dense detection, and referring-expression grounding. Its results on COCO, LVIS, Dense200, VisDrone, HumanRef, and RefCOCOg demonstrate the applicability of this design across localization tasks and visual domains. The VLM interface also allows the localization request to specify attributes, spatial relations, and detailed descriptions directly in language, as illustrated by the referring-expression evaluations.

Locus retains the backbone architecture and predicts boxes through the existing autoregressive decoder. VES and the auxiliary geometric losses are used during training; inference reads continuous coordinates from the decoder's output distributions. For the dedicated-module comparison on RefCOCOg test, Table R3e reports measured throughput of 2.1 boxes per second for Locus-4B and 0.9 for OMG-LLaVA-7B, together with mean F1 scores of 82.4% and 65.7%, respectively. These results characterize the practical accuracy–throughput trade-off of the evaluated systems for box localization, while OMG-LLaVA's visual decoder also provides pixel-level mask outputs.

5. Generalization across VLM families and model sizes

We evaluated Locus on Rex-Omni-3B, InternVL3-2B, and Qwen3-VL-8B. For each checkpoint, we perform paired fine-tuning from the same official initialization using the same Objects365-based data containing detection and referring-expression annotations. “Standard SFT” denotes the supervised fine-tuning baseline, and “SFT + VES/CCP” adds the two Locus components. Tables R2a and R2b present the resulting comparisons separately for COCO and RefCOCOg test.

Table R2a. Paired fine-tuning comparison on COCO. All scores are percentages.

ModelFine-tuning objective[email protected][email protected]Mean F1
Rex-Omni-3BStandard SFT73.516.953.2
Rex-Omni-3BSFT + VES/CCP77.518.256.1
InternVL3-2BStandard SFT60.513.244.8
InternVL3-2BSFT + VES/CCP63.816.149.2
Qwen3-VL-8BStandard SFT63.816.246.5
Qwen3-VL-8BSFT + VES/CCP66.220.951.8

Table R2b. Paired fine-tuning comparison on RefCOCOg test. All scores are percentages.

ModelFine-tuning objective[email protected][email protected]Mean F1
Rex-Omni-3BStandard SFT87.235.574.8
Rex-Omni-3BSFT + VES/CCP89.737.177.7
InternVL3-2BStandard SFT85.329.470.5
InternVL3-2BSFT + VES/CCP88.531.873.5
Qwen3-VL-8BStandard SFT89.232.173.2
Qwen3-VL-8BSFT + VES/CCP92.234.276.3

Locus improves mean F1 on COCO by 2.9, 4.4, and 5.3 percentage points for Rex-Omni-3B, InternVL3-2B, and Qwen3-VL-8B, respectively. The corresponding gains on RefCOCOg test are 2.9, 3.0, and 3.1 percentage points. The paired comparisons show consistent improvements across the evaluated VLM families and model sizes on both detection and referring-expression grounding.

The implementation requirements follow directly from the two objectives. VES requires access to the decoder's query and key projections and a mapping from visual tokens to image patches, so that phrase-conditioned affinities can be associated with annotated regions. CCP requires an autoregressive coordinate-token interface with an ordered mapping between token bins and spatial values, together with access to the corresponding logits. These interfaces let the same supervision principles operate on each model's existing representations and coordinate distributions. We will make these architectural requirements explicit in the revised method description.

6. Qualitative comparisons and table formatting

We will add predictions from the original Qwen3-VL-4B model alongside Locus on the same images and referring expressions in Fig. 3, using consistent colors for ground truth and model predictions. This side-by-side presentation will make the localization differences directly visible. We will also update Table 1 so that bold and underlined values consistently denote the best and second-best results across both the COCO and LVIS columns.

I thank the authors for their detailed response and additional experiments. In particular, the results across different VLM architectures and model sizes address my main concern regarding the generalizability of Locus. I also appreciate the clarifications regarding coverage regularization, geometric consistency, and inference efficiency. I encourage the authors to incorporate these additional results and clarifications into the revised manuscript. Overall, I maintain my original rating of weak acceptance

Reviewer_n6uwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes to enhance current VLM's grounding capability by adding specific visual evidence supervision that re-distributes the attentions inside LLM layers toward target regions and avoids few patch tokens receiving high attentions, together with a continuous coordinate prediction module that integrates three losses. The experimental results are good, surpassing various state-of-the-art VLMs on visual grounding.

Critical strengths

The method is simple, the idea of attention redistribution and encouraging model to focus on real patches that belong to target referent to boost precise localization is well motivated, but meanwhile, being very effective, as shown in Table 1-3. The proposed two losses both contributes to the final performance with non-negligible margins.

Critical weaknesses
  1. The writing could be substantially improved, especially on mathematical notations and their formal definitions, the equations are with many superscripts and subscripts, very hard to read, and some symbols miss formal definitions, leaving readers only to guess their meaning. And some sentences have grammar mistakes, I highly suspect that it is AI-written and authors don't have enough time to polish.
  2. There is no ablation on Eq. 10, the continuous coordinate supervision loss, authors should compare it with cross-entropy loss applied on discrete tokens.
  3. The continuous coordinate prediction component, though with multiple losses, each loss is not new. GIoU loss is well developed and improves a lot as shown in Table. 4b. The distance-aware loss in Eq. 13 has been studied in several previous works with similar ideas including but not limited to: 1) Regress, Don’t Guess – A Regression-like Loss on Number Tokens for Language; 2) Teaching Metric Distance to Discrete Autoregressive Language Models; 3) Enhancing Numerical Prediction of MLLMs with Soft Labeling. This reduces the novelty of this paper.
Questions for authors
  1. Please clarify the difference between CCP component and previous works that studies distance-aware coordinate prediction and switching from discrete token prediction to continuous coordinate prediction.
  2. Please ablate over discrete token prediction versus continuous coordinate prediction in Eq. 10.
3 comments

Thank you for highlighting the coordinate-supervision ablation and the connections to numerical-token learning. We clarify the formulation and inference procedure, provide a matched comparison of hard-label and two-bin coordinate supervision, and explain how Locus connects visual evidence with continuous box geometry.

1. Notation and presentation

The formulation follows a phrase–region–coordinate correspondence. In Section 3.2, a phrase–instance pair is denoted by g=(r,i)g=(r,i), and IgI_g contains the image patches overlapping its target box. The phrase supplies the query representation, while the instance specifies the target region. For a fixed pair, the affinity and coverage quantities can be written with fewer subscripts. The weight wg=1/(∣R∣Nr)w_g=1/(\lvert\mathcal R\rvert N_r) preserves the averaging over instances within each phrase and then over phrases.

Section 3.3 distinguishes three distributions: the interpolated coordinate target qq, the full-vocabulary prediction PVP_{\mathcal V}, and the coordinate-only distribution p(k∣C)p(k\mid\mathcal C). Eq. (10) supervises coordinate positions over the full vocabulary; the geometric objectives operate on the ordered coordinate distribution. This separates the requirement to generate a coordinate token from the spatial structure of its predicted value.

Section 3.4 specifies the two operations at inference: the generated discrete token is fed into the autoregressive decoder, and the corresponding coordinate distribution is used to recover the continuous value reported in the final box. This makes the connection between token generation and continuous localization explicit.

2. Hard-label cross-entropy versus the two-bin target in Eq. (10)

Table R1 compares hard-label coordinate cross-entropy with the two-bin soft target using the same backbone, training data, and VES configuration. All four variants use expectation readout at inference, so the comparison evaluates the training targets under a common continuous-coordinate readout. The geometry-loss group comprises Wasserstein, validity, and GIoU supervision.

Table R1. Coordinate-supervision ablation on COCO. All scores are percentages.

Coordinate supervisionGeometry losses[email protected][email protected]Mean F1
Hard-label CEOff64.212.546.9
Two-bin soft target, Eq. (10)Off67.815.650.1
Hard-label CEOn72.819.555.4
Two-bin soft target, Eq. (10)On74.322.557.5

The two-bin target improves mean F1 by 3.2 percentage points without geometry losses and 2.1 points with them. Its advantage is also visible at the strict IoU threshold: [email protected] increases from 12.5 to 15.6 and from 19.5 to 22.5, respectively.

The interpolation preserves the continuous target in the supervision itself. For example, a target coordinate of 527.3 assigns probabilities 0.7 and 0.3 to bins 527 and 528, yielding the exact target expectation. Thus, the target retains fractional boundary information while remaining compatible with coordinate-token prediction. The results show that this information improves localization both before and after adding box geometry supervision.

3. CCP and its connection to prior numerical-token objectives

Regress, Don't Guess develops regression-style objectives for numerical tokens, including Wasserstein supervision. DIST²Loss converts token distances into reward-weighted targets, and Enhancing Numerical Prediction of MLLMs with Soft Labeling assigns probability to nearby numerical values. The latter two also demonstrate the value of distance-aware supervision for visual grounding. Generalized Focal Loss, discussed in our related work, provides a precedent for adjacent-bin interpolation and distribution-based coordinate readout.

Locus advances this direction by jointly supervising visual correspondence, boundary distributions, and the geometry of the assembled box within autoregressive grounding. These levels address connected requirements: the phrase must identify the intended region, each boundary must be placed precisely, opposite boundaries must be ordered consistently, and all four boundaries must form a well-aligned box.

CCP implements this geometric structure through complementary objectives:

  1. Fractional boundary supervision. Two-bin interpolation retains the continuous position of each target boundary.
  2. Ordered distribution supervision. Wasserstein distance penalizes displacement of probability mass along the coordinate axis.
  3. Boundary-order supervision. The validity objective encourages consistent left–right and top–bottom ordering across boundary distributions.
  4. Whole-box supervision. GIoU directly optimizes the spatial overlap and configuration of the box recovered from the four expected coordinates; the two axis pairs are sorted when computing this loss.

Whole-box geometry adds information beyond individual coordinate errors. Equal absolute errors at the four boundaries can shift, shrink, or expand a box, producing different overlap with the target. By optimizing the assembled box, CCP trains the coordinate distributions according to the localization quality they jointly produce. VES connects this geometric learning to the phrase-conditioned visual regions from which the boundaries should be inferred.

The ablations support both levels of supervision. With geometry losses enabled, two-bin interpolation improves mean F1 from 55.4 to 57.5 in Table R1. With Wasserstein and validity enabled, adding GIoU improves COCO [email protected] from 18.6 to 22.5, a 3.9-point gain in manuscript Table 4(b). Together, these results establish the value of preserving fractional coordinates and directly supervising box geometry in the autoregressive framework.

Dear Reviewer,

Thank you again for your thoughtful review. With one day remaining in the discussion period, we would appreciate the opportunity to hear your thoughts on our rebuttal, particularly whether our responses have clarified your concerns.

If any questions remain or further clarification would be helpful, we would be happy to follow up before the discussion closes.

Thank you for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.