- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes Locus, a fine-tuning recipe for autoregressive VLM-based detection and visual grounding, built on Qwen3-VL-4B, that keeps the coordinate-token interface unchanged. It adds two groups of auxiliary objectives. Visual Evidence Supervision (VES) combines an attention concentration loss, which maximizes phrase-conditioned attention mass on the visual patches overlapping the ground-truth box, with a coverage regularizer that penalizes attention collapsing onto few patches. Continuous Coordinate Prediction (CCP) guides the model how to predict coordinate tokens. The paper reports gains on COCO, LVIS, Dense200, VisDrone, HumanRef and RefCOCOg, together with ablations of each component.
Critical strengths
- The method requires no architectural changes and preserves the autoregressive generation interface.
- The evaluation covers a broad set of detection, dense-detection, and referring benchmarks, with component ablations on three of them.
Critical weaknesses
1.The motivation is unclear and not justified. The introduction states that token-level supervision does not constrain the correspondence between the expression and its supporting visual evidence, but this is asserted rather than demonstrated. A well-trained VLM should learn to condition each coordinate token on the relevant visual context, as language models do for next-token prediction, and it is unclear why the existing mechanisms are insufficient. Likewise, if the predicted coordinates are correct, it is unclear why the supporting evidence requires separate supervision. The only evidence offered consists of two examples in Fig. 1, whose caption does not explain the box colors, and two attention maps in Fig. 5. The claimed limitation should be justified experimentally. Another point that was unclear to me is that attention concentration may conflict with relational grounding. Localizing a referent among similar objects (e.g., "the second boy from the left" in Fig. 2) plausibly requires attending to distractors and context, not only to the target region. In addition the absence of a dedicated module is presented as an advantage without justification. Decoder-based approaches such as OMG-LLaVA (and OMG-Seg-style decoders) typically add small modules and often yield solid gains. The paper neither shows that the end-to-end design is better in a quality/complexity trade-off nor compares against such approaches. The claim that quantized coordinates "reduce prediction complexity" is also not supported by a citation or an experiment.
- The "zero-shot" claims and comparisons are unclear. Locus is trained with the box-generation objective on a large detection and referring corpus, and "zero-shot" is never defined. It is unclear whether the training data overlaps with COCO or LVIS (the referring datasets in Table 5 are built on COCO images). Table 1 compares models with different training data and budgets, so the state-of-the-art claim cannot be attributed to the method alone. The only controlled comparison is the autoregressive baseline in the ablations.
Questions for authors
- What experiment shows that standard token-level training leaves the visual evidence insufficiently constrained?
- Would that make sense to compare L_coord + L_W with a simple L1/regression loss on the coordinates to simplify the objective?
- Would that make sense to compare Locus to an autoregressive baseline that uses post-hoc box sorting instead of L_valid?
- How is "zero-shot" defined, and does the training data overlap with COCO or LVIS images?
- How does Locus compare with a dedicated-module approach (such as OMG-LLaVA) in terms of quality and complexity?
8 comments
3. Comparison with L1 regression
We compare our coordinate objective with L1 regression on expected coordinates. Both settings retain coordinate-token type supervision, validity supervision, and GIoU, so that both learn to generate coordinate tokens and optimize box geometry.
The key distinction is what the coordinate objective supervises. L1 evaluates the expected location; different distributions can share that expectation. For example, equal mass at two distant bins can have the correct mean despite assigning no mass near the target. Two-bin supervision instead preserves the fractional target location, and Wasserstein supervision penalizes displacement of probability mass along the ordered bins. This provides a learning signal beyond the accuracy of the mean alone.
Table R3c. Coordinate-supervision comparison on COCO. Both settings retain coordinate-token type supervision, validity, and GIoU. Scores are percentages.
| Coordinate supervision | [email protected] | [email protected] | Mean F1 |
|---|---|---|---|
| L1 regression | 71.1 | 17.5 | 54.2 |
| 74.3 | 22.5 | 57.5 |
Distributional supervision improves Mean F1 by 3.3 points and [email protected] by 5.0 points, supporting its contribution to precise localization beyond expected-coordinate regression.
4. Validity supervision versus post-hoc sorting
These approaches intervene at different stages. Post-hoc sorting corrects boundary order after generation. The validity objective supplies gradients during training that encourage the coordinate distributions to respect and . This is especially useful alongside GIoU: GIoU evaluates the assembled box after its expected boundaries are ordered, whereas validity supervision directly encourages the predicted boundary roles to remain consistent.
In manuscript Table 4(b), adding validity to Wasserstein supervision improves COCO [email protected] from 69.9% to 70.5% and [email protected] from 17.8% to 18.6%. We additionally compare the complete model with a variant trained without validity supervision that applies sorting at inference.
Table R3d. Validity supervision and post-hoc sorting on COCO. Scores are percentages.
| Method | [email protected] | [email protected] | Mean F1 |
|---|---|---|---|
| Without + inference-time sorting | 73.1 | 21.3 | 55.2 |
| Locus with | 74.3 | 22.5 | 57.5 |
The gains of 2.3 Mean-F1 points and 1.2 [email protected] points support learning consistent boundary ordering during training, beyond repairing the final boundary order.
5. Definition of zero-shot evaluation and data overlap
Following the evaluation usage in Rex-Omni and LocateAnything, “zero-shot” in our COCO and LVIS comparisons denotes evaluation without training on those evaluation images. We checked the fine-tuning corpus for image-level overlap with both evaluation splits and excluded overlapping images from training.
This check applies across data sources, including the COCO-derived referring datasets listed in Table 5. The relevant separation is therefore defined at the image level across the combined training corpus, rather than by dataset name alone. Locus learns grounding from the remaining detection and referring annotations and is evaluated on the excluded benchmark images. We will make this definition explicit alongside Table 1. The benchmark results and matched ablations provide complementary evidence for Locus. Table 1 establishes its performance relative to existing detection and grounding systems; under the same training setting, manuscript Table 4(c) shows that Locus improves COCO Mean F1 from 46.5% for the autoregressive baseline to 57.5%, an 11.0-point gain. The paired cross-model experiments further show consistent gains when applying VES and CCP to the same initialization and fine-tuning data. These comparisons connect the benchmark performance to the proposed training objectives.
6. Comparison with a dedicated-module approach
We evaluate OMG-LLaVA, which produces segmentation masks through a dedicated visual decoder, and convert its masks to boxes for RefCOCOg test evaluation. Locus-4B reaches 82.4% Mean F1 and 43.5% [email protected], with a measured throughput of 2.1 boxes per second.
Table R3e. RefCOCOg test comparison. F1 scores are percentages; BPS is measured boxes per second.
| Model | [email protected] | [email protected] | Mean F1 | BPS |
|---|---|---|---|---|
| OMG-LLaVA-7B | 80.0 | 31.5 | 65.7 | 0.9 |
| Qwen3-VL-8B (manuscript Table 3 checkpoint) | 88.6 | 33.8 | 75.2 | 1.5 |
| Locus-4B | 92.8 | 43.5 | 82.4 | 2.1 |
Qwen3-VL-8B, whose nominal size is closer to OMG-LLaVA-7B, also achieves higher Mean F1 (75.2% versus 65.7%) and measured throughput (1.5 versus 0.9 BPS). This row uses the Qwen3-VL-8B checkpoint reported in manuscript Table 3; Tables R2a and R2b report the additional paired fine-tuning experiment on Objects365-based data.
The designs offer different localization interfaces. OMG-LLaVA generates a [SEG] representation that conditions a visual decoder; its mask is then resized, thresholded, and converted into a box. Locus predicts boundaries through the autoregressive decoder and reads continuous coordinates from its output distributions without an additional localization model pass. VES and CCP strengthen this direct path by jointly supervising phrase-to-region correspondence and box geometry. The results demonstrate a favorable accuracy–throughput trade-off for Locus on box localization, while OMG-LLaVA's decoder additionally supports pixel-level mask prediction.
Thank you for the detailed questions about visual evidence, coordinate supervision, and evaluation. We provide additional experiments on target-region affinity, L1 regression, post-hoc sorting, and a dedicated-module approach, and clarify our data protocol. Together, these results support Locus's central contribution: jointly supervising visual evidence and continuous box geometry through the autoregressive decoder.
1. Motivation for visual evidence supervision
Precise localization requires the model to identify the target and translate its visual evidence into box boundaries. Standard token-level supervision directly optimizes the generated coordinates, while the correspondence between the expression and its supporting visual regions is learned indirectly. Previous work has shown that VLMs may encode relevant visual evidence yet fail to use it effectively when producing an answer (Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs, ICLR 2026).
This issue is particularly relevant when multiple objects are generated sequentially. As the output grows, later predictions are conditioned on an increasingly long generated prefix and may rely more heavily on preceding text. Zhao et al. document language bias in VLMs (Looking Beyond Text: Reducing Language Bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance, EMNLP 2025), and Sun et al. observe declining reliance on image inputs during long multimodal reasoning (Mitigating Visual Forgetting via Take-along Visual Conditioning for Multi-modal Long CoT Reasoning, ACL 2025). These findings motivate testing whether explicit visual supervision helps when generating multiple boxes.
We argue that each object and its boundaries require evidence from the image. Related work also shows that adding region-based attention supervision to next-token training improves visual localization tasks (Direct Visual Grounding by Directing Attention of Visual Tokens, arXiv:2511.12738). This supports an explicit training signal linking referring phrases to their corresponding visual regions. VES provides this signal by supervising phrase-conditioned visual affinities at selected response positions in the final six decoder layers. The decoder's native attention retains access to the full image, including at the supervised layers. This allows contextual cues to support referent identification while encouraging attention to the target region for localization. The coverage term further encourages spatially distributed support within that region, providing evidence for precise boundary prediction.
We test VES directly by adding it to the CCP-equipped model. On COCO, target-region affinity increases from 24.3% to 67.8%, alongside a Mean F1 increase from 52.1% to 57.5%. On RefCOCOg test, the corresponding changes are 21.9% to 65.4% and 79.8% to 82.4%. Thus, explicit evidence supervision improves localization even when coordinate and box geometry are already supervised.
Table R3a. VES ablation on COCO.
Table R3b. VES ablation on RefCOCOg test.
In Tables R3a and R3b, target affinity is the mass assigned to box-overlapping patches by the phrase-conditioned, image-normalized affinity in Eqs. 2–3. F1 scores are percentages. The appendix attention maps illustrate the corresponding spatial changes in two examples.
We will also revise Figure 1's caption to explain the box colors.
2. Why use quantized coordinate tokens?
Atomic coordinate tokens provide a compact autoregressive interface: each boundary occupies one coordinate token, rather than a potentially longer digit sequence. This representation is used by Rex-Omni (arXiv:2510.12798) and LocateAnything. In Locus, the discrete bins also provide an ordered support for learning a coordinate distribution. The discrete interface and continuous output serve complementary purposes: discrete tokens drive autoregressive generation, while expectation readout recovers fractional coordinates from their probabilities. Consequently, the reported boundary need not coincide with an integer bin.