- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies referring expression comprehension. The authors argue that current VLMs often pick the wrong object or answer without checking. They propose to treat REC as a multi-step process in which the model uses tools to collect evidence before it answers. The main challenges they address are the lack of step-by-step training data and the difficulty of teaching a model to use tools well with RL. To this end, they build a trajectory dataset, construct a set of visual tools, and train the model with SFT followed by GRPO with a new tool-aware reward. They report state-of-the-art results on several REC benchmarks.
Critical strengths
S1. Step-by-step checking clearly helps on generalized REC.
- It is much better than prior methods at rejecting sentences that match no object and at finding several targets (Table 2).
- The examples in Fig. 3 show that it checks each condition before answering.
Why this matters to my score: This is the strongest support for the paper's main idea.
S2. The tool reward is a simple and useful idea.
- Tools are rewarded only when they help, and short answers only when they are correct (Sec. 3.4).
- This leads to fewer tool calls and shorter outputs with no loss in accuracy (Table 4).
Why this matters to my score: Agents trained with RL often overuse tools, and this fix is easy to reuse.
Critical weaknesses
W1. The experiments do not show that the gains come from the agent itself.
- Some tools call an unnamed VLM (App. D), and the training data comes from a 27B model (Sec. 3.2). Part of the gain may come from these larger models.
- The base model seems under-tested (Table 3). Key baselines are missing: general VLMs, the 27B model with the same tools, and RL without tools (Tables 1 and 3).
- Tool cost is not counted (Table 4), and the closely related Rex-Thinker (ICLR 2026) is not discussed.
Why this matters to my score: The gains may come from larger models or weak baselines rather than from the agent. This is the main reason for my score.
W2. The test setup may leak test data, and key details are missing.
- Ref-L4 val is used for training (App. A.1) but is also part of the reported test results (Sec. 4.1, Table 2). Overlap is checked per example, not per image (App. A).
- The main reward is never defined, and Eq. 9 gives a failed tool call a higher score than a valid one that does not help (Sec. 3.4).
- No variance is reported, although many ablation gaps are small (Table 3). The paper also contradicts itself in places (e.g., Sec. 3.2 vs. App. A.4).
Why this matters to my score: I cannot tell whether the results are free of test data, reproduce the reward, or trust the ablations.
Questions for authors
- Which model answers the QLA, REPHRASE, VISUAL_GROUNDING, and VISION_SELF_CHECK calls during data collection, during RL, and during testing? How do accuracy and run time change if the 4B policy answers these calls itself?
- Are the Ref-L4 results in Table 2 on the test split only? How many images are shared between the training data and each test split?
- How do these baselines perform on the same benchmarks?
- (a) the 27B model with the same tools;
- (b) the tools without a trained policy;
- (c) GRPO with an IoU reward and no tools, on the same 3.2K examples.
- How is RecAgent different from Rex-Thinker (ICLR 2026)?
- No tool adjusts box coordinates, and RecAgent's Ref-L4 [email protected] is below CogVLM-Grounding's (47.50 vs. 48.35). What evidence shows that RecAgent draws better boxes for objects it has already identified correctly?
3 comments
Q3: Performance of the requested baselines.
The table above summarizes performance under the different settings.
- (a) 27B + same tools: As clarified in Q1, all VLM-based tools use the model executing the task, so both the policy and these tools use the 27B model in this setting. Formatting and parsing failures substantially reduce its end-to-end performance, which also motivates the additional trajectory-correction steps used during data construction. Row (d) reports performance after excluding these failures. The higher scores suggest stronger grounding capability when the larger model produces a valid answer; this is a conditional subset rather than a full-sample evaluation. Using a larger model also entails greater computational cost. Our central objective is to improve a given VLM through tool-augmented training, so this model-scale comparison should be considered alongside the fixed-backbone 4B ablations.
- (b) Tools without a trained policy: Table 4 already includes an SFT-trained tool-use policy without RL. Row (b) additionally considers the 4B model without task-specific training. Its lower scores than direct grounding are consistent with difficulties adapting to the tool interface, motivating the SFT cold start.
- (c) GRPO without tools: The no-tool SFT + RL baseline is reported in Table 3(c), and its scores are reproduced in row (c) above.
| ID | Setting | RefCOCO | RefCOCO+ | RefCOCOg |
|---|---|---|---|---|
| (a) | 27B + same tools | 44.52 | 43.68 | 56.63 |
| (b) | Untrained 4B + tools | 70.24 | 63.97 | 62.68 |
| (c) | 4B + SFT + RL, no tools | 87.02 | 80.11 | 82.67 |
| (d) | 27B + same tools (excluding formatting and parsing errors) | 92.50 | 92.18 | 93.51 |
| (e) | Ours | 89.43 | 83.47 | 85.86 |
Q4: How does RecAgent differ from Rex-Thinker?
Both methods use explicit reasoning, SFT, and GRPO to improve REC. They differ primarily in their research focus, evidence acquisition, and input requirements.
(a) Research focus: Rex-Thinker emphasizes interpretable and verifiable candidate selection; RecAgent learns how to acquire grounding evidence through tool use. Rex-Thinker aims to make grounding decisions interpretable and visually verifiable, while supporting abstention when no object matches the expression. RecAgent focuses on deciding what evidence is still needed, which tools can provide it, and when there is enough evidence to stop. Accordingly, our training targets both localization and the interaction policy.
(b) Execution: Rex-Thinker detects candidates before reasoning; RecAgent invokes tools during reasoning and adapts subsequent actions to their outputs. Rex-Thinker applies Planning–Action–Summarization reasoning to the image, box hints, and referring expression. Its official single-image implementation first detects candidates and then generates the reasoning and answer in a single model call. RecAgent instead interleaves policy decisions with tool execution. After observing a tool's output, it can request a crop, check attributes or spatial relations, invoke a tool again, or produce the final answer. This allows subsequent actions to address unresolved constraints without prescribing a fixed tool sequence. For example, if color is established but relative depth remains ambiguous, the policy can request depth evidence; when sufficient evidence is available, it can stop.
(c) Inputs: Rex-Thinker additionally receives target-category labels in its RefCOCOg evaluation; RecAgent starts with only the image and referring expression. In the reported RefCOCOg experiments, Rex-Thinker uses the original COCO category label to prompt GroundingDINO for candidate detection. RecAgent's policy generates the entity descriptions and retrieval parameters from the image and expression. Such category annotations may be unavailable in deployment, in which case the former pipeline requires an additional category-inference step. RecAgent incorporates the generation of retrieval descriptions into its policy. The distinction is therefore whether the target-category prior is supplied externally or the policy derives a retrieval description from the task input.
Q5: Does RecAgent produce more precise boxes once the correct object is identified?
Thank you for this insightful question. Our current tool suite does not include a tool that directly adjusts bounding-box coordinates.
- Changing the prompt can alter the predicted box. For example, rewording a query through REPHRASE can lead the grounding model to produce a different box. However, how the model arrives at these adjustments is not fully interpretable. A tool that explicitly refines coordinates would therefore be a valuable addition. In practice, though, we have found that models struggle with fine-grained numerical adjustments: they can often identify a reasonable region, but precisely describing and correcting small boundary errors through text is much harder. This may partly reflect the limitations of generating spatial coordinates autoregressively as language tokens. With this in mind, a promising direction is to incorporate semantic segmentation for finer refinement. For example, the predicted box could serve as a region proposal for segmentation, with the resulting mask then used to adjust the box boundaries.
- Another complication is noise in the datasets themselves, which we have found particularly noticeable in RefCOCO and RefCOCO+. In some cases, a model may produce a visually more precise box yet receive a lower score because it differs from the ground-truth annotation. For example, these datasets contain many part-to-whole references: an expression such as “blue shirt” may be annotated with a box around the person wearing the shirt, rather than the shirt alone. We regard such discrepancies as a form of dataset bias that warrants careful consideration when interpreting grounding performance. They also raise a broader question: whether revisiting the annotations and cleaning the existing datasets could yield a better benchmark for evaluating precise localization. Although this is not the focus of our work, we consider it a worthwhile direction for future research.
Rebuttal for Reviewer Jvxm
Q1: Which models execute the tools at each stage.
We thank the reviewer for their patience, thorough review, and thoughtful feedback. We sincerely apologize for the ambiguity in our wording and the resulting misunderstanding. All tools requiring VLM inference use the model currently executing the task: Qwen3.8-27B during offline data construction, and the 4B policy during RL and testing. The 27B teacher is therefore used for offline supervision, not online tool execution. The terms “current Qwen backend,” “same backend,” and “contextual Qwen vision model” in Appendix D all refer to the model used at the corresponding stage.
This design reuses the VLM’s existing capabilities, such as REPHRASE and VISUAL_GROUNDING, as modular operations that the policy can invoke on demand. During SFT, the 4B model learns from teacher-generated trajectories containing the 27B model’s tool observations, providing demonstrations of how to interpret and act on these observations. More broadly, the aim is to organize VLM reasoning into explicit, task-specific skills that can be optimized and selectively invoked.
For example, during RL and testing, VISION_SELF_CHECK draws numbered candidate boxes on the image and returns the annotated image to the 4B model, which assesses which candidates match the referring expression. This verification is performed entirely by the 4B model itself, without any calls to the 27B model.
Q2: the problem of Ref-L4.
Table 2 reports results on Ref-L4 val+test, consistent with Table 4 of the original Ref-L4 paper and the cited baselines. We apologize for the ambiguous reference to a “disjoint test split” in Appendix A.1. To cover compositional attributes and relational reasoning in long expressions, we sampled 800 candidate examples from the Ref-L4 validation split. Since some baselines could not be fully reproduced, we retained the full val+test evaluation set to maintain comparability with their published results. To reduce direct textual overlap between adaptation and evaluation queries, we used an LLM to rewrite the sampled queries while preserving their referential meaning and constraints. This changes the wording while retaining the original images and target annotations.
Example of Query Rewriting:
For a fairer comparison, we evaluate both our model and CogVLM-Grounding, the strongest reported Ref-L4 baseline, after excluding the 800 adaptation-source examples from the evaluation set. Both models show minimal changes in performance.