acceptodds
Under review as a conference paper at ICLR 2027

RecAgent: Reinforcing Referring Expression Comprehension through Agentic Tool-Augmentation

Abstract

Referring expression comprehension (REC) requires localizing the image region described by a free-form expression, and multimodal large language models (MLLMs) provide a flexible interface for this task while remaining prone to three recurring failures: they bind compositional constraints inconsistently, adopt a candidate before sufficient visual evidence is gathered, and translate a correct semantic decision into an imprecise box. We present RecAgent, an agentic framework that recasts REC as active evidence acquisition and optimizes it with tool-augmented reinforcement learning. RecAgent is trained in two stages. A supervised cold start on RefCoT-4K, a self-generated corpus of executable chain-of-thought trajectories, equips the policy to decompose an expression, hypothesize candidate referents, and verify each hypothesis. At inference, the policy queries a diagnostic tool library comprising an Expression Decomposer, a Visual Grounder, an Attribute Matcher, and a Spatial Reasoner, selecting which tools to call and in what order according to the ambiguity at hand. To make this behavior learnable, we propose a verification-gated composite reward that couples continuous IoU-weighted localization feedback with format validity, tool-usage quality, and a trajectory-efficiency term activated on verified-correct answers, rewarding trajectories that are short, correct, and well formed. Across standard (RefCOCO/+/g) and challenging generalization (gRefCOCO, Ref-L4) benchmarks, RecAgent attains state-of-the-art results, demonstrating the effectiveness and efficiency of active visual reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision Language Model, Agentic Reinforcement Learning, Referring Expression Comprehension, Chain-of-Thought

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JvxmOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies referring expression comprehension. The authors argue that current VLMs often pick the wrong object or answer without checking. They propose to treat REC as a multi-step process in which the model uses tools to collect evidence before it answers. The main challenges they address are the lack of step-by-step training data and the difficulty of teaching a model to use tools well with RL. To this end, they build a trajectory dataset, construct a set of visual tools, and train the model with SFT followed by GRPO with a new tool-aware reward. They report state-of-the-art results on several REC benchmarks.

Critical strengths

S1. Step-by-step checking clearly helps on generalized REC.

  • It is much better than prior methods at rejecting sentences that match no object and at finding several targets (Table 2).
  • The examples in Fig. 3 show that it checks each condition before answering.

Why this matters to my score: This is the strongest support for the paper's main idea.

S2. The tool reward is a simple and useful idea.

  • Tools are rewarded only when they help, and short answers only when they are correct (Sec. 3.4).
  • This leads to fewer tool calls and shorter outputs with no loss in accuracy (Table 4).

Why this matters to my score: Agents trained with RL often overuse tools, and this fix is easy to reuse.

Critical weaknesses

W1. The experiments do not show that the gains come from the agent itself.

  • Some tools call an unnamed VLM (App. D), and the training data comes from a 27B model (Sec. 3.2). Part of the gain may come from these larger models.
  • The base model seems under-tested (Table 3). Key baselines are missing: general VLMs, the 27B model with the same tools, and RL without tools (Tables 1 and 3).
  • Tool cost is not counted (Table 4), and the closely related Rex-Thinker (ICLR 2026) is not discussed.

Why this matters to my score: The gains may come from larger models or weak baselines rather than from the agent. This is the main reason for my score.

W2. The test setup may leak test data, and key details are missing.

  • Ref-L4 val is used for training (App. A.1) but is also part of the reported test results (Sec. 4.1, Table 2). Overlap is checked per example, not per image (App. A).
  • The main reward is never defined, and Eq. 9 gives a failed tool call a higher score than a valid one that does not help (Sec. 3.4).
  • No variance is reported, although many ablation gaps are small (Table 3). The paper also contradicts itself in places (e.g., Sec. 3.2 vs. App. A.4).

Why this matters to my score: I cannot tell whether the results are free of test data, reproduce the reward, or trust the ablations.

Questions for authors
  1. Which model answers the QLA, REPHRASE, VISUAL_GROUNDING, and VISION_SELF_CHECK calls during data collection, during RL, and during testing? How do accuracy and run time change if the 4B policy answers these calls itself?
  2. Are the Ref-L4 results in Table 2 on the test split only? How many images are shared between the training data and each test split?
  3. How do these baselines perform on the same benchmarks?
    • (a) the 27B model with the same tools;
    • (b) the tools without a trained policy;
    • (c) GRPO with an IoU reward and no tools, on the same 3.2K examples.
  4. How is RecAgent different from Rex-Thinker (ICLR 2026)?
  5. No tool adjusts box coordinates, and RecAgent's Ref-L4 [email protected] is below CogVLM-Grounding's (47.50 vs. 48.35). What evidence shows that RecAgent draws better boxes for objects it has already identified correctly?
3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer Jvxm

Q1: Which models execute the tools at each stage.

We thank the reviewer for their patience, thorough review, and thoughtful feedback. We sincerely apologize for the ambiguity in our wording and the resulting misunderstanding. All tools requiring VLM inference use the model currently executing the task: Qwen3.8-27B during offline data construction, and the 4B policy during RL and testing. The 27B teacher is therefore used for offline supervision, not online tool execution. The terms “current Qwen backend,” “same backend,” and “contextual Qwen vision model” in Appendix D all refer to the model used at the corresponding stage.

This design reuses the VLM’s existing capabilities, such as REPHRASE and VISUAL_GROUNDING, as modular operations that the policy can invoke on demand. During SFT, the 4B model learns from teacher-generated trajectories containing the 27B model’s tool observations, providing demonstrations of how to interpret and act on these observations. More broadly, the aim is to organize VLM reasoning into explicit, task-specific skills that can be optimized and selectively invoked.

For example, during RL and testing, VISION_SELF_CHECK draws numbered candidate boxes on the image and returns the annotated image to the 4B model, which assesses which candidates match the referring expression. This verification is performed entirely by the 4B model itself, without any calls to the 27B model.

Q2: the problem of Ref-L4.

Table 2 reports results on Ref-L4 val+test, consistent with Table 4 of the original Ref-L4 paper and the cited baselines. We apologize for the ambiguous reference to a “disjoint test split” in Appendix A.1. To cover compositional attributes and relational reasoning in long expressions, we sampled 800 candidate examples from the Ref-L4 validation split. Since some baselines could not be fully reproduced, we retained the full val+test evaluation set to maintain comparability with their published results. To reduce direct textual overlap between adaptation and evaluation queries, we used an LLM to rewrite the sampled queries while preserving their referential meaning and constraints. This changes the wording while retaining the original images and target annotations.

Example of Query Rewriting:

  • Original query: A fully ripe muskmelon lies on a cutting board next to a tray of white eggs, beneath the suspended wire fruit baskets.
  • Rewritten query: The fully ripe muskmelon resting on a cutting board beside a tray containing white eggs, below the hanging wire fruit baskets.

For a fairer comparison, we evaluate both our model and CogVLM-Grounding, the strongest reported Ref-L4 baseline, after excluding the 800 adaptation-source examples from the evaluation set. Both models show minimal changes in performance.

ModelFull set ([email protected] / .75 / .9 / mAcc)Excluding 800 examples ([email protected] / .75 / .9 / mAcc)
CogVLM-Grounding81.70 / 70.77 / 48.35 / 66.0981.63 / 70.64 / 48.32 / 66.05
Ours85.38 / 71.76 / 47.50 / 67.2485.27 / 71.70 / 47.54 / 67.19

Q3: Performance of the requested baselines.

The table above summarizes performance under the different settings.

  • (a) 27B + same tools: As clarified in Q1, all VLM-based tools use the model executing the task, so both the policy and these tools use the 27B model in this setting. Formatting and parsing failures substantially reduce its end-to-end performance, which also motivates the additional trajectory-correction steps used during data construction. Row (d) reports performance after excluding these failures. The higher scores suggest stronger grounding capability when the larger model produces a valid answer; this is a conditional subset rather than a full-sample evaluation. Using a larger model also entails greater computational cost. Our central objective is to improve a given VLM through tool-augmented training, so this model-scale comparison should be considered alongside the fixed-backbone 4B ablations.
  • (b) Tools without a trained policy: Table 4 already includes an SFT-trained tool-use policy without RL. Row (b) additionally considers the 4B model without task-specific training. Its lower scores than direct grounding are consistent with difficulties adapting to the tool interface, motivating the SFT cold start.
  • (c) GRPO without tools: The no-tool SFT + RL baseline is reported in Table 3(c), and its scores are reproduced in row (c) above.
IDSettingRefCOCORefCOCO+RefCOCOg
(a)27B + same tools44.5243.6856.63
(b)Untrained 4B + tools70.2463.9762.68
(c)4B + SFT + RL, no tools87.0280.1182.67
(d)27B + same tools (excluding formatting and parsing errors)92.5092.1893.51
(e)Ours89.4383.4785.86

Q4: How does RecAgent differ from Rex-Thinker?

Both methods use explicit reasoning, SFT, and GRPO to improve REC. They differ primarily in their research focus, evidence acquisition, and input requirements.

(a) Research focus: Rex-Thinker emphasizes interpretable and verifiable candidate selection; RecAgent learns how to acquire grounding evidence through tool use. Rex-Thinker aims to make grounding decisions interpretable and visually verifiable, while supporting abstention when no object matches the expression. RecAgent focuses on deciding what evidence is still needed, which tools can provide it, and when there is enough evidence to stop. Accordingly, our training targets both localization and the interaction policy.

(b) Execution: Rex-Thinker detects candidates before reasoning; RecAgent invokes tools during reasoning and adapts subsequent actions to their outputs. Rex-Thinker applies Planning–Action–Summarization reasoning to the image, box hints, and referring expression. Its official single-image implementation first detects candidates and then generates the reasoning and answer in a single model call. RecAgent instead interleaves policy decisions with tool execution. After observing a tool's output, it can request a crop, check attributes or spatial relations, invoke a tool again, or produce the final answer. This allows subsequent actions to address unresolved constraints without prescribing a fixed tool sequence. For example, if color is established but relative depth remains ambiguous, the policy can request depth evidence; when sufficient evidence is available, it can stop.

(c) Inputs: Rex-Thinker additionally receives target-category labels in its RefCOCOg evaluation; RecAgent starts with only the image and referring expression. In the reported RefCOCOg experiments, Rex-Thinker uses the original COCO category label to prompt GroundingDINO for candidate detection. RecAgent's policy generates the entity descriptions and retrieval parameters from the image and expression. Such category annotations may be unavailable in deployment, in which case the former pipeline requires an additional category-inference step. RecAgent incorporates the generation of retrieval descriptions into its policy. The distinction is therefore whether the target-category prior is supplied externally or the policy derives a retrieval description from the task input.

Q5: Does RecAgent produce more precise boxes once the correct object is identified?

Thank you for this insightful question. Our current tool suite does not include a tool that directly adjusts bounding-box coordinates.

  1. Changing the prompt can alter the predicted box. For example, rewording a query through REPHRASE can lead the grounding model to produce a different box. However, how the model arrives at these adjustments is not fully interpretable. A tool that explicitly refines coordinates would therefore be a valuable addition. In practice, though, we have found that models struggle with fine-grained numerical adjustments: they can often identify a reasonable region, but precisely describing and correcting small boundary errors through text is much harder. This may partly reflect the limitations of generating spatial coordinates autoregressively as language tokens. With this in mind, a promising direction is to incorporate semantic segmentation for finer refinement. For example, the predicted box could serve as a region proposal for segmentation, with the resulting mask then used to adjust the box boundaries.
  2. Another complication is noise in the datasets themselves, which we have found particularly noticeable in RefCOCO and RefCOCO+. In some cases, a model may produce a visually more precise box yet receive a lower score because it differs from the ground-truth annotation. For example, these datasets contain many part-to-whole references: an expression such as “blue shirt” may be annotated with a box around the person wearing the shirt, rather than the shirt alone. We regard such discrepancies as a form of dataset bias that warrants careful consideration when interpreting grounding performance. They also raise a broader question: whether revisiting the annotations and cleaning the existing datasets could yield a better benchmark for evaluating precise localization. Although this is not the focus of our work, we consider it a worthwhile direction for future research.
Reviewer_49ahOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RecAgent, a tool-augmented agentic framework that reformulates referring expression comprehension as an active process of evidence acquisition and candidate verification. The method first distills executable tool-use trajectories into RefCoT-4K for supervised cold start, then applies GRPO with localization, format, tool-use quality, and verification-gated efficiency rewards. Experiments report results on RefCOCO/+/g, gRefCOCO, and Ref-L4, together with component ablations and an efficiency analysis.

Critical strengths
  1. Clear and Well-Structured Framework. The overall framework is clear and easy to follow. The paper presents a well-organized pipeline consisting of trajectory construction, supervised fine-tuning, and reinforcement learning. Figure 2 and Section 3 provide a clear view of the whole training procedure.
Critical weaknesses
  1. Insufficient and unclear baseline comparisons. Tables 1 and 2 would benefit from reporting the backbone sizes of the compared methods to facilitate a fair comparison. Furthermore, it is unclear why the two tables adopt largely different sets of baselines and why many entries are left blank. Several recent and relevant approaches, such as WeDetect-Ref [1] and SAM 3 Agent [2], are absent from the comparison. These omissions make it difficult to assess the competitiveness of RecAgent against recent methods, especially given the paper's claims of state-of-the-art performance.

  2. The necessity of tool-augmented reinforcement learning is not sufficiently demonstrated. While the paper identifies limitations of direct VLM grounding, it does not convincingly establish why REC requires a tool-augmented agentic framework rather than simpler alternatives such as single-pass CoT reasoning. the current experiments do not sufficiently disentangle the contributions of the SFT–RL training process from those of the external tools. More controlled comparisons are needed to determine whether the performance gains stem primarily from the training strategy, the external tools, or their combination. As a result, the additional complexity of the proposed framework is not sufficiently justified.

[1] Fu, Shenghao, et al. "WeDetect: Fast open-vocabulary object detection as retrieval." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

[2] Carion, Nicolas, et al. "Sam 3: Segment anything with concepts." International conference on learning representations. Vol. 2026. 2026.

Questions for authors
  1. Please clarify the composition of the training data used for SFT and RL, especially data splits. Table 2 reports results on Ref-L4 val+test, while Appendix A.1 states that Ref-L4 validation data are used as an adaptation source. This raises concerns about potential data leakage.

  2. Reward formulation details. Please provide the exact definition of the localization reward (RlocaR_{\mathrm{loca}}), particularly for multi-target and no-target cases. Additionally, how is the RankScore in Eq. (10) calculated, and how is the no-tool baseline (τ0\tau_0) in Eq. (7) constructed? These details are important for understanding and reproducing the proposed reward design.

  3. The framework incorporates nine tools, it is unclear whether such a large tool suite is necessary for REC. Please provide the distribution of tool calls across these tools and justify whether all nine tools are indeed necessary.

  4. Although Table 4 provides some efficiency analysis, including the average number of tool calls and generated tokens, it would be helpful to report the per-tool latency and compare the overall inference time against direct grounding baselines. This would help assess whether the performance gains justify the additional computational overhead introduced by the tool-augmented framework.

  5. Missing appendix reference. Line 463 contains an unresolved reference (“Appendix ??”). Please correct this reference.

2 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer 49ah

Q1: Problem of Ref-L4.

A: Table 2 reports results on Ref-L4 val+test, consistent with Table 4 of the original Ref-L4 paper and the cited baselines. We apologize for the ambiguous reference to a “disjoint test split” in Appendix A.1. To cover compositional attributes and relational reasoning in long expressions, we sampled 800 candidate examples from the Ref-L4 validation split. Since some baselines could not be fully reproduced, we retained the full val+test evaluation set to maintain comparability with their published results. To reduce direct textual overlap between adaptation and evaluation queries, we used an LLM to rewrite the sampled queries while preserving their referential meaning and constraints. This changes the wording while retaining the original images and target annotations.

Example of Query Rewriting:

  • Original query: A fully ripe muskmelon lies on a cutting board next to a tray of white eggs, beneath the suspended wire fruit baskets.
  • Rewritten query: The fully ripe muskmelon resting on a cutting board beside a tray containing white eggs, below the hanging wire fruit baskets.

For a fairer comparison, we evaluate both our model and CogVLM-Grounding, the strongest reported Ref-L4 baseline, after excluding the 800 adaptation-source examples from the evaluation set. Both models show minimal changes in performance.

ModelFull set ([email protected] / .75 / .9 / mAcc)Excluding 800 examples ([email protected] / .75 / .9 / mAcc)
CogVLM-Grounding81.70 / 70.77 / 48.35 / 66.0981.63 / 70.64 / 48.32 / 66.05
Ours85.38 / 71.76 / 47.50 / 67.2485.27 / 71.70 / 47.54 / 67.19

Q2: Reward formulation details.

We thank the reviewer for the careful review and address each question below.

(1) RlocaR_{\rm loca} is a continuous IoU-based localization reward. Let PP and GG denote the predicted and ground-truth box sets, respectively, and let M∗M^* be the one-to-one matching that maximizes their total IoU. We define:

\left\\{ \begin{array}{ll} 1, & P=G=\varnothing,\\\\ 0, & \text{any predicted box is invalid or exactly one set is empty},\\\\ \dfrac{\sum\limits_{(p,g)\in M^*}\operatorname{IoU}(p,g)}{\max(|P|,|G|)}, & \text{otherwise}. \end{array} \right.$$ For a single predicted box and a single ground-truth box, this reward reduces to their IoU. For multiple targets, normalization by the larger of the two set sizes penalizes both missing and extra predictions. For no-target examples, only a valid empty prediction receives a reward of 1. **(2) Definition of RankScore.** RankScore rewards more concise reasoning among trajectories that produce correct answers. Within the rollout group for the same input, we compare generation lengths only among trajectories with valid output formats and correct localization ([email protected] = 1). Let $C$ denote this subset, $n=|C|$, and $L_i$ the number of model-generated tokens in trajectory $i$, excluding tool observations. We rank these trajectories from shortest to longest, with ranks starting at 1, and denote trajectory $i$'s rank by $r_i$. Ties receive the average of the ranks they occupy. For $i\in C$, we define: $$\operatorname{RankScore}(-L_i)= \begin{cases} \dfrac{n-r_i}{n-1}, & n>1, 0.5, & n=1. \end{cases}$$ For example, three correct trajectories with lengths of 100, 200, and 300 tokens receive ranks of 1, 2, and 3, yielding scores of 1, 0.5, and 0. If their lengths are 100, 100, and 300 tokens, the first two share an average rank of 1.5, yielding scores of 0.75, 0.75, and 0. When all correct trajectories have the same length, each receives 0.5. Trajectories with invalid output formats or incorrect localization are excluded from the ranking and receive an efficiency reward $R_{\rm verf}$ of 0. If $C$ is empty, all trajectories in the group receive an efficiency reward of 0. **(3) No-tool reference.** We generate $\tau_0$ using the same image, query, and model, but with tools disabled, and compute $\Delta_i=R_{\rm loca}(\tau_i)-R_{\rm loca}(\tau_0)$. Specifically, we omit the tool schemas from this separate no-tool branch, so the model has no tools available and instead reasons about the query before producing its answer directly.

Q3: Why are nine tools needed?

This is a valuable question. To organize the nine tools, we group them into four functional categories: ED (QLA, REPHRASE), VG (FIND, VISUAL_GROUNDING), AM (CLIP_MATCH, IMAGE_CROP, VISION_SELF_CHECK), and SR (SPATIAL_JUDGE, DEPTH_ESTIMATE). In principle, the tools within each category could be consolidated into a single tool: ED handles query interpretation and reformulation, VG performs localization, AM supports fine-grained attribute comparison, and SR assesses spatial relations. We expose these operations separately to give the model greater flexibility in selecting the specific operation it needs. FIND and VISUAL_GROUNDING, for example, serve the same broad localization purpose but draw on different capabilities: FIND incorporates external priors, whereas VISUAL_GROUNDING reuses the VLM's own grounding capability. We clarify this distinction in our response to Reviewer Jvxm's Q1.

Table 3(e–h) reports ablations that remove each of the four tool categories. Below, we additionally provide the distribution of calls across individual tools:

ToolShare of tool calls (%)
QLA12
REPHRASE8
FIND25
VISUAL_GROUNDING15
CLIP_MATCH12
IMAGE_CROP14
VISION_SELF_CHECK9
SPATIAL_JUDGE3
DEPTH_ESTIMATE2

VG tools account for the largest share of calls, as expected for a grounding task. SR tools are used less frequently; we attribute this to the relative scarcity of queries requiring explicit spatial-relation reasoning in the dataset.

Q4: Per-tool latency and comparison with direct grounding.

Table 4 reports the number of tool calls and generated tokens. We additionally report per-tool latencies for the current implementation, measured on a single H100 with a request batch size of 32 over 1,000 samples drawn from the dataset.

ToolMean (ms)Median (ms)p95 (ms)
QLA19.4016.2433.22
REPHRASE8.777.4511.55
FIND85.9583.8690.92
VISUAL_GROUNDING9.648.2412.77
CLIP_MATCH26.7824.0930.08
IMAGE_CROP75.4970.99108.51
VISION_SELF_CHECK109.77106.78148.65
SPATIAL_JUDGE10.819.0113.89
DEPTH_ESTIMATE54.6651.8758.26

Latency is measured from dispatching a tool call to receiving and decoding its result. VISION_SELF_CHECK has the highest latency, which we attribute primarily to its greater I/O overhead.

We also compare our method with the direct grounding baseline under the same batch 32 setting. Their inference latencies are 4.24 s and 6.48 s, respectively. The baseline's longer latency is largely attributable to the 4B model's extended chain-of-thought reasoning. In particular, we observe that its reasoning traces repeatedly revisit coordinate transformations and local visual details, increasing generation time and slowing inference.

Q5: Missing appendix reference.

This part refers to B.1, and we will fix it in the camera-ready version.

Reviewer_A2R9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RecAgent, a tool-augmented reinforcement learning framework for referring expression comprehension (REC) that integrates expression decomposition, visual grounding, attribute matching, and spatial reasoning to improve object localization. The framework employs a two-stage training pipeline, consisting of supervised fine-tuning on synthesized tool-use reasoning trajectories followed by GRPO with a verification-gated composite reward that encourages accurate localization, effective tool use, and efficient reasoning. Experiments on standard and challenging REC benchmarks demonstrate competitive or state-of-the-art performance, although the individual techniques largely build on prior work, with the main contribution lying in their integration and REC-specific reward design.

Critical strengths
  1. My favorite aspect of this paper is the technically well-designed RL training pipeline. In particular, the supervised cold start, tool-augmented GRPO, and verification-gated reward provide a coherent approach to training an agent for effective and efficient tool use.

  2. Comprehensive integration of existing techniques. Although many individual components have precedents in prior work, RecAgent brings them together into a unified framework for referring expression comprehension. I summarized the novelty of its components with the assistance of ChatGPT in the following table. I encourage the authors to include a similar comparison to clarify their contributions and improve readability:

RecAgent componentClosely related precedentsWhat's potentially new
Expression decompositionVisProg (2023), ViperGPT (2023), Pelican (2024), VIRO (2026)Integration into a learned REC policy
Visual Grounder (GroundingDINO / VLM)ViperGPT, VisProg, VIROChoice and coordination of grounding tools
Attribute matching (CLIP / crops)VisProg, ViperGPT, PelicanAdaptive verification of REC candidates
Spatial reasoning (geometry / depth)ViperGPT, VIROIntegration with other tools in RL rollouts
Synthetic tool-use CoT for SFTTACO, Tool-StarREC-specific RefCoT dataset
Tool-augmented GRPOReVPT, VTool-R1REC-specific reward design
SFT followed by tool-augmented RLReTool, Tool-StarAdaptation to visual grounding
Reward for tool-use qualityTool-Star, PyVision-RLImprovement relative to a no-tool baseline
Reward for efficient correct trajectoriesEarlier RL efficiency formulationsVerification-gated ranking of trajectory length
Critical weaknesses
  1. The authors identified 3 fundamental limitations: 1) Constraint-Inconsistent Binding; 2) Evidence-Insufficient Selection; 3) Semantic-Spatial Misalignment. The latter 2 seem to be special cases of 1, constraint-inconsistent binding. However, after closer examination I think it's more of improper naming of the limitations. Better names of the limitations could be: 1) Compositional Reasoning Failure; 2) Premature Candidate Selection; 3) Fine-Grained Visual Grounding Failure.
  2. The three examples in Figure 1 appear to be deliberately constructed challenging cases, yet how they are generated is unclear. Section 3.2 (lines 246–252) describes sampling existing benchmark examples and verifying their annotations, rather than synthesizing new images or referring expressions. The authors should clarify whether the examples in Figure 1 originate from existing datasets, were manually constructed, or were synthetically generated. More importantly, since these examples motivate the three claimed fundamental limitations, it would be valuable to provide empirical evidence that these failure modes occur systematically in representative benchmark data, rather than only in selected illustrative cases.
  3. VIRO (CVPR 2026) is a closely related work that already investigates compositional reasoning, visual verification, and spatial grounding for REC using specialized operators. Although RecAgent introduces a more comprehensive tool library and an RL-based adaptive tool-selection policy, the manuscript does not sufficiently establish the benefits of these additions relative to existing neuro-symbolic pipelines. A direct numerical comparison with VIRO may be difficult because its official implementation has not been released and the training and evaluation settings differ. Nevertheless, a controlled comparison with a non-RL tool-use baseline, using the same backbone and tool library, would help isolate the contribution of learned adaptive tool selection. Furthermore, wall-clock inference latency and total tool-execution costs should be reported to substantiate the claimed efficiency improvements.
Questions for authors

N/A

1 comment
AuthorsOpenReview ↗

Rebuttal for Reviewer A2R9

We appreciate the reviewer's recognition of this work, as well as their careful review and constructive suggestions.

Q: Naming the three failure modes.

We thank the reviewer for this helpful suggestion and agree that the proposed terminology describes these failure modes more clearly and accurately. In the camera-ready version, we will adopt Compositional Reasoning Failure, Premature Candidate Selection, and Fine-Grained Visual Grounding Failure, and update Figure 1 and §3.1 accordingly. We will also replace “fundamental limitations” with “failure modes” and clarify that these failure modes can co-occur.

Q2: Origin of the Figure 1 examples.

We thank the reviewer for raising this point. The examples in Figure 1 were synthesized to illustrate the three failure modes more clearly. Appendix B.2 (Figure 3) provides additional real examples, contrasting direct Qwen3.5-4B predictions with RecAgent's tool-use trajectories.

Q3: Non-RL baseline and comparison with VIRO.

A: Table 4 already includes a non-RL tool-use baseline: the SFT agent uses the same 4B backbone and tool library but has not undergone RL optimization. Compared with this SFT-only baseline, RecAgent improves [email protected] on RefCOCOg val-g from 81.18% to 85.59%, while reducing the average number of tool calls from 1.64 to 1.21 and generated tokens from 286 to 221. This provides a controlled comparison of accuracy and interaction costs before and after RL optimization. We agree that VIRO is closely related to our work. However, its official implementation has not been released. This makes it difficult to verify implementation-level details of program generation and validation, operator execution, and model configuration, or to faithfully reproduce the method under matched backbone and tool settings. Per-tool latencies and the latency comparison with the direct grounding baseline are addressed in our response to Reviewer 49ah's Q4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.