acceptodds
Under review as a conference paper at ICLR 2027

ReferFlow: Beyond Final-Target Localization for Complex Referring Expressions

Abstract

Multimodal Large Language Models have made significant progress in referring expression comprehension (REC), yet most REC benchmarks still focus on final-target localization. Complex queries often involve intermediate visual references with different dependencies on prior grounding, but final-only evaluation leaves this process unexamined. To address this gap, we formulate trajectory-level REC, in which models receive an ordered sequence of grounding steps constructed from a single complex query and localize the target regions associated with each step. We instantiate this setting in ReferFlow-Bench, which characterizes structured region transitions through four trajectory types, namely Refinement, Shift, Reset, and Mixture. Experiments show that current models struggle to consistently ground intermediate visual references across successive steps. We further propose ReferFlow, which decouples interleaved text–latent trajectory generation from coordinate decoding. Specifically, Stateful Region Latent Reasoning represents each target region as an object-indexed state composed of continuous latents, with explicit visual and spatial supervision to preserve region information for subsequent grounding. Trajectory-Aware Parallel Decoding then recovers all coordinate blocks in parallel through four geometry-constrained refinement rounds, with each block conditioned on its corresponding trajectory prefix, thereby reducing within-box sequential dependencies and eliminating cross-box autoregression. We further apply Grounding-Aware GRPO to promote consistent grounding across steps and learn adaptive trajectory generation from complex REC queries. ReferFlow achieves leading performance on general REC benchmarks and ReferFlow-Bench.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
referring expression comprehension, visual grounding

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ZXHCOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ReferFlow for trajectory-level REC, extending evaluation from final-target localization to intermediate grounding steps. The authors introduce ReferFlow-Bench with four trajectory types: Refinement, Shift, Reset, and Mixture. ReferFlow combines continuous SRLR, TAPD, and Grounding-Aware GRPO to improve multi-step grounding. Experiments on ReferFlow-Bench and standard REC benchmarks demonstrate strong performance, supported by extensive comparisons and ablation studies.

Critical strengths

1.Novel task formulation and benchmark. 2.Very strong performance and thorough validation.

Critical weaknesses

None

Questions for authors
  1. Benchmark Evaluation: ReferFlow-Bench provides models with predefined grounding steps, while ReferFlow also claims to support adaptive trajectory generation. Could the authors evaluate the model directly from the original complex queries without providing intermediate steps? This would better demonstrate its end-to-end reasoning capability.

  2. Fair Comparison: ReferFlow is trained on 1.6M samples, including 400K newly constructed trajectories. Could the authors provide a baseline trained with the same data and training budget, but without the proposed modules? This would help distinguish the contributions of the framework from those of additional training data.

  3. Benchmark Generalization: Both ReferFlow-Train and ReferFlow-Bench are constructed using the same automated annotation pipeline. Could the authors provide evidence that the improvements generalize to independently constructed or human-authored grounding trajectories, rather than relying on patterns specific to the generated data?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ZXHC Part 1/3

We appreciate the reviewer’s positive assessment of our task formulation and benchmark, as well as the recognition of our strong results and thorough experiments. We address the reviewer’s questions below.

Q1: Evaluation from original complex queries

Thank you for suggesting this evaluation. ReferFlow can directly localize a target or generate intermediate grounding steps as needed from an original complex query. Our general REC experiments in Table 2 already evaluate this capability given only the image and the original query, with no intermediate steps provided at inference time. ReferFlow achieves state-of-the-art results on eight benchmarks, including the reasoning-oriented GroundingME and PR-Bench.

Each sample in ReferFlow-Bench includes an original complex query and its decomposition into ordered grounding steps. We evaluate ReferFlow and four comparison models on the full dataset under both input formats, using the same checkpoint for each model across both settings. In the original-query setting, models receive only the image and the original query, without intermediate steps. The original query refers to the target of the final annotated step, so we report F-Acc for both settings using the same final target.

ModelInputRefinementShiftResetMixtureOverall
Qwen3-VL-2BAnnotated ordered steps29.928.330.929.229.6
Qwen3-VL-2BOriginal query28.523.531.925.427.3
Rex-OmniAnnotated ordered steps50.642.836.834.041.1
Rex-OmniOriginal query46.233.637.226.435.9
GLM-4.6V-FlashAnnotated ordered steps42.036.831.342.738.2
GLM-4.6V-FlashOriginal query39.830.229.936.334.1
Qwen3.5-27BAnnotated ordered steps50.640.637.149.544.5
Qwen3.5-27BOriginal query48.134.236.243.240.4
ReferFlowAnnotated ordered steps65.862.263.460.463.0
ReferFlowOriginal query63.156.964.255.059.8

Given only the image and the original query, ReferFlow achieves 59.8% overall F-Acc, outperforming all comparison models, including Qwen3.5-27B at 40.4%. These results demonstrate ReferFlow’s end-to-end grounding capability without provided intermediate steps.

With annotated steps, ReferFlow achieves 63.0% overall F-Acc. The gains are larger on Shift and Mixture, where the provided steps specify intermediate visual references and cross-step region dependencies. Refinement also benefits from the progression from broader to narrower target sets. On Reset, the original-query setting performs slightly better (64.2% vs. 63.4%), consistent with its independently specified targets.

We have added this paired evaluation to Section 5.3 of the revised manuscript (Figure 7, lines 468–477).

AuthorsOpenReview ↗

Response to Reviewer ZXHC Part 2/3

Q2: Fair comparison under matched training conditions

Thank you for highlighting the importance of a matched baseline. The Text baseline in Table 3 provides this comparison. It uses the same Qwen3-VL-2B backbone, SFT data (1.6M examples, including 400K ReferFlow-Train trajectories), and optimization budget as the models with the proposed modules. It represents target regions using bounding-box coordinate tokens without SRLR or TAPD. All models in the architectural comparisons in Tables 3 and 4(a) are trained without RL. Details of the training data are provided in Appendix C.1 and Table 5.

Table 3 compares the Text baseline with SRLR under the same next-token prediction (NTP) decoding scheme, evaluating the region-state representation and its visual and spatial supervision. Table 4(a) keeps SRLR fixed and compares coordinate decoding strategies to evaluate TAPD.

Under these matched SFT conditions, SRLR and TAPD improve RefCOCOg accuracy from 87.9% to 92.0% and GroundingME accuracy from 32.4% to 39.1%. On ReferFlow-Bench, Q-Acc increases from 52.6% to 60.8% and T-Acc from 17.9% to 25.6%. These results demonstrate the contributions of the proposed modules under the same training data and optimization budget. Grounding-Aware GRPO provides further improvements, reported separately in the last row of the table below.

ModelTrainingRefCOCOgGroundingMEReferFlow-Bench Q-AccReferFlow-Bench T-AccReferFlow-Bench F-Acc
Text baselineSFT87.932.452.617.948.2
SRLR + NTPSFT90.537.456.921.353.9
SRLR + TAPDSFT92.039.160.825.659.4
SRLR + TAPDSFT + RL92.943.063.928.463.0

We have clarified the Text baseline and shared training settings in the caption of Table 3 in the revised manuscript (lines 441–443).

AuthorsOpenReview ↗

Response to Reviewer ZXHC Part 3/3

Q3: Generalization beyond the automated annotation pipeline

Thank you for this insightful question. Appendix E.7 (Appendix E.8 in the revised manuscript) reports a human evaluation of ReferFlow’s self-generated grounding trajectories on 500 randomly sampled GroundingME samples, checking whether the predicted intermediate boxes correctly ground the corresponding step descriptions. To examine generalization beyond our automated annotation pipeline, we use the same 500 images and original queries to construct grounding trajectories independently written by human annotators.

One annotator writes grounding steps and annotates intermediate target boxes, retaining the original final-target annotations. Two other annotators conduct two rounds of review to check consistency with the original queries and box accuracy. Disagreements are resolved through discussion. Thirteen samples with unresolved annotation ambiguity are excluded before model evaluation, leaving 487 human-authored trajectories. Annotators do not refer to model outputs or trajectories generated by our automated pipeline, and all steps use only localization cues present in the original queries.

We evaluate ReferFlow and four comparison models on these 487 samples without additional training. Each model is evaluated separately with the human-authored ordered steps and the original query. With ordered steps, we follow the ReferFlow-Bench protocol and report Q-Acc, T-Acc, and F-Acc. With original queries, we report final-target accuracy following the GroundingME evaluation protocol.

As shown in the table below, ReferFlow achieves 57.2% Q-Acc, 24.2% T-Acc, and 43.5% F-Acc on the human-authored trajectories, outperforming all four comparison models. Qwen3.5-27B, the strongest comparison model, achieves 52.6%, 20.1%, and 39.8%, respectively. These results support the generalization of ReferFlow’s gains to independently human-authored grounding trajectories. Given only the original queries, ReferFlow also achieves the highest final-target accuracy at 40.2%, compared with 38.0% for Qwen3.5-27B.

ModelOriginal Query AccHuman-Authored Steps Q-AccHuman-Authored Steps T-AccHuman-Authored Steps F-Acc
Qwen3-VL-2B26.737.89.928.1
Rex-Omni29.242.414.431.8
GLM-4.6V-Flash27.739.410.530.0
Qwen3.5-27B38.052.620.139.8
ReferFlow40.257.224.243.5
Reviewer_3RqUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies trajectory-level referring expression comprehension, evaluating the localization of intermediate references rather than only the final target. It introduces ReferFlow-Bench with four types of region transitions and proposes a system combining supervised region latents, parallel coordinate decoding, and trajectory-aware reinforcement learning. The approach improves both trajectory-level evaluation and conventional referring expression comprehension.

Critical strengths
  1. Evaluating intermediate grounding is a useful extension of final-target evaluation. The trajectory categories and length-dependent results expose failures that a single final box can conceal; image-level splitting and human review strengthen the benchmark.
  2. Tables 3–4 and the appendices provide substantial mechanism analysis, including spatial/visual supervision, decoding choices, incorrect or oracle intermediate boxes, region-state interventions, and coordinate regression.
Critical weaknesses
  1. The attribution of improvements to architecture versus training remains unclear. The system uses substantial new trajectory supervision, and the Text baseline already achieves 52.6 Q-Acc and 17.9 T-Acc. Please identify which comparisons match training examples, optimization budget, and RL, so that data and architectural gains can be distinguished. (pp. 6, 9, 17–18)
  2. The benchmark supplies ordered grounding steps, whereas practical use starts from an undecomposed expression. Although adaptive trajectory generation and general REC evaluations are included, a paired comparison on the same examples is needed to distinguish executing a supplied decomposition from constructing one. (Appendices B and E)
Questions for authors
  1. Does the Table 3 Text baseline share all SFT examples, the RL stage, and the optimization budget with the complete system? If not, which comparisons isolate each source of improvement?
  2. On a shared subset, how much does performance change between the original complex expression and the annotated ordered steps?
  3. Can the NTP/TAPD comparison report end-to-end latency, memory, and accuracy as a function of the number of regions, including text and latent generation?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 3RqU Part 1/2

We sincerely thank you for your thoughtful review. Your positive comments on our step-level evaluation and our analyses of ReferFlow’s design are greatly appreciated. We also value your suggestions for clarifying the experimental comparisons and address each point below.

W1 and Q1: Contributions of training data, architecture, and RL

We appreciate the reviewer’s question about the training setup. The Text baseline and the other models in Table 3 and 4(a) use the same backbone, SFT data (1.6M examples, including 400K ReferFlow-Train trajectories), and optimization budget, and are trained without RL. Details of the training data are provided in Appendix C.1 and Table 5. The following comparisons distinguish the contributions of architecture, trajectory supervision, and RL:

  1. SRLR (Table 3). We keep NTP decoding fixed and compare the Text baseline with region latents trained without explicit region supervision, then add spatial and visual supervision separately and jointly. This evaluates the region-state representation and the contributions of its supervision under matched training conditions.

  2. TAPD (Table 4(a)). We fix the best-performing SRLR setting from Table 3 and compare coordinate decoding strategies, isolating the contribution of TAPD under the same SFT data and optimization budget.

  3. SFT data formats (Table 4(b)). We compare three data formats to determine the default SFT setting. Starting from Adaptive Trajectory, which combines REC data with trajectory-level REC data, we remove trajectory supervision from the 400K ReferFlow-Train examples for Direct Grounding and generate trajectories for the 1.2M REC examples for Trajectory Only. We train separate models while keeping the backbone, images, source queries, final target annotations, and optimization budget fixed. Adaptive Trajectory achieves the best overall performance, supporting our choice of the default data format.

  4. RL rewards (Table 4(b)). We initialize all RL runs from the same SFT checkpoint trained with SRLR, TAPD, and the default Adaptive Trajectory data format. We compare different combinations of the trajectory reward RTRR_{\mathrm{TR}} and the multi-step reward RmultiR_{\mathrm{multi}}, keeping all other RL settings fixed.

The table below summarizes these comparisons and the results before and after RL. Under matched SFT conditions, SRLR and TAPD improve Q-Acc from 52.6% to 60.8% and T-Acc from 17.9% to 25.6%. Grounding-Aware GRPO further improves these metrics to 63.9% and 28.4%, respectively.

ModelTrainingRefCOCOgGroundingMEReferFlow-Bench Q-AccReferFlow-Bench T-AccReferFlow-Bench F-Acc
Text baselineSFT87.932.452.617.948.2
SRLR + NTPSFT90.537.456.921.353.9
SRLR + TAPDSFT92.039.160.825.659.4
SRLR + TAPDSFT + RL92.943.063.928.463.0
AuthorsOpenReview ↗

Response to Reviewer 3RqU Part 2/2

W2 and Q2: Grounding from original queries versus annotated steps

Thank you for suggesting this paired comparison. Each sample in ReferFlow-Bench includes an original complex query and its decomposition into ordered grounding steps. We provide target-region annotations at each step to evaluate grounding under different cross-step region dependencies. These annotations support step-level evaluation with Q-Acc and trajectory-level evaluation with T-Acc, and allow us to identify where localization errors occur.

We conduct a paired comparison on the full ReferFlow-Bench dataset using ReferFlow and four comparison models. For each model, we use the same checkpoint under both input formats. The original query refers to the target of the final annotated step, so we report F-Acc for both settings. Given only the original query, ReferFlow can directly localize the target or generate intermediate grounding steps as needed.

ModelInputRefinementShiftResetMixtureOverall
Qwen3-VL-2BAnnotated ordered steps29.928.330.929.229.6
Qwen3-VL-2BOriginal query28.523.531.925.427.3
Rex-OmniAnnotated ordered steps50.642.836.834.041.1
Rex-OmniOriginal query46.233.637.226.435.9
GLM-4.6V-FlashAnnotated ordered steps42.036.831.342.738.2
GLM-4.6V-FlashOriginal query39.830.229.936.334.1
Qwen3.5-27BAnnotated ordered steps50.640.637.149.544.5
Qwen3.5-27BOriginal query48.134.236.243.240.4
ReferFlowAnnotated ordered steps65.862.263.460.463.0
ReferFlowOriginal query63.156.964.255.059.8

All five models achieve higher overall F-Acc with annotated steps. For ReferFlow, overall F-Acc changes from 63.0% with annotated steps to 59.8% with the original query. The decreases are larger on Shift and Mixture, from 62.2% to 56.9% and from 60.4% to 55.0%, respectively. Without provided steps, models must resolve the references and their dependencies directly from the original expression. Refinement also benefits from the provided progression from broader to narrower target sets, with F-Acc changing from 65.8% to 63.1%. On Reset, the original-query setting performs slightly better (64.2% vs. 63.4%), consistent with its independently specified targets.

We have added this paired evaluation to Section 5.3 of the revised manuscript (Figure 7, lines 468–477).

Q3: End-to-end comparison of NTP and TAPD as the number of regions increases

Thank you for suggesting this additional analysis. We compare NTP and TAPD using the same SRLR setting, SFT data, and training budget. We measure end-to-end latency over the full generation process, including text and latent generation as well as coordinate decoding.

We evaluate both methods on ReferFlow-Bench and HumanRef, covering trajectory grounding and multi-object grounding from a single expression. For ReferFlow-Bench, we fix the trajectory length to three steps, group samples by the total number of ground-truth boxes across those steps, and report Q-Acc. For HumanRef, we group expressions by the number of ground-truth boxes and report DF1. Each comparison uses the same samples and inference configuration.

The table below shows that TAPD has lower end-to-end latency than NTP in every group, consistent with its parallel coordinate decoding design. Latency is 1.9–3.0 s on ReferFlow-Bench and 0.9–2.4 s on HumanRef, compared with 3.3–5.2 s and 1.4–4.0 s for NTP, respectively. Peak GPU memory is comparable, with slightly higher usage for TAPD in most groups. TAPD also achieves better grounding performance in every group, with larger gains as the number of regions increases.

MetricDecoding strategyReferFlow-Bench 3–4ReferFlow-Bench 5–8ReferFlow-Bench ≥9HumanRef 1HumanRef 2–4HumanRef 5–8HumanRef ≥9
End-to-end latency (s) ↓NTP3.34.65.21.42.02.94.0
TAPD1.92.63.00.91.21.72.4
Peak GPU memory (GB) ↓NTP4.54.94.94.54.54.54.5
TAPD4.94.95.04.64.64.64.6
Grounding performance (%) ↑NTP60.454.243.578.470.660.842.9
TAPD63.258.750.480.874.370.456.0

Column labels indicate the number of ground-truth regions. Grounding performance is measured by Q-Acc on ReferFlow-Bench and DF1 on HumanRef.

We have added this analysis to Appendix E.1 of the revised manuscript (Table 9, lines 1270–1276).

Reviewer_k8aPOpenReview ↗
Rating
3
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ReferFlow, which aims to make up for the missing intermediate evaluation in Referring Expression Compression (REC). It claims that existing methods only concentrate on the final results and overlook the correctness of intermediate visual references. The paper proposes a ReferFlow-Bench constructed with long narrative and structured region transitions through four trajectory types. It further propose ReferFlow, which paralleled decodes all the targets in the query from VLM. Extensive experiments prove that ReferFlow achieves impressive performance.

Critical strengths
  1. The paper is easy to understand and follow.

  2. The motivation is clear. The paper proposes a more completed perspective for evaluation of REC with intermediate status.

  3. ReferFlow-Bench concludes multiple and extensive trajectory types to enrich the diversity of evaluation.

  4. Extensive experiments shows impressive improvements and effectiveness to support the claims.

Critical weaknesses
  1. Claims. The kernel of ReferFlow is argumentaion of the intermediate status in REC. However, it's unclear that how important are those intermediate states for reasoning about target. As shown on the Fig.14, even for GroundingME, which haves the most trajectory-based data, Qwen3.5 also shows close results in Tab.2 without the grounding-specialized settings. The necessity of assessing intermediate states is a matter of concern to me.
Questions for authors
  1. Can ReferFlow automatically provide the plan for locate the target or it only receives the fixed inputs? If it appears like the latter sitiuation, ReferFlow is more like a paralleled decoder for grounding multiple objects, which has been explored in PNG [1].

  2. In Tab.8, the inference speed of ReferFlow is higher than Qwen3-VL is easy to understand in GroundingME due to the way of decoding targets. However, in RefCOCOg, which is a dataset with single object, why ReferFlow is also faster than Qwen3-VL?

  3. Are there any experiments designed for disambiguation? It seems the primary utility of an intermediate state lies in helping to disambiguate targets that might otherwise have overlapping or redundant semantics.

  4. Intermediate states are still represented using multiple targets. Could the use of multiple targets affect the final grounding? Furthermore, if these multiple targets were instead broken down into separate sub-tasks for localization, would that lead to any change in performance, both for ReferFlow and for existing methods?

[1] Panoptic Narrative Grounding, ICCV 23

4 comments
AuthorsOpenReview ↗

Response to Reviewer k8aP Part 1/4

We appreciate the reviewer’s thoughtful feedback and positive remarks on our motivation for evaluating intermediate grounding, the diversity of trajectory types in ReferFlow-Bench, and our empirical results. We also greatly value the suggestions regarding additional experiments, and we address each point below.

W1: The role of intermediate grounding

  1. Importance of evaluating intermediate grounding.

Thank you for raising this question. Complex queries often specify a target through other objects and their relationships. Final-target accuracy leaves these intermediate visual references unexamined. Evaluating intermediate grounding allows us to examine whether these references are correctly localized and identify where grounding errors occur.

In the human evaluation on 500 randomly sampled GroundingME samples (Appendix E.8 of the revised manuscript), approximately 73% of multi-step trajectories with incorrect final predictions contain at least one intermediate grounding error. Examining these predictions helps identify the incorrectly localized references and where grounding first goes wrong. ReferFlow-Bench supports systematic evaluation through ordered grounding steps, target-region annotations, and four trajectory types with different cross-step region dependencies.

  1. Intermediate region states support subsequent grounding.

SRLR retains target-specific visual content and spatial information in intermediate region states, which remain available as references for subsequent grounding. Table 3 shows that explicitly preserving this information through visual and spatial supervision improves grounding, particularly on longer trajectories.

Figure 11 (Figure 12 in the revised manuscript) further examines whether these states affect subsequent target localization. Replacing one historical region state with that of another object substantially reduces F-Acc on Refinement, Shift, and Mixture, while Reset remains relatively stable. These results show that intermediate states contribute to final localization when subsequent targets depend on previously grounded regions.

  1. Effectiveness of trajectory-based training.

Table 4(b) compares separately trained models with the same backbone, images, source queries, final target annotations, and optimization budget. Adaptive Trajectory combines REC data with trajectory-level REC data. Direct Grounding removes trajectory supervision from the 400K ReferFlow-Train examples, while Trajectory Only extends trajectory supervision to the 1.2M REC examples using the same annotation pipeline.

On GroundingME, Direct Grounding, Trajectory Only, and Adaptive Trajectory achieve 35.2%, 37.9%, and 39.1% accuracy, respectively. These results show that incorporating grounding trajectories improves final localization on complex queries. Adaptive Trajectory achieves the best overall performance by allowing direct localization or intermediate grounding steps as needed. We include the grounding-mode results from Table 4(b) below for reference.

Grounding modeRefCOCOgGroundingMEReferFlow-Bench Q-AccReferFlow-Bench T-Acc
Direct Grounding91.135.2––
Trajectory Only88.737.960.325.0
Adaptive Trajectory92.039.160.825.6
  1. Comparison with Qwen3.5.

We agree that Qwen3.5-27B achieves final-target accuracy close to ReferFlow on GroundingME (42.2% vs. 43.0%). ReferFlow is built on Qwen3-VL-2B. We therefore examine whether trajectory-based grounding can provide further gains within the Qwen3.5 family.

Following Table 4(b), we train separate Direct Grounding and Adaptive Trajectory models with Qwen3.5-4B, keeping the training samples and optimization budget fixed. Adaptive Trajectory improves GroundingME accuracy from 42.7% to 46.2% over Direct Grounding, showing that trajectory-based grounding also benefits a stronger backbone.

All fine-tuned models in the table are evaluated after SFT without RL.

BackboneSettingRefCOCOgGroundingMEQ-AccT-Acc
Qwen3-VL-2BAdaptive Trajectory92.039.160.825.6
Qwen3.5-4BOriginal88.439.639.18.7
Qwen3.5-4BDirect Grounding92.242.7––
Qwen3.5-4BAdaptive Trajectory92.746.265.829.2
AuthorsOpenReview ↗

Response to Reviewer k8aP Part 2/4

Q1: Generating grounding steps from raw queries and the relationship to PNG

Thank you for raising this question. ReferFlow can both generate grounding steps from an original referring expression and follow a provided sequence of grounding steps. Given only an image and the original query, it can directly localize the target or generate intermediate grounding steps as needed. On GroundingME, ReferFlow generates multi-step trajectories for 74.5% of queries (Figure 14; Figure 15 in the revised manuscript).

We appreciate the reviewer’s comparison with PNG. PNG and ReferFlow-Bench formulate different grounding tasks. PNG associates noun phrases in image narratives with pixel-level regions covering both things and stuff. ReferFlow-Bench evaluates intermediate localization within complex referring expressions, providing ordered grounding steps and target-region annotations at each step. These steps require models to narrow an existing target set, use a previously grounded region to locate a related target, or introduce an independent target. Thus, PNG evaluates noun-phrase grounding in narratives, whereas ReferFlow-Bench evaluates consistent intermediate grounding across ordered steps with different region transitions and cross-step dependencies.

The model designs also differ. The PNG baseline uses a cross-modality Transformer to match noun phrases to candidate panoptic segmentation regions. ReferFlow uses Stateful Region Latent Reasoning (SRLR) to sequentially generate object-indexed region states that retain target-specific visual and spatial information for subsequent grounding. Trajectory-Aware Parallel Decoding (TAPD) then decodes the corresponding bounding boxes in parallel. Its trajectory-aware mask conditions each coordinate block on the corresponding trajectory prefix, enabling each box to use historical text and region states during parallel decoding while preventing attention between coordinate blocks.

Q2: Inference speed comparison between ReferFlow and Qwen3-VL in single-target grounding

We appreciate the reviewer’s question. ReferFlow reduces sequential decoding steps even for a single target through parallel coordinate decoding and a compact output format.

Coordinate decoding. Under the assumptions in Appendix E.1 (lines 1242–1262), Qwen3-VL represents coordinates in [0,999][0,999] with one token per digit. Generating the four coordinates autoregressively therefore requires an estimated 4–12 sequential decoding steps. TAPD predicts the three digits of each coordinate in parallel and recovers one coordinate per round, decoding all four coordinates in four rounds.

Output format. Qwen3-VL generates a coordinate list such as [x1,y1,x2,y2][x_1,y_1,x_2,y_2], including two square brackets and three commas. ReferFlow represents the target with an object-indexed region state consisting of two region latents enclosed by <|latent_start|> and <|latent_end|>. TAPD then recovers the coordinates without generating the brackets or commas.

The table below summarizes the estimated sequential steps for single-box prediction, including region latents and formatting or delimiter tokens.

ModelRegion latentsCoordinate decodingFormatting or delimiter tokensTotal sequential steps
Qwen3-VL-2B—4–1259–17
ReferFlow2428

This shorter decoding path helps explain ReferFlow’s lower inference latency on RefCOCOg. As reported in Table 8, ReferFlow achieves an average inference latency of 0.8 s, compared with 1.5 s for Qwen3-VL-2B. As the number of boxes increases, TAPD maintains a fixed coordinate-decoding depth of four rounds, widening its advantage over autoregressive decoding in terms of sequential decoding steps.

AuthorsOpenReview ↗

Response to Reviewer k8aP Part 3/4

Q3: Experiments on target disambiguation

We appreciate the reviewer’s suggestion. Table 2 reports ReferFlow’s overall localization accuracy on Ref-Adv. This benchmark includes same-category distractors, some of which satisfy only part of the referring expression, requiring models to distinguish the target from several plausible candidates. We further analyze this benchmark to evaluate ReferFlow’s disambiguation performance.

We compare ReferFlow with a coordinate-token baseline. The baseline represents target regions as bounding-box coordinate tokens, while ReferFlow uses object-indexed region states with explicit visual and spatial supervision and TAPD for coordinate decoding.

For each method, we train separate models under two SFT settings. Direct Grounding removes intermediate trajectory supervision and retains only final-target supervision. Adaptive Trajectory uses the default SFT data format, combining REC data with trajectory-level REC data, allowing the model to localize directly or generate intermediate grounding steps as needed. Within each setting, both methods use the same backbone, training data, and optimization budget. All models are evaluated after SFT without RL. We evaluate all four models on the same queries using final-target [email protected] and report results grouped by the number of distractors.

MethodGrounding modes2–3 distractors4–6 distractors≥7 distractorsOverall
Coordinate-token baselineDirect Grounding56.248.339.552.1
Coordinate-token baselineAdaptive Trajectory56.752.444.254.1
ReferFlowDirect Grounding58.251.443.454.6
ReferFlowAdaptive Trajectory63.359.452.761.0

Adaptive Trajectory improves overall accuracy from 52.1% to 54.1% for the coordinate-token baseline and from 54.6% to 61.0% for ReferFlow. ReferFlow’s advantage is larger under Adaptive Trajectory (61.0% vs. 54.1%) than under Direct Grounding (54.6% vs. 52.1%). Under Adaptive Trajectory, this advantage also grows with the number of distractors. These results show that ReferFlow benefits more from intermediate grounding and achieves stronger disambiguation performance among semantically similar candidates.

We have added this analysis to Appendix E.5 of the revised manuscript (Table 11, lines 1341–1349).

AuthorsOpenReview ↗

Response to Reviewer k8aP Part 4/4

Q4: Effects of multiple intermediate targets and task decomposition

We thank the reviewer for the question. Each grounding step in ReferFlow-Bench contains a single referring expression. When a step has multiple target boxes, all boxes correspond to objects matching that same expression, rather than separate targets described by different expressions. ReferFlow maintains a separate region state for each target as context for subsequent grounding. To examine whether multiple intermediate targets affect final grounding, we analyze how intermediate target count, trajectory length, and grounding quality relate to final localization. We also compare ReferFlow and existing methods when grounding steps are presented together or as separate localization subtasks.

  1. Effects of intermediate target count and trajectory length.

For ReferFlow, we fix the trajectory length at three steps and group samples by the total number of ground-truth boxes before the final step. F-Acc decreases from 66.9% with 2–3 intermediate targets to 64.5% with 4–6 targets and 58.7% with at least seven targets. We further examine trajectory length on samples containing 4–6 intermediate targets. F-Acc decreases from 64.5% at three steps to 62.8% at four steps and 58.2% at five steps. Samples with more intermediate targets or longer trajectories have lower F-Acc, highlighting the difficulty of long-horizon trajectory-level grounding.

Intermediate targets (three-step trajectories)2–34–6≥7
F-Acc (%)66.964.558.7
Trajectory length (4–6 intermediate targets)345
F-Acc (%)64.562.858.2
  1. Effects of intermediate grounding quality (Appendix E.2 / Figure 11; Figure 12 in the revised manuscript).

To examine how incorrect intermediate region states affect final grounding, we replace the region state of one intermediate target with that of another object in the same image (Mismatched) and rerun the remaining grounding steps. The table below summarizes the results from Figure 11 (Figure 12 in the revised manuscript). F-Acc drops substantially on Refinement, Shift, and Mixture, with the largest decrease on Shift, where an incorrect intermediate state can provide a misleading reference for locating the next target. Reset remains relatively stable, suggesting that ReferFlow limits interference from historical states when localizing independently specified targets. These results show that intermediate region states affect final grounding according to subsequent steps’ dependence on prior regions.

SettingRefinementShiftResetMixture
Original F-Acc (%)65.862.263.460.4
Mismatched F-Acc (%)40.617.862.638.4
F-Acc decrease25.244.40.822.0
  1. Effects of sequential localization subtasks (Appendix D.2 / Figure 9; Figure 10 in the revised manuscript).

We examine step-level decomposition using a multi-turn setting that provides one grounding query per turn. Each turn predicts all target instances matching that query. The original image and full dialogue history, including previously predicted boxes, remain available, preserving cross-step region dependencies. We compare this setting with our default single-turn evaluation, where the model receives all ordered steps at once, using the same images and grounding steps. This comparison separates the trajectory into step-level subtasks while retaining the complete target set within each step.

The table below summarizes selected F-Acc results from Figure 9 (Figure 10 in the revised manuscript) and additionally reports results for Qwen3-VL-2B. All five models achieve higher F-Acc when grounding steps are presented as separate localization subtasks, with ReferFlow performing best in both settings.

ModelSingle-turn F-Acc (%)Multi-turn F-Acc (%)
Qwen3-VL-2B29.638.7
Rex-Omni41.149.6
GLM-4.6V-Flash38.247.3
Qwen3.5-27B44.554.2
ReferFlow63.066.5

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response

We sincerely thank all reviewers for the time and effort devoted to evaluating our work and for their constructive and insightful comments. We are encouraged by their recognition of the motivation and value of evaluating intermediate grounding beyond final-target localization (@k8aP, @3RqU), the novel task formulation (@ZXHC), and the comprehensive benchmark design (@k8aP, @3RqU, @ZXHC). We also appreciate their positive assessment of ReferFlow’s strong empirical performance (@k8aP, @ZXHC) and the extensive experiments, ablation studies, and mechanism analyses supporting our findings (@k8aP, @3RqU, @ZXHC).

At the core of our work is the evaluation of intermediate visual grounding, an important process for extracting visual evidence. Complex queries often specify a target through other objects and their relationships, introducing multiple visual references. ReferFlow-Bench organizes these references into annotated grounding trajectories, revealing intermediate localization errors hidden by final-target evaluation and evaluating whether models maintain consistent grounding as cross-step region dependencies change. ReferFlow combines object-indexed region states with trajectory-aware parallel coordinate decoding, supporting consistent grounding across steps while reducing coordinate-decoding costs. Our work therefore offers a new perspective on evaluating intermediate visual evidence and may provide insights for the community’s future research on long-horizon visual grounding and reasoning tasks.

We have carefully considered each remark and have responded accordingly. Following the reviewers’ constructive suggestions, we have devoted considerable effort to further improving our work. Our responses and revisions address four main areas:

  1. Autonomous trajectory generation and generalization. We clarify ReferFlow’s ability to generate grounding trajectories from original queries and follow provided steps, supported by a paired evaluation on the full ReferFlow-Bench dataset. An additional evaluation on independently human-authored trajectories examines generalization beyond the automated annotation pipeline. We also clarify the differences from PNG in task formulation and model design.
  2. The value of intermediate grounding. New disambiguation experiments and error analyses, alongside existing supervision ablations and region-state interventions, examine the value of evaluating intermediate references. We further investigate the effects of target count, trajectory length, and step-level task decomposition, and demonstrate the benefits of trajectory-based training with Qwen3.5-4B.
  3. Fair comparisons and sources of improvement. We clarify the shared training conditions and explain which comparisons evaluate the contributions of SRLR, TAPD, trajectory supervision, and RL reward components.
  4. End-to-end inference efficiency. We compare the end-to-end efficiency and grounding performance of NTP and TAPD across different numbers of regions and explain ReferFlow’s speed advantage in single-target grounding.

Finally, we have revised the main text and appendices to incorporate key experimental results and clarify the points discussed in our responses. Newly added content is highlighted in blue for ease of reference. We sincerely appreciate the reviewers’ thoughtful guidance, which has helped strengthen our experimental validation and improve the clarity of the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.