- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes ReferFlow for trajectory-level REC, extending evaluation from final-target localization to intermediate grounding steps. The authors introduce ReferFlow-Bench with four trajectory types: Refinement, Shift, Reset, and Mixture. ReferFlow combines continuous SRLR, TAPD, and Grounding-Aware GRPO to improve multi-step grounding. Experiments on ReferFlow-Bench and standard REC benchmarks demonstrate strong performance, supported by extensive comparisons and ablation studies.
Critical strengths
1.Novel task formulation and benchmark. 2.Very strong performance and thorough validation.
Critical weaknesses
None
Questions for authors
-
Benchmark Evaluation: ReferFlow-Bench provides models with predefined grounding steps, while ReferFlow also claims to support adaptive trajectory generation. Could the authors evaluate the model directly from the original complex queries without providing intermediate steps? This would better demonstrate its end-to-end reasoning capability.
-
Fair Comparison: ReferFlow is trained on 1.6M samples, including 400K newly constructed trajectories. Could the authors provide a baseline trained with the same data and training budget, but without the proposed modules? This would help distinguish the contributions of the framework from those of additional training data.
-
Benchmark Generalization: Both ReferFlow-Train and ReferFlow-Bench are constructed using the same automated annotation pipeline. Could the authors provide evidence that the improvements generalize to independently constructed or human-authored grounding trajectories, rather than relying on patterns specific to the generated data?
5 comments
Response to Reviewer ZXHC Part 2/3
Q2: Fair comparison under matched training conditions
Thank you for highlighting the importance of a matched baseline. The Text baseline in Table 3 provides this comparison. It uses the same Qwen3-VL-2B backbone, SFT data (1.6M examples, including 400K ReferFlow-Train trajectories), and optimization budget as the models with the proposed modules. It represents target regions using bounding-box coordinate tokens without SRLR or TAPD. All models in the architectural comparisons in Tables 3 and 4(a) are trained without RL. Details of the training data are provided in Appendix C.1 and Table 5.
Table 3 compares the Text baseline with SRLR under the same next-token prediction (NTP) decoding scheme, evaluating the region-state representation and its visual and spatial supervision. Table 4(a) keeps SRLR fixed and compares coordinate decoding strategies to evaluate TAPD.
Under these matched SFT conditions, SRLR and TAPD improve RefCOCOg accuracy from 87.9% to 92.0% and GroundingME accuracy from 32.4% to 39.1%. On ReferFlow-Bench, Q-Acc increases from 52.6% to 60.8% and T-Acc from 17.9% to 25.6%. These results demonstrate the contributions of the proposed modules under the same training data and optimization budget. Grounding-Aware GRPO provides further improvements, reported separately in the last row of the table below.
| Model | Training | RefCOCOg | GroundingME | ReferFlow-Bench Q-Acc | ReferFlow-Bench T-Acc | ReferFlow-Bench F-Acc |
|---|---|---|---|---|---|---|
| Text baseline | SFT | 87.9 | 32.4 | 52.6 | 17.9 | 48.2 |
| SRLR + NTP | SFT | 90.5 | 37.4 | 56.9 | 21.3 | 53.9 |
| SRLR + TAPD | SFT | 92.0 | 39.1 | 60.8 | 25.6 | 59.4 |
| SRLR + TAPD | SFT + RL | 92.9 | 43.0 | 63.9 | 28.4 | 63.0 |
We have clarified the Text baseline and shared training settings in the caption of Table 3 in the revised manuscript (lines 441–443).
Response to Reviewer ZXHC Part 3/3
Q3: Generalization beyond the automated annotation pipeline
Thank you for this insightful question. Appendix E.7 (Appendix E.8 in the revised manuscript) reports a human evaluation of ReferFlow’s self-generated grounding trajectories on 500 randomly sampled GroundingME samples, checking whether the predicted intermediate boxes correctly ground the corresponding step descriptions. To examine generalization beyond our automated annotation pipeline, we use the same 500 images and original queries to construct grounding trajectories independently written by human annotators.
One annotator writes grounding steps and annotates intermediate target boxes, retaining the original final-target annotations. Two other annotators conduct two rounds of review to check consistency with the original queries and box accuracy. Disagreements are resolved through discussion. Thirteen samples with unresolved annotation ambiguity are excluded before model evaluation, leaving 487 human-authored trajectories. Annotators do not refer to model outputs or trajectories generated by our automated pipeline, and all steps use only localization cues present in the original queries.
We evaluate ReferFlow and four comparison models on these 487 samples without additional training. Each model is evaluated separately with the human-authored ordered steps and the original query. With ordered steps, we follow the ReferFlow-Bench protocol and report Q-Acc, T-Acc, and F-Acc. With original queries, we report final-target accuracy following the GroundingME evaluation protocol.
As shown in the table below, ReferFlow achieves 57.2% Q-Acc, 24.2% T-Acc, and 43.5% F-Acc on the human-authored trajectories, outperforming all four comparison models. Qwen3.5-27B, the strongest comparison model, achieves 52.6%, 20.1%, and 39.8%, respectively. These results support the generalization of ReferFlow’s gains to independently human-authored grounding trajectories. Given only the original queries, ReferFlow also achieves the highest final-target accuracy at 40.2%, compared with 38.0% for Qwen3.5-27B.
| Model | Original Query Acc | Human-Authored Steps Q-Acc | Human-Authored Steps T-Acc | Human-Authored Steps F-Acc |
|---|---|---|---|---|
| Qwen3-VL-2B | 26.7 | 37.8 | 9.9 | 28.1 |
| Rex-Omni | 29.2 | 42.4 | 14.4 | 31.8 |
| GLM-4.6V-Flash | 27.7 | 39.4 | 10.5 | 30.0 |
| Qwen3.5-27B | 38.0 | 52.6 | 20.1 | 39.8 |
| ReferFlow | 40.2 | 57.2 | 24.2 | 43.5 |
Response to Reviewer ZXHC Part 1/3
We appreciate the reviewer’s positive assessment of our task formulation and benchmark, as well as the recognition of our strong results and thorough experiments. We address the reviewer’s questions below.
Q1: Evaluation from original complex queries
Thank you for suggesting this evaluation. ReferFlow can directly localize a target or generate intermediate grounding steps as needed from an original complex query. Our general REC experiments in Table 2 already evaluate this capability given only the image and the original query, with no intermediate steps provided at inference time. ReferFlow achieves state-of-the-art results on eight benchmarks, including the reasoning-oriented GroundingME and PR-Bench.
Each sample in ReferFlow-Bench includes an original complex query and its decomposition into ordered grounding steps. We evaluate ReferFlow and four comparison models on the full dataset under both input formats, using the same checkpoint for each model across both settings. In the original-query setting, models receive only the image and the original query, without intermediate steps. The original query refers to the target of the final annotated step, so we report F-Acc for both settings using the same final target.
Given only the image and the original query, ReferFlow achieves 59.8% overall F-Acc, outperforming all comparison models, including Qwen3.5-27B at 40.4%. These results demonstrate ReferFlow’s end-to-end grounding capability without provided intermediate steps.
With annotated steps, ReferFlow achieves 63.0% overall F-Acc. The gains are larger on Shift and Mixture, where the provided steps specify intermediate visual references and cross-step region dependencies. Refinement also benefits from the progression from broader to narrower target sets. On Reset, the original-query setting performs slightly better (64.2% vs. 63.4%), consistent with its independently specified targets.
We have added this paired evaluation to Section 5.3 of the revised manuscript (Figure 7, lines 468–477).