- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper analyzes a common challenge in multimodal Retrieval-Augmented Generation (RAG), where queries regarding multimodal entities are often ambiguous. This issue essentially stems from a lack of human alignment within Vision-Language Models (VLMs), specifically, challenges in internal image-text alignment (identifying the specific entity within the visual content) and output alignment (generating a precise textual representation of an entity rather than an ambiguous one). While post-training can partially mitigate such issues, the inherent ambiguity in raw human queries, combined with the fact that such errors cannot be entirely eliminated, makes this an important area for research. The authors also introduce a dataset and a training-free recursive query method, propose intervention techniques, namely "Oracle" and "Disturbance", as well as an Retrieval-Involved Trajectory Efficiency (RTE) criterion, and provide a detailed analysis of the proposed approach.
Critical strengths
- The issue raised is highly relevant to practical circumstances and is of significant importance.
Critical weaknesses
-
As I mentioned in the paper summary, this problem seems closely related to VLM alignment. A VLM that can ground visual content and describe entities accurately could reduce or perhaps even resolve it. Related work such as [1] uses RL post-training to improve a VLM's querying behavior. Could the authors discuss how much of the problem remains with such methods? Also, the prompts are detailed but all zero-shot. Would adding a few examples help?
-
My main concern is the evaluation. The experiments are limited to the authors' RefAmb benchmark, which has not yet been released. Task completion is assessed only through a rubric scored by an LLM judge, as acknowledged in the limitations. The evaluation also lacks external baselines, including on RefAmb. Comparisons with related methods, particularly RL-based approaches, would help clarify how much of the problem they already address and what the proposed method adds.
-
RTE seems reasonable, though I wonder how it handles cases where the model introduces incorrect facts and uses them in later reasoning, as RTE would presumably remain high in such cases despite the factual errors. Could the authors clarify whether there is a check on the correctness of the reused facts? Since RTE relies on LLM annotation, it would also be useful to discuss how the judge model's preferences might affect the score.
[1] Wang Q, Ding R, Zeng Y, et al. Vrag-rl: Empower vision-perception-based rag for visually rich information understanding via iterative reasoning with reinforcement learning[J]. Advances in Neural Information Processing Systems, 2026, 38: 57133-57160.
Questions for authors
Please see the weaknesses above. I also have the following concerns and questions.
-
The cost analysis only compares with a standard ReAct baseline. Given the repeated reflection steps, the proposed method may still be expensive in absolute terms, even if it costs less than this baseline. Could the authors report the absolute cost and include a comparison with an efficient method?
-
IIGCR introduces extra components and a gating threshold, tau_s, but it is unclear how these choices carry over to a new benchmark. How was tau_s chosen in the current experiments?
I would be willing to revise my rating if most of these concerns are addressed.
5 comments
Thanks for the additional experiments. MMSearch-R1 is cited twice, as a 2025 arXiv paper and a 2026 ACL paper, which caused my confusion. Please merge these references.
The new results on OK-VQA and A-OKVQA address my concern that evaluation was limited to RefAmb. However, RefAmb remains unavailable, so the reproducibility concern remains. The RTE response also does not directly address the case where incorrect facts are reused and receive high scores. Agreement with human annotations alone does not establish factual correctness. It is also unclear whether threshold selection used a development split separate from the final evaluation. While I appreciate the additional evidence, I maintain my original rating.
We sincerely thanks a lot for your patient response.
W1: Redundant references and Availability of RefAmb
We removed the newly added redundant references to MMSearch-R1. Besides, we have open-sourced the RefAmb test split dataset, which is provided in https://github.com/anonymous-artifact-2026831/anonymous-supplementary-code/tree/main/data
W2: Can RTE assign high scores to trajectories that absorb incorrect facts?
We thank the reviewer for raising this concern. Theoretically, RTE can assign a high score to a step that absorbs incorrect evidence, because its Utility factor measures uptake in subsequent reasoning rather than independently verified factual correctness. RTE originally stands for diagnoses novel information acquisition and subsequent use to study trajectory evolution and stagnation. We have clarified this scope and added a reference-grounded factuality check and correctness-constrained sensitivity analysis.
Blinded factuality audit. We audit Qwen2.5-VL-7B on MCSearch (since this source has golden retrieval annotations) using 150 retrieval steps: five conditions (ReAct, Oracle, Disturbance, and IIGCR), three automatic-Utility strata , and ten steps per condition–stratum cell. The GPT-5.1 receives the question, current sub-question, query, retrieved/extracted evidence, subsequent reasoning, and benchmark reference facts, but not condition identities or automatic Utility scores. It identifies explicitly used retrieved claims and assesses their correctness and relevance. A claim is classified as incorrect when contradicted by the supplied evidence or reference facts; claims that cannot be verified from the available materials are classified separately as unverifiable. Irrelevant use denotes use of a non-incorrect claim that does not support the current sub-question, with incorrect use taking precedence.
| Automatic Utility (U_t) | Sub-question Support | Incorrect use | Irrelevant use | Unverifiable use |
|---|---|---|---|---|
| 0 | 10.0% | 44.0% | 0.0% | 2.0% |
| 0.5 | 74.0% | 26.0% | 0.0% | 4.0% |
| 1 | 92.0% | 10.0% | 0.0% | 0.0% |
Percentages indicate the proportion of audited steps with each property. Thus, high Utility is associated with greater solution support and less incorrect use in this sample, but does not completely guarantee factual correctness: 5 of the 50 steps at (U_t=1) still use incorrect claims. This directly confirms the reviewer’s concern while quantifying its occurrence in the audited sample.
Correctness-constrained sensitivity. We additionally design a correctness-constrRTE$\Delta F_t \cdot U_t \cdot \mathbb{I}(\text{at least one correct and relevant retrieved claim is used}).$This gate retains a step’s original score only when verified correct and relevant uptake is identified; otherwise, it assigns zero. It is a supplementary sensitivity analysis, not a redefinition of RTE. Raw and checked scores below are balanced step-level means over the same samples per condition.
| Intervention run | Raw score | Checked score | Incorrect use | Irrelevant use | Unverifiable use |
|---|---|---|---|---|---|
| Legacy ReAct | 0.324 | 0.286 | 20.0% | 0.0% | 3.3% |
| Oracle | 0.377 | 0.337 | 10.0% | 0.0% | 6.7% |
| Disturbance | 0.305 | 0.258 | 46.7% | 0.0% | 0.0% |
Under the correctness constraint, the intervention scores follow Oracle > ReAct > Disturbance (0.337 > 0.286 > 0.258), supporting the expected direction of referential intervention effects in this sample. We ran the RTE metric using the revised RTE on the full IIGCR dataset again, and the results are as follows. Although the values have changed, the partial order of the performance metrics remains unchanged.
| Method | Raw score | Checked score |
|---|---|---|
| ReAct | 0.311 | 0.259 |
| IIGCR | 0.389 | 0.323 |
IIGCR has a higher checked score than ReAct, indicating the core observation under the old metric is not reversed.The underlying reason for this consistency is also consistent with the model’s role in evidence selection. VLMs are not passive recipients of retrieval results: their prior knowledge, visual understanding, and assessment of the current question can help them reject irrelevant evidence and identify information that fills a reasoning gap. Consequently, explicit evidence uptake can reflect this internal filtering process.
Revisions
We will document these supplementary results in the revised manuscript Appendix E.4
We sincerely thank the reviewer for your patience and suggestions.
W1.1: Complementarity with RL post-training
We agree that post-trained visual grounding and query policies can alleviate referential under-specification. However, Oracle grounding does not recover all failures (Section 3.4), motivating repair of subsequent actions and accumulated trajectories.
We tested MMSearch-R1-7B with/without IIGCR on the CRAG subset of RefAmb:
IIGCR yields significant gains of 14.23 F1/5.75 LJ points, with 30.9% fewer retrievals and 0.97s additional latency. This directly supports that RL still has room for improvement and the complementarity between RL and online repair for this backbone (more details in lines 1598–1615).
We also evaluated the released VRAG-RL checkpoint, obtaining average RTE/LJ/F1 of 35.66/23.80/24.36 (Table 2). Due to computational constraints, we did not retrain it on RefAmb. This is a transfer evaluation: differences between its visually rich document setting and RefAmb’s entity-centric tasks may affect performance, rather than indicate an inherent limitation of RL (more details in line 1564–1597).
W1.2: Few-shot prompting
Following your advice, we added three demonstrations from an independent development set:
Examples improve ReAct’s F1/LJ but not EM/RTE. Zero-shot IIGCR outperforms three-shot ReAct across quality metrics, while demonstrations do not improve IIGCR. Thus, examples help but do not replace online repair in this setting (lines 1339–1349).
W2: Evaluation and external validity
Data availability. RefAmb draws on public OVEN, InfoSeek, CRAG-MM, and MC-Search data, with construction and annotation checks in Appendices C–D. After rebuttal, we will release indices, task assignments, annotations, and construction/evaluation code, subject to source-data licenses.
External benchmarks. We evaluated OK-VQA and A-OKVQA without RefAmb’s referential filtering and with unchanged thresholds and repair settings:
IIGCR improves F1/LJ over both baselines; gains over Ambiguity Rewrite are significant at p<.05. These results support transfer beyond RefAmb without target-specific tuning (added in lines 483–500).
Evaluation/baselines. LJ uses reference-grounded correctness judgments (Appendix L.5); Table 2 now additionally reports F1. The original paper already includes MMSearch-R1. The added VRAG-RL evaluation and paired MMSearch-R1 experiment broaden the RL comparison and test additional repair benefits.
W3: RTE and factual correctness
RTE measures novel information acquisition and subsequent uptake to diagnose trajectory evolution and stagnation, rather than factual correctness. However, human agreement reaches high κ for step Utility and high ρ and τ for trajectory quality (Table 8). This consistency may reflect VLMs’ use of prior knowledge and visual understanding to select evidence they consider correct, making subsequent uptake more likely to involve genuinely correct information(lines 1179–1196). RTE therefore complements, rather than replaces, final-answer evaluation through LJ (Appendix E.3).
Q1: Absolute cost
Table 5 reports higher absolute token counts and latency, indicating that reflection increases output tokens. Under the same evaluation setting:
Compared with MMSearch-R1, which achieves a better trade-off between accuracy and efficiency, IIGCR incurs slightly higher latency while improving answer quality, offering an accuracy–cost trade-off rather than universally lower latency.
Q2: Threshold selection and transfer
We selected τs=.90 and τf=.20 through hyperparameter search on RefAmb. Sensitivity analyses in Figures 13–14 show LJ around .32–.34 for τf=.05–.40 and above .33 for τs=.50–.90. This indicates that within the tested range, threshold variations do not lead to significant performance degradation. The unchanged configuration also improves OK-VQA and A-OKVQA, providing direct transfer evidence.