acceptodds
Under review as a conference paper at ICLR 2027

Trajectory Analysis and Optimization of Agentic Multi-modal RAG Under Referential Definiteness Modulation

Abstract

Agentic multimodal Retrieval-Augmented Generation (RAG) systems rely on Vision-Language Models (VLMs) to formulate retrieval queries. However, these queries often contain under-specified references to visual entities, causing the retriever to return semantically related but functionally non-resolving evidence, thereby impairing downstream multi-hop reasoning. While existing work on multimodal disambiguation largely assumes static pipelines or manually clarified references, to the best of our knowledge we provide the first systematic study of how such referential under-specification propagates through iterative retrieval trajectories and impacts final answers. To enable rigorous diagnosis, we construct a multi-dataset benchmark and introduce Retrieval-Involved Trajectory Efficiency (RTE), a metric that quantifies the novelty and utility of retrieved evidence. Controlled interventions using oracle entity grounding and entity obfuscation across various VLM backbones reveal that obfuscation increases factual novelty at the expense of utility, while oracle grounding yields diminishing returns as backbone capability grows. Notably, final answers exhibit surprising robustness to ambiguous entity references compared to their intermediate reasoning trajectories. We further identify trajectory stagnation as a recurring failure mode where naive query rewriting offers limited remedy. Motivated by these findings, we propose IIGCR, a training-free optimization mechanism that detects stagnation online and recovers degraded trajectories through context reconstruction and referent-aware rewriting. Extensive experiments demonstrate that IIGCR effectively leverages trajectory-level failure signals for online repair, consistently enhancing both final answer quality and trajectory efficiency. Our code will be publicly available upon acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic RAG, Agents, LLMs, LVLMs, Information Retrieval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BBYQOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper analyzes a common challenge in multimodal Retrieval-Augmented Generation (RAG), where queries regarding multimodal entities are often ambiguous. This issue essentially stems from a lack of human alignment within Vision-Language Models (VLMs), specifically, challenges in internal image-text alignment (identifying the specific entity within the visual content) and output alignment (generating a precise textual representation of an entity rather than an ambiguous one). While post-training can partially mitigate such issues, the inherent ambiguity in raw human queries, combined with the fact that such errors cannot be entirely eliminated, makes this an important area for research. The authors also introduce a dataset and a training-free recursive query method, propose intervention techniques, namely "Oracle" and "Disturbance", as well as an Retrieval-Involved Trajectory Efficiency (RTE) criterion, and provide a detailed analysis of the proposed approach.

Critical strengths
  1. The issue raised is highly relevant to practical circumstances and is of significant importance.
Critical weaknesses
  1. As I mentioned in the paper summary, this problem seems closely related to VLM alignment. A VLM that can ground visual content and describe entities accurately could reduce or perhaps even resolve it. Related work such as [1] uses RL post-training to improve a VLM's querying behavior. Could the authors discuss how much of the problem remains with such methods? Also, the prompts are detailed but all zero-shot. Would adding a few examples help?

  2. My main concern is the evaluation. The experiments are limited to the authors' RefAmb benchmark, which has not yet been released. Task completion is assessed only through a rubric scored by an LLM judge, as acknowledged in the limitations. The evaluation also lacks external baselines, including on RefAmb. Comparisons with related methods, particularly RL-based approaches, would help clarify how much of the problem they already address and what the proposed method adds.

  3. RTE seems reasonable, though I wonder how it handles cases where the model introduces incorrect facts and uses them in later reasoning, as RTE would presumably remain high in such cases despite the factual errors. Could the authors clarify whether there is a check on the correctness of the reused facts? Since RTE relies on LLM annotation, it would also be useful to discuss how the judge model's preferences might affect the score.

[1] Wang Q, Ding R, Zeng Y, et al. Vrag-rl: Empower vision-perception-based rag for visually rich information understanding via iterative reasoning with reinforcement learning[J]. Advances in Neural Information Processing Systems, 2026, 38: 57133-57160.

Questions for authors

Please see the weaknesses above. I also have the following concerns and questions.

  1. The cost analysis only compares with a standard ReAct baseline. Given the repeated reflection steps, the proposed method may still be expensive in absolute terms, even if it costs less than this baseline. Could the authors report the absolute cost and include a comparison with an efficient method?

  2. IIGCR introduces extra components and a gating threshold, tau_s, but it is unclear how these choices carry over to a new benchmark. How was tau_s chosen in the current experiments?

I would be willing to revise my rating if most of these concerns are addressed.

5 comments

We sincerely thank the reviewer for your patience and suggestions.

W1.1: Complementarity with RL post-training

We agree that post-trained visual grounding and query policies can alleviate referential under-specification. However, Oracle grounding does not recover all failures (Section 3.4), motivating repair of subsequent actions and accumulated trajectories.

We tested MMSearch-R1-7B with/without IIGCR on the CRAG subset of RefAmb:

MethodF1LJRetrievalsTime(s)
MMSearch-R119.6819.801.7510.15
+ IIGCR33.91***25.55**1.2111.12

IIGCR yields significant gains of 14.23 F1/5.75 LJ points, with 30.9% fewer retrievals and 0.97s additional latency. This directly supports that RL still has room for improvement and the complementarity between RL and online repair for this backbone (more details in lines 1598–1615).

We also evaluated the released VRAG-RL checkpoint, obtaining average RTE/LJ/F1 of 35.66/23.80/24.36 (Table 2). Due to computational constraints, we did not retrain it on RefAmb. This is a transfer evaluation: differences between its visually rich document setting and RefAmb’s entity-centric tasks may affect performance, rather than indicate an inherent limitation of RL (more details in line 1564–1597).

W1.2: Few-shot prompting

Following your advice, we added three demonstrations from an independent development set:

MethodKF1EMLJRTE
ReAct525.319.3325.67.311
ReAct + 3-shot527.508.6729.40.264
IIGCR531.0111.0733.47.390
IIGCR + 3-shot531.3511.3331.73.353

Examples improve ReAct’s F1/LJ but not EM/RTE. Zero-shot IIGCR outperforms three-shot ReAct across quality metrics, while demonstrations do not improve IIGCR. Thus, examples help but do not replace online repair in this setting (lines 1339–1349).

W2: Evaluation and external validity

Data availability. RefAmb draws on public OVEN, InfoSeek, CRAG-MM, and MC-Search data, with construction and annotation checks in Appendices C–D. After rebuttal, we will release indices, task assignments, annotations, and construction/evaluation code, subject to source-data licenses.

External benchmarks. We evaluated OK-VQA and A-OKVQA without RefAmb’s referential filtering and with unchanged thresholds and repair settings:

DatasetMethodF1LJRTETime(s)
OK-VQAReAct13.5639.00.4115.74
Ambiguity Rewrite12.9637.00.4027.67
IIGCR18.55*44.50*.4225.28
A-OKVQAReAct16.8845.00.2515.20
Ambiguity Rewrite17.0742.00.3056.53
IIGCR19.15*49.50*.3144.48

IIGCR improves F1/LJ over both baselines; gains over Ambiguity Rewrite are significant at p<.05. These results support transfer beyond RefAmb without target-specific tuning (added in lines 483–500).

Evaluation/baselines. LJ uses reference-grounded correctness judgments (Appendix L.5); Table 2 now additionally reports F1. The original paper already includes MMSearch-R1. The added VRAG-RL evaluation and paired MMSearch-R1 experiment broaden the RL comparison and test additional repair benefits.

W3: RTE and factual correctness

RTE measures novel information acquisition and subsequent uptake to diagnose trajectory evolution and stagnation, rather than factual correctness. However, human agreement reaches high κ\kappa for step Utility and high ρ\rho and τ\tau for trajectory quality (Table 8). This consistency may reflect VLMs’ use of prior knowledge and visual understanding to select evidence they consider correct, making subsequent uptake more likely to involve genuinely correct information(lines 1179–1196). RTE therefore complements, rather than replaces, final-answer evaluation through LJ (Appendix E.3).

Q1: Absolute cost

Table 5 reports higher absolute token counts and latency, indicating that reflection increases output tokens. Under the same evaluation setting:

MethodTime(s)#Retrievals
ReAct10.023.01
MMSearch-R110.151.75
IIGCR11.192.60

Compared with MMSearch-R1, which achieves a better trade-off between accuracy and efficiency, IIGCR incurs slightly higher latency while improving answer quality, offering an accuracy–cost trade-off rather than universally lower latency.

Q2: Threshold selection and transfer

We selected τs=.90\tau_s=.90 and τf=.20\tau_f=.20 through hyperparameter search on RefAmb. Sensitivity analyses in Figures 13–14 show LJ around .32–.34 for τf=.05–.40\tau_f =.05–.40 and above .33 for τs=.50–.90\tau_s=.50–.90. This indicates that within the tested range, threshold variations do not lead to significant performance degradation. The unchanged configuration also improves OK-VQA and A-OKVQA, providing direct transfer evidence.

Thanks for the additional experiments. MMSearch-R1 is cited twice, as a 2025 arXiv paper and a 2026 ACL paper, which caused my confusion. Please merge these references.

The new results on OK-VQA and A-OKVQA address my concern that evaluation was limited to RefAmb. However, RefAmb remains unavailable, so the reproducibility concern remains. The RTE response also does not directly address the case where incorrect facts are reused and receive high scores. Agreement with human annotations alone does not establish factual correctness. It is also unclear whether threshold selection used a development split separate from the final evaluation. While I appreciate the additional evidence, I maintain my original rating.

We sincerely thanks a lot for your patient response.

W1: Redundant references and Availability of RefAmb

We removed the newly added redundant references to MMSearch-R1. Besides, we have open-sourced the RefAmb test split dataset, which is provided in https://github.com/anonymous-artifact-2026831/anonymous-supplementary-code/tree/main/data

W2: Can RTE assign high scores to trajectories that absorb incorrect facts?

We thank the reviewer for raising this concern. Theoretically, RTE can assign a high score to a step that absorbs incorrect evidence, because its Utility factor measures uptake in subsequent reasoning rather than independently verified factual correctness. RTE originally stands for diagnoses novel information acquisition and subsequent use to study trajectory evolution and stagnation. We have clarified this scope and added a reference-grounded factuality check and correctness-constrained sensitivity analysis.

Blinded factuality audit. We audit Qwen2.5-VL-7B on MCSearch (since this source has golden retrieval annotations) using 150 retrieval steps: five conditions (ReAct, Oracle, Disturbance, and IIGCR), three automatic-Utility strata Ut∈0,0.5,1U_t\in{0,0.5,1}, and ten steps per condition–stratum cell. The GPT-5.1 receives the question, current sub-question, query, retrieved/extracted evidence, subsequent reasoning, and benchmark reference facts, but not condition identities or automatic Utility scores. It identifies explicitly used retrieved claims and assesses their correctness and relevance. A claim is classified as incorrect when contradicted by the supplied evidence or reference facts; claims that cannot be verified from the available materials are classified separately as unverifiable. Irrelevant use denotes use of a non-incorrect claim that does not support the current sub-question, with incorrect use taking precedence.

Automatic Utility (U_t)Sub-question SupportIncorrect useIrrelevant useUnverifiable use
010.0%44.0%0.0%2.0%
0.574.0%26.0%0.0%4.0%
192.0%10.0%0.0%0.0%

Percentages indicate the proportion of audited steps with each property. Thus, high Utility is associated with greater solution support and less incorrect use in this sample, but does not completely guarantee factual correctness: 5 of the 50 steps at (U_t=1) still use incorrect claims. This directly confirms the reviewer’s concern while quantifying its occurrence in the audited sample.

Correctness-constrained sensitivity. We additionally design a correctness-constrRTE$\Delta F_t \cdot U_t \cdot \mathbb{I}(\text{at least one correct and relevant retrieved claim is used}).$This gate retains a step’s original score only when verified correct and relevant uptake is identified; otherwise, it assigns zero. It is a supplementary sensitivity analysis, not a redefinition of RTE. Raw and checked scores below are balanced step-level means over the same samples per condition.

Intervention runRaw scoreChecked scoreIncorrect useIrrelevant useUnverifiable use
Legacy ReAct0.3240.28620.0%0.0%3.3%
Oracle0.3770.33710.0%0.0%6.7%
Disturbance0.3050.25846.7%0.0%0.0%

Under the correctness constraint, the intervention scores follow Oracle > ReAct > Disturbance (0.337 > 0.286 > 0.258), supporting the expected direction of referential intervention effects in this sample. We ran the RTE metric using the revised RTE on the full IIGCR dataset again, and the results are as follows. Although the values have changed, the partial order of the performance metrics remains unchanged.

MethodRaw scoreChecked score
ReAct0.3110.259
IIGCR0.3890.323

IIGCR has a higher checked score than ReAct, indicating the core observation under the old metric is not reversed.The underlying reason for this consistency is also consistent with the model’s role in evidence selection. VLMs are not passive recipients of retrieval results: their prior knowledge, visual understanding, and assessment of the current question can help them reject irrelevant evidence and identify information that fills a reasoning gap. Consequently, explicit evidence uptake can reflect this internal filtering process.

Revisions

We will document these supplementary results in the revised manuscript Appendix E.4

Reviewer_aPgmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the problem of referential under-specification in agentic multimodal RAG. When VLMs generate retrieval queries, they may use vague expressions such as "the object in the image", making it difficult for retrievers to find useful information. Rather than evaluating only the final answers, the paper studies how such ambiguous queries affect the entire retrieval trajectory. To this end, it constructs RefAmb, a benchmark containing 1,500 samples, and introduces Retrieval-Involved Trajectory Efficiency (RTE) to measure the novelty and utility of retrieved information. The paper then uses oracle and disturbance interventions to analyze the effects of referential ambiguity on retrieval trajectories. Based on these findings, it proposes IIGCR, a training-free method that detects retrieval stagnation and repairs trajectories through query rewriting, visual grounding, reflection, and context pruning. The experiments demonstrate improvements in retrieval efficiency and final answer quality on multiple VLM backbones.

Critical strengths
  1. This paper investigates an interesting problem in agentic multimodal RAG. While VLMs can understand the visual context, text retrievers only receive the generated queries, which may contain ambiguous references. I think studying how this mismatch affects the entire retrieval trajectory, rather than just the final answer, is meaningful. The oracle and disturbance interventions also provide useful insights into how referential ambiguity influences retrieval behavior.

  2. The evaluation is relatively comprehensive. The authors construct a benchmark with 1,500 samples from four existing datasets and evaluate multiple VLM backbones. The proposed IIGCR is training-free and achieves improvements over several baselines. The paper also includes ablation studies and efficiency analysis, which help demonstrate the effectiveness of the proposed components.

Critical weaknesses
  1. I have some concerns about the proposed RTE metric. The paper relies on an LLM to evaluate retrieval utility based on the model's subsequent reasoning text. However, this evaluation does not directly verify whether the retrieved evidence is correct or actually contributes to answering the question. A model may consider retrieved information useful even when it is incorrect or irrelevant. Although the paper reports agreement with human judgments, it remains unclear whether RTE accurately reflects the actual contribution of retrieved evidence. Since RTE is central to the paper's analysis, this limitation may affect the reliability of the reported findings.

  2. The experiments do not clearly establish whether the improvements of IIGCR specifically come from addressing referential under-specification. IIGCR incorporates several general trajectory repair techniques, including repeated-query detection, reflection, and context pruning, which may also benefit retrieval trajectories without ambiguous references. However, the paper does not compare IIGCR with a general stagnation-repair baseline or separately evaluate its performance on ambiguous and non-ambiguous cases. As a result, it is difficult to determine whether the observed improvements are specifically attributable to resolving referential ambiguity or to more general improvements in retrieval planning.

Questions for authors
  • How reliable is RTE in measuring the actual contribution of retrieved evidence? Can the authors provide human evaluation that directly checks whether the retrieved information is correct and useful?

  • Can the authors separately evaluate IIGCR on trajectories with and without ambiguous references? How does it compare with a general trajectory repair method?

  • Why are the IIGCR results for Qwen2.5-VL-7B different between Table 2 and Table 13? Are they evaluated under different settings?

2 comments

W1: Validity and reliability of RTE

We sincerely thank the reviewer for highlighting the distinction between evidence uptake, factual correctness, and contribution to the final answer. RTE diagnoses whether the ReAct retrieval–feedback loop introduces new information and absorbs it into subsequent reasoning, enabling analysis of how referential definiteness affects trajectory evolution and stagnation. It is not a direct measure of factual correctness or causal contribution to the final answer. Incorrect evidence can still receive high Utility when absorbed. This does not change RTE’s intended target—new information acquisition and subsequent use—but explains why it complements rather than replaces LJ (Appendix E.3). We will clarify this scope in the revision (line 145-146).

To test whether high Utility may reflect absorption of incorrect information, we added a blinded evidence audit of approximately 200 trajectories covering five task categories and high/low RTE. Annotators examined evidence and subsequent reasoning with method identities and automatic scores hidden. ESS measures whether evidence provides at least partial support for the current subproblem; IEA measures whether reasoning relies on incorrect or irrelevant retrieved evidence.

U_tESS (%)IEA (%)
030.39.1
0.569.710.3
170.610.1

The audit supports the intended interpretation of Utility: the proportion of steps providing solution support rises from 30.3% at (U_t=0) to 69.7% and 70.6% at (U_t=0.5) and (U_t=1), respectively. In contrast, incorrect or irrelevant evidence uptake remains approximately stable at 9.1–10.3%. Thus, higher Utility is associated with substantially more task-supporting evidence, without a corresponding increase in erroneous evidence uptake. This directly addresses the concern that the score merely rewards a model’s willingness to incorporate retrieved content regardless of its usefulness.

The underlying reason for this consistency is also consistent with the model’s role in evidence selection. VLMs are not passive recipients of retrieval results: their prior knowledge, visual understanding, and assessment of the current question can help them reject irrelevant evidence and identify information that fills a reasoning gap. Detailed discussion has added in line 1179-1196.

W2: Attribution of IIGCR’s improvements

We added fixed ambiguity-based subgroup evaluation and generic repair controls to distinguish where gains occur and whether isolated general interventions explain them.

We group examples solely by whether the original ReAct trajectory ever issues an under-specified text query, keeping assignments fixed across methods to avoid intervention-dependent grouping.

ReAct subgroupMethodF1EMLJ
Under-specifiedReAct22.685.2617.34
IIGCR27.655.8822.60
Gain+4.97+0.62+5.26
SpecifiedReAct30.6611.7232.54
IIGCR31.6911.2135.68
Gain+1.02−0.51+3.14

The larger F1/LJ gains on under-specified trajectories support IIGCR’s intended role. Gains on the remaining examples also reflect general recovery benefits. More details have added in line 1617-1629.

We also compare isolated controllers under the same five-turn budget (added in line 1669-1676). Reflection tests general replanning; history-aware and random pruning test whether history reconstruction or deletion alone explains the gains.

ControllerF1EMLJRTETime(s)
None24.949.3325.27.31110.02
Local repair30.599.4031.13.29411.71
Global reflection30.299.8731.87.28011.43
History-aware pruning30.3110.0732.07.32411.34
Random pruning29.889.8730.53.31711.58
Full IIGCR31.8111.0733.43.38411.19

Full IIGCR exceeds every isolated controller across all quality metrics. Against history-aware pruning, the strongest isolated LJ baseline, it gains 1.50/1.00/1.36 F1/EM/LJ points and increases RTE from .324 to .384.

These findings align with Section 3: Oracle grounding does not recover every failure, motivating local entity/visual repair together with reflection and pruning of accumulated failure states. Gains are larger on under-specified trajectories, and full recovery adds value beyond isolated generic controls.

Q1

See W1 for RTE reliability and the blinded evidence audit.

Q2

See W2 for subgroup evaluation and generic repair controls.

Q3: Tables 2 and 13

We sincerely appreciate your careful and thorough reading of our paper. Table 13 (now Table 18)’s ablations used a 500-example RefAmb subset, whereas Table 2 used the full dataset. Their LJ/RTE values therefore reflect different evaluation sets. We will explicitly state the difference in Table 18.

The authors have addressed most of my concerns through additional experiments and clarifications. However, the attribution of IIGCR's improvements remains somewhat unclear, as the current experiments do not fully rule out the benefits of combining general-purpose trajectory repair strategies. Nevertheless, the rebuttal has substantially strengthened the paper, so I am inclined to raise my rating.

Reviewer_6qhtOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the problem of trajectory stagnation caused by referential under-specification in retrieval queries in multimodal Agentic RAG. The authors construct the diagnostic benchmark RefAmb and the process metric RTE (combining factual novelty and evidence utility), and analyze the evolutionary patterns of trajectories across different models through entity-clarification and obfuscation interventions; furthermore, they propose a training-free online repair method, IIGCR, which uses query repetition and low information increment signals to trigger local rewriting, visual grounding and cropping, as well as global reflection and context pruning. Experiments show that IIGCR can improve the trajectory efficiency and final answer quality of small- and medium-sized open-source models.

Critical strengths
  1. It achieves noticeable performance improvements on small- and medium-sized open-source models.
  2. The paper is easy to follow.
Critical weaknesses
  1. No gains on strong models: On GPT-5.1, the final answer quality LJ of IIGCR (51.13) shows a negligible improvement compared to ReAct (50.73) (tying or dropping on 3 out of 5 tasks), and is noticeably lower than Oracle Rewrite (51.60). In addition, the maximum number of retrieval turns in the experiments is set to only 5 turns, which seems somewhat contradictory to the claimed "long-horizon trajectory stagnation and stack-based context pruning."
  2. IIGCR is essentially a pipeline concatenation of multi-stage heuristic rules, with relatively limited algorithmic novelty. This method chains together several standard modules, cosine similarity thresholding for duplicate detection, prompt-guided query rewriting, Grounding DINO region cropping for image-to-image search, and deleting low-gain dialogue history, making the overall design lean toward rule-based engineering.
Questions for authors

See Weaknesses.

1 comment

We thank the reviewer for your meaningful insight and highlighting these concerns.

W1.1: Gains on strong models

We would like to clarify that Section 4.1 explicitly states that IIGCR maintains "comparable answer quality on GPT-5.1" (lines 427-428), rather than claiming a substantial LJ gain. Besides, Table 2 shows a significant RTE improvement from 24.12 to 25.85, with gains across all five tasks. Higher RTE reflects less redundant evidence acquisition and greater uptake in subsequent reasoning. As discussed in Sections 3.2/4.1 and Appendix E.3, stronger models can partly compensate for inefficient retrieval through parametric knowledge and visual understanding; final-answer quality can therefore mask inefficient trajectories. RTE and LJ provide complementary process and outcome evaluations.

Oracle Rewrite uses annotated entity information, providing an idealized upper-bound condition for referential definiteness that practical baselines cannot access accurately (Section 3.1). On the stronger Qwen3-VL-32B, IIGCR improves both RTE (33.12→35.66) and LJ (37.73→45.40). We will clarify these backbone-dependent benefits in the paper (line 427-436).

W1.2: Five-turn budget

As a supplementary, we additionally varied the maximum budget K over 3/5/8/10/15 using Qwen2.5-VL-7B on the same 1,500 RefAmb examples:

Table. Retrieval-budget scaling on task-balanced RefAmb (Qwen2.5-VL-7B; 1,500 examples).

KReAct LJIIGCR LJ
329.1331.07
529.1332.73
829.0733.53
1029.0733.40
1528.8033.53

Increasing the budget does not improve ReAct's answer quality, whereas IIGCR retains its advantage, including +4.33 and +4.73 LJ points at K=10, 15.Thus, broader turn budget does not substitute for trajectory repair. These results establish budget robustness, while the original ablation establishes pruning's benefit within five turns. We will distinguish larger allowed budgets from realized trajectory lengths and qualify the "long-horizon" wording accordingly (Caption of Fig.3), and strenghthen the analysis on long horizon study in the paper (line 451-482).

Besides, context pruning addresses the persistent influence of unproductive retrieval states on subsequent decisions; its necessity does not require trajectories to reach a fixed number of turns. Section 3.4/Figure 3 already show declining novelty and plateauing effective contributions within five turns. Besides, ablation in Table 18 supports this design: removing pruning reduces RTE from 36.05 to 34.61.

W2: Novelty beyond standard modules.

Our contribution is a diagnosis-driven recovery policy specifying when to intervene, at what level, and which historical states to retain or remove. Its design follows Section 3's findings that referential under-specification propagates through retrieval feedback and can persist beyond the original query defect. Therefore, the true contribution of this method lies in targeted measures proposed based on rigorous experimental observations, which in turn verify the reliability of the observed phenomena.

Each intervention addresses a diagnosed failure: (1) queries can change without introducing new evidence, motivating both pre-retrieval repetition and post-retrieval low-gain detection; (2) Oracle grounding does not recover all failures, motivating escalation from local action repair to global recovery; (3) unresolved visual clues require ROI anchoring when textual reformulation is insufficient; (4) reflection revises future plans, while targeted pruning removes feedback and thoughts sustaining failed plans. The modules are conditionally activated, rather than executed serially for every example (Section 4, Algorithms 1–3). We also demonstrated the synergistic effects of different components in Table 18.

We further added incremental ablation on RefAmb examples with Qwen2.5-VL-7B and the same five-turn budget:

Table. Matched-compute controller-composition ablation on task-balanced RefAmb.

ControllerF1LJRTETime(s)
None25.3125.67.31110.02
+Local repair30.5931.13.29411.71
+Global reflection30.2931.87.28011.43
+History-aware pruning30.3132.07.32411.34
+ Random pruning29.8830.53.31711.58
Full IIGCR31.0133.47.39011.19

Full IIGCR outperforms every isolated controller across F1/LJ/RTE, with only 1.17s additional latency over ReAct. History-aware pruning also outperforms random pruning, supporting selection of failed states rather than arbitrary history deletion. We add a more detailed analysis to the paper (line 1669-1690). Together, the diagnosis, query-level baseline, and composition controls support the coordinated recovery design beyond any individual standard tool. We will make this connection explicit in the paper(line 97-103).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.