acceptodds
Withdrawn from ICLR 2027

InterEvolve: Self-Evolving Agentic Search via Retrieval–Understanding Interaction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

State assessment in self-evolving agentic search poses significant challenges, as inaccurate evaluations can mislead the evolutionary trajectory and trap the system in redundant or mis-evolved loops. In this paper, we observe that humans mitigate such failures via retrieval-understanding interaction. During multi-turn search, they use prior understanding to form expectations about retrieval results and assess the consistency between results and expectations to resist misdirection from search outputs. Inspired by this behavior, we propose InterEvolve, an agentic search framework that improves the accuracy of state assessment without requiring additional model training or task-specific fine-tuning. Specifically, InterEvolve comprises two main components: the Retrieval–Understanding Progress Assessor (RUPA) and the Query Refinement Validator (QRV). RUPA leverages retrieval–understanding consistency to assess whether each search iteration progresses in a productive direction. Moreover, QRV constrains retrieval discrepancies between the original and refined queries, ensuring reliable query refinement and preventing misdirected evolution. Extensive experiments on the MMEB-v2 benchmark (78 datasets) across 4 popular retrieval backbones demonstrate that InterEvolve consistently outperforms the naive agentic search baseline in all model combinations. Notably, agentic search relying on weaker models like Qwen3.8-27B for state assessment suffers from misdirected evolution, degrading from 74.96 to 74.14 below single-turn retrieval. InterEvolve recovers this degradation and further improves it to 75.48, demonstrating its robustness safeguard. The code will be released if the paper is accepted.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agentic Search, multi-modal retrieval, LVLM

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZXR9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose InterEvolve, a training-free loop for multimodal retrieval in which an LVLM judges the current Top-1 and rewrites the query if it looks wrong. A rewrite is accepted only if it passes two checks: RUPA, which asks the LVLM to answer the question directly and requires the answer to become more similar to the new Top-1, and QRV, which limits how much the score distribution over the current Top-10 changes. If either check fails, the search stops. On MMEB-v2 the method improves four embedding models by 0.4 to 2.0 points on average, and beats a naive rewrite loop by 0.61 to 1.69 points on the Image subset.

Critical strengths
  • The paper shows a real problem clearly: a naive rewrite loop can do worse than single-pass retrieval, for example 74.14 vs. 74.96 with a weaker LVLM (Table 3). The fix is simple, needs no training, and reuses the existing retriever and LVLM, so it is easy to adopt.
  • The method section is precise, and the authors are honest that RUPA measures agreement with the LVLM rather than correctness.
Critical weaknesses
  • Some results don't match the algorithm as described. A failed check returns the Top-1 of the last accepted state, so if no rewrite is ever accepted, the output is just the initial Top-1. At δ_RUPA = 1, where almost no rewrite can pass, results should therefore be close to single-pass retrieval (74.96 for Qwen3-VL-2B). Yet in Fig. 3a the curves stay near their best values (about 76.7 for Kimi).
  • There are also a few number errors. In Fig. 3b, r_0 = 0.6311 and r_1 = 0.5731, so by Eq. 4 Δr_0 should be −0.058, but it is printed as 0.1311. In Table 4 (Qwen3-VL-8B), Ret goes from 78.97 to 80.62 but is marked +0.56, and GD goes from 92.92 to 93.22 but is marked +1.65.
  • It isn't clear where the gain comes from. The only agentic baseline is the naive loop, while RUPA asks a strong LVLM to answer the question directly, which may itself inject useful knowledge. Simple baselines with the same LVLM, such as retrieving with the LVLM's answer or reranking the Top-10, are missing. The cost is also not reported, even though Algorithm 1 can make up to 13 LVLM calls per query. Thresholds were chosen on the same benchmark used for evaluation, with single runs, and the full method is only 0.11 above RUPA alone in Table 2.
  • Reproducibility is also a concern: the LVLM prompts and the softmax temperature τ are not given, and there is no Reproducibility Statement, which ICLR strongly encourages.
Questions for authors

see weaknesses

Reviewer_ezWQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

InterEvolve is a training-free framework for iterative multimodal retrieval. RUPA checks improvements in answer–candidate consistency, while QRV constrains changes in retrieval distributions to control query refinement. Experiments on 78 MMEB-v2 datasets show aggregate improvements across four retrieval backbones.InterEvolve is a training-free framework for iterative multimodal retrieval. RUPA checks improvements in answer–candidate consistency, while QRV constrains changes in retrieval distributions to control query refinement.

Critical strengths
  1. The framework separates query proposal from acceptance and reuses frozen models, offering a practical way to control harmful rewrites without additional training.
  2. Evaluation covers multiple modalities and retrieval backbones, with results showing recovery from degradation caused by naive iterative rewriting.
Critical weaknesses
  1. Since both the query and expected answer change, higher answer–candidate agreement may reflect a consistent but incorrect interpretation of the original request.
  2. QRV only checks the current Top-10. It may miss a wrong result from outside this set becoming the new Top-1.
  3. The gains may partly come from conservative stopping. More evidence is needed to show whether the method makes better judgments or just avoids further bad rewrites.
  4. The paper does not report LVLM calls, token usage, or latency, so it is hard to judge whether the gains are worth the added cost.
  5. The full method beats RUPA alone by 0.11 and QRV alone by 0.25 . Without showing how much the results vary, this is not enough to conclude that both are necessary.
Questions for authors
  1. How often do the checks accept harmful rewrites or reject beneficial ones?
  2. How does InterEvolve compare with same-LVLM reranking or answer-guided retrieval under similar inference budgets?
  3. What temperature, prompts, and decoding settings were used? Were the thresholds selected on a separate validation set?
  4. Please clarify why the consistency gain in Fig 3b does not match the difference between its displayed scores. 0.6311-0.5713=?
  5. Some gains in Table 4 do not match the scores. For Qwen3-VL-8B, the listed scores imply Ret and GD gains of +1.65 and +0.30, not +0.56 and +1.65.
Reviewer_3Q7QOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

InterEvolve is a training-free framework for iterative multimodal retrieval. It addresses harmful query rewriting by separating rewrite generation from acceptance, using frozen retrieval and vision-language models.

Its main contribution is a joint acceptance rule: RUPA requires increased consistency between the retrieved top result and a model-generated expected answer, while QRV constrains changes in retrieval-score distributions over the current top-10 candidates. If either check fails, the system stops and retains the previous result. “Self-evolution” therefore refers to query refinement within a task, rather than model learning.

Validation uses MMEB-v2’s 78 datasets spanning images, videos, and visual documents, with four retrieval backbones. Full-benchmark gains range from 0.4–2.0 percentage points. Image-only ablations examine both components, thresholds, and three understanding models; notably, the method reverses degradation caused by naive rewriting with a weaker model.

The results support improved aggregate retrieval performance, but do not establish broader agent capabilities or directly validate state-assessment accuracy. Missing uncertainty estimates and inference-cost measurements, alongside numerical inconsistencies, limit the strength of the conclusions.

Critical strengths
  • A concrete, training-free mechanism for controlling harmful query refinement. InterEvolve separates proposing a rewrite from accepting it, combining an answer–retrieval consistency check with a local retrieval-distribution constraint. This addresses a consequential failure mode: additional search iterations can reduce accuracy. Its use of frozen models makes the approach practical to incorporate into existing retrieval systems.
  • Consistent aggregate improvements, supported by targeted ablations. Across four retrieval backbones, InterEvolve improves full MMEB-v2 averages by 0.4–2.0 percentage points. Particularly compelling is the weak-understanding-model setting: naive iteration reduces image performance from 74.96 to 74.14, while InterEvolve raises it to 75.48. Together with improvements from each component individually, this provides substantive evidence that the proposed controls help mitigate harmful rewriting across multiple configurations.
Critical weaknesses
  • The experiments do not isolate the claimed improvement in state assessment. Results measure final retrieval accuracy, without evaluating whether RUPA/QRV reliably distinguish beneficial from harmful rewrites or comparing against matched-budget, simpler stopping rules. Consequently, the gains could reflect more conservative termination rather than better state assessment. The experiments do not directly establish whether state assessment becomes more accurate, nor do they rule out the possibility that the gains come from more conservative stopping. Moreover, combining both modules improves over RUPA alone by only 0.11 percentage points, without uncertainty estimates, providing limited evidence of meaningful complementarity between the two components.
Questions for authors
  • Clarification on baselines: How does InterEvolve compare with fixed-round rewriting and simpler stopping policies under matched inference budgets? Can you provide evidence that its gains arise from better rewrite selection rather than earlier stopping or additional computation?
  • Clarification on the claimed mechanism: On a shared set of proposed rewrites, how reliably do RUPA and QRV reject harmful transitions (correct → incorrect) while retaining beneficial ones (incorrect → correct)? Do you have measurements supporting improved state assessment beyond final retrieval accuracy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.