- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of evidence selection in multimodal agentic retrieval. Existing retrieval control methods can help agents discover missing information through query refinement or filtering, but they may also exclude or replace previously useful evidence, leading to incomplete support for downstream answering. The authors aim to improve evidence completeness and reduce evidence loss while allowing agents to perform flexible retrieval operations.
To achieve this, the authors propose EGAL-SC, a framework that separates retrieval control from evidence admission. It first performs both unrestricted and controlled retrieval to obtain candidate documents. A state-conditioned admission mechanism then selects evidence based on four signals: query relevance, alignment with accumulated evidence, novelty, and a learned document-error score. The framework also incorporates evidence sufficiency verification and iterative retrieval refinement to decide whether to answer, continue searching, or stop.
The authors evaluate EGAL-SC on two multimodal QA datasets, MultiModalQA and WebQA, using both evidence-selection experiments and end-to-end agent evaluations. The evidence-selection experiments compare against MMR and a no-risk variant, while the end-to-end evaluation compares seven system configurations, including six baselines. Results show that EGAL-SC improves evidence completeness and retention over MMR, and achieves higher overall answer F1 (21.93% vs. 19.52%) than Agentic Hybrid. However, the improvements come with higher computational costs, and the overall F1 gain is primarily driven by increased answer coverage rather than better quality on jointly answered questions.
Critical strengths
-
The paper provides a useful analysis of performance, answer coverage, and computational cost. I especially appreciate the effort to understand where the performance gains come from, rather than only reporting the overall F1.
-
Twin retrieval is an interesting idea. Keeping both unrestricted and controlled retrieval results provides more options for evidence selection and may help avoid losing useful evidence.
-
The paper evaluates evidence quality beyond final answer accuracy. It includes metrics such as complete evidence, support loss, and retained support units. These metrics are helpful for understanding how well the retrieval process preserves useful information.
-
The experimental results show improvements in evidence completeness and overall answer F1 over the baselines, although the gains come with higher computational costs.
Critical weaknesses
-
The technical novelty seems somewhat limited. The main ideas, including twin retrieval, state-conditioned evidence scoring, and iterative verification, are relatively straightforward and closely related to existing retrieval and agentic RAG approaches. The scoring function is also a simple weighted combination of existing signals. While integrating these components is useful, the paper could better clarify what is technically new.
-
The comparison with strong baselines is insufficient. Although the paper evaluates seven systems, several baselines are relatively simple. It would be helpful to include stronger methods from recent work, especially those focusing on agentic retrieval and evidence selection. Also, the implementation of Agentic Hybrid, the main baseline, is not clearly described.
-
The evaluation of individual components is insufficient. The main evidence-selection experiment uses Full–Rank without the admission threshold, rather than the actual admission mechanism used in EGAL-SC. There is also no detailed ablation of the relevance, alignment, and novelty terms. As a result, it is unclear how much each component contributes to the overall improvements.
-
The presentation could be improved. Some important experimental results, such as the detailed seven-system comparison, are only provided in the appendix. Also, the notation and motivation behind Equations 3 and 4 are not very intuitive. In particular, the distinction between alignment, gap, and novelty could be explained more clearly.
Questions for authors
-
How were the weights in Equation 4 chosen? Were they set heuristically or tuned on a development set? How sensitive are the results to these weights?
-
Could the authors clarify how Agentic Hybrid is implemented? What are the main differences between Agentic Hybrid and EGAL-SC, especially in retrieval, evidence selection, and stopping?
-
The evidence-selection experiments use Full–Rank without the admission threshold of 0.65. How was this threshold chosen? Why doesn't Table 1 include the actual threshold-based admission used in EGAL-SC?
-
Could the authors provide a more detailed ablation of the relevance, alignment, and novelty terms? In particular, how much does alignment help beyond novelty, since both are based on similarity to existing evidence?
Thank you for the detailed response and additional experiments. I appreciate the new History–MMR baseline, component ablations, and threshold analysis, which address several of my concerns. However, the technical novelty remains limited. Also, the new results show that the 0.65 admission threshold reduces evidence completeness and retention compared with Full–Rank, while its benefit to downstream F1 remains unestablished.
Overall, I appreciate the improvements, but my main concerns remain. I will keep my rating unchanged at 2.