acceptodds
Under review as a conference paper at ICLR 2027

EGAL-SC: State-Conditioned Evidence Admission for Multimodal Retrieval Agents

Abstract

Retrieval controls can recover missing evidence while excluding support needed to answer the same question. Effective retrieval therefore requires deciding which candidates should enter the answering context. We introduce EGAL-SC, a multimodal retrieval framework that pairs controlled and unrestricted searches with a state-conditioned evidence admission rule. The rule combines relevance, alignment with accumulated evidence, novelty, and a learned document-error signal. On 562 held-out first-control states from MultiModalQA and WebQA, threshold-free ranking of the same candidate pools under a common five-document cap yields 272 complete-evidence outcomes, compared with 239 for maximal marginal relevance, and retains 49 additional support units. The complete-evidence difference passes a four-test Holm correction; the smaller difference from removing the error signal does not. In a seven-system comparison on two 500-question panels, EGAL-SC obtains 21.933% all-question F1 versus 19.520% for Agentic Hybrid, with broader answer coverage and higher computation. On jointly answered questions, its F1 is lower. These results support state-conditioned evidence ranking and show that system-level answer gains must be assessed together with answer selection and cost.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal retrieval-augmented generation, Agentic retrieval, Evidence admission, Logical retrieval, Evidence retention, Selective question answering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nFu7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the problem of evidence selection in multimodal agentic retrieval. Existing retrieval control methods can help agents discover missing information through query refinement or filtering, but they may also exclude or replace previously useful evidence, leading to incomplete support for downstream answering. The authors aim to improve evidence completeness and reduce evidence loss while allowing agents to perform flexible retrieval operations.

To achieve this, the authors propose EGAL-SC, a framework that separates retrieval control from evidence admission. It first performs both unrestricted and controlled retrieval to obtain candidate documents. A state-conditioned admission mechanism then selects evidence based on four signals: query relevance, alignment with accumulated evidence, novelty, and a learned document-error score. The framework also incorporates evidence sufficiency verification and iterative retrieval refinement to decide whether to answer, continue searching, or stop.

The authors evaluate EGAL-SC on two multimodal QA datasets, MultiModalQA and WebQA, using both evidence-selection experiments and end-to-end agent evaluations. The evidence-selection experiments compare against MMR and a no-risk variant, while the end-to-end evaluation compares seven system configurations, including six baselines. Results show that EGAL-SC improves evidence completeness and retention over MMR, and achieves higher overall answer F1 (21.93% vs. 19.52%) than Agentic Hybrid. However, the improvements come with higher computational costs, and the overall F1 gain is primarily driven by increased answer coverage rather than better quality on jointly answered questions.

Critical strengths
  1. The paper provides a useful analysis of performance, answer coverage, and computational cost. I especially appreciate the effort to understand where the performance gains come from, rather than only reporting the overall F1.

  2. Twin retrieval is an interesting idea. Keeping both unrestricted and controlled retrieval results provides more options for evidence selection and may help avoid losing useful evidence.

  3. The paper evaluates evidence quality beyond final answer accuracy. It includes metrics such as complete evidence, support loss, and retained support units. These metrics are helpful for understanding how well the retrieval process preserves useful information.

  4. The experimental results show improvements in evidence completeness and overall answer F1 over the baselines, although the gains come with higher computational costs.

Critical weaknesses
  1. The technical novelty seems somewhat limited. The main ideas, including twin retrieval, state-conditioned evidence scoring, and iterative verification, are relatively straightforward and closely related to existing retrieval and agentic RAG approaches. The scoring function is also a simple weighted combination of existing signals. While integrating these components is useful, the paper could better clarify what is technically new.

  2. The comparison with strong baselines is insufficient. Although the paper evaluates seven systems, several baselines are relatively simple. It would be helpful to include stronger methods from recent work, especially those focusing on agentic retrieval and evidence selection. Also, the implementation of Agentic Hybrid, the main baseline, is not clearly described.

  3. The evaluation of individual components is insufficient. The main evidence-selection experiment uses Full–Rank without the admission threshold, rather than the actual admission mechanism used in EGAL-SC. There is also no detailed ablation of the relevance, alignment, and novelty terms. As a result, it is unclear how much each component contributes to the overall improvements.

  4. The presentation could be improved. Some important experimental results, such as the detailed seven-system comparison, are only provided in the appendix. Also, the notation and motivation behind Equations 3 and 4 are not very intuitive. In particular, the distinction between alignment, gap, and novelty could be explained more clearly.

Questions for authors
  1. How were the weights in Equation 4 chosen? Were they set heuristically or tuned on a development set? How sensitive are the results to these weights?

  2. Could the authors clarify how Agentic Hybrid is implemented? What are the main differences between Agentic Hybrid and EGAL-SC, especially in retrieval, evidence selection, and stopping?

  3. The evidence-selection experiments use Full–Rank without the admission threshold of 0.65. How was this threshold chosen? Why doesn't Table 1 include the actual threshold-based admission used in EGAL-SC?

  4. Could the authors provide a more detailed ablation of the relevance, alignment, and novelty terms? In particular, how much does alignment help beyond novelty, since both are based on similarity to existing evidence?

1 comment

Thank you for the detailed response and additional experiments. I appreciate the new History–MMR baseline, component ablations, and threshold analysis, which address several of my concerns. However, the technical novelty remains limited. Also, the new results show that the 0.65 admission threshold reduces evidence completeness and retention compared with Full–Rank, while its benefit to downstream F1 remains unestablished.

Overall, I appreciate the improvements, but my main concerns remain. I will keep my rating unchanged at 2.

Reviewer_3VAmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EGAL-SC pairs unrestricted and controlled retrieval, then selects evidence using query relevance, alignment with accumulated evidence, novelty, and a learned document-error score. On 562 held-out states, its threshold-free ranking improves complete-evidence coverage over MMR. Full-system experiments show higher overall F1 and answer coverage, alongside higher cost and lower F1 on jointly answered questions.

Critical strengths
  1. The fixed-state comparison controls candidate pools, histories, rendering, and document budgets. Full–Rank completes 33 more states than tuned MMR, with the difference surviving Holm correction. This provides useful evidence for the proposed ranking score.
  2. The paper carefully separates evidence completeness, answer coverage, and answer quality. Reporting the decline on jointly answered questions and the increased computation makes the system-level gains easier to interpret.
Critical weaknesses
  1. The main experiment does not establish which parts of the admission design are needed. Full–Rank removes the admission threshold, while its MMR baseline uses within-call redundancy rather than accumulated evidence. A simple baseline using the same cross-turn memory and an alignment ablation would help identify the contribution. The risk signal’s incremental gain does not survive Holm correction, and the full-system comparisons do not isolate admission from stopping and repair.
  2. The central evidence measurements are difficult to audit. Experiments rely on private model configurations and laboratory-maintained support annotations, while the supplement provides aggregates rather than question-level evidence mappings or predictions. The agreement study evaluates output correctness, not whether reference support is actually visible in the selected context. This leaves uncertainty about the measurements underlying the main result.
Questions for authors
  1. How does Full–Rank compare with a relevance–redundancy baseline using the same accumulated memory, and with a version without the alignment term?
  2. What are the completeness and support-loss results with the actual 0.65 admission threshold? How many additional system answers come from normal verification versus draft fallback?
  3. Can you provide anonymized support-to-context mappings and agreement measurements for those annotations? Does the main result hold after accounting for shared sources?
Reviewer_gvDMOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents EGAL-SC, a multimodal retrieval framework designed to improve evidence selection in retrieval-augmented agents. The main idea is to separate retrieval control from evidence admission, allowing the agent to consider both unrestricted and controlled search results before deciding which documents should enter the answering context. The proposed admission mechanism combines query relevance, alignment with previously accumulated evidence, novelty, and a learned document-error score. The framework also includes evidence verification, retrieval repair, and stopping decisions. Authors evaluate the method on MultiModalQA and WebQA. A held-out analysis involving 562 retrieval states shows improvements in complete-evidence coverage over MMR, while a separate seven-system comparison reports higher overall answer F1 and coverage than Agentic Hybrid. However, the improvements come with additional computational cost, and performance on jointly answered questions is lower. Overall, the paper explores evidence-aware retrieval control, with the main contribution being the integration and evaluation of several existing retrieval and selection signals.

Critical strengths
  1. I agree with the authors that retrieving additional documents does not necessarily improve the evidence available for answering. A controlled search may introduce useful information while excluding previously retrieved support. Separating retrieval from evidence admission is therefore a reasonable design choice.

  2. One aspect I appreciate is the use of matched plain and controlled retrieval requests in Section 4.1. This allows the system to consider alternative candidates generated under different retrieval constraints rather than relying exclusively on a potentially restrictive search action. The design is straightforward, but it provides a sensible way to reduce the dependence on a single retrieval decision.

  3. Interestingly, the paper does not simply present every result as an improvement. It acknowledges that the contribution of the learned document-error signal is not significant after multiple-testing correction and that EGAL-SC performs worse on jointly answered questions despite having higher overall F1.

Critical weaknesses
  1. My main concern is that the core admission mechanism largely combines familiar retrieval signals using a fixed weighted score. Equation (4) integrates relevance, alignment, novelty, and a learned error estimate, but these ingredients are closely related to existing reranking, diversity-aware retrieval, and relevance-feedback techniques. The weights are also manually specified rather than learned jointly. While the combination is reasonable, I struggle to identify a sufficiently distinct methodological contribution beyond applying established ideas to a multimodal agent setting.

  2. Another issue is that the proposed selector and the MMR baseline do not operate with exactly the same information. As described in Appendix C, MMR considers redundancy among documents selected within the current call, whereas EGAL-SC uses accumulated cross-turn evidence memory. Consequently, the performance improvement may partly come from access to historical information rather than the proposed scoring mechanism itself. A history-aware MMR baseline, or a simpler reranker using the same evidence memory, would be important for establishing whether the new admission rule offers a meaningful advantage.

  3. I am also concerned about the relationship between the evaluation and the actual admission mechanism. The statistically significant comparison in Table 2 is obtained using Full–Rank, which explicitly removes the 0.65 admission threshold. However, the deployed EGAL-SC system uses this threshold to decide which documents are admitted. Furthermore, Section 3 notes that the full five-document list may be replaced, so there is no guaranteed preservation of previously selected evidence. The fallback procedure can also return drafts without passing the normal sufficiency or citation checks. Although the authors define safety narrowly in terms of support loss, the results do not isolate whether the proposed admission gate itself delivers the intended protection.

  4. The complete-system evaluation raises another important question. EGAL-SC improves all-question F1 from 19.520% to 21.933% compared with Agentic Hybrid, but the gain largely comes from answering more questions. On the 307 jointly answered questions, EGAL-SC actually performs worse, with 55.863% F1 compared with 58.697%. At the same time, mean latency increases from 365.42 to 485.62 seconds, and token consumption rises from 15,835 to 22,011 per question. Since the complete-system comparison does not isolate retrieval, admission, verification, and stopping, it is unclear how much of the improvement should be attributed to the proposed selector.

  5. Finally, the experimental validation relies heavily on private laboratory configurations, including the Qwen2.5-family backbone, custom feature interfaces, corpus resources, and reference annotations. The authors acknowledge that full execution requires access to these resources. The held-out ranking analysis is also restricted to 562 usable first-control states reached by the existing controller, rather than a broader distribution of retrieval states. Although the paper provides substantial implementation details, it remains difficult to independently reproduce the complete framework or determine whether the reported improvements generalize to other models, retrieval systems, and datasets.

Questions for authors
  1. Could the authors clarify what they consider to be the main methodological novelty beyond combining existing relevance, diversity, and error-prediction signals? In particular, what would be lost if the proposed weighted admission rule were replaced by a simpler history-aware reranking method?

  2. Since the main ranking comparison uses the threshold-free Full–Rank variant, could the authors provide evidence that the actual admission threshold of 0.65 improves support preservation or downstream answering? How sensitive are the results to this threshold?

  3. Regarding the learned error signal, could the authors provide its standalone evaluation performance on question-disjoint data and explain how its scores are calibrated? Given the relatively small and statistically inconclusive incremental improvement, I would like to understand whether this component is necessary for the proposed framework.

  4. Can the authors clarify how much of the end-to-end F1 improvement comes specifically from evidence admission rather than differences in answer coverage, verification, or stopping behavior? A controlled comparison that changes only the admission module within the complete system would materially strengthen the main claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.