acceptodds
Under review as a conference paper at ICLR 2027

Beyond Relevance: Broad-to-Deep Contextual Evidence Reasoning for Retrieval-Augmented Medical Question Answering

Abstract

Medical question answering increasingly relies on retrieval-augmented generation to provide large language models with external evidence, yet most existing methods still treat evidence construction as a largely static relevance-ranking problem. Inspired by how people often approach unfamiliar problems, first surveying a broad range of potentially useful references, then organizing the most informative evidence, and finally following concepts that remain unclear, we formulate evidence acquisition as a broad-to-deep process. Towards this end, we propose a novel approach named Broad-to-deep Contextual Evidence Reasoning (BRACE) for medical question answering with two complementary stages. The broad stage constructs a compact evidence context through determinantal resolution packing, where candidate documents are assessed by both their relevance to the question and the uncertainty they resolve, while the determinantal objective preserves complementary information across the selected set. Building on this context, the deep stage performs progressive uncertainty resolution by iteratively localizing high-uncertainty concepts within the evolving question and evidence context and converting them into targeted retrieval cues. Newly acquired evidence reshapes the context and may expose more specific information needs, allowing retrieval to progressively move from broad semantic coverage toward deeper supporting knowledge. In this way, BRACE lets the evolving context determine both which evidence should be retained and where subsequent retrieval should proceed. Experiments on five medical question-answering benchmarks with three open-source readers consistently demonstrate the effectiveness of our framework over strong retrieval, generation, and hybrid knowledge-augmentation baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical Question Answering, Evidence Sufficiency, Uncertainty-Guided Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_BBVsOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the "relevance ≠ utility" problem in retrieval-augmented medical question answering by formulating evidence construction as a broad-to-deep process (BRACE):

  • Broad stage — Determinantal Resolution Packing (DRP): selects a compact evidence set from a pool of retrieved and LLM-generated documents using a determinantal point process (DPP). The quality term combines cross-encoder relevance with the document's uncertainty-resolution gain (U(∅)−U(D), where U is the reader's conditional token NLL over the question). The determinantal term preserves complementarity. The authors prove that a shifted version of the εI-regularized objective is monotone submodular, yielding a (1−1/e) greedy guarantee.
  • Deep stage — Progressive Uncertainty Resolution (PUR): computes the reader's residual token-level uncertainty over the current evidence context, takes tokens above the top-ρ quantile, and asks an auxiliary LLM selector to organize them into retrieval queries; the process runs for T=3 rounds, progressively replacing evidence.
Critical strengths

1,The per-document flipping experiment directly quantifies that relevance does not track usefulness: 11.3% of top-10 documents flip correct answers to wrong, while 9.6% of rank-21–30 documents flip wrong answers to correct, with no significant difference across rank buckets (paired t-test, p=0.5584). This is a compelling, evidence-backed motivation rather than a purely intuitive argument. 2,The quality×diversity DPP decomposition (Eq.6) is elegant, and the uncertainty-resolution quality term is a natural extension of standard DPP retrieval. The (1−1/e) guarantee is essentially a standard instantiation of Nemhauser (1978) on a regularized DPP (limited theoretical novelty), but the proof is correct (the matrix-determinant-lemma derivation checks out), and the guarantee, regularization coefficient, and algorithm are consistently tied together. 3,15 comparisons (5 benchmarks × 3 readers) with BRACE leading in the vast majority, averaging +4.03pp over the strongest baseline (I verified: (3.75+4.09+4.24)/3 = 4.03 ✓). The method is training-free and reader-agnostic; ablations (R-DRP / U-DRP / w/o PUR) support each component; reproducibility details (seeds, temperatures, ε_dpp, normalization stabilizers) are complete.

Critical weaknesses

1,PUR's core idea—using the model's low-confidence tokens to formulate retrieval queries—is highly similar to FLARE (Jiang et al., 2023) and related to adaptive/active retrieval lines such as Adaptive-RAG and DeepRAG, none of which are cited or compared. PUR differs in that it uses the reader's residual NLL over evidence+question rather than generation-side confidence, and runs a fixed number of offline iterations, but these distinctions must be made explicit (ideally with empirical comparison) for the novelty claim to hold. This is the most likely point of attack by other reviewers. 2,U(D) is the NLL of question tokens conditioned on evidence, not uncertainty over the answer. Question NLL largely reflects linguistic/surface familiarity; its link to knowledge gaps rests only on the correlation plot in Figure 2 (right), which shows a single fitted line on two datasets without confidence intervals or cross-reader validation. Suggested additions: comparison with answer entropy/self-consistency, and a causal ablation (e.g., queries built from randomly selected tokens). 3,"Training-free" is a selling point, but the cost shifts to inference: ~30 single-document forward passes per question (for U({d_i})), token-level logprob computation over the full context each round, and one selector API call per round (T=3). No latency/FLOPs/token-cost comparison against MedRGAG or MedGENIE is reported; the conclusion only says it "incurs additional computation." Numbers are needed to judge practicality. 4,Only the number of rounds T is analyzed; there is no ablation for λ (uncertainty scaling), β (the cap on the uncertainty contribution), or the ρ=10% quantile. With β=0.1, uncertainty contributes at most 0.1 relative to relevance r_i∈[0,1]—uncertainty is a deliberately weak signal, a design choice that needs empirical support.

Questions for authors

see Weaknesses

Reviewer_JPCvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines evidence selection for retrieval-augmented medical question answering. Its starting point is that highly ranked documents do not always help the reader answer correctly. BRACE addresses this with two stages: initial evidence construction and uncertainty-guided refinement.

In the broad stage, Determinantal Resolution Packing scores retrieved and generated passages using relevance and reader-conditioned uncertainty. A determinantal objective favors complementary passages, and a selector retains a smaller context. In the deep stage, Progressive Uncertainty Resolution uses uncertain spans from the current context to build follow-up retrieval queries. The context is updated with the new evidence after each round. The paper also proves a greedy approximation guarantee for the shifted DRP objective.

The authors evaluate BRACE on five medical QA benchmarks with three open-source readers. In Table 1, it has the highest average accuracy for each reader, with a mean gain of 4.03 percentage points over MedRGAG.

Critical strengths
  1. The motivation is supported by a concrete analysis of evidence utility.

Appendix B goes beyond the usual assumption that higher-ranked documents are more useful. Table 2 shows that 11.3% of documents ranked 1–10 change a previously correct answer into an incorrect one, while 9.6% of documents ranked 21–30 correct a previously wrong answer. These results motivate the use of reader-conditioned uncertainty in DRP. DRP combines relevance, uncertainty reduction, and complementarity when selecting the initial context. PUR seeks additional evidence where uncertainty remains after the reader has seen that context. Figure 3 also provides an example of later retrieval correcting a misleading initial context.

  1. BRACE improves mean accuracy across all three reader models.

BRACE has the highest average accuracy for Qwen2.5-7B, Llama-3.1-8B, and Ministral-8B (Table 1). Its gains over MedRGAG range from 3.75 to 4.24 percentage points. Figure 2 also compares the full method with relevance-only and uncertainty-only DRP variants. Both R-DRP and U-DRP score below the full method in the reported ablations. Theorem 3.1 gives a (1−1/e) approximation guarantee for greedy selection under the shifted DRP objective. That guarantee concerns the selection objective, rather than final QA accuracy.

Critical weaknesses
  1. Lack of methodological novelty and end-to-end learning mechanisms

The proposed BRACE framework is largely an engineering-heavy, training-free pipeline that combines off-the-shelf components (such as BM25, fixed LLM readers, and a black-box LLM selector) with handcrafted heuristic scoring functions (e.g., manually tuned DPP weights and token-uncertainty thresholds). It lacks a principled, end-to-end learning mechanism to jointly optimize evidence selection and multi-turn reasoning. Relying purely on prompting, heuristic score aggregation, and multi-step pipeline engineering without learnable modules significantly limits the algorithmic depth and novelty of the work. This makes the contribution more suited to an application-oriented or medical informatics venue rather than ICLR, which typically values deep methodological innovations and learning-based architectures.

  1. The PUR ablation does not isolate the benefit of uncertainty-guided queries.

The w/o PUR setting in Section 4.3 removes iterative refinement entirely. The full system retrieves five additional passages per round for up to three rounds (Section 4.1). The selector can also revise the context after each round (Section 3.4). Figure 2 therefore compares systems with different opportunities to retrieve and reconsider evidence. The accuracy gap suggests that refinement helps, but leaves the role of the uncertainty-based query unclear. Ordinary iterative retrieval might produce part of the same gain. A stronger control would keep the rounds, passage count, and selector updates fixed while replacing the uncertainty cues with the original question or standard query expansion. Without that comparison, I cannot attribute the improvement specifically to PUR's uncertainty signal.

  1. The main experiments omit a closely related iterative retrieval method.

Section 2 discusses i-MedRAG, but it is not included in Table 1. This matters because i-MedRAG also retrieves evidence in follow-up steps. PUR builds its queries from high-uncertainty spans in the current context, while i-MedRAG uses follow-up questions. The experiments do not directly test the advantage of that difference. Figure 3 and Appendix D give useful examples in which refinement corrects a wrong answer. Those examples cannot tell us whether PUR would outperform another iterative strategy. A direct comparison with i-MedRAG under matched experimental settings is still needed to assess PUR's contribution. If that comparison is impractical, the authors should explain why.

Questions for authors
  1. Does PUR outperform ordinary iterative retrieval with the same budget?

Could the authors compare PUR with a baseline that retrieves five passages per round for up to three rounds, using the original question or standard query expansion? The reader and selector should remain the same. Please report whether the gain persists when retrieval rounds, passage counts, and selector updates are matched.

  1. How stable are the smaller reported gains?

Appendix C.3 gives a random seed and nonzero temperatures for generation and selection, but Table 1 reports no run-to-run variation. On PubMedQA* with Qwen2.5-7B, BRACE scores 49.40% and MedGENIE scores 49.20%. Were these results obtained from a single run? Could the authors report variation across runs or confidence intervals, especially for comparisons with such small differences?

Reviewer_eDxiOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BRACE, a training-free framework for medical question answering built on a MedRGAG-style retrieval, generation, and evidence-selection pipeline. It first uses Determinantal Resolution Packing (DRP) to select complementary documents based on query relevance and reader-conditioned question likelihood. It then applies Progressive Uncertainty Resolution (PUR), which extracts high-surprisal spans from the current context to expand retrieval queries and iteratively update the evidence through an LLM selector. I understand its main contribution as using reader-derived signals to connect initial evidence selection with subsequent retrieval. This is a reasonable extension of existing evidence-construction methods, but the methodological advance appears incremental: the underlying retrieval–generation pipeline, DPP-based selection, and iterative retrieval are established techniques. The distinctive value therefore depends on whether the proposed uncertainty signal provides a meaningful advantage over simpler selection and query-expansion strategies. Experiments on five medical QA benchmarks with three readers report improved average accuracy, and the ablations provide evidence that the proposed components are useful within the full pipeline. However, these results establish the effectiveness of the combined system more clearly than the specific advantages of its mechanisms. Direct reranking and budget-matched iterative retrieval comparisons are missing, while the candidate-budget alignment with the reimplemented MedRGAG requires clarification. Thus, the empirical results are promising, but the distinct contribution and the proposed uncertainty-resolution explanation remain insufficiently validated.

Critical strengths

1.The paper addresses a meaningful limitation of relevance-based evidence selection. Query–document relevance does not necessarily capture a document’s usefulness to a particular reader, and independently ranked documents may provide redundant information. Combining reader-derived signals with set-level complementarity is a reasonable approach to this problem. The preliminary document-utility analysis provides useful motivation.

2.The reported improvements extend across multiple datasets and reader models. BRACE achieves higher average accuracy than the compared baselines with all three readers, and the component ablations provide evidence that both initial evidence composition and subsequent refinement contribute within the proposed pipeline. This breadth makes the empirical results worth investigating further.

Critical weaknesses

1.The candidate-pool budget is not clearly controlled across methods. BRACE uses 15 retrieved and 15 generated documents, whereas the original MedRGAG uses five of each. Although the paper aligns auxiliary-model configurations, it does not clarify whether the reimplemented MedRGAG uses the same candidate budget or identical candidate documents. Improved evidence coverage therefore remains a possible confound, rather than a confirmed experimental mismatch.

2.Additional retrieval and selector computation are not adequately controlled. PUR introduces extra retrieval opportunities and repeated selector calls. Using the same selector model and final context size does not equalize these resources. Comparisons under matched retrieval, selector-call, or token budgets are needed to determine whether the gains arise from the proposed refinement strategy or additional computation.

3.The ablations do not isolate the contribution of the determinantal objective. R-DRP and U-DRP both retain DPP, while removing DRP changes both scoring and subset selection. A direct top-k baseline using the same combined quality scores would more clearly establish whether determinantal selection provides benefits beyond the proposed document-quality scoring.

4.Comparisons with alternative iterative retrieval and query-expansion strategies are missing. Self-RAG and i-MedRAG are discussed but not evaluated. Moreover, the paper does not compare uncertainty-guided expansion against alternatives such as entity-based expansion or LLM-generated follow-up queries under matched budgets. Consequently, the specific advantage of PUR over other ways of obtaining additional evidence remains unclear.

5.The uncertainty score is not sufficiently validated as a measure of missing knowledge. Question-token negative log-likelihood measures textual predictability, which need not imply answer correctness or factual reliability. Likewise, high-surprisal spans may reflect rare terminology rather than actionable knowledge gaps. The heatmap and ablations provide indirect support, but stronger validation is needed to justify interpreting these signals as knowledge deficits and their reduction as resolution.

6.The refinement analysis does not establish reliable progress across rounds. Figure 2 shows fluctuating PubMedQA results and occasional degradation in later rounds. This does not invalidate PUR, but weakens the evidence for progressive resolution of information needs. Per-round uncertainty–accuracy relationships, correct-to-incorrect transitions, and repeated-run variability would clarify when refinement helps or harms. The DRP approximation guarantee applies to its selection objective and does not establish these behavioral properties.

Questions for authors
  1. Are PUR's uncertainty scores actually conditioned on the question?
    Under X=(Dt,Q)X=(D_t,Q) in Eq. (3), document-token probabilities cannot depend on the subsequent question or later documents. Why are these scores interpreted as question-specific residual uncertainty after reading the evidence? What input ordering is used in the implementation?

  2. Can lower NLL distinguish useful evidence from question repetition?
    Would a document that merely repeats the question, without providing answer-relevant knowledge, also reduce U([di])U([d_i])? How often does NLL decrease while the answer changes from correct to incorrect?

  3. Does the uncertainty ablation confound informative scoring with a positive offset?
    The full method adds approximately 0.06070.0607–0.10.1 to rir_i, whereas R-DRP removes this term entirely. Does the uncertainty signal still improve performance compared with qi=ri+cq_i=r_i+c or randomly permuted uncertainty scores?

  4. Does DRP outperform direct ranking with identical quality scores?
    With candidate documents, qiq_i, selection size, and the downstream selector held fixed, what is the accuracy difference between DRP and selecting the top-kk documents by qiq_i?

  5. What candidate budget was used for the reimplemented MedRGAG?
    Did it use the original 5+55+5 documents or BRACE's 15+1515+15? Did both methods receive exactly the same retrieved and generated documents for each question?

  6. Do PUR's gains come from uncertainty guidance rather than additional retrieval and selection?
    With refinement rounds, new documents per round, and selector calls held fixed, how does PUR compare with replacing high-NLL spans with medical entities or LLM-generated follow-up queries?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.