- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses the "relevance ≠ utility" problem in retrieval-augmented medical question answering by formulating evidence construction as a broad-to-deep process (BRACE):
- Broad stage — Determinantal Resolution Packing (DRP): selects a compact evidence set from a pool of retrieved and LLM-generated documents using a determinantal point process (DPP). The quality term combines cross-encoder relevance with the document's uncertainty-resolution gain (U(∅)−U(D), where U is the reader's conditional token NLL over the question). The determinantal term preserves complementarity. The authors prove that a shifted version of the εI-regularized objective is monotone submodular, yielding a (1−1/e) greedy guarantee.
- Deep stage — Progressive Uncertainty Resolution (PUR): computes the reader's residual token-level uncertainty over the current evidence context, takes tokens above the top-ρ quantile, and asks an auxiliary LLM selector to organize them into retrieval queries; the process runs for T=3 rounds, progressively replacing evidence.
Critical strengths
1,The per-document flipping experiment directly quantifies that relevance does not track usefulness: 11.3% of top-10 documents flip correct answers to wrong, while 9.6% of rank-21–30 documents flip wrong answers to correct, with no significant difference across rank buckets (paired t-test, p=0.5584). This is a compelling, evidence-backed motivation rather than a purely intuitive argument. 2,The quality×diversity DPP decomposition (Eq.6) is elegant, and the uncertainty-resolution quality term is a natural extension of standard DPP retrieval. The (1−1/e) guarantee is essentially a standard instantiation of Nemhauser (1978) on a regularized DPP (limited theoretical novelty), but the proof is correct (the matrix-determinant-lemma derivation checks out), and the guarantee, regularization coefficient, and algorithm are consistently tied together. 3,15 comparisons (5 benchmarks × 3 readers) with BRACE leading in the vast majority, averaging +4.03pp over the strongest baseline (I verified: (3.75+4.09+4.24)/3 = 4.03 ✓). The method is training-free and reader-agnostic; ablations (R-DRP / U-DRP / w/o PUR) support each component; reproducibility details (seeds, temperatures, ε_dpp, normalization stabilizers) are complete.
Critical weaknesses
1,PUR's core idea—using the model's low-confidence tokens to formulate retrieval queries—is highly similar to FLARE (Jiang et al., 2023) and related to adaptive/active retrieval lines such as Adaptive-RAG and DeepRAG, none of which are cited or compared. PUR differs in that it uses the reader's residual NLL over evidence+question rather than generation-side confidence, and runs a fixed number of offline iterations, but these distinctions must be made explicit (ideally with empirical comparison) for the novelty claim to hold. This is the most likely point of attack by other reviewers. 2,U(D) is the NLL of question tokens conditioned on evidence, not uncertainty over the answer. Question NLL largely reflects linguistic/surface familiarity; its link to knowledge gaps rests only on the correlation plot in Figure 2 (right), which shows a single fitted line on two datasets without confidence intervals or cross-reader validation. Suggested additions: comparison with answer entropy/self-consistency, and a causal ablation (e.g., queries built from randomly selected tokens). 3,"Training-free" is a selling point, but the cost shifts to inference: ~30 single-document forward passes per question (for U({d_i})), token-level logprob computation over the full context each round, and one selector API call per round (T=3). No latency/FLOPs/token-cost comparison against MedRGAG or MedGENIE is reported; the conclusion only says it "incurs additional computation." Numbers are needed to judge practicality. 4,Only the number of rounds T is analyzed; there is no ablation for λ (uncertainty scaling), β (the cap on the uncertainty contribution), or the ρ=10% quantile. With β=0.1, uncertainty contributes at most 0.1 relative to relevance r_i∈[0,1]—uncertainty is a deliberately weak signal, a design choice that needs empirical support.
Questions for authors
see Weaknesses