- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies reasoning-intensive retrieval for underspecified queries whose text alone does not determine relevance. It proposes CPR, which requires no additional training and treats corpus documents as partial candidates for the latent intent, averaging corpus-based relevance predictions under a prior built from semantic and semantic-lexical evidence. Scoring combines semantic neighbor relations with lexical coverage relative to a corpus background and weights predictions by candidate support. Experiments on BRIGHT and R2MED with three frozen retrievers show improvements over direct retrieval, leading results among the compared LLM-free methods, and complementarity with query expansion methods such as HyDE and ThinkQE.
Critical strengths
1. Improvements across the full evaluation suite CPR improves the task-average nDCG@10 of direct retrieval on all three frozen retrievers (pooled mean 20.30 to 27.04) and leads every compared LLM-free method across 11 BRIGHT and 8 R2MED tasks, with corresponding recall gains. This matters because these results, spanning 19 tasks and three retrievers, form the paper's central empirical case.
2. Demonstrated complementarity with query expansion Composing CPR with HyDE or ThinkQE improves over both standalone methods and recovers 7.20% and 6.89% of Base misses that neither recovers, and the HyDE composition improves on both methods on every backbone. The paper also separates recovery of missed queries from ranking gains on retrieved queries. This matters because it identifies a practical complementarity between corpus evidence and generated query evidence, which is the paper's most transferable finding.
Critical weaknesses
1. Limited novelty relative to closely related retrieval methods CPR combines components that are individually close to established retrieval techniques, including document-side evidence estimation, rank-based neighbor weighting, and multiplicative semantic and lexical coverage. The paper compares with recent corpus-guided and probabilistic methods such as LexBoost, ECLIPSE-PRF, RDIME-PRF, SemRank, Parry et al., PQR, and QuDAR. All of these comparisons remain at the full-pipeline level, and no ablation isolates what corpus-conditioned prediction adds beyond the neighbor and lexical signals already used by these methods. The paper also lacks a matched-budget control that fuses the same signals without the intent mixture, so the source of the reported gains remains difficult to identify. For these reasons, the boundary between CPR and prior work is unclear.
2. The core empirical claim remains unsupported The headline comparison against dense retrieval and the baselines reports no significance tests or confidence intervals, and the intervals that do appear cover only component and diagnostic analyses. The gains also decline as the backbone strengthens (E5 +8.76, Qwen3 +6.64, ReasonIR +4.80), HyDE surpasses CPR on ReasonIR nDCG (31.33 vs 28.96), and task-level outcomes include losses (TheoremQA-Theorems -2.71, MedXpertQA-Exam -1.25, TheoremQA-Questions -0.40) alongside large gains. The abstract presents a negative observation about prior rankings as a central finding, and the paper reports no latency, memory, or aggregation cost even though it acknowledges that corpus-wide relations are expensive. The main empirical claim of a robust and practical advantage over direct retrieval is therefore not established.
3. Unverified design choices and inconclusive diagnostics The method depends on many design choices that receive no ablation or sensitivity analysis, including the setting k = ⌈√N⌉, the CSLS-style density correction, reciprocal ranks, the rank product, equal weighting of the two priors, alternating folds, top-h truncation, IDF weighting, and the forced retention of the top-1 document. No development set or tuning protocol is reported, so benchmark-specific tuning cannot be ruled out. The diagnostic analyses are circular because ΔL and the coverage-gain correlations are computed post hoc with relevance labels from the same tasks and with weights derived from the quantity under evaluation. Formal issues also remain. The normalization of u(d) over C is combined with sums over Aq in Eq. 2 and Eq. 10, ρs denotes two different rank functions, Eq. 10 is justified only by intuition, and an uncalibrated ratio is treated as a valid binary likelihood on the strength of a range argument. These problems prevent the reported gains from being attributed to the proposed design and raise serious concerns about reproducibility.
Questions for authors
What is the minimal essential difference between CPR and the corpus-guided methods it already compares with, and how much does CPR add over them at matched budget? Do the headline gains survive significance testing and remain robust beyond E5, particularly on ReasonIR where HyDE leads? What latency, memory, and aggregation cost does CPR add relative to Base and HyDE, and were the hyperparameters selected on a held-out development set? Please align the posterior-predictive framework with the actual estimators and provide a derivation. Report query-weighted results, additional backbones, and measured efficiency. Replace the label-dependent diagnostics with label-free validation or state their interpretation limits explicitly.
3 comments
Online cost. We measured generation, query-side encoding, and retrieval as applicable across the 19 tasks. The following values are batch-amortized seconds per query, not isolated single-query latencies.
| Method | E5 | Qwen3 | ReasonIR |
|---|---|---|---|
| Base | 0.015 | 0.041 | 0.046 |
| ECLIPSE-PRF | 0.016 | 0.043 | 0.048 |
| CPR | 0.316 | 0.380 | 0.409 |
| HyDE | 7.985 | 8.178 | 8.433 |
| ThinkQE | 50.330 | 50.564 | 50.839 |
| CPR+HyDE | 8.068 | 8.260 | 8.542 |
CPR is more costly than LLM-free retrieval baselines but avoids query-time LLM generation. In the measured setup, adding CPR to HyDE costs approximately 0.08–0.11 additional seconds/query, compared with a +6.262 nDCG@10 gain. Recorded failed ThinkQE generation attempts are excluded. CPR retrieval was timed using four AMD EPYC 7313 CPU cores (FP64); generation, query encoding, and other methods' final retrieval used RTX 6000 Ada GPUs. Model loading and corpus preprocessing are excluded.
Million-document scalability. On one million unique documents with E5 and 2,157 cached queries, search time is 4.826 ms/query for Base, 841.716 ms/query for CPR, and 1,069.006 ms/query for CPR+HyDE. CPR requires 25.096 GiB peak GPU allocation, 28.688 GiB peak CPU RSS, and 24.052 GiB of relation-index storage; offline index construction takes 281.897 seconds. These cached-input measurements exclude generation and encoding, and retrieval accuracy on the combined million-document corpus was not evaluated. They quantify the added storage and aggregation costs rather than establishing a cost advantage over direct retrieval.
The statistical significance results are reported in Section 4 and Appendix D, while the computational analysis is presented in Section 8 and Appendix H. We will additionally report the BGE-M3 and query-weighted results in Appendix D.
W3
We first clarify the definition of in Section 2. Our intention was to allow to be a configurable subset of the corpus, ranging from a restricted set of retrieved documents to the full eligible corpus. The original description did not clearly distinguish these cases, and we have revised it accordingly. Importantly, and play different roles: provides the corpus-wide background , whereas determines which sources contribute to the relevance prediction. We make this distinction explicit in the derivation of Equations (2) and (10).
Design sensitivity. We varied one design choice at a time across 19 tasks, 2,157 queries, and E5/Qwen3/ReasonIR while retaining the original evaluation inputs. Below, differences are alternative configuration minus the manuscript CPR baseline in task-macro nDCG@10, with paired 95% bootstrap intervals.
| Changed design choice | ΔnDCG@10 [95% CI] |
|---|---|
| Prior mixture weight α=0.25 instead of 0.5 | −1.378 [−1.788, −0.968] |
| Prior mixture weight α=0.75 instead of 0.5 | −0.247 [−0.593, 0.109] |
| Density neighborhood k=0.5√N | +0.147 [−0.028, 0.322] |
| Density neighborhood k=2√N | −0.302 [−0.463, −0.145] |
| Remove CSLS-style density correction | −1.533 [−1.891, −1.181] |
| Prior reciprocal-rank exponent β=0.5 | −1.316 [−1.566, −1.067] |
| Replace semantic–lexical rank product with RRF60 | −0.409 [−0.669, −0.147] |
| Top-h at 25%, with head-only renormalization | +0.208 [0.110, 0.308] |
| Replace query-term IDF with uniform weighting | +0.004 [−0.069, 0.078] |
| Remove forced top-1 retention | −1.002 [−1.373, −0.639] |
Random 2- and 4-fold alternatives yield changes between −0.090 and +0.077 nDCG@10 across the tested seeds, with intervals including zero. The default equal-weight prior mixture (α=0.5) attains 27.035 nDCG@10, compared with 25.658 at α=0.25 and 26.789 at α=0.75. Removing density correction and top-1 retention decreases performance, whereas the top-h experiment identifies a modest improvement over the default. These tests support several design choices without implying that every default is optimal.
Diagnostic interpretation. The ΔL measure uses benchmark relevance labels to compare query-term coverage in relevant documents and semantic competitors, but does not use CPR's predicted gains to define ΔL. In the source-supported terminology analysis, documents used to construct the relevant/competitor term sets are excluded from the sources being examined. Both analyses are post-hoc, label-dependent associations, not independent performance validation or causal evidence. The main comparisons and controlled ablations, reported above, provide the performance evidence.
The prior-mixture sensitivity analysis is presented in Section 5.1 and Appendix E.1, while the scoring controls and additional sensitivity analyses are provided in Appendix E.4. We clarify the role of in Section 2 and discuss the interpretation limits of the label-dependent diagnostics in Section 6.1 and Appendix F.4.
Dear Reviewer qdmF,
Thank you again for your thoughtful feedback. We would greatly appreciate it if you could review our responses and let us know whether they have addressed your concerns or if any questions remain. Thank you for your time and consideration.
Best regards, The Authors
We sincerely thank the reviewer for their thoughtful and constructive feedback. We greatly appreciate the time and effort devoted to reviewing our work. We have revised the manuscript to clarify the formulation and provide additional experimental evidence addressing the concerns. We address each point below.
W1
Contribution. Underspecified queries may not express the information needed to judge document relevance. Rather than simply augmenting the query or ranking corpus evidence, CPR uses eligible corpus documents as partial evidence sources and combines their source-conditioned relevance predictions. A document that matches the query closely need not be the most informative source for predicting other documents' relevance. This distinction between direct ranking quality and predictive contribution is the central contribution of CPR.
Controlled comparisons. Across 19 tasks and three backbones, applying corpus-conditioned prediction improves prior-only ranking by +1.273 nDCG@10 (95% CI [0.921, 1.638]). Holding the candidate prior, semantic and lexical evidence, source scope, and candidate set fixed, CPR outperforms same-signal RRF60 by +1.437 [1.208, 1.660]. Under the same resource configuration (one RTX 6000 Ada and four CPU cores) and a retrieval-time cap of 2T, the gain over RRF60 remains +1.436 [1.207, 1.663] across all 19 tasks; CPR also exceeds ECLIPSE-PRF by +11.522 [10.688, 12.360] within that cap. Here, T is the measured full-CPR retrieval time for the corresponding task and backbone, and configuration selection does not use relevance labels.
The RRF control isolates the final combination of fixed signals; it does not, by itself, remove source aggregation. Replacing summed source contributions with the strongest contribution per candidate yields a smaller, statistically inconclusive difference (+0.124 [−0.018, 0.266]). We therefore attribute the controlled gains to CPR's predictive scoring and signal combination, without claiming that every aspect of the mixture aggregation is individually indispensable.
The contribution of predictive scoring is discussed in Section 5.3 and Appendix E.3, with same-signal comparisons reported in Appendix E.4. We will further include the matched computational budget comparisons in Appendix H.
W2
Significance and backbone robustness. We reanalyzed 19 tasks and 2,157 queries with 10,000 paired within-task bootstrap resamples. CPR improves over Base, and CPR+HyDE improves over HyDE, with 95% confidence intervals excluding zero for all three evaluated backbones.
CPR also exceeds ECLIPSE-PRF by +5.947 [5.255, 6.645] and RDIME-PRF by +6.246 [5.559, 6.938]. The gains persist with query-weighted aggregation (Base 20.370 → CPR 26.909; HyDE 26.581 → CPR+HyDE 32.423). On an additional BGE-M3 backbone evaluated over all 19 tasks, CPR − Base is +6.050 [5.279, 6.820] and CPR+HyDE − HyDE is +6.432 [5.622, 7.265].
Limits of the improvements. The gain over Base decreases with stronger backbones. Standalone HyDE exceeds CPR on ReasonIR (31.335 vs. 28.960) and BGE-M3 (22.518 vs. 19.340), and CPR loses to Base on TheoremQA-Theorems (−2.71), MedXpertQA-Exam (−1.25), and TheoremQA-Questions (−0.40) in task-macro nDCG@10. Nonetheless, CPR+HyDE improves over HyDE on both ReasonIR and BGE-M3. We will state these task- and backbone-dependent limitations explicitly instead of claiming uniform superiority.
Meaning of the prior-ranking observation. The abstract's observation distinguishes a prior's direct ranking performance from its predictive usefulness: Semantic–Lexical ranks better initially, whereas Semantic achieves better final ranking after prediction. We will clarify this interpretation in the abstract.