- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a two-stage method for retrieving evidence sets for multi-hop QA. ParaSet uses precomputed passage embeddings to score candidate sets cheaply, allowing a broader beam search. SetCE then uses a more expressive cross-encoder to score the proposals. Training preferences are constructed by perturbing gold evidence sets, with coverage taking priority over precision. Experiments compare the method with passage-level and listwise rerankers on three QA datasets using two first-stage retrievers. The paper also tests combinations of retrievers, measures inference cost, and examines how gains vary with evidence expansion ratio.
Critical strengths
-
A useful evidence set may include passages that receive low individual relevance scores. The two-stage design makes it practical to explore such sets: a cheap cached scorer proposes candidates before the expensive cross-encoder is applied. The attention-cache details and inference breakdown explain where computation is saved. Using a fixed downstream answer model also helps keep the evaluation focused on retrieval.
-
Table 1 shows that the full pipeline improves over CE in all six dataset–retriever combinations while returning fewer passages on average. The combined system is compared with deeper CE retrieval and a union of independently parameterized passage rerankers. These controls help test whether the gains come simply from retrieving more passages or adding a second model. The paper also reports where SetCE alone falls short of CE.
Critical weaknesses
-
The comparisons change more than set-level supervision. On p. 8, the ListCE–SetCE comparison is described as controlled because the models share an architecture and inference procedure. However, SetCE receives extra set-level fine-tuning, and Table 6 gives the methods different candidate-pool settings in several conditions. The amount of training and the available evidence therefore change along with the objective. It is also unclear how much ParaSet needs set interactions beyond what a similarly trained additive or pooled scorer could provide. Matching candidate pools, training exposure, and search budgets would help establish which gains come from set scoring itself.
-
Some standalone gains are small, such as 23.9 versus 23.8 on Contriever/2Wiki, without run-level or paired-query uncertainty. The combined system has larger gains, but this does not establish the reliability of the standalone results. The comparison with SetR (ACL 2025) also needs to be interpreted narrowly: Appendix L uses a prompt-style version, rather than the original trained system. Finally, Table 2 shows that the end-to-end speed advantage is smaller than the retrieval-stage advantage once generation is included. These distinctions affect how large a practical improvement the paper establishes.
Questions for authors
- Why is the ListCE–SetCE comparison described as controlled when Table 6 uses different candidate pools and SetCE receives additional fine-tuning? Are results available with the same pools, training exposure, and search budgets?
- Is there an existing comparison between ParaSet and an additive or simple pooled scorer trained on the same set preferences? This would help assess whether the interactions are needed. An extensive new architecture study is not requested for the rebuttal.
- What run-level variation or paired-query uncertainty is available for Table 1? Please also clarify that the SetR comparison applies to the implemented prompt-style baseline, and distinguish retrieval speed from total answer latency.
3 comments
W2 / Q3. SetR comparison and retrieval latency
| The comparison with SetR (ACL 2025) also needs to be interpreted narrowly: Appendix L uses a prompt-style version, rather than the original trained system. / Table 2 shows that the end-to-end speed advantage is smaller than the retrieval-stage advantage once generation is included
We appreciate the opportunity to clarify these two interpretations further. The SetR-CoT&IRI-style method described in our original appendix is a prompting-based implementation rather than the original trained SetR system. This distinction is already stated in the manuscript, and we will make it more prominent in the revised version.
Regarding efficiency, we report both retrieval-stage and end-to-end latency precisely to distinguish the efficiency of evidence selection from the cost of downstream answer generation. In our IRCoT comparison, evidence-selection latency decreases from 6.18 s to 0.421 s (14.7× speedup; 93.2% reduction), while end-to-end latency decreases from 8.32 s to 2.641 s (3.15× speedup; 68.3% reduction). Importantly, evidence selection accounts for approximately 74% of the original pipeline's latency, compared with only 16% in ours. Thus, our method substantially reduces the retrieval bottleneck, even though the overall speedup is naturally moderated by the remaining answer-generation cost.
More fundamentally, retrieval efficiency is valuable independently of the latency of a particular downstream QA model. Evidence selection is not necessarily followed by immediate answer generation. Retrieved evidence may instead be evaluated for sufficiency or reliability, used to determine whether to answer or abstain, or passed to additional retrieval and verification stages. For example, recent work such as [1] demonstrates the practical value of lightweight evidence screening before invoking an expensive language model. Such modular pipelines highlight why evidence-selection efficiency is a meaningful objective in its own right, rather than solely a component of one fixed retrieve-then-generate workflow.
Furthermore, as downstream generators become smaller or more optimized, retrieval can account for a larger fraction of total inference cost. Efficient evidence selection is also particularly valuable in iterative retrieval and agentic systems, where retrieval may be invoked repeatedly before a final answer is produced.
Our contribution is therefore not limited to accelerating a particular end-to-end QA pipeline, but to providing an efficient and reusable set-level retrieval component. We include end-to-end measurements to provide a complete picture of the evaluated QA pipeline, while reporting retrieval-stage latency separately to characterize the efficiency of the proposed retrieval method itself. We will further clarify this distinction in the revised manuscript.
[1] Sieve and Sage: Efficient Distraction Filtering for Reliable RALM Abstention (Won et al.)
Dear Reviewer,
We would like to kindly follow up on our rebuttal response. We have provided additional experiments and clarifications to address your concerns, and we would greatly appreciate your feedback on whether these sufficiently resolve the issues you raised.
If any concerns remain, we would be happy to provide further clarification during the discussion period.
Thank you again for your time and thoughtful review.
We sincerely thank the reviewer for the careful assessment of our experimental controls, statistical reliability, and efficiency claims. Your suggestions have helped us strengthen the evidence in three complementary ways.
Controlled comparisons: In addition to the original independently validation-tuned configurations, a new fixed-pool comparison shows that SetCE achieves better F5 and Perfect Recall than ListCE under the same first-stage candidate pool and search procedure. Our shared ParaSet ablations also demonstrate an advantage over uniform pooling.
Reliability: Three independent training/evaluation runs preserve the main QA EM trends, including the higher mean EM of ParaSet→SetCE over CE and SetCE over ListCE.
Interpretation and efficiency: We further clarify the scope of our SetR-CoT&IRI-style comparison and distinguish the large evidence-selection speedup from the smaller but still meaningful end-to-end latency improvement.
W1 / Q1. Matched-pool ListCE versus SetCE
| The amount of training and the available evidence therefore change along with the objective.
We appreciate the opportunity to provide a more detailed controlled evaluation. Our original Table 1 selected each method's candidate-pool size independently on the development set, which allows the methods to be compared under their best validation-selected configurations. Following the reviewer's suggestion, we additionally fix Qwen3-Embedding-0.6B, kb=50, and the same inference/search procedure for ListCE and SetCE.
SetCE improves both F5 and Perfect Recall in all three matched candidate-pool settings.
Both ListCE and SetCE rely on the same underlying gold supporting-evidence annotations, with no additional human labeling. Their distinction is in how they use the negative supervision: ListCE reduces the scores of irrelevant next passages in a locally supervised sequential retrieval task, while SetCE constructs perturbed evidence sets and directly reduces the scores of incomplete or noisy set candidates. ListCE is evaluated using its best validation-selected checkpoint under its own objective, and the fixed-pool comparison examines the effect of this additional set-level preference learning under matching search inputs.
W1 / Q2. Simple pooling versus ParaSet
| How much ParaSet needs set interactions beyond what a similarly trained additive or pooled scorer could provide / Is there an existing comparison between ParaSet and an additive or simple pooled scorer trained on the same set preferences?
We additionally investigate whether the learned ParaSet attention aggregation improves over uniform pooling trained on the same set preferences. As detailed in the General Response and new Appendix N, ParaSet obtains higher mean F$_5$ and Perfect Recall across all three datasets. These results support the value of ParaSet's learned aggregation for scoring evidence sets under this fixed candidate-pool setting.
W2 / Q3. Reliability of the standalone comparisons
| Some standalone gains are small, such as 23.9 versus 23.8 on Contriever/2Wiki, without run-level or paired-query uncertainty. The combined system has larger gains, but this does not establish the reliability of the standalone results. / What run-level variation or paired-query uncertainty is available for Table 1?
We thank the reviewer for highlighting the importance of run-to-run stability. We retrained and reevaluated the standalone methods across three independent runs. The table below reports mean ± standard deviation of downstream QA EM (%).
The principal trends remain reproducible: ParaSet→SetCE has higher mean QA EM than CE in all six settings, while SetCE exceeds ListCE in all six. We also retain the Evidence Expansion Ratio analysis showing that the relative benefit of set-level retrieval tends to grow when fewer supporting passages are directly connected to the query. New Appendix Q contains the standalone three-run results.