acceptodds
Under review as a conference paper at ICLR 2027

Scoring a Set, Not Summing Passage Scores: Effective and Efficient Set Retrieval

Abstract

Multi-hop question answering requires retrieving multiple evidence passages whose usefulness often depends on one another. Conventional retrievers either rank passages independently or construct evidence sequentially through locally supervised next-passage decisions. Sequential conditioning captures some cross-passage dependencies, but its local extension scores do not provide a common criterion for comparing complete evidence sets of different compositions and sizes. A natural alternative is to rank evidence sets directly, which requires an explicit scorer over arbitrary query-set pairs. We therefore formulate multi-hop retrieval as direct ranking of candidate evidence sets through a learned query-set compatibility score . Following an energy-based structured prediction perspective, we learn this score by contrasting complete evidence sets against incomplete and noisy alternatives in the combinatorial set space. Once a gold evidence set is available, such alternatives can be generated automatically by adding, removing, or replacing passages, providing rich set-level supervision without additional annotation. To make inference over this combinatorial space practical, we introduce ParaSet, a lightweight scorer over precomputed passage representations for efficient set exploration, and SetCE, an expressive cross-encoder for set reranking. Across three multi-hop QA benchmarks and two first-stage retrievers, set-level retrieval consistently provides a complementary signal to passage-level relevance, becoming relatively more effective when less of the required evidence is directly connected to the query. Motivated by this complementarity, combining the two signals further improves downstream QA performance, outperforming both deeper passage-level retrieval and an ensemble of distinct passage-level retrievers. Finally, without invoking an LLM during retrieval, our method remains competitive with LLM-assisted evidence-selection pipelines while achieving up to faster evidence selection.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Natural Language Processing, Retrieval, Retrieval-Augmented Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_W58tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a two-stage method for retrieving evidence sets for multi-hop QA. ParaSet uses precomputed passage embeddings to score candidate sets cheaply, allowing a broader beam search. SetCE then uses a more expressive cross-encoder to score the proposals. Training preferences are constructed by perturbing gold evidence sets, with coverage taking priority over precision. Experiments compare the method with passage-level and listwise rerankers on three QA datasets using two first-stage retrievers. The paper also tests combinations of retrievers, measures inference cost, and examines how gains vary with evidence expansion ratio.

Critical strengths
  1. A useful evidence set may include passages that receive low individual relevance scores. The two-stage design makes it practical to explore such sets: a cheap cached scorer proposes candidates before the expensive cross-encoder is applied. The attention-cache details and inference breakdown explain where computation is saved. Using a fixed downstream answer model also helps keep the evaluation focused on retrieval.

  2. Table 1 shows that the full pipeline improves over CE in all six dataset–retriever combinations while returning fewer passages on average. The combined system is compared with deeper CE retrieval and a union of independently parameterized passage rerankers. These controls help test whether the gains come simply from retrieving more passages or adding a second model. The paper also reports where SetCE alone falls short of CE.

Critical weaknesses
  1. The comparisons change more than set-level supervision. On p. 8, the ListCE–SetCE comparison is described as controlled because the models share an architecture and inference procedure. However, SetCE receives extra set-level fine-tuning, and Table 6 gives the methods different candidate-pool settings in several conditions. The amount of training and the available evidence therefore change along with the objective. It is also unclear how much ParaSet needs set interactions beyond what a similarly trained additive or pooled scorer could provide. Matching candidate pools, training exposure, and search budgets would help establish which gains come from set scoring itself.

  2. Some standalone gains are small, such as 23.9 versus 23.8 on Contriever/2Wiki, without run-level or paired-query uncertainty. The combined system has larger gains, but this does not establish the reliability of the standalone results. The comparison with SetR (ACL 2025) also needs to be interpreted narrowly: Appendix L uses a prompt-style version, rather than the original trained system. Finally, Table 2 shows that the end-to-end speed advantage is smaller than the retrieval-stage advantage once generation is included. These distinctions affect how large a practical improvement the paper establishes.

Questions for authors
  1. Why is the ListCE–SetCE comparison described as controlled when Table 6 uses different candidate pools and SetCE receives additional fine-tuning? Are results available with the same pools, training exposure, and search budgets?
  2. Is there an existing comparison between ParaSet and an additive or simple pooled scorer trained on the same set preferences? This would help assess whether the interactions are needed. An extensive new architecture study is not requested for the rebuttal.
  3. What run-level variation or paired-query uncertainty is available for Table 1? Please also clarify that the SetR comparison applies to the implemented prompt-style baseline, and distinguish retrieval speed from total answer latency.
3 comments

We sincerely thank the reviewer for the careful assessment of our experimental controls, statistical reliability, and efficiency claims. Your suggestions have helped us strengthen the evidence in three complementary ways.

  1. Controlled comparisons: In addition to the original independently validation-tuned configurations, a new fixed-pool comparison shows that SetCE achieves better F5 and Perfect Recall than ListCE under the same first-stage candidate pool and search procedure. Our shared ParaSet ablations also demonstrate an advantage over uniform pooling.

  2. Reliability: Three independent training/evaluation runs preserve the main QA EM trends, including the higher mean EM of ParaSet→SetCE over CE and SetCE over ListCE.

  3. Interpretation and efficiency: We further clarify the scope of our SetR-CoT&IRI-style comparison and distinguish the large evidence-selection speedup from the smaller but still meaningful end-to-end latency improvement.


W1 / Q1. Matched-pool ListCE versus SetCE

| The amount of training and the available evidence therefore change along with the objective.

We appreciate the opportunity to provide a more detailed controlled evaluation. Our original Table 1 selected each method's candidate-pool size independently on the development set, which allows the methods to be compared under their best validation-selected configurations. Following the reviewer's suggestion, we additionally fix Qwen3-Embedding-0.6B, kb=50k_b=50, and the same inference/search procedure for ListCE and SetCE.

DatasetListCE F5SetCE F5ListCE PRSetCE PR
HotpotQA29.6333.7612.2018.60
2Wiki29.3630.419.6010.40
MuSiQue32.3332.7812.8013.20

SetCE improves both F5 and Perfect Recall in all three matched candidate-pool settings.

Both ListCE and SetCE rely on the same underlying gold supporting-evidence annotations, with no additional human labeling. Their distinction is in how they use the negative supervision: ListCE reduces the scores of irrelevant next passages in a locally supervised sequential retrieval task, while SetCE constructs perturbed evidence sets and directly reduces the scores of incomplete or noisy set candidates. ListCE is evaluated using its best validation-selected checkpoint under its own objective, and the fixed-pool comparison examines the effect of this additional set-level preference learning under matching search inputs.


W1 / Q2. Simple pooling versus ParaSet

| How much ParaSet needs set interactions beyond what a similarly trained additive or pooled scorer could provide / Is there an existing comparison between ParaSet and an additive or simple pooled scorer trained on the same set preferences?

We additionally investigate whether the learned ParaSet attention aggregation improves over uniform pooling trained on the same set preferences. As detailed in the General Response and new Appendix N, ParaSet obtains higher mean F$_5$ and Perfect Recall across all three datasets. These results support the value of ParaSet's learned aggregation for scoring evidence sets under this fixed candidate-pool setting.


W2 / Q3. Reliability of the standalone comparisons

| Some standalone gains are small, such as 23.9 versus 23.8 on Contriever/2Wiki, without run-level or paired-query uncertainty. The combined system has larger gains, but this does not establish the reliability of the standalone results. / What run-level variation or paired-query uncertainty is available for Table 1?

We thank the reviewer for highlighting the importance of run-to-run stability. We retrained and reevaluated the standalone methods across three independent runs. The table below reports mean ± standard deviation of downstream QA EM (%).

First-stageDatasetBiCEListCESetCEParaSet→SetCE
ContrieverHotpotQA35.0 ± 0.02937.7 ± 0.3835.2 ± 0.8638.1 ± 0.4038.9 ± 0.65
Contriever2Wiki20.4 ± 0.0023.7 ± 0.3417.1 ± 0.2218.8 ± 0.6024.1 ± 0.28
ContrieverMuSiQue4.8 ± 0.035.2 ± 0.304.9 ± 0.296.3 ± 0.295.8 ± 0.06
Qwen3-EmbHotpotQA35.6 ± 0.06737.5 ± 0.4936.1 ± 0.3837.9 ± 0.2838.8 ± 0.60
Qwen3-Emb2Wiki25.0 ± 0.03926.5 ± 0.3425.3 ± 0.1726.5 ± 0.1127.3 ± 0.12
Qwen3-EmbMuSiQue11.3 ± 0.0310.9 ± 0.3010.5 ± 0.5911.8 ± 0.6011.9 ± 0.29

The principal trends remain reproducible: ParaSet→SetCE has higher mean QA EM than CE in all six settings, while SetCE exceeds ListCE in all six. We also retain the Evidence Expansion Ratio analysis showing that the relative benefit of set-level retrieval tends to grow when fewer supporting passages are directly connected to the query. New Appendix Q contains the standalone three-run results.

W2 / Q3. SetR comparison and retrieval latency

| The comparison with SetR (ACL 2025) also needs to be interpreted narrowly: Appendix L uses a prompt-style version, rather than the original trained system. / Table 2 shows that the end-to-end speed advantage is smaller than the retrieval-stage advantage once generation is included

We appreciate the opportunity to clarify these two interpretations further. The SetR-CoT&IRI-style method described in our original appendix is a prompting-based implementation rather than the original trained SetR system. This distinction is already stated in the manuscript, and we will make it more prominent in the revised version.

Regarding efficiency, we report both retrieval-stage and end-to-end latency precisely to distinguish the efficiency of evidence selection from the cost of downstream answer generation. In our IRCoT comparison, evidence-selection latency decreases from 6.18 s to 0.421 s (14.7× speedup; 93.2% reduction), while end-to-end latency decreases from 8.32 s to 2.641 s (3.15× speedup; 68.3% reduction). Importantly, evidence selection accounts for approximately 74% of the original pipeline's latency, compared with only 16% in ours. Thus, our method substantially reduces the retrieval bottleneck, even though the overall speedup is naturally moderated by the remaining answer-generation cost.

More fundamentally, retrieval efficiency is valuable independently of the latency of a particular downstream QA model. Evidence selection is not necessarily followed by immediate answer generation. Retrieved evidence may instead be evaluated for sufficiency or reliability, used to determine whether to answer or abstain, or passed to additional retrieval and verification stages. For example, recent work such as [1] demonstrates the practical value of lightweight evidence screening before invoking an expensive language model. Such modular pipelines highlight why evidence-selection efficiency is a meaningful objective in its own right, rather than solely a component of one fixed retrieve-then-generate workflow.

Furthermore, as downstream generators become smaller or more optimized, retrieval can account for a larger fraction of total inference cost. Efficient evidence selection is also particularly valuable in iterative retrieval and agentic systems, where retrieval may be invoked repeatedly before a final answer is produced.

Our contribution is therefore not limited to accelerating a particular end-to-end QA pipeline, but to providing an efficient and reusable set-level retrieval component. We include end-to-end measurements to provide a complete picture of the evaluated QA pipeline, while reporting retrieval-stage latency separately to characterize the efficiency of the proposed retrieval method itself. We will further clarify this distinction in the revised manuscript.

[1] Sieve and Sage: Efficient Distraction Filtering for Reliable RALM Abstention (Won et al.)

Dear Reviewer,

We would like to kindly follow up on our rebuttal response. We have provided additional experiments and clarifications to address your concerns, and we would greatly appreciate your feedback on whether these sufficiently resolve the issues you raised.

If any concerns remain, we would be happy to provide further clarification during the discussion period.

Thank you again for your time and thoughtful review.

Reviewer_hKchOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes two set-level rerankers, ParaSet and SetCE, for multi-hop retrieval. Given annotated supporting facts, the authors use an energy-based ranking objective to train models that score the quality of retrieved passage subsets. To avoid exhaustive scoring of all possible subsets, the lightweight ParaSet model performs beam search, while the more powerful SetCE reranks the selected candidates. The authors report improved answer accuracy on multi-hop QA benchmarks.

Critical strengths
  1. Simple and elegant training approach. The authors propose a straightforward way to construct training data for set-level rerankers by perturbing gold evidence sets. They also introduce a reasonable ordering of candidate sets based on evidence coverage and precision. The resulting training approach is simple, intuitive, and generally outperforms the sequential ListCE baseline.

  2. Interesting architectural design of ParaSet. To the best of my knowledge, ParaSet represents a relatively original architectural solution for efficient set-level reranking. Its lightweight attention mechanism and reuse of cached bi-encoder representations allow it to score many candidate sets with low computational overhead, making it particularly interesting from an architectural perspective.

Critical weaknesses
  1. Unclear experimental setup and weak baseline comparisons. The reported answer EM scores on HotpotQA, 2Wiki, and MuSiQue are substantially lower than those in related work. Beam Retrieval [1] reports 72.7% EM on HotpotQA and 88.5% on 2Wiki. Q-RAG [2] reports 59% on HotpotQA and 37% on MuSiQue (OOD); moreover, its Table 4 reports up to 49.3% HotpotQA EM with Qwen3-4B, compared to 38.9% for ParaSet→SetCE and 41.4% for the combined method using Qwen3-8B. While these results may use different evaluation settings, the paper does not sufficiently clarify the retrieval corpus and protocol, particularly whether retrieval is performed over dataset-provided distractors or a larger Wikipedia corpus. Search-R1 [3], a more recent LLM-based method, achieves 43.3% / 38.2% / 19.6% EM on HotpotQA / 2Wiki / MuSiQue, outperforming the reported results on average. It would provide a more relevant comparison than the relatively old IRCoT baseline. Finally, despite the availability of gold supporting evidence, the main experiments focus on answer EM rather than directly comparing retrieval quality.

  2. Insufficient ablation of ParaSet. It is unclear whether ParaSet provides better candidate-set selection or whether the gains mainly result from evaluating more candidate sets with SetCE. This could be tested by replacing ParaSet proposals with randomly sampled sets, or by increasing the beam width of standalone SetCE. A comparison under matched computational budgets would better isolate the contribution of ParaSet.

References

[1] Zhang, J., et al. (2024). End-to-End Beam Retrieval for Multi-Hop Question Answering. NAACL 2024. arXiv:2308.08973.

[2] Sorokin, A., et al. (2026). Q-RAG: Long Context Multi-Step Retrieval via Value-Based Embedder Training. ICLR 2026. arXiv:2511.07328.

[3] Jin, B., et al. (2025). Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. COLM 2025. arXiv:2503.09516.

Questions for authors
  1. Could the authors report the standalone performance of ParaSet under the same evaluation setup as ListCE and SetCE? It would also be useful to compare ParaSet→SetCE against alternatives with (a) randomly sampled candidate sets from the top-10 or top-20 retrieved passages and (b) a wider beam search using standalone SetCE, preferably under comparable computational budgets.

  2. Could the authors clarify the retrieval corpus and evaluation protocol, particularly whether they use dataset-provided distractors or a larger Wikipedia corpus? Could they also explain why the reported answer EM scores are substantially lower than those in related work, such as Beam Retrieval, Q-RAG, and Search-R1?

2 comments

We sincerely thank the reviewer for the encouraging assessment of our training objective and ParaSet architecture, and for the constructive suggestions on evaluation clarity and ParaSet ablations.

We are pleased to address the two main concerns.

  1. We clarify that our retrieval is performed over a substantially larger dataset-wide corpus, rather than the question-specific distractor candidates used by two cited works which is much easier, thus displaying higher scores (Beam Retrieval and Q-RAG), and distinguish our frozen-answer-generator setting from RL-trained search-and-reasoning systems (Search-R1).
  2. Our new controlled ablations show that ParaSet produces better evidence proposals than random selection under the same SetCE reranking budget; we also provide standalone ParaSet retrieval measurements.

W1 / Q2. Retrieval corpus and comparison with prior work

| Unclear experimental setup and weak baseline comparisons. / Could the authors clarify the retrieval corpus and evaluation protocol, particularly whether they use dataset-provided distractors or a larger Wikipedia corpus? Could they also explain why the reported answer EM scores are substantially lower than those in related work, such as Beam Retrieval, Q-RAG, and Search-R1?

Thank you for the opportunity to clarify our evaluation setting and its differences from the studies cited in the review.

We construct a dataset-wide retrieval corpus for each benchmark, aggregating passage texts available across dataset splits and removing duplicates. The resulting retrieval collections contain 509,315 HotpotQA, 430,225 2WikiMultiHopQA, and 117,534 MuSiQue passages. The first-stage bi-encoder searches the corresponding full collection for each query; retrieval is not restricted to the small set of distractors accompanying that question. We have included the details in the revised manuscript (new Appendix M and the main experimental setup).

This distinction matters for the reported answer EM comparisons:

  • Beam Retrieval selects supporting evidence from question-specific pools of 10 passages (HotpotQA/2Wiki) or 20 passages (MuSiQue), and its high answer EM is reported with a task-supervised reader.

  • Q-RAG reports distractor-based retrieval over question-specific contexts, with different retrieval/answering configurations (including QwQ-32B in its main experiments).

  • Search-R1 retrieves from a large Wikipedia corpus, but its strongest results come from an RL-trained LLM that learns search-query generation, iterative retrieval, and answer generation together. Our retrieval component is evaluated with a frozen Qwen3-8B answer model instead.

These differences in candidate-space construction and in the training/inference of the answer model make absolute cross-paper answer EM difficult to interpret as a direct comparison of retrieval quality. Our set retriever may also complement LLM-based multi-step reasoning pipelines by supplying more informative initial evidence sets.


W2 / Q1. Does ParaSet contribute useful candidate-set selection?

We thank the reviewer for proposing a comparison with random candidate sets. We conducted exactly this diagnostic, matching the first-stage top-20 passage pool, the number of proposals (20), the proposals' cardinalities, and the frozen SetCE reranker. The resulting improvements are reported in our General Response (Ablation Study of ParaSet) and new Appendix N.

ParaSet-generated proposals achieve higher mean F5 and Perfect Recall than set-size-matched random proposals on all three datasets with Qwen3-Embedding-0.6B at kb=20k_b=20 and kp=20k_p=20. The standalone ParaSet performance (without SetCE) is also included in Appendix N's controlled pooling comparison, where learned aggregation outperforms uniform pooling at kb=50k_b=50 with Qwen3-Embedding-0.6B. This directly demonstrates the effectiveness of ParaSet's proposal content under the same SetCE candidate budget.

[Edit: Clarification on SetCE evaluation budget]

We would also like to emphasize that ParaSet→SetCE actually evaluates fewer candidate sets with the expensive SetCE model than standalone SetCE beam search, rather than benefiting from more SetCE evaluations.

Specifically, with kb=20k_b=20 and beam width B=2B=2, standalone SetCE evaluates 20 singleton sets and 37 distinct two-passage sets within just the first two search depths, already requiring 57 SetCE set evaluations, with further evaluations at subsequent depths. In contrast, ParaSet→SetCE performs candidate exploration using the lightweight ParaSet scorer and evaluates only kp=20k_p=20 proposed sets with SetCE in total.

Therefore, our framework does not gain an advantage by allocating more expensive cross-encoder evaluations to candidate exploration. Instead, it decouples broad, inexpensive set exploration from selective, expressive SetCE reranking.

Dear Reviewer,

We would like to kindly follow up on our rebuttal response. We have provided additional experiments and clarifications to address your concerns, and we would greatly appreciate your feedback on whether these sufficiently resolve the issues you raised.

If any concerns remain, we would be happy to provide further clarification during the discussion period.

Thank you again for your time and thoughtful review.

Reviewer_erscOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a set-level retrieval framework for multi-hop question answering, addressing the limitations of conventional passage-level and sequential retrieval methods in capturing dependencies among evidence passages. Instead of scoring passages independently or relying on locally supervised next-passage decisions, the authors learn a query–set compatibility function that directly evaluates complete evidence sets. The model is trained using structured negative perturbations and a margin-based ranking objective that prioritizes evidence completeness. To efficiently search the combinatorial set space, the authors introduce a two-stage framework consisting of ParaSet, a lightweight scorer for candidate-set exploration, and SetCE, an expressive cross-encoder for set reranking. Experiments on three multi-hop QA benchmarks demonstrate improvements in downstream QA performance, particularly when combining set-level and passage-level retrieval, while achieving competitive efficiency compared with LLM-assisted retrieval methods.

Critical strengths
  1. The paper addresses an important limitation of existing retrieval methods by directly learning query–set compatibility rather than relying on independent passage scores or locally supervised sequential decisions. This formulation is particularly relevant to multi-hop QA, where evidence usefulness depends on relationships among passages.

  2. The proposed ParaSet→SetCE architecture balances computational efficiency and scoring expressiveness. ParaSet reuses precomputed passage representations for efficient candidate exploration, while SetCE performs more expressive reranking. This design makes searching a combinatorial evidence space computationally practical.

  3. The structured negative perturbations and fine-grained ranking objective provide supervision over complete, incomplete, and noisy evidence sets without requiring additional annotations. The shared objective can also be applied to different scoring architectures.

Critical weaknesses
  1. ParaSet uses attention-weighted pooling of independently encoded passages without explicit passage–passage interactions. It is unclear whether this design captures the complex evidence dependencies motivating the paper. Comparisons with simpler pooling methods and more expressive set encoders are needed.

  2. The addition-only beam search with a narrow beam width (B = 2) cannot revise earlier selections through deletion or replacement. Early pruning may also discard promising evidence combinations, limiting the claimed advantage over sequential retrieval.

  3. The contributions of some core design like recall-first ordering is not isolated. Ablations against other set-scoring architectures are also missing, leaving the effectiveness of individual design choices unclear.

  4. The paper discusses several closely related methods, including SetR, ARM, and OptiSet, but does not include direct comparisons with their original implementations in the main results table.

Questions for authors
  1. How does ParaSet model cross-passage dependencies without explicit passage–passage attention?
  2. Can the authors provide separate evaluations on other set-based retrieval method and demonstrate the advantage of the method design in paper?
4 comments

We sincerely thank the reviewer for the thoughtful and insightful feedback. We particularly appreciate the four suggestions regarding (W1) comparisons with alternative ParaSet architectures, (W2) our beam-search algorithm, (W3) recall-first set-quality ordering, and (W4) original implementations of LLM-based set retrieval methods. Each has helped us strengthen the experimental evidence or clarify an important aspect of the approach.

We are pleased to summarize our responses as follows:

  1. ParaSet architecture: Our additional controlled ablation shows that ParaSet better captures useful evidence-set dependencies than a comparable uniform-pooling baseline. Its learned proposals also outperform set-size-matched random selection.

  2. Beam-search clarification: We acknowledge that our addition-only expansion does not explicitly support deletion or replacement. However, our search retains all evaluated sets, including candidates excluded from subsequent expansion, for final selection without additional set-scoring evaluations beyond those required by the beam search. Hence earlier, smaller, or unsuccessful expansion candidates can still be selected when their global set scores are higher.

  3. Recall-first ordering: We additionally evaluate precision-first and F1-based set orderings, and find that the original recall-first ordering performs best in most settings, supporting the choice made in the paper.

  4. LLM-based comparisons: We investigated the suggested original implementations; code/checkpoint availability and issues in released implementations prevented complete reproduction within the rebuttal period. We nevertheless provide an additional comparison using the API-based GPT-4o-mini model for the SetR-CoT&IRI-style baseline, under which our approach remains competitive.


W1 / Q1. Cross-passage dependencies without explicit passage–passage attention

| It is unclear whether this design captures the complex evidence dependencies motivating the paper / How does ParaSet model cross-passage dependencies without explicit passage–passage attention?

Thank you for the opportunity to clarify the distinction between independently encoded passage features and independent passage scores. ParaSet does not score each passage as an independent scalar and sum those scores. It gathers their vector representations using learned attention aggregation and applies a nonlinear readout to the resulting set representation. As a result, adding a passage can affect the final score differently depending on the other passages already present, enabling non-additive preferences over evidence sets without explicit pairwise passage attention.

As shown in our General Response (Ablation Study of ParaSet) and new Appendix N, ParaSet improves retrieval quality over uniform pooling. Its generated proposals also consistently outperform set-size-matched random proposals with the same SetCE. These controlled findings provide direct empirical support for the effectiveness of its set-scoring architecture and exploration behavior.

W2. Addition-only beam search and earlier selections

| The addition-only beam search with a narrow beam width (B = 2) cannot revise earlier selections through deletion or replacement. Early pruning may also discard promising evidence combinations, limiting the claimed advantage over sequential retrieval.

We appreciate the observation that addition-only expansion cannot explicitly revise a retained hypothesis by deletion or replacement. We would like to clarify an important property of our procedure.

Our beam search distinguishes between candidates retained for further expansion and candidates considered for final selection. At each depth, the active beam retains only the highest-scoring sets for subsequent expansion. However, all evaluated candidate sets, including those excluded from the active beam, are preserved in a separate accumulated pool. The final evidence set is selected from this entire pool, rather than exclusively from the last beam.

This distinction is particularly meaningful because our global set-scoring function directly evaluates complete evidence sets without imposing monotonicity across search depths. For example, consider three nested sets, Sℓ⊂Sℓ+1⊂Sℓ+2S_\ell \subset S_{\ell+1} \subset S_{\ell+2}. It is possible that s(q,Sℓ)>s(q,Sℓ+1)s(q,S_\ell) > s(q,S_{\ell+1}), while s(q,Sℓ+2)>s(q,Sℓ)s(q,S_{\ell+2}) > s(q,S_\ell). Our scorer can directly reflect these differences in set compatibility, rather than relying on accumulated scores from local extension decisions. Importantly, every set that has been evaluated remains eligible for final selection, regardless of whether its subsequent expansions improve or degrade its score.

This capability requires no additional set-scoring evaluations beyond those already performed during beam-search exploration. By retaining and reusing previously evaluated candidates, our method can select earlier, smaller, or unsuccessful expansion candidates without additional search operations.

While addition-only expansion does not explicitly support deletion or replacement, and active-beam pruning still limits which combinations can be explored, these are potential directions for further extending our search algorithm. Our current framework already demonstrates the practical benefit of global set scoring: as shown in our original experiments, the proposed ParaSet$\rightarrow$SetCE pipeline outperforms both independent top-kk passage retrieval and locally supervised sequential ListCE in downstream QA.

We will further emphasize this distinction in the revised manuscript to clarify how global set scoring enables more flexible final-set selection within an efficient beam-search procedure.


W3. Recall-first set-quality ordering

| The contributions of some core design like recall-first ordering is not isolated.

Following the reviewer's suggestion, we compared recall-first, precision-first, and F$_1$-based preferences using Qwen3-Embedding-0.6B as the first-stage retriever, with the other training/evaluation configurations held fixed. The downstream QA EM results (SetCE and ParaSet → SetCE)were averaged across three runs.

DatasetPrecision-firstRecall-first (ours)F$_1$-based
HotpotQA34.0 / 38.637.9 / 38.837.7 / 38.8
2Wiki24.5 / 16.226.5 / 27.325.9 / 19.7
MuSiQue12.0 / 11.811.8 / 11.912.2 / 11.7

Downstream QA EM (%). Each entry is SetCE / ParaSet→SetCE. New Appendix O contains the same comparison.

While alternative set orderings also achieve competitive results, recall-first achieves the strongest or tied-strongest performance in most comparisons. These findings suggest that set-level preference learning is useful under different ordering strategies, while prioritizing the recovery of complete supporting evidence provides the most effective supervision overall for our multi-hop retrieval setting. This additional ablation further supports the recall-first design adopted in our original paper.

W4 / Q2. Comparisons with original LLM-based set retrieval implementations

| The paper discusses several closely related methods, including SetR, ARM, and OptiSet, but does not include direct comparisons with their original implementations in the main results table / Can the authors provide separate evaluations on other set-based retrieval method and demonstrate the advantage of the method design in paper?

We thank the reviewer for suggesting direct reproduction of SetR, ARM, and OptiSet. We investigated their available implementations and attempted to reproduce the comparisons under our setting. For OptiSet, publicly implementation resources are not available yet. For SetR, the released code does not include the trained checkpoints; reproducing the full method would require generating training responses using GPT and then fine-tuning the LLM, a pipeline difficult to complete within the rebuttal period. For ARM, we have been working to reproduce the method following its official repository. However, we encountered execution issues that require additional debugging, making it difficult to complete the reproduction within the rebuttal period.

To provide additional evidence in the meantime, we extend our SetR-CoT&IRI-style evaluation using GPT-4o-mini, an API-based closed-source model, in place of the 8B-scale LLM used in the earlier baseline.

MethodHotpotQA2WikiMuSiQue
SetR-CoT&IRI-style (GPT-4o-mini)40.129.79.1
Our combined retrieval (Qwen3-8B)42.229.66.0

Our retrieval pipeline remains competitive under this additional LLM-based evaluation, achieving higher answer EM on HotpotQA and comparable performance on 2Wiki, without invoking an LLM during evidence retrieval.

Beyond answer accuracy, we believe that achieving competitive performance at substantially lower computational cost is itself an important advantage of our approach. As reported in our original evaluation (Table 2), our method achieves approximately 5.0× faster evidence selection and 4.4× lower peak GPU memory usage than the 8B-based SetR-CoT&IRI-style baseline.

Furthermore, the lightweight nature of ParaSet enables flexible combinations of retrieval models. For example, multiple ParaSet models could independently generate candidate evidence sets in parallel, with their combined proposals subsequently reranked by a single SetCE model. Such a design offers a natural way to exploit complementary retrieval signals while retaining the efficiency advantages of lightweight set scoring, without introducing LLM inference into the retrieval stage.

We believe these results highlight an important contribution of our framework: learned set-level retrieval can achieve competitive performance with LLM-based approaches while offering substantially greater computational efficiency and flexibility for combining multiple retrieval models.

Dear Reviewer,

We would like to kindly follow up on our rebuttal response. We have provided additional experiments and clarifications to address your concerns, and we would greatly appreciate your feedback on whether these sufficiently resolve the issues you raised.

If any concerns remain, we would be happy to provide further clarification during the discussion period.

Thank you again for your time and thoughtful review.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response — Ablation Study of ParaSet

We sincerely thank the reviewers for their insightful suggestions regarding the effectiveness of ParaSet. Following these suggestions, we conducted two additional controlled ablations: (A) learned attention aggregation versus uniform pooling, and (B) learned ParaSet proposals versus set-size-matched random proposals passed to the same SetCE reranker.

We are pleased to report two complementary findings. ParaSet achieves higher retrieval performance than uniform pooling at kb=50k_b=50 with Qwen3-Embedding-0.6B, and ParaSet-generated proposals consistently yield better final evidence sets than set-size-matched random proposals under the same SetCE proposal budget. We have incorporated the detailed setups and results into the revised paper (new Appendix N, Ablation Study of ParaSet).

A. ParaSet versus Uniform Pooling

To evaluate the contribution of ParaSet's attention-based aggregation, we replace it with uniform averaging while preserving the same frozen first-stage bi-encoder representations, set-level preference training examples and objective, optimization exposure, nonlinear readout, and search settings. The results are averaged across three independent training runs. For retrieval, we use Qwen3-Embedding-0.6B, fix kb=50k_b=50, beam width B=2B=2, and maximum set size Tmax⁡=20T_{\max}=20; SetCE is not used. Each scorer returns its own highest-scoring visited set.

Retrieval performance (%), mean ± standard deviation across three seeds; 500 evaluation queries per dataset. PR means Perfect Recall (Perfect Recall means that the whole gold set is included in the retrieval.).

DatasetUniform F$_5$ParaSet F$_5$Uniform PRParaSet PR
HotpotQA36.13 ± 1.1140.01 ± 0.3824.67 ± 1.7028.93 ± 1.45
2Wiki35.35 ± 0.5239.16 ± 0.2714.00 ± 0.5318.40 ± 0.80
MuSiQue24.87 ± 0.6230.09 ± 1.865.40 ± 0.358.53 ± 1.47

ParaSet obtains higher mean F$_5$ and PR on all three datasets at kb=50k_b=50 with Qwen3-Embedding-0.6B. For example, on 2Wiki the mean PR improves from 14.00% to 18.40%. This experiment supports the benefit of learned attention aggregation over a comparable uniform-pooling architecture under this fixed candidate-pool setting.

B. ParaSet Proposals versus Randomly Selected Proposals

We next test whether ParaSet actively identifies useful evidence combinations rather than merely providing SetCE with many candidates. Both methods use the same top-2020 first-stage passages, provide exactly 20 unique proposed sets per query, and use the same frozen SetCE for final selection. The random baseline preserves the set size of each ParaSet proposal and randomly resamples passage identities from that pool, without gold information. Thus, both proposal count and set sizes are matched.

Retrieval performance of the final SetCE-selected set (%), mean ± standard deviation over three trained ParaSet seeds. Both variants use the same Qwen3 first-stage top-20 passages, frozen SetCE, and 20 set-size-matched proposals per query.

DatasetRandom→SetCE F$_5$ParaSet→SetCE F$_5$Random→SetCE PRParaSet→SetCE PR
HotpotQA25.26 ± 0.9254.64 ± 0.5311.52 ± 0.9236.47 ± 1.01
2Wiki39.36 ± 4.4348.96 ± 0.9221.23 ± 4.6727.33 ± 5.06
MuSiQue26.50 ± 1.7735.51 ± 0.9710.32 ± 1.5510.80 ± 1.00

ParaSet-generated proposals outperform set-size-matched random proposals on mean F$_5$ and PR across all three Qwen3-Embedding configurations. The gains are especially pronounced on HotpotQA (PR 11.52% → 36.47%). With kb=20k_b=20, the search space contains 220−1=1,048,5752^{20}-1=1{,}048{,}575 nonempty subsets, yet SetCE receives only 20 candidate sets. These results show that ParaSet's learned proposal content is valuable independently of the number and sizes of candidate sets passed to SetCE.

Taken together, these experiments provide direct evidence for ParaSet's lightweight learned aggregation and informed candidate-set exploration. Full methodological details and all three Qwen3-Embedding dataset comparisons for each ablation are in revised Appendix N.

AuthorsOpenReview ↗

Summary of Author Responses and Additional Experimental Results (1/3)

We sincerely thank all reviewers for their thoughtful and constructive feedback. As the discussion period concludes, we would like to summarize the key strengths recognized in the reviews and the additional evidence provided during our rebuttal.


1. Key Strengths Recognized by the Reviewers

We are particularly encouraged that the reviewers recognized several important aspects of our work:

  • Novel problem formulation and set-level learning (ersc): The reviewer highlighted the importance of directly learning query–set compatibility for multi-hop retrieval, where evidence usefulness depends on relationships among passages. The reviewer also recognized the value of our structured negative perturbations, which provide rich set-level supervision without requiring additional annotations.

  • Simple and original methodological design (hKch): The reviewer described our training approach as "simple and elegant" and ParaSet as a "relatively original architectural solution" for efficient set-level reranking. In particular, the reviewer appreciated the lightweight attention mechanism and reuse of cached bi-encoder representations, which enable efficient scoring of candidate sets.

  • Practical effectiveness and computational efficiency (W58t): The reviewer recognized that useful evidence sets can contain passages with low individual relevance scores, motivating our set-level retrieval formulation. The reviewer also highlighted the practical value of combining inexpensive candidate exploration with expressive reranking, and noted that our full pipeline improves over passage-level CE across all six dataset–retriever configurations while returning fewer passages on average.

We believe these observations collectively highlight the central contribution of our work: introducing direct set-level compatibility learning as a practical retrieval paradigm, together with an efficient architecture for searching the combinatorial evidence-set space.

Building on these recognized strengths, we conducted several additional experiments to address the remaining concerns and further validate our methodological choices.


2. Additional Experimental Evidence

A. Validating ParaSet's architecture and candidate exploration (Appendix N)

We conducted two controlled ablations to isolate the contribution of ParaSet:

  • ParaSet vs. uniform pooling: Under matched training supervision and search settings, ParaSet achieves higher mean F5 and Perfect Recall across all three datasets with Qwen3-Embedding-0.6B at kb=50k_b=50.
  • ParaSet proposals vs. random proposals: With identical first-stage pools, proposal counts, set cardinalities, and the same frozen SetCE reranker, ParaSet-generated proposals achieve higher mean F5 and Perfect Recall across all three datasets.

Importantly, ParaSet→SetCE does not benefit from evaluating more sets with the expensive SetCE model. Instead, lightweight ParaSet performs candidate exploration, and only a limited number of proposals are passed to SetCE. Our random-proposal ablation matches this SetCE reranking budget.

These findings directly support both aspects of ParaSet's contribution: effective learned set aggregation and efficient identification of informative evidence combinations.

B. Validating set-level learning and recall-first supervision (Appendices O and P)

We further investigated the effectiveness of our set-level learning formulation:

  • Recall-first ordering: We compared recall-first, precision-first, and F1-based preferences. While alternative orderings also achieve competitive results, our original recall-first ordering achieves the strongest or tied-strongest downstream QA performance in most comparisons.
  • Matched-pool ListCE vs. SetCE: Under identical first-stage candidate pools (kb=50k_b=50, Qwen3-Embedding-0.6B) and inference procedures, SetCE achieves higher F5 and Perfect Recall across all three datasets. On HotpotQA, F5 increases from 29.63 to 33.76, and Perfect Recall increases from 12.20% to 18.60%.

These results provide further empirical support for learning compatibility over complete evidence sets, rather than relying exclusively on locally supervised next-passage decisions.

AuthorsOpenReview ↗

Summary of Author Responses and Additional Experimental Results (2/3)

C. Confirming experimental reliability across independent runs (Appendix Q)

Following the reviewers' suggestions, we repeated model training and downstream QA evaluation across three independent runs, reporting mean and standard deviation.

The primary performance trends are consistently reproduced:

  • ParaSet→SetCE achieves higher mean QA EM than passage-level CE in all six dataset–retriever configurations.
  • SetCE achieves higher mean QA EM than ListCE in all six configurations.

These additional results strengthen the empirical reliability of our original findings and demonstrate that the observed advantages of set-level retrieval persist across independent training runs.

D. Clarifying evaluation protocols and LLM-based comparisons (Appendices M, L, and R)

We clarified that our experiments use dataset-wide, deduplicated retrieval corpora, comprising 509,315 passages for HotpotQA, 430,225 for 2WikiMultiHopQA, and 117,534 for MuSiQue, rather than small question-specific distractor pools.

We also explained important differences in retrieval protocols and answer-model training relative to Beam Retrieval, Q-RAG, and Search-R1.

For LLM-based set retrieval, we investigated the available implementations of SetR, ARM, and OptiSet and clarified the limitations encountered in reproducing their original systems. We additionally extended our SetR-CoT&IRI-style comparison using GPT-4o-mini, showing that our method remains competitive with an LLM-based evidence-selection approach without invoking an LLM during retrieval.

Regarding computational efficiency, we explicitly distinguish retrieval-stage and end-to-end measurements. Relative to IRCoT, our framework achieves:

  • 14.7× faster evidence selection, and
  • 3.15× faster end-to-end QA.

These results emphasize the practical value of our framework as a lightweight and reusable retrieval component, particularly for iterative retrieval, evidence verification, and systems with efficient downstream generators.

E. Clarifying the advantages of global set scoring during search

We further clarified an important property of our beam-search algorithm: all evaluated candidate sets remain eligible for final selection, including those excluded from subsequent beam expansion.

Because our global compatibility function directly compares evidence sets of different cardinalities, earlier or smaller sets can still be selected when subsequent expansions are less compatible. This requires no additional set-scoring evaluations beyond the existing search traversal.

While the addition-only search procedure remains approximate, this property illustrates how global set scoring enables flexible final-set selection without relying on accumulated local extension scores.

AuthorsOpenReview ↗

Summary of Author Responses and Additional Experimental Results (3/3)

3. Concluding Remarks

Taken together, the original results, the strengths identified by the reviewers, and the additional rebuttal experiments provide complementary evidence for three central contributions:

  1. A new retrieval formulation: Direct query–set compatibility learning enables evaluation of complete evidence sets, addressing limitations of independent passage relevance and locally supervised sequential retrieval.

  2. An effective and efficient set retrieval framework: ParaSet enables lightweight exploration of a combinatorial evidence-set space, while SetCE provides expressive reranking. Our additional architecture and proposal ablations directly validate the contributions of this design.

  3. Practical benefits beyond standalone retrieval accuracy: Set-level compatibility provides a complementary signal to passage-level relevance, improving downstream QA while enabling computationally efficient evidence selection without LLM inference during retrieval.

We believe the additional experiments substantially strengthen the empirical foundation of our paper and reinforce the value of direct set-level scoring as an effective, efficient, and reusable retrieval paradigm.

We sincerely thank the reviewers again for their constructive feedback, which helped us clarify our contributions and strengthen the experimental evidence. Detailed results are available in our General Response, individual reviewer responses, and revised manuscript (Appendices M–R).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.