acceptodds
Under review as a conference paper at ICLR 2027

Beyond Relevance-Centric Retrieval: Rubrics-Oriented Document Set Selection and Ranking

11 views

Abstract

In RAG and deep research, answer quality depends on the entire evidence set a model obtains, and a top-k of individually relevant documents does not necessarily constitute a set that satisfies a complex information need. Yet evaluation has unfolded along the single axis of relevance, with discounted aggregates such as nDCG collapsing set quality into a sum of per-document relevance. Redundancy, conflict, and complementarity, properties that surface only at the level of the set, therefore never enter the measurement, though they are precisely what determines whether one set is better than another. We introduce SetwiseEvalKit, a document set evaluation benchmark organized as nine rubric dimensions across three levels, spanning both short-form and long-form scenarios and built from approximately 28K query-specific rubrics. A systematic evaluation of 12 rerankers yields three diagnoses: the strongest method covers fewer than 46% of the rubrics; the cross-document coordination dimensions are uniformly the weakest; and no single method leads in both scenarios. These diagnoses characterize what current methods lack, yet leave open whether the nine dimensions can conversely guide selection. We probe this with Rubric4Setwise, an exploratory configuration that supplies query-specific rubrics as selection criteria to Qwen3-8B. Without any fine-tuning, the resulting sets outperform many purpose-trained rerankers on downstream generation, draw on fewer documents and search rounds, and rank among the strongest in both scenarios. This corroborates the validity of the nine-dimension design and suggests that rubrics can pass from criteria for judgment into signals for selection, a direction that merits further study.

Primary area:
datasets and benchmarks
Keywords:
RAG, Deep Research, Setwise Evaluation, Rubric

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZuNMOpenReview ↗
Rating
1
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SetwiseEvalKit, a set-level document evaluation framework for RAG and Deep Research designed to overcome the limitations of traditional relevance-based evaluation. The authors define nine evaluation dimensions across three levels—including Relevance, Complementarity, Redundancy, Conflict, Completeness, and Reachability. Leveraging ~28K query-specific rubrics, they evaluate 12 rerankers. Additionally, the authors present Rubric4Setwise, an exploratory method using rubrics to guide Qwen3-8B in document set selection, reporting superior downstream generation performance compared to several existing rerankers.

Critical strengths
  1. Categorizing document-set quality into three tiers and nine dimensions equips SetwiseEvalKit with fine-grained diagnostic capability. For instance, Complementarity isolates information overlap, while Reachability evaluates whether the evidence set supports a complete reasoning chain. This nuanced breakdown provides richer analytical utility than a single aggregated relevance score.

  2. The evaluation covers traditional ad-hoc reranking, reasoning-enhanced reranking, and setwise reranking across single-turn RAG and multi-turn deep research. The authors complement automated scoring with human quality audits of the rubrics, LLM judge stability checks, and downstream generation evaluations, reinforcing the empirical scope of the benchmark.

Critical weaknesses
  1. The framework aims to measure how well a document set supports downstream generation, yet certain dimension definitions conflict with this goal.Most notably, Appendix D.6 defines Complementarity: when a single document contains all necessary information to produce a correct answer, Complementarity can still be scored as 0 due to a lack of cross-document division of labor. Consequently, a complete, accurate, and concise single-document set gets penalized for lacking multi-document synergy.Furthermore, equal weighting across all nine dimensions lacks justification. Is the benchmark rewarding document sets that truly improve answer generation, or merely enforcing the authors' preconceived "ideal set structure"? This directly threatens construct validity.

  2. The authors validate the benchmark via correlation between Rubric Coverage and downstream generation quality, reporting Pearson r=0.92r = 0.92 (Figure 4). However, this correlation is calculated on method-level aggregated results, not on individual query-document set pairs.Aggregated correlation fails to prove:Higher rubric scores for a specific query yield better answers.Nine-dimensional metrics predict downstream performance better than traditional relevance metrics.Set-level dimensions add independent predictive value beyond relevance.Incremental predictive validity over traditional relevance baselines remains unproven.

Questions for authors
  1. Rubric4Setwise uses rubrics derived from reference answers unavailable to other rerankers. Can the authors include a baseline utilizing reference answers (or key facts) without the nine-dimensional rubric? Otherwise, how do you isolate rubric-oriented selection gains from simple answer leakage?

  2. If a single document fully and accurately answers a query, why can its Complementarity score be 0 (Appendix D.6)? Does this design systematically penalize compact, high-quality, low-redundancy document sets? How do you justify equal-weight averaging across all nine dimensions?

  3. Figure 4 presents method-level aggregated correlations. Can you evaluate query-level predictive power for nDCG, Relevance-only scores, and Nine-dimensional Rubric scores against downstream answer quality? If the nine-dimensional score does not significantly beat a relevance-only baseline, what is its core value over traditional evaluation?

2 comments
AuthorsOpenReview ↗

Summary of Our Rebuttal to Reviewer ZuNM

We sincerely thank you for your constructive feedback. In response to your concerns, we conducted additional experiments and provided clarifications:

  • (1) Answer leakage (Q1): We added an answer-only baseline that gives the same reference answer to the selector without any rubric. The query-answer rubric outperforms this baseline, showing that the gain comes from rubric guidance rather than answer access.
  • (2) Complementarity and equal weighting (W1/Q2): We clarified that Complementarity measures only the complementary structure among documents, and that single-document selections are rare (0% for Rank4Gen, 3.64% for SetR). Reranker rankings are almost unchanged under importance weighting, and removing any dimension lowers downstream performance, supporting equal weighting.
  • (3) Query-level value of nine dimensions (W2/Q3): At the query level, all nine dimensions outperform Relevance alone in every setting, and they pinpoint failure modes that relevance alone cannot reveal.

We hope these results address your concerns, and would sincerely appreciate it if you could reconsider your score in light of the new evidence.

Thank you again for your valuable time and feedback.

Thank you for the additional experiments, which address some of my secondary concerns. However, my main concern regarding construct validity remains unresolved.

First, the low frequency of single-document selections does not justify assigning a score of 0 to a complete, accurate single document. Including this penalty in the Overall score conflicts with the benchmark's stated objective of measuring how well a document set supports downstream generation. Second, stable rankings (Tab 3) and leave-one-dimension-out results (Tab 4) still do not justify assigning equal weights to all nine dimensions.

Given these unresolved issues in the metric design, I will maintain my original rating.

Reviewer_R4pBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces SetwiseEvalKit, a benchmark that evaluates retrieved document sets using query-specific rubrics across nine dimensions. It evaluates twelve rerankers and explores rubric-guided document selection. The work has value in organizing evaluation criteria and constructing a benchmark, but I just feel its conceptual novelty is limited, and the validity of the proposed metrics requires stronger evidence.

Critical strengths
  1. The problem is promising: it introduces SetwiseEvalKit, a benchmark that evaluates retrieved document sets using query-specific rubrics across nine dimensions.
  2. The benchmark covers both short-form QA and long-form research and evaluates twelve rerankers, providing a useful resource for studying retrieval quality across different settings.
Critical weaknesses
  1. Insufficient justification for the nine dimensions. The limitations of relevance-only evaluation are already well established. The paper should explain why these particular dimensions are necessary, how overlapping dimensions differ, and why their aggregate is more informative than simpler alternatives. Notably, Relevance alone correlates with downstream performance at (r=0.94), compared with (r=0.92) for the overall score. This does not demonstrate additional predictive value from the full framework.

  2. Inconclusive validation of set-level metrics. In Appendix B.3, none of the three set-level dimensions has a statistically significant correlation with downstream performance. The authors attribute this to limited variation among rerankers, but alternative explanations include insufficient coverage of relevant failure modes, weak scoring discrimination, or limitations of the metrics themselves. Controlled experiments introducing redundancy, missing evidence, and contradictions would help distinguish these possibilities.

  3. Unclear advantage over conventional LLM judging. “Rubric-Judge” is itself a form of LLM-as-Judge, making the distinction in Table 1 unclear. The key missing comparison is the same model scoring the nine generic dimensions directly versus scoring with generated query-specific rubrics, given identical queries, reference answers, and documents. The paper should establish whether rubric generation improves human agreement, stability, or diagnostic accuracy.

Small things:

  1. Missing rubric sensitivity analysis. How do scores and system rankings change when rubric requirements are removed, added, or reformulated? Which requirements are essential for RAG, and do they differ across tasks? Actual rubric ablations are needed; human estimates of the consequences of removing a rubric do not replace these experiments.
  2. Fonts in several figures and tables are too small, and some text has overly tight line spacing, particularly in Figure 6.
  3. Missing related work: Joren et al. Sufficient Context: A New Lens on Retrieval-Augmented Generation Systems. ICLR 2025. https://arxiv.org/abs/2411.06037
Questions for authors
  1. Why do you propose those nine dimensions and not others? Any specific principles? What is the relationship between those nine?
  2. The set-level dimensions are not significant correlation; are they really useful?
  3. What is the advantage over conventional LLM judging? “Rubric-Judge” is itself a form of LLM-as-Judge? What is the technical novelty here?
1 comment
AuthorsOpenReview ↗

Summary of Our Rebuttal to Reviewer R4pB

We sincerely thank you for your constructive feedback. In response to your concerns, we conducted additional experiments and provided clarifications:

  • (1) Rationale for nine dimensions (W1/Q1): We explained the three-level design principle and how the dimensions relate. Ablating to Relevance only shows that all nine dimensions outperform Relevance alone in every setting.
  • (2) Set-level dimensions (W2/Q2): Removing set-level dimensions, individually or together, lowers downstream performance in all settings, with the largest drop when all three are removed.
  • (3) Advantage over generic LLM judging (W3/Q3): The novelty lies in multi-level, multi-dimensional set evaluation, not in the rubric judge itself. Compared with a generic meta-rubric, query-specific rubrics are closer to human scores, more stable, and more diagnostic.
  • (4) Rubric sensitivity (W4): We removed each dimension and recomputed results. Downstream performance drops in all settings, and the selected sets change for 25–61% of queries.
  • (5) Presentation and related work (W5/W6): We added the suggested citation and discussion in the Introduction, and promise to enlarge fonts and loosen line spacing (including Fig. 6) in the next version.

We hope these results address your concerns, and would sincerely appreciate it if you could reconsider your score in light of the new evidence.

Thank you again for your valuable time and feedback.

Reviewer_45faOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces SetwiseEvalKit, a benchmark evaluating retrieved document sets through nine rubric dimensions spanning document quality, cross-document interactions, and overall evidence utility. It evaluates rerankers in short-form QA and long-form research settings. It also explores Rubric4Setwise, which provides answer-derived rubrics to an off-the-shelf model for document selection.

Critical strengths

Relevant problem: Document relevance alone does not capture redundancy, contradiction, or missing reasoning links in RAG evidence. Useful diagnostic framework: The nine dimensions provide interpretable explanations of retrieval failures across two complementary scenarios.

Critical weaknesses

Limited validation of the judge: The paper measures repeated scoring by the same model, but does not report agreement across different judges or direct alignment with human document-set scores. Human evaluation assesses rubric quality rather than scoring accuracy. Unclear incremental value over relevance: Relevance alone correlates strongly with downstream quality (r=0.94r=0.94), while several central set-oriented dimensions show no statistically significant correlation. The reported analyses do not establish that nine-dimensional aggregation improves prediction over relevance alone. Subjective omission analysis: The reported impact of removing rubrics reflects annotators’ expectations, rather than an actual leave-one-rubric-out experiment measuring ranking changes. Restricted interpretation of selection gains: Rubric4Setwise receives reference-answer information. Its gains therefore do not establish effectiveness under realistic deployment conditions.

Questions for authors

With fixed rubrics and document sets, how consistent are scores and reranker rankings across different judge models? How closely do automatic scores agree with independent human scores of document-set quality? Does the nine-dimensional aggregate predict downstream quality better than relevance alone? Could you provide an ablation or incremental prediction analysis? Have you actually removed individual rubrics and recomputed rankings, beyond asking annotators to estimate the effect? How does Rubric4Setwise perform with rubrics generated from the query alone, without access to the reference answer?

3 comments
AuthorsOpenReview ↗

Summary of Our Rebuttal to Reviewer 45fa

We sincerely thank you for your constructive feedback. In response to your concerns, we conducted additional experiments and provided clarifications:

  • (1) Judge validation (W1/Q1/Q2): We added two judge models and human scoring. Rankings are highly consistent across judges, and judge scores are close to human scores.
  • (2) Value of nine dimensions (W2/Q3/Q4): Ablating to Relevance only shows that all nine dimensions outperform Relevance alone in every setting.
  • (3) Actual rubric omission (W3/Q5): We removed each dimension and recomputed results. Downstream performance drops in all settings, and the selected sets change for 25–61% of queries.
  • (4) Reference-free setting (W4/Q6): Query-only and meta-rubric variants retain most of the gains without reference answers, staying well above all other rerankers.

We hope these results address your concerns, and would sincerely appreciate it if you could reconsider your score in light of the new evidence.

Thank you again for your valuable time and feedback.

Thanks for your reply,these additional results resolve my concerns.

Thank you for your thoughtful feedback and the time and effort you dedicated to reviewing our paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

We sincerely appreciate your time and insightful feedback. We are glad that reviewers 45fa(R1)\text{45fa(R1)}, R4pB(R2)\text{R4pB(R2)}, and ZuNM(R3)\text{ZuNM(R3)} recognized that our work tackles a relevant and promising problem (R1, R2\text{R1, R2}), that the nine-dimension framework provides fine-grained, interpretable diagnosis of document-set quality (R1, R3\text{R1, R3}), and that the benchmark has a broad evaluation scope, covering short-form QA and long-form deep research with twelve rerankers (R2, R3\text{R2, R3}).

The main concerns were the value of the nine dimensions, the reliability of the scoring, and the use of reference answers. We addressed each with new experiments and clarifications, summarized below. Point-by-point responses are posted under each review.

  • New experiments:
    • Nine dimensions vs. Relevance alone: all nine dimensions are better in every setting. (R1, R2, R3\text{R1, R2, R3})
    • Removing each dimension in turn: performance drops in every setting. (R1, R2, R3\text{R1, R2, R3})
    • Importance-weighted rankings: almost unchanged, supporting equal weighting. (R3\text{R3})
    • Reference-free variants and an answer-only baseline: reference-free variants retain most of the gains, and the full rubric beats the answer-only baseline. (R1, R3\text{R1, R3})
    • Two additional judge models: rankings are highly consistent across judges. (R1\text{R1})
    • Generic meta-rubric baseline: query-specific rubrics are closer to human scores and more stable. (R2\text{R2})
  • New human study:
    • Human scoring on 100 queries: judge scores are close to human scores. (R1, R2\text{R1, R2})
  • Clarifications and revisions:
    • Design principle of the nine dimensions, and the novelty over generic LLM-as-a-judge. (R2\text{R2})
    • Definition of Complementarity: single-document selections are rare, so the score-0 case has a negligible effect. (R3\text{R3})
    • Added the suggested related work, and we promise to enlarge fonts and loosen line spacing in the next version. (R2\text{R2})

We warmly encourage you to review our responses. We hope the new results and clarifications address your concerns, and we are happy to clarify any remaining questions.

Best regards,

Authors of Submission 16581

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.