- Rating
- 1
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SetwiseEvalKit, a set-level document evaluation framework for RAG and Deep Research designed to overcome the limitations of traditional relevance-based evaluation. The authors define nine evaluation dimensions across three levels—including Relevance, Complementarity, Redundancy, Conflict, Completeness, and Reachability. Leveraging ~28K query-specific rubrics, they evaluate 12 rerankers. Additionally, the authors present Rubric4Setwise, an exploratory method using rubrics to guide Qwen3-8B in document set selection, reporting superior downstream generation performance compared to several existing rerankers.
Critical strengths
-
Categorizing document-set quality into three tiers and nine dimensions equips SetwiseEvalKit with fine-grained diagnostic capability. For instance, Complementarity isolates information overlap, while Reachability evaluates whether the evidence set supports a complete reasoning chain. This nuanced breakdown provides richer analytical utility than a single aggregated relevance score.
-
The evaluation covers traditional ad-hoc reranking, reasoning-enhanced reranking, and setwise reranking across single-turn RAG and multi-turn deep research. The authors complement automated scoring with human quality audits of the rubrics, LLM judge stability checks, and downstream generation evaluations, reinforcing the empirical scope of the benchmark.
Critical weaknesses
-
The framework aims to measure how well a document set supports downstream generation, yet certain dimension definitions conflict with this goal.Most notably, Appendix D.6 defines Complementarity: when a single document contains all necessary information to produce a correct answer, Complementarity can still be scored as 0 due to a lack of cross-document division of labor. Consequently, a complete, accurate, and concise single-document set gets penalized for lacking multi-document synergy.Furthermore, equal weighting across all nine dimensions lacks justification. Is the benchmark rewarding document sets that truly improve answer generation, or merely enforcing the authors' preconceived "ideal set structure"? This directly threatens construct validity.
-
The authors validate the benchmark via correlation between Rubric Coverage and downstream generation quality, reporting Pearson (Figure 4). However, this correlation is calculated on method-level aggregated results, not on individual query-document set pairs.Aggregated correlation fails to prove:Higher rubric scores for a specific query yield better answers.Nine-dimensional metrics predict downstream performance better than traditional relevance metrics.Set-level dimensions add independent predictive value beyond relevance.Incremental predictive validity over traditional relevance baselines remains unproven.
Questions for authors
-
Rubric4Setwise uses rubrics derived from reference answers unavailable to other rerankers. Can the authors include a baseline utilizing reference answers (or key facts) without the nine-dimensional rubric? Otherwise, how do you isolate rubric-oriented selection gains from simple answer leakage?
-
If a single document fully and accurately answers a query, why can its Complementarity score be 0 (Appendix D.6)? Does this design systematically penalize compact, high-quality, low-redundancy document sets? How do you justify equal-weight averaging across all nine dimensions?
-
Figure 4 presents method-level aggregated correlations. Can you evaluate query-level predictive power for nDCG, Relevance-only scores, and Nine-dimensional Rubric scores against downstream answer quality? If the nine-dimensional score does not significantly beat a relevance-only baseline, what is its core value over traditional evaluation?
2 comments
Thank you for the additional experiments, which address some of my secondary concerns. However, my main concern regarding construct validity remains unresolved.
First, the low frequency of single-document selections does not justify assigning a score of 0 to a complete, accurate single document. Including this penalty in the Overall score conflicts with the benchmark's stated objective of measuring how well a document set supports downstream generation. Second, stable rankings (Tab 3) and leave-one-dimension-out results (Tab 4) still do not justify assigning equal weights to all nine dimensions.
Given these unresolved issues in the metric design, I will maintain my original rating.
Summary of Our Rebuttal to Reviewer ZuNM
We sincerely thank you for your constructive feedback. In response to your concerns, we conducted additional experiments and provided clarifications:
We hope these results address your concerns, and would sincerely appreciate it if you could reconsider your score in light of the new evidence.
Thank you again for your valuable time and feedback.