- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Current LLM evaluations heavily rely on pairwise judges, AUC, Elo scores, and win rates. However, these metrics are not equivalent to whether the final selected set preserves diverse and valid content. This paper elaborates on the underlying causes, presents an actionable decomposition framework, and demonstrates various failure modes through controlled tasks and the Natural-50 benchmark.
Critical strengths
The framework is clearly laid out, the experimental design includes proper controls, metric definitions are rigorous, and the sensitivity analysis is comprehensive.
Critical weaknesses
The work exhibits limited external validity, as the Natural-50 benchmark is relatively small in scale. In addition, the paper lacks comparisons against strong set-level baseline methods.
Questions for authors
reference Weaknesses
3 comments
Thank you for your response. After considering it, I have decided to maintain my original score.
Thank you very much for your time, thoughtful feedback, and continued consideration of our work. We sincerely appreciate your constructive comments.
If you have any further questions, concerns, or suggestions, or if any aspect of our work would benefit from additional clarification or evidence, please feel free to let us know. We would be more than happy to provide further details and will do our utmost to address them thoroughly.
Thank you again for your valuable feedback and engagement.
Expanded Evaluation and Diagnostic Evidence
Thank you for recognizing the framework and experimental rigor. We address the sample-size and strong-baseline concerns with expanded evaluation and controls tracing pairwise evidence to set utility.
1. Expanded complete-set evaluation (Natural200)
Natural200 contains 200 queries and 1,200 responses, with six responses per query. Five are unchanged generations from MoDA’s public Infinity-Chat-derived cache (
yuancarrieyjy/tulu3_infinite-chats-taxonomy_mix_10k_no_thinking), and one is the historical Infinity-Chat first response.Two annotators independently labeled each query, followed by adjudication. The human reference and primary protocol were frozen before scoring the new test sets. Four exclusions under the frozen rules leave 196 eligible queries, 193 with nontrivial selection space at K=3, compared with 12/30 in the original Natural-50 final test. This proportion describes the constructed candidate sets. We average all 720 orders within each query before query-level inference and enumerate all 20 three-response subsets to obtain each query’s same-budget ceiling.
2. Strong set-level baselines
We added farthest-first (FF), facility-location, length, and representative-selection controls. The frozen Natural200 analysis includes 24 combinations of six scoring interfaces and four selectors, plus First-K and Longest-K. The primary metric, Avoidable Coverage Loss (pp), is mean attainable coverage minus mean selected coverage.
Direct set-level selection reduces loss relative to order- and length-based selection. Jaccard/facility-location still incurs 8.53 pp of loss against the exhaustive same-budget optimum (95% CI [7.04, 10.15]). This quantifies omissions of content already available among the candidates.
Separately, post-hoc equal-quality k-DPP MAP comparisons cover the original controlled tasks and Natural-50. With Qwen3-U, k-DPP and FF reach zero loss in every tested controlled condition. On Natural-50, Jaccard/k-DPP loses 5.31 pp, compared with 7.13 for order-averaged FF and 8.23 for content-tied facility-location. The paired coverage difference between k-DPP and FF is 1.82 pp (95% CI [-0.94, 5.15]). We use a regularized PSD kernel and exact fixed-K MAP selection. These comparisons are reported separately from Natural200.
3. Information and mechanism diagnostics
We added post-hoc content-aware controls to examine what changes when selection has explicit functional-content information. DeepSeek-V4-Pro predicts validity and functional units from the query and six responses, without test human labels as inputs. Greedy and exact selectors share these predictions and tie rules. Greedy selects maximum marginal predicted content, while exact enumerates budget-feasible subsets. Both filter predicted-invalid responses and may select fewer than K. These are additional-information diagnostics, using richer inputs than pairwise-only selectors.
On Natural200, greedy and exact incur 7.98 and 7.83 pp loss. Exact improves coverage over First-K by 13.77 pp (descriptive paired 95% CI [10.92, 16.69]) and over Longest-K by 4.96 pp [2.23, 7.84]. Comparisons with Jaccard/facility-location and shared-prediction greedy have paired intervals including zero. Sharing predictions allows us to assess the contribution of search complexity under fixed content information.
In 48 exploratory controlled tasks with multi-unit overlap, exact content selection reduces loss by 8.68 pp relative to identically validity-filtered facility-location (descriptive 95% CI [6.25, 11.11]). Shared-prediction greedy matches exact. Provenance controls also affect the comparison. Requiring retention of the historical response reduces facility-location’s coverage advantage over matched First-K to 0.42 pp (95% CI [-0.47, 1.32]). These findings are condition-specific.
4. What the expanded evaluation reveals
Intended-Use Alignment tests whether pairwise scores capture use-relevant distinctions. Fixed-Threshold Transfer asks whether development-calibrated decisions hold on new sets. Selection Compatibility traces how grouping and budgeted selection affect retained alternatives. The added direct selectors and information controls extend these checks, while query-level coverage and same-budget ceilings establish whether differences lead to avoidable omissions.
Together, the expanded natural evidence and complementary baselines reveal when simple selection is sufficient, where losses occur, and how scoring information interacts with downstream selection. They provide a more comprehensive and traceable basis for diagnosing where pairwise evaluation succeeds, where downstream selection fails, and whether those failures result in avoidable loss of valid content.