acceptodds
Under review as a conference paper at ICLR 2027

From Pairwise Evidence to Set Utility: A Controlled Meta-Evaluation of Homogeneity Measures for LLM Response Selection

Abstract

Selecting a small subset of large language model responses requires removing redundancy while preserving distinct valid alternatives. However, strong pairwise homogeneity prediction for the intended use does not guarantee good subset selection; even with perfect pairwise rankings, the selection pipeline can still incur avoidable coverage loss under a fixed selection budget. To explain this gap, we introduce a controlled meta-evaluation framework that holds responses fixed while varying their intended use across four task families with verifiable answers. We trace how decision thresholds fixed on development data interact with clustering and budgeted selection to shape final coverage. When evaluation pools different uses, a purpose-only control can achieve high AUC without inspecting response content, yet cannot distinguish alternatives within the target use. Input-order enumeration further shows that incorrect merges or splits reduce coverage only under certain repetition patterns, orderings, and budgets. On Natural-50, a collection of open-ended responses with human-annotated functional units, we evaluate coverage when retaining three of six responses per query. Transferred to 30 test queries, the pipeline selected by pooled AUC on controlled tasks incurs a mean per-query coverage shortfall of 8.36 percentage points relative to the optimum attainable from the same candidates under the same budget (95% CI: 1.94–16.03), with losses concentrated in six queries.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Response Selection, Homogeneity Evaluation, Pairwise Evaluation, Diversity Evaluation, Set Utility, Functional Coverage, Meta-Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_otHuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Current LLM evaluations heavily rely on pairwise judges, AUC, Elo scores, and win rates. However, these metrics are not equivalent to whether the final selected set preserves diverse and valid content. This paper elaborates on the underlying causes, presents an actionable decomposition framework, and demonstrates various failure modes through controlled tasks and the Natural-50 benchmark.

Critical strengths

The framework is clearly laid out, the experimental design includes proper controls, metric definitions are rigorous, and the sensitivity analysis is comprehensive.

Critical weaknesses

The work exhibits limited external validity, as the Natural-50 benchmark is relatively small in scale. In addition, the paper lacks comparisons against strong set-level baseline methods.

Questions for authors

reference Weaknesses

3 comments
AuthorsOpenReview ↗

Expanded Evaluation and Diagnostic Evidence

Thank you for recognizing the framework and experimental rigor. We address the sample-size and strong-baseline concerns with expanded evaluation and controls tracing pairwise evidence to set utility.

1. Expanded complete-set evaluation (Natural200)

Natural200 contains 200 queries and 1,200 responses, with six responses per query. Five are unchanged generations from MoDA’s public Infinity-Chat-derived cache (yuancarrieyjy/tulu3_infinite-chats-taxonomy_mix_10k_no_thinking), and one is the historical Infinity-Chat first response.

Two annotators independently labeled each query, followed by adjudication. The human reference and primary protocol were frozen before scoring the new test sets. Four exclusions under the frozen rules leave 196 eligible queries, 193 with nontrivial selection space at K=3, compared with 12/30 in the original Natural-50 final test. This proportion describes the constructed candidate sets. We average all 720 orders within each query before query-level inference and enumerate all 20 three-response subsets to obtain each query’s same-budget ceiling.

2. Strong set-level baselines

We added farthest-first (FF), facility-location, length, and representative-selection controls. The frozen Natural200 analysis includes 24 combinations of six scoring interfaces and four selectors, plus First-K and Longest-K. The primary metric, Avoidable Coverage Loss (pp), is mean attainable coverage minus mean selected coverage.

SelectorLoss (pp) ↓
First-K21.60
Longest-K12.79
Jaccard/FF9.40
Jaccard/facility-location8.53

Direct set-level selection reduces loss relative to order- and length-based selection. Jaccard/facility-location still incurs 8.53 pp of loss against the exhaustive same-budget optimum (95% CI [7.04, 10.15]). This quantifies omissions of content already available among the candidates.

Separately, post-hoc equal-quality k-DPP MAP comparisons cover the original controlled tasks and Natural-50. With Qwen3-U, k-DPP and FF reach zero loss in every tested controlled condition. On Natural-50, Jaccard/k-DPP loses 5.31 pp, compared with 7.13 for order-averaged FF and 8.23 for content-tied facility-location. The paired coverage difference between k-DPP and FF is 1.82 pp (95% CI [-0.94, 5.15]). We use a regularized PSD kernel and exact fixed-K MAP selection. These comparisons are reported separately from Natural200.

3. Information and mechanism diagnostics

We added post-hoc content-aware controls to examine what changes when selection has explicit functional-content information. DeepSeek-V4-Pro predicts validity and functional units from the query and six responses, without test human labels as inputs. Greedy and exact selectors share these predictions and tie rules. Greedy selects maximum marginal predicted content, while exact enumerates budget-feasible subsets. Both filter predicted-invalid responses and may select fewer than K. These are additional-information diagnostics, using richer inputs than pairwise-only selectors.

On Natural200, greedy and exact incur 7.98 and 7.83 pp loss. Exact improves coverage over First-K by 13.77 pp (descriptive paired 95% CI [10.92, 16.69]) and over Longest-K by 4.96 pp [2.23, 7.84]. Comparisons with Jaccard/facility-location and shared-prediction greedy have paired intervals including zero. Sharing predictions allows us to assess the contribution of search complexity under fixed content information.

In 48 exploratory controlled tasks with multi-unit overlap, exact content selection reduces loss by 8.68 pp relative to identically validity-filtered facility-location (descriptive 95% CI [6.25, 11.11]). Shared-prediction greedy matches exact. Provenance controls also affect the comparison. Requiring retention of the historical response reduces facility-location’s coverage advantage over matched First-K to 0.42 pp (95% CI [-0.47, 1.32]). These findings are condition-specific.

4. What the expanded evaluation reveals

Intended-Use Alignment tests whether pairwise scores capture use-relevant distinctions. Fixed-Threshold Transfer asks whether development-calibrated decisions hold on new sets. Selection Compatibility traces how grouping and budgeted selection affect retained alternatives. The added direct selectors and information controls extend these checks, while query-level coverage and same-budget ceilings establish whether differences lead to avoidable omissions.

Together, the expanded natural evidence and complementary baselines reveal when simple selection is sufficient, where losses occur, and how scoring information interacts with downstream selection. They provide a more comprehensive and traceable basis for diagnosing where pairwise evaluation succeeds, where downstream selection fails, and whether those failures result in avoidable loss of valid content.

Thank you for your response. After considering it, I have decided to maintain my original score.

Thank you very much for your time, thoughtful feedback, and continued consideration of our work. We sincerely appreciate your constructive comments.

If you have any further questions, concerns, or suggestions, or if any aspect of our work would benefit from additional clarification or evidence, please feel free to let us know. We would be more than happy to provide further details and will do our utmost to address them thoroughly.

Thank you again for your valuable feedback and engagement.

Reviewer_CrcROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors motivate their problem by asking if there was a pool of solutions and one had to deduplicate it, then do good pairwise equivalent or distinct scorers yield a good K-subset of LLM responses that retains the most distinct valid content under a budget of K. The paper is a meta-evaluation. The authors argue that equivalence depends on how the responses are being used. They introduce two notions (a) feasibility confirmation (which is more about equivalence based on the validity of responses) and (b) alternative-preservation (which is more about the equivalence of the strings/responses).

Their proposed pipeline has the following stages: (a) get a scorer that scores pairs (eg: Jaccard, TF-IDF, LLM as a judge etc), (b) find a threshold on the dev-set by maximizing the bacc (c)convert the pairs into groups / sets / clusters. They run their experiments on two datasets: (a) a controlled/synthetic set (b) Natural-50, a subset of infinichat.

The results show that pairwise measures that score quality (and then you calculate AUC) does not guarantee good/distinct subset selection. On the controlled synthetic tasks, the control baseline beats a perfect scorer on pooled AUC (a metric the authors introduce) while being at chance within the target use. Intended-use AUC loses a little coverage combining pairwise scores into clusters. On Natural-50, the pipeline selected by pooled AUC on the controlled tasks loses versus the attainable optimum, identical to the First-K baseline.

Critical strengths
  1. Careful evaluation of the problem
  2. A lot of details in the paper that make it reproducible and transparent
Critical weaknesses
  1. The problem is weakly motivated, the only main motivation the authors give is the artificial hivemind paper (in line #40). The introduction also has a lot of jargon that the authors use before really introducing those phrases (Eg: the notion of “units”/”functional unit” in line # 68, or functional coverage in line 69) and the definitions are then found deep in the appendix. This also makes the paper very dense and hard to follow.
  2. The “pooled AUC” metric in Section 3 is not grounded in any work. And the authors spend most of section 4 showing that it doesn’t work.
  3. The paper is about selecting distinct alternatives, which is only the alternative-preservation use. Feasibility is not includded in retention or coverage experiments (line #146 ). This is only used for pooled AUC.
  4. The test instances/pairs are very less (for the four tasks it is 128 instances and Natural-50 has 30 test queries). This is also an issue with the dev set, which is only 64 controlled dev cases and applied to different sets.
  5. The failures are treated as “transfer” problems. But balanced accuracy based thresholds on easy four tasks can be poorly calibrated, and on Natural-50 the controlled threshold is applied without any adaptation.
  6. Picking the earliest response per cluster in input order (line # 175-176) and averaging over permutations, builds order sensitivity into the method.
  7. The paper is missing a lot of citations eg: in 3.4 for NoveltyBench and Jaccard, TF-IDF, STS, NoveltyBench and Qwen3, Natural-50 is missing the dataset name.
  8. Figure 2 is adding no new information, it is just repeating what table 1 has done
Questions for authors
  1. Is there any evidence that practitioners or benchmarks select scorers by pooled AUC?
  2. Since feasibility is not included in retention, what does it add beyond enabling pooled AUC?
  3. Can you evaluate on more queries?
  4. How do results change with score-based selectors (farthest-first, MMR, DPP that you cite in the related works #104-105) on the controlled tasks, and with a representative rule that isn’t input-order-based?
3 comments
AuthorsOpenReview ↗

Motivation, Intended Use, and Set-Level Diagnostic Evidence

Thank you for the detailed review. Multiple LLM answers do not necessarily give users multiple useful options. Different wording may repeat one solution; novelty may be invalid; and interchangeability depends on the user's goal. This raises the first question: what counts as a meaningful alternative for the intended use?

A second question follows when only K answers can be shown. A repetitive final set may mean that no distinct valid options were generated, or that available options were discarded during selection. They require different remedies. We isolate selection-side loss by holding candidates fixed. Pairwise AUC measures ranking, not retained content; final coverage alone cannot identify the decisions behind omissions. We therefore trace thresholding, grouping, and selection, comparing the selected set with the best same-candidate, same-budget subset. This separates avoidable loss from budget limits within observed candidates and exposes intermediate failures.

1. Intended use and pooled AUC (Q1–Q2). NoveltyBench uses AUC to compare equivalence classifiers before grouping responses. We use cross-use pooling as a controlled aggregation diagnostic; its widespread adoption has not been established. Changing purpose weights moves Purpose-Only pooled AUC from 0.500 to 0.982, with target-use AUC fixed at 0.500 and coverage at 83.33% (complete linkage, K=2). Qwen3-P remains selected at all tested nonzero feasibility weights: aggregate credit shifts without improved target-use discrimination.

Feasibility matters independently of pooling: valid solutions can be interchangeable for confirming existence yet distinct when preserving options. On fully scored three-answer subsets of 128 tasks (two valid alternatives, one invalid), candidates and Qwen3-P/facility-location were held fixed across uses. At K=1, feasibility input selects a valid answer in 96.88% of tasks versus 84.38% under alternative input (paired +12.50 pp [6.25, 18.75]). At K=2, feasibility success saturates, while alternative input preserves 4.69 pp [1.56, 8.20] more valid alternatives; both trail First-K on alternative coverage. Thus intended use affects downstream selection outcomes independently of pooled AUC in these post-hoc subsets.

2. Expanded natural evidence (Q3). Natural200 has 200 six-response queries: 196 eligible under pre-scoring frozen rules, 193 with nontrivial K=3 selection space (versus 12/30 in Natural-50). Independent annotations were adjudicated. Averaging 720 orders within query, Jaccard/facility-location has mean Avoidable Coverage Loss of 8.53 pp [7.04, 10.15] relative to the exhaustive same-budget ceiling. Loss affects 128 queries; the largest two contribute 7.01% of total loss. Each set uses five MoDA cached generations and one historical Infinity-Chat answer; conclusions concern this construction.

3. Direct selectors, order, and calibration (Q4; W5–W6). On single-object controlled tasks, Qwen3-U has AUC=1 yet single linkage loses alternatives; farthest-first (FF) and equal-quality k-DPP MAP reach the ceiling. With identical Qwen3-P scores and K=3 on Natural-50, order-averaged losses are 8.05 pp for transferred single linkage, 5.89 for FF, 5.19 for facility-location, and 4.60 for k-DPP MAP. These descriptive values do not establish a stable ranking.

Holding partitions, cluster priority, and filling fixed, medoids give 8.29 pp loss versus 8.05 for earliest representatives (paired coverage difference -0.23 pp [-1.89, 1.45]). Canonical-text ordering removes order variation but still loses 8.69 pp: order stability does not ensure content retention. Constant-relevance MMR equals FF under matched initialization/ties; independent relevance scores were unavailable. Changing only to a natural-development threshold improves order-averaged coverage by 2.56 pp [0.09, 5.64] under the historical purpose input.

4. What the evaluation adds. Natural200 broadens evidence of omissions; Natural-50 illustrates their mechanisms. At the transferred threshold, 25/30 queries collapse into one cluster, explaining near-First-K behavior. With those partitions fixed, separate human-reference interventions on representative choice and empty-slot filling reduce fixed-order loss from 8.36 pp to 4.08 and 1.39 pp. These diagnostic effects are not additive. The three checks—Intended-Use Alignment, Fixed-Threshold Transfer, and Selection Compatibility—link meaningful differences, operating decisions, and retained content. Together they locate decisions affecting retained content, examine why omissions occur, and quantify avoidable loss against the same-budget ceiling.

We agree that earlier definitions, nearby citations, and a less redundant Fig. 2 would improve readability. Natural200's protocol was frozen before scoring; other new diagnostics are post-hoc, with descriptive query/task-bootstrap CIs.

AuthorsOpenReview ↗

Case-Level Evidence on Decision-Level Coverage Loss

Our November 14 response presented the motivation, aggregate results, and stage-level interventions. We now add trace-level evidence from all 30 frozen Natural-50 test queries to show what those checks reveal about actual selection decisions. The two cases below both end at 50% coverage, but differ in attainable coverage and recovery paths. All case results use the original fixed input order, K=3, and frozen human references.

Q026 — a recoverable omission. The six valid game-controller descriptions contain four human-annotated functional units under the use of preserving distinct visual content:

AnswerExact excerpt from frozen responseUnit
A1“two joysticks, a directional pad, and multiple buttons”U1
A2“symmetrical arrangement of thumbsticks, triggers, and colorful buttons”U1
A3“a glowing touchpad in the center”U2
A4“two symmetrical analog sticks on the left”U3
A5“Wires snake out from its base, connecting it to the console.”U4
A6“a directional pad, two analog sticks, and an assortment of buttons”U1

All 15 Qwen3-P pair scores exceed the frozen threshold of 0.49708748 (range: 0.534–1.000), forming one single-linkage cluster. The original selector takes A1 as representative and fills with A2 and A3. Because A1 and A2 repeat U1, the set retains 2/4 units (50%). Exhaustively checking all 20 three-response subsets gives a same-budget ceiling of 3/4 (75%): 25 percentage points (pp) are unavoidable under K=3, while another 25 pp are avoidable coverage loss (ACL) from this selection.

Matched diagnostic interventions (original fixed order):

InterventionQ026: coverage / ACLQ028: coverage / ACL
Original50% / 25 pp50% / 16.67 pp
Reference-relation correction75% / 0 ppN/A (partial overlap)
Representative-only oracle50% / 25 pp50% / 16.67 pp
Filling-only oracle75% / 0 pp50% / 16.67 pp
Joint representative + filling oracle75% / 0 pp66.67% / 0 pp
Exhaustive same-budget optimum75% / 0 pp66.67% / 0 pp

In Q026, correcting unambiguous reference relations yields four clusters—{A1,A2,A6}, {A3}, {A4}, {A5}—and selects A1/A3/A4, reaching 75%. Alternatively, keeping the original one-cluster partition and representative A1 fixed, replacing the redundant fill A2 with A4 also recovers U3 and reaches 75%. Changing the representative alone provides no gain. These are different ways to recover the same omission, not additive contributions.

Q028 — joint-decision dependence. For this song-recommendation query, the original set A1/A5/A2 retains 3/6 units (50%); the same-budget ceiling is 4/6 (66.67%). With the partition and cluster priority fixed, neither representative-only nor filling-only optimization improves coverage. Their joint optimization selects A2/A5/A3 and reaches the ceiling. Thus, testing either decision independently would miss a recoverable 16.67-pp shortfall.

Full-query check. The analysis retains all 30 queries, including zero-loss cases. Six have positive fixed-order ACL. Representative-only optimization improves four and reaches the ceiling in one; filling-only optimization improves five and reaches it in four. Q028 is the only loss-bearing query where neither individual change helps but their combination does. Q026 was selected as the positive-loss case closest to the median (query-ID tie-break); Q028 illustrates the complementary interaction pattern.

What this adds to our first response. Equal final coverage can conceal different attainable ceilings and different recovery structures. Beyond assigning an aggregate ACL or ranking selectors, the evaluation connects which answers occupy limited slots, which functional units they repeat or omit, and which controlled changes recover available content. The common candidate pool and budget distinguish omissions caused by selection from limits imposed by K. Human-reference interventions serve as diagnostic probes, not deployable selection algorithms or a unique additive attribution. These traces make the explanatory contribution of linking pairwise evidence to retained set utility concrete.

AuthorsOpenReview ↗

Follow-up and Willingness to Address Further Concerns

Thank you again for your time and thoughtful review of our work. We sincerely appreciate your detailed feedback and the opportunity to address your concerns through our responses and additional analyses.

As the discussion period is approaching its conclusion, we would like to kindly ask whether you have any remaining concerns, questions, or suggestions that we could further address.

We would greatly welcome any additional feedback and would be more than happy to provide further clarification, evidence, or analysis wherever helpful. We remain fully committed to addressing your concerns as thoroughly as possible.

Thank you once again for your valuable time and consideration.

Reviewer_VmMYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

When a model can only show a few responses, the selector has to drop duplicates without dropping real alternatives. The paper asks how well existing pairwise homogeneity scores do this once they pass through a threshold fixed on development data, clustering and a budget. It measures the valid content left against the best subset of the same size.

The tests use synthetic tasks with checkable answers, where the same responses are relabelled under different intended uses, and Natural-50, open-ended responses annotated by humans for functional units. The authors argue that pooled AUC can reward cues about the use, that perfect ranking can still lose coverage, and that a pipeline chosen by pooled AUC loses coverage on Natural-50. They recommend reporting pairwise performance, coverage and avoidable loss together.

Critical strengths

1. Metrics are judged by what ends up in the selected set.

  • If only three of six responses can be shown, a good pairwise AUC doesn't tell me which options survive. The paper scores what is left after threshold, clustering and budget.

  • Avoidable Coverage Loss compares that subset with the best subset of the same size from the same candidates. So loss the budget forces isn't blamed on the selector.

2. Intended use is made explicit while the responses stay fixed.

  • The same responses are labelled under different uses. 73 and 89 are interchangeable if you need any valid prime and different if you want options.

  • The Natural-50 annotations record validity and partial overlap. They don't force every pair into equivalent or distinct, and others could reuse them.

3. A scorer that never reads the responses can win on pooled AUC.

  • The Purpose-Only Control looks only at the use, yet beats Qwen3-U on pooled AUC, 0.875 to 0.750. Its Intended-Use AUC is 0.5 against 1.0, and its coverage 83.33% against 100%.

  • Appendix A.1 caps any use-blind scorer at one quarter plus half its within-use AUC, so 0.75 here. I checked the derivation and it holds.

  • The cap is specific to this design, but any benchmark that pools several conditions into one AUC runs the same risk. The Flip and Hold split in Table F3 is a cheap way to catch it.

Critical weaknesses

1. The 8.36 point headline says little about pooled AUC.

  • The abstract correctly calls 8.36 a shortfall from the same-budget optimum. Most readers will still take it as the cost of choosing a pipeline by pooled AUC.

  • Table F13 gives that pipeline the same coverage as First-K, which just keeps the first three responses. Figure 4 shows the same rounded loss on every query.

  • Refitting the threshold on Natural-50 gains 3.72 points, or 2.01 with complete linkage, and both intervals touch zero. At most this is about a transferred threshold.

2. The Natural-50 results rest on very few queries.

  • Avoidable loss can't go below zero, so an interval above zero only says some queries lost something. Here that is six of 30, with over half the loss in two.

  • Three of those six lose the same amount under all five schemes. Everything also comes from one frozen order, though the controlled section shows that order matters.

  • The Qwen gap behind the AUC and coverage reversal is 1.30 points from one query. AUC is also pooled across queries while selection happens within one.

3. It is unclear how much of the loss is the scorer's fault.

  • The selector keeps the first response and represents each cluster by its earliest member. Some of the loss may be out of reach for any pairwise scorer.

  • Appendix F.6 counts mechanisms but doesn't split the loss in points. There is no oracle label arm or validity filter to separate the causes.

4. The controlled evidence is weaker than it looks.

  • Perfect ranking loses coverage only because a threshold sits on top of a fixed budget, and only with single linkage. Intended-Use AUC isn't reported for the three-alternative tasks where this happens.

  • With one valid object per response, farthest-first on a perfect ranking reaches the optimum at any K. The paper only tries that rule post hoc on Natural-50.

  • Among real scorers the pooled AUC pick trails the Intended-Use pick by just 0.65 and 1.60 points. Even that gap rests on a development tie with TF-IDF broken by name order.

Questions for authors
  1. On Natural-50, how many clusters does the pooled AUC scheme form per query at its controlled threshold, and how does its order averaged loss compare with First-K?

  2. How much of the 8.36 points remains if the same selector gets oracle relation labels, or if invalid responses are filtered out first?

  3. What is Intended-Use AUC on the three-alternative tasks that lost coverage, and does Table 2 change without name-order tie breaking and with TF-IDF fitted out of sample?

3 comments
AuthorsOpenReview ↗

Mechanism-Level Evidence and Diagnostic Scope

Thank you for checking our derivation and raising precise questions. Using frozen responses, human references, and cached scores, we examined how controlled interventions change retained content. New bootstrap 95% CIs resample independent queries/tasks; post-hoc comparisons are descriptive.

Q1. Clusters, order, and 8.36 pp. At the controlled-development threshold, Qwen3-P/single linkage forms 1, 2, 3, and 5 clusters on 25, 3, 1, and 1 Natural-50 queries. Under the frozen order, 27/30 subsets match First-K and all 30 yield identical coverage. Averaging all 720 orders within each query gives Avoidable Coverage Loss of 8.05 pp [4.07, 12.38] versus First-K's 7.76 pp [3.97, 12.04]; paired coverage difference: -0.29 pp [-0.87, 0.17]. Frequent single-cluster outcomes explain this near-First-K behavior. Under the historical annotated-purpose input, adapting only the threshold on natural development improves order-averaged coverage by 2.56 pp [0.09, 5.64].

The original 8.36 pp is the frozen-order shortfall from the exhaustive same-budget optimum, not the causal cost of pooled-AUC scheme selection. Separately, the Purpose-Only Control and AUC bound show how pooled AUC can reward use cues without within-use discrimination.

Q2. Relations, validity, and downstream decisions. Partial overlap and uncertain validity prevent a complete binary oracle for every natural query. On 20 queries admitting a strict six-response binary reference, replacing predicted relations lowers frozen-order loss from 8.37 to 7.12 pp. Across all 30, functional-unit equality projection leaves 7.52 pp; human-validity filtering leaves 6.69 pp; and both together leave 5.86 pp (baseline: 8.36 pp). A partial oracle correcting only unambiguous pairs also leaves 7.52 pp; ambiguous pairs retain model decisions. All use the original coverage denominator and ceiling.

Holding original partitions and cluster priority fixed, optimizing representatives using human functional units while preserving the filling rule leaves 4.08 pp (paired gain: 4.27 pp [0.56, 8.91]). Optimizing only empty-slot filling leaves 1.39 pp (gain: 6.97 pp [1.39, 14.17]); joint optimization attains the ceiling. These controlled oracle interventions reveal that correcting pairwise relations alone does not recover all attainable content. Even with partitions fixed, representative selection and slot filling substantially affect coverage. Their effects can interact, underscoring why pairwise decisions and set utility need separate checks.

Q3. Perfect ranking and development ties. Qwen3-U has task-level Intended-Use AUC=1 on every tested three-alternative task, yet single linkage at K=3 loses coverage on 2/32 balanced and 3/32 imbalanced tasks. Traces identify accepted cross-alternative edges and omissions; complete linkage and farthest-first (FF) attain the ceiling. Perfect ranking therefore need not guarantee coverage after a transferred threshold and single linkage, while simpler selection suffices in this regime.

Development ties matter too. In a post-hoc sensitivity check, uniformly averaging schemes tied under the original 64-task Intended-Use-AUC criterion gives 94.73% test coverage versus 100% for the original name-based choice. In four further checks, TF-IDF is fitted on 48 tasks and thresholds/schemes selected on 16 disjoint tasks, keeping the representation fixed on test. TF-IDF development AUC is 0.819–0.892 and no longer ties for best; Qwen3-U and Qwen3-P still tie in three of four splits. Coverage across legal tied choices spans 98.44–100%. Development size also changes, so differences cannot be attributed to TF-IDF fitting alone.

Broader natural evidence. Natural200 has 196 eligible six-response queries, 193 with nontrivial K=3 selection space. Jaccard/facility-location incurs order-averaged loss on 128 queries (mean 8.53 pp [7.04, 10.15]); the top two contribute only 7.01% of total loss. Each set combines five unchanged MoDA cached generations with one historical Infinity-Chat response. These results extend coverage-omission evidence across a larger constructed collection. The natural AUC–coverage ranking reversal disappears under the tested original-query, generic-use input. This sensitivity highlights Intended-Use Alignment; strictly within-query AUC is estimable for only a small subset, so we limit the ranking observation to its evaluated conditions.

What these checks reveal. Intended-Use Alignment detects purpose-related confounding; Fixed-Threshold Transfer separates ranking from operating decisions; Selection Compatibility traces grouping, representative choice, and budget filling into retained content. Together, controlled interventions and same-budget ceilings localize omissions, explain how downstream decisions affect them, and quantify their consequences, while identifying when simple selectors suffice.

AuthorsOpenReview ↗

Additional Evidence Beyond the Original Clustering Pipeline

We would like to complement our first response with additional Natural200 results that extend its mechanism-level findings.

Our earlier diagnostics identified how the frozen threshold, cluster formation, representative choice, and budget filling affect retained content. We have now examined a further question: what happens when the original clustering pipeline is replaced by stronger, direct score-based set selectors?

1. Strong set-level comparisons

We completed the six-interface k-DPP MAP extension alongside the original 24 scorer-selector combinations, using the same 196 eligible Natural200 queries, six responses per query, K=3, frozen scores, and human coverage reference.

Under Jaccard scoring, order-averaged Avoidable Coverage Loss is 9.40 pp for farthest-first (FF), 8.53 pp for facility-location, and 9.45 pp for k-DPP MAP, against the 74.73% exhaustive same-budget coverage ceiling.

Facility-location improves Functional Coverage over First-K by 13.07 pp [10.29, 15.91] and over Longest-K by 4.26 pp [1.84, 6.87]. These results demonstrate that stronger set-level selection can recover substantial valid content omitted by simpler rules. At the same time, the remaining avoidable loss shows that improving the selector does not eliminate the need to evaluate what the selected set actually preserves.

2. Scoring evidence and selection rules jointly affect coverage

The complete comparison reveals an additional pattern: the relative effectiveness of selection methods changes with the scoring interface.

With Qwen3-U, facility-location improves coverage over FF by 6.88 pp [3.04, 10.88], compared with 0.88 pp [-0.10, 1.86] under Jaccard.

The facility-location and k-DPP comparison also changes across interfaces. Under TF-IDF, their avoidable losses are 16.49 and 13.90 pp, respectively; under Qwen3-U, they are 16.80 and 24.76 pp.

Thus, neither a strong pairwise scorer nor a sophisticated set-selection objective, considered independently, determines the content ultimately retained. Their combination matters.

3. What this adds to our original diagnosis

The first response established how the original pipeline's decisions contributed to coverage loss. These new comparisons establish that the evaluation question extends beyond that particular pipeline: even direct, score-based selection methods exhibit different coverage outcomes and measurable gaps from the same-budget optimum.

This strengthens the central motivation for Selection Compatibility. Pairwise evaluation tells us how candidate responses are scored; set-level evaluation tells us what those scores accomplish after selection. Examining both makes it possible to identify which combinations preserve meaningful alternatives, where omissions remain, and how much valid content could still have been retained under the same budget.

Together, the original interventions and expanded comparisons move the analysis from diagnosing a particular failure to systematically evaluating how pairwise evidence translates into set utility.

These additional comparisons are post-hoc and use query-level paired bootstrap intervals, with input orders averaged within queries. The k-DPP implementation retains its fixed PSD kernel construction and selection rules without test tuning.

AuthorsOpenReview ↗

Follow-up on Our Responses and Additional Evidence

Thank you again for your careful review and constructive feedback. We sincerely appreciate your insightful questions, which have helped us strengthen both our experimental analysis and the presentation of our findings.

With the discussion period approaching its conclusion, we would like to kindly ask whether our responses and supplementary evidence have addressed your main concerns, or whether there are any remaining questions or issues that would benefit from further clarification.

We would warmly welcome any additional comments and would be very happy to provide further explanations, analyses, or supporting evidence. We will do our utmost to respond promptly and thoroughly.

Thank you again for your time, consideration, and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.