- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies LLM benchmark compression using prompt embeddings and graph-based subset selection. It constructs proximity graphs and applies exact or heuristic maximum independent set (MIS) solvers to select mutually separated prompts. For a fixed configuration, subset construction requires neither evaluated-model responses nor category labels. The empirical study covers four HELM benchmark subsets, 66 LLMs, and 2,558 configurations spanning embeddings, distances, thresholds, and solvers. The evaluation distinguishes fidelity to full-benchmark rankings from consistency of model rankings across selection seeds, and additionally examines absolute-score error, overlap among top-ranked models, and category redistribution. At , the paper reports mean Spearman correlation of 0.967 with approximately 74–82% prompt reduction. It recommends ReduMIS with Qwen-8B embeddings and standardized-Euclidean distance, identifying seed consistency and coverage as its main advantages over sampling.
Critical strengths
- The study provides a useful empirical analysis of benchmark sensitivity to compression and interactions between embeddings and distance measures. Table 9 identifies a useful mild-compression regime and quantifies degradation under aggressive pruning. Reporting random sampling’s stronger aggregate fidelity and lower MAE makes the study informative about the limitations of semantic selection (Tables 16 and 19).
- The selection procedure does not require a pre-existing matrix of model responses (Section 3 and Appendix B). The independent-set constraint provides an explicit pairwise-separation property in the chosen embedding space, and the resulting subset can be reused for subsequent model evaluations. This is practically attractive when historical model evaluations are unavailable.
Critical weaknesses
- The reported baseline results contain numerical inconsistencies. Because the graph-selection principle builds on prior work such as SELECTOR (Cenikj et al., 2022), the paper’s contribution depends primarily on the reliability of its empirical findings and practical guidance. The “reliable and faithful” quadrant is defined by and . Page 18 reports that 44% of stratified configurations occupy this quadrant, whereas Table 13 reports 61.1%. Applying the stated thresholds to Table 16 gives 10/18 = 55.6% for both stratified and uniform-stratified sampling; Table 13 reports 50.0% for the latter. Furthermore, GPQA//uniform-stratified is reported as on p. 18 and in Table 16, placing it in different quadrants. These discrepancies cannot be explained by rounding and affect method-specific rates and conclusions about failure modes. Table 16 still shows higher for MIS in the reported comparisons, but these inconsistencies leave the magnitude of this advantage uncertain.
- The coverage comparison suffers from a separate budget confound. Baseline subset sizes are matched to the average MIS subset size for each threshold–benchmark combination, rather than to the size of the specific MIS subset being evaluated. Specifically, Appendix F, p. 23, acknowledges that Figures 13–15 compare MIS subsets of 116, 284, and 332 prompts with baseline subsets of 82, 204, and 187, respectively. Because category bars are normalized by full-benchmark size, larger bars can reflect the larger retained subset. These figures cannot establish an equal-budget coverage advantage. Figure 1 uses a 2% uniform-reference line when comparing the full-benchmark and subset distributions. This reference is appropriate for the subset, but the corresponding reference for the full benchmark is approximately 7.14%. The figure therefore does not directly show that the subset has a more uniform category distribution. A comparison at the same subset size is needed to substantiate the recommended configuration’s claimed coverage advantage over sampling.
- The behavioral analysis does not establish that semantically similar prompts provide redundant information about model performance. Figure 5 reports only weak correlations between the difficulty of retained prompts and that of discarded prompts, approximately –. Statistical significance does not establish interchangeability, and prompts with similar average difficulty may still differ in their ability to distinguish between models. Appendix K/Figure 29 finds essentially no relationship between embedding distance and response-vector similarity within the selected neighbor pairs, yet Section 5 treats this as evidence that the threshold identifies evaluative redundancy. A flat relationship within the selected distance range does not establish that the threshold separates redundant from nonredundant pairs. The analysis lacks a suitable control comparison—for example, non-neighbor or random pairs—that accounts for category and marginal difficulty. The authors should either support the mechanistic conclusions with a control analysis using the existing response matrix or narrow those conclusions.
Questions for authors
- Which baseline results are authoritative, and how are ties handled when computing ? Please reconcile the results on p. 18, in Figure 11, and in Tables 13 and 16. Would resolving these discrepancies or applying standard tie corrections change the main findings?
- How does the recommended configuration compare with sampling at the same subset size in fidelity, MAE, ranking consistency, and coverage? Please distinguish retained category counts from within-subset proportions.
- Do selected neighbor pairs show greater similarity in model predictions than suitable control pairs? A comparison accounting for category and marginal difficulty would clarify whether evaluative redundancy is supported.
- What is the prompt-similarity annotation study mentioned in the Ethics Statement? Page 10 refers to a study conducted by the authors’ team, but I could not locate its methods or results. If it supports the reported conclusions, please identify the evidence; otherwise, clarify its role.
4 comments
Q2: Recommended configuration vs. same-size sampling
Question: How does the recommended configuration compare with sampling at the same subset size in fidelity, MAE, ranking consistency, and coverage? Please distinguish retained category counts from within-subset proportions.
The results show that at equal size, the recommended configuration is as faithful as sampling and provides more balanced category coverage without using labels. We compared it with 1,000 draws each of random, stratified, and uniform-stratified sampling at exactly its subset size. At p20, ρ is within 0.007 of random and stratified sampling on all four benchmarks (e.g., 0.975 vs. 0.980 on GPQA and 0.996 vs. 0.996 on Omni-MATH). Category coverage, measured on within-subset proportions by normalized entropy (1 = all categories equally represented), is 0.974 vs. 0.936 for random sampling on GPQA, 0.978 vs. 0.969 on MMLU-Pro and 0.810 vs. 0.775 on Omni-MATH; the MIS subset is more balanced than 96%, 92% and over 99% of same-size random subsets, respectively, and more balanced than the median random subset in 17 of 18 benchmark–threshold cells. MAE is higher for MIS (e.g., 4.8 vs. 3.0 points on GPQA) because redistributing coverage also changes the difficulty mix. Kendall’s W is 0.999–1.000 for MIS and 0.964–0.995 for the baselines, which reflects that ReduMIS returns the same subset across seeds rather than higher fidelity. We now report retained counts and within-subset proportions separately for every category: for example, on GPQA the MIS subset contains 32 Physics, 52 Chemistry and 32 Biology prompts (28%, 45% and 28% of the subset), compared with 48.4, 47.4 and 20.3 (42%, 41% and 18%) for random sampling.
In the revised manuscript, this comparison is described in Section 5 (Table 2 and “Category redistribution”) and in Appendix G (Tables 21–23 and Figures 14–16); Table 22 reports retained counts and within-subset proportions separately.
Q3: Neighbor pairs vs. matched control pairs
Question: Do selected neighbor pairs show greater similarity in model predictions than suitable control pairs? A comparison accounting for category and marginal difficulty would clarify whether evaluative redundancy is supported.
In the revised manuscript, we report these correlations separately for each embedding model, distance metric, and distance threshold. The results show that the correlations vary between configurations but not systematically. Across all 631 ReduMIS configurations (six embeddings, three distances, six thresholds, and all four benchmarks, now including IFEval), the difficulty correlation between each discarded prompt and its nearest retained prompt is positive in 89% of the configurations and positive on average for every embedding and distance, whereas random prompt pairs show no correlation (r = 0.00). No embedding is consistently strongest across benchmarks, and large decoders and small encoders reach similar values on average (at p20: 0.10–0.23 vs. 0.10–0.21 across the four benchmarks). Standardized Euclidean gives the highest or tied-highest correlation on three of the four benchmarks, and the alignment tends to decrease at aggressive thresholds. The same pattern holds for the benchmarks’ own difficulty annotations, which involve no model evaluations: at p20, the official difficulty ratings of neighbors correlate at r = 0.44 on Omni-MATH, and the question writers’ ratings at r = 0.22 on GPQA. Because no embedding yields markedly stronger alignment, this analysis does not change the recommended pipeline.
We also added the control comparison suggested by the reviewer, accounting for both category and marginal difficulty. For each discarded prompt and its nearest retained neighbor, we constructed a control pair: the same discarded prompt paired with a random prompt from the same category whose difficulty matches that of the retained neighbor (mean absolute difficulty difference 0.004). Comparing the two pairs for the same discarded prompt isolates the effect of embedding proximity, and we ran this comparison for all 469 configurations on the three category-labeled benchmarks. Embedding neighbors share a category far more often than random prompt pairs (at p20: 95% vs. 37% on GPQA, 59% vs. 8% on MMLU-Pro and 60% vs. 23% on Omni-MATH). Once category and difficulty are matched, however, neighbors agree on model outcomes as often as the control pairs: at p20, 58.2% vs. 57.7% on GPQA, 64.2% vs. 64.0% on MMLU-Pro and 62.5% vs. 62.5% on Omni-MATH, with differences within ±1.3 percentage points for every embedding.
The behavioral similarity between embedding neighbors is therefore accounted for by shared category and difficulty, which MIS recovers from the prompt text without labels or model evaluations; the benchmarks’ own difficulty annotations show the same (neighbors’ official Omni-MATH difficulty ratings correlate at r = 0.44). We have narrowed our claims accordingly: we no longer describe MIS as removing evaluative redundancy beyond what category and difficulty explain, but as removing semantic redundancy that captures category and difficulty structure without labels.
In the revised manuscript, this analysis is described in Section 5 (“Does embedding proximity imply evaluative redundancy?”, Table 3) and in Appendix L (“Matched controls”, Tables 38 and 39, and Figure 30b).
Q4: The annotation study mentioned in the Ethics Statement
Question: What is the prompt-similarity annotation study mentioned in the Ethics Statement? Page 10 refers to a study conducted by the authors’ team, but I could not locate its methods or results. If it supports the reported conclusions, please identify the evidence; otherwise, clarify its role.
Thank you for flagging this. The Ethics Statement referred to a planned study in which subject-matter experts would annotate prompt similarity. The study is still incomplete, and the original wording incorrectly implied that it had been completed. No data from the study were used in the reported analyses or conclusions. We have removed the reference from the revised Ethics Statement.
The revised Ethics Statement no longer refers to this study.
Q1: Authoritative baseline results and tie handling
Question: Which baseline results are authoritative, and how are ties handled when computing W? Please reconcile the results on p. 18, in Figure 11, and in Tables 13 and 16. Would resolving these discrepancies or applying standard tie corrections change the main findings?
Thank you for identifying the discrepancies. We rechecked the reported values against the experimental results and corrected the discrepancies in the revised manuscript. The discrepancies arose because the script that produced Tables 12–15, Figure 11, and the corresponding text on p. 18 redrew the stratified and uniform-stratified baseline samples with a different random stream from the analysis behind Table 16. Table 16 is authoritative. We regenerated Tables 12–15, Figure 11, and the text from the same draws as Table 16: stratified and uniform-stratified sampling are both reliable-and-faithful in 55.6% of configurations, no baseline configuration is reliable-but-reshaped, and the only baseline configuration below ρ = 0.70 is random sampling on GPQA at p90 (ρ = 0.6999). The GPQA p95 uniform-stratified case described on p. 18 (ρ = 0.685, W = 0.549) came from the redrawn samples; in the final run, this configuration has ρ = 0.743 and W = 0.607. In the same check, we found that Table 5 omitted the standardized-Euclidean configurations because of a naming mismatch; the corrected table changes the mean overlaps by at most 0.04 and leaves the conclusions unchanged.
Ties. Rankings use average ranks for tied scores, and Spearman’s ρ is computed on these ranks. Kendall’s W was computed without tie correction, which slightly underestimates concordance when rankings contain ties (identical rankings with ties give W slightly below 1, e.g. 0.9993). With the standard tie correction, W increases mainly for very small subsets, where many models tie: the share of stochastic MIS configurations with W ≥ 0.90 rises from 84.9% to 89.1% (mean W from 0.950 to 0.967), 54 of the 1,262 configurations move from a seed-unstable to the corresponding seed-stable quadrant, and one baseline configuration changes quadrant (uniform-stratified, GPQA p50: W = 0.898 to 0.903). Because the correction can only increase W, it strengthens rather than changes the concordance results, and the main findings remain unchanged. We now report tie-corrected W alongside the original values.
In the revised manuscript, the definition of Kendall’s W with and without tie correction is given in Section 3, the corrected baseline results are in Appendix G (Tables 16, 17 and 19), and the tie-correction analysis is in Appendix D (Tables 7 and 13).