- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
The paper explains Expected Shortfall (ES) and how to choose the smallest ES difference among K models. It compares “synchronized evaluation” and “independent evaluation”. The paper proves a matching minimax rate for expected regret. There are corresponding experiments on MMLU-Pro and ARC-Challenge.
Critical strengths
- The minimax characterization and its formulation, derivation is interpretable and explicit.
- Theorem 1 has both upper bound and lower bound, and they both depend on logK, which is strong result.
Critical weaknesses
My main concern is that the evidence does not directly support the paper’s main claim. Also, the visualization of numbers and figures is not very good. Details are in the questions.
Questions for authors
- Conventional model selection in ML compares mean performance such as accuracy. The experiments reach that regime by pairing unbounded losses (correct-answer NLL / absolute error) with a 0.95-ES criterion, but the paper motivates ES only in general terms and does not explain when practitioners would select models this way. In what instances do we choose a model with this strategy?
- The introduction explains that the paper adaptively chooses when to evaluate and stop, and I found the related derivation in Appendix I. However, there seems to be no experiment of the optimally chosen algorithm/heuristic value.
- Figure readability is poor. The style of the figure makes it hard to read. (Fig. 1’s a, b bolded black marker, blue and orange title, different style of (a)(b) in Fig. 2, Fig. 2 (b) showing slope difference but not clear of interpretation)
- Fig. 3 is also hard to figure out why this figure is important. The graph of independent and synchronized does not show a tendency between the four graphs. Should I interpret that orange is constantly bigger than blue, or in some setting the gap is bigger? Linear y-axis and slope, no marker; the figure is talking about Kn, but the writing is talking about n, which makes it hard to interpret.
- Table 1 is the only table that contains real data results, but it is hard to interpret. PCS 0.144 → 0.831, regret 4.227 → 0.126 is not sufficient to support the main content’s claim.
- How big are the constants in the real experiments? Are the bounds informative at that scale?
- The fixed-budget lower bounds require K >= 8. Does the separation also hold for K=2, the common case of comparing two models?
- Can you generalize this tendency to another setting and explain better why this is an important problem?
5 comments
Thank you for clarifying the K=2 case and for adding the proof. I still have three concerns.
-
I still find Figure 2, especially panel (b), difficult to interpret. I understand that a positive slope difference means the required evaluation budget grows faster under independent sampling as the gap shrinks. What should the reader learn from seeing 'all six', 'smallest four', and 'largest four' side by side? Current figures 1,2,3 all are hard to interpret by reading the paper.
-
In the medical claims experiment, synchronization increases the probability of selecting the best model. However, the best and second-best models differ by <1$ in the 0.95-ES of ABS prediction error. I remain unclear about the practical importance of distinguishing models. Why is this decision important?
-
I found the intuition behind the experiments hard to follow in the main text. Several passages report estimates and 95% intervals in succession, leaving the reader to infer which comparison matters most. It would help to state the question each experiment answers and its main finding in plain language before presenting the numerical details.
I appreciate the revisions, but these concerns remain, so I am keeping my current rating.
Follow-up to Reviewer WKdc
Thank you for the follow-up. We address the three concerns below.
1. Interpretation of Figure 2(b).
“All six” is the main fit using all six ES gaps. “Smallest four” excludes the two largest gaps, and “largest four” excludes the two smallest. These overlapping fits check whether the conclusion depends on the gap range used for fitting. All three simultaneous 95% intervals lie above zero, showing that the positive slope difference persists after either end of the tested range is excluded. We plan to make this purpose explicit in the labels and caption.
2. Importance of the medical comparison.
You are correct that the best and second-best models differ by only 0.530 dollars in ES. This case study does not establish that distinguishing these two models has substantial economic value.
However, the practical evaluation question is whether we can reliably select a model with near-minimal tail prediction error under a limited evaluation budget, even when the two leading models are treated as equivalent. In the without-replacement experiment with 8,192 records per candidate, the estimated probability of selecting a model within 1 dollar of the best ES is 96.9% under synchronization versus 29.6% under independent sampling, at the same evaluation cost. Thus, the gain in selection reliability persists without requiring us to distinguish between the two leading models.
3. Presentation of the experiments.
We agree that main findings should come before the numerical details. In Section 5.1, we plan to bring the budget-growth finding forward. In Section 5.3, we will first summarize where synchronization improves selection and where the difference is inconclusive. We will make the medical tolerance result more prominent in Section 5.2. The estimates and confidence intervals will follow these main findings.
Remaining soundness concern.
Finally, your original review assigned a soundness score of 1. We would appreciate clarification on whether any specific concern about the correctness of our theoretical results or the validity of our experiments remains. We would be glad to address it directly. Thank you.
Response to Reviewer WKdc
Thanks for your feedback. We understand your concern about how the experiments support our main claim. We address your questions below.
Main claim and evidence. To clarify, Theorem 1 establishes that the observation protocol itself changes the minimax ES-selection rate: shared inputs achieve a faster rate than independent evaluation under the same moment bounds and evaluation cost. It predicts that, as ES gaps shrink, independent evaluation requires budgets that grow faster than synchronized evaluation.
Since these are worst-case rates over a class of distributions, this difference in budget scaling need not appear on every distribution. We therefore examine the predicted scaling on the hard families used to establish the lower bounds. Figure 2(b) shows that, as the ES gap shrinks, the number of evaluations needed to reach the same probability of correct selection grows faster under independent evaluation, consistent with the theory.
Because these experiments use different hard families, we then compare the two protocols on the same loss distribution, with the same selector and budget. Synchronization still improves selection, providing further evidence for the benefit of shared inputs. We finally show that this benefit also appears in real evaluations using medical claims and language-model benchmarks.
We have revised the opening of Section 5 to explain the purpose of each experiment and make its connection to the theoretical result clearer.
Why ES and what Table 1 shows. As you note, model selection usually focuses on average performance. However, a model with good average performance can still make severe errors on a small fraction of cases. ES is useful when these errors matter for the decision, as studied in risk-sensitive learning (Holland & Haress, 2021). For example, in medical-cost forecasting, large prediction errors can affect budgeting and reserve planning. In language-model evaluation, ES allows us to assess performance on questions where the correct answer receives very low probability. Our study examines how to select models efficiently when this tail-risk criterion matters.
For Table 1, we compare both protocols on the same medical data, using the same models, ES criterion, selection rule, and evaluation budget. The key difference is whether the models share evaluation inputs. Since large medical claims can affect all models' errors, sharing inputs makes their comparisons more reliable. The higher PCS and lower regret provide real-data evidence for this benefit at a fixed budget. Section 5.2 now also reports PCS across budgets: as the number of records per candidate increases 64-fold, synchronized PCS rises to 0.968, while independent PCS remains near the 1/7 chance level.
The K=2 case. Thanks for raising this question. The separation also holds for K=2. We have added a binary-testing proof in Appendix G.4 and summarized the result in Section 3. The minimax expected regret rates are n−1/2 under synchronized evaluation and n−γ(p,r) under independent evaluation.
Adaptive stopping and constants. Our main result concerns fixed-budget minimax selection, which is also the focus of our experiments. The adaptive procedures extend the theory by establishing expected evaluation-cost guarantees when the ES gaps are unknown. The constants in both the fixed-budget regret bounds and the adaptive expected-cost guarantees have not been numerically calibrated. We have clarified the role of the adaptive procedures in Section 4 and Appendix I.
Readability of Figures 1 and 2. We have standardized the panel labels in Figure 1 and clarified the interpretation of Figure 2(b)'s slope difference in its caption.
Figure 3 interpretation. For Figure 3, the main purpose is to show whether sharing inputs improves model selection at the same evaluation cost on real language-model benchmarks. Each panel compares the two protocols within one benchmark and reference setting. A lower mean regret means the selected model is closer to the best model in ES. The benefit is clear on ARC-Challenge and on MMLU-Pro with OpenOrca excluded. On MMLU-Pro with Mistral excluded, the confidence interval for the difference includes zero.
To make the comparison easier to follow, we added markers at the tested budgets and clarified the panel titles. We also clarified that n is the number of questions per candidate, while the horizontal axis shows the total number of candidate–question evaluations across all 27 candidates.
Generalizability. Beyond medical claims and LLM benchmarks, our framework can also be useful for selecting among fixed policies by tail risk using common simulated scenarios, such as reinforcement-learning policies evaluated using shared random seeds.
We hope these clarifications and revisions address your concerns.