- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper is written extremely badly, so it is difficult to decode its contribution. As best I can tell, it is an extended elaboration on the following simple idea:
Suppose an ML model has a finite number of parameter settings that affect its performance on different kinds of examples. In the simplest case, there are two kinds of examples (A and B), and a model (call it model 1) has two modes. The first mode gets 0% error on A examples but 100% error on B examples, and the second mode is the opposite. Model 2 gets 40% error on all examples. We need to choose a model, but don't know the fraction of type A examples at deployment, which will be p. Suppose we know that p is uniform in [0,1]. For any -fixed- parameterization of Model 1, its expected error is 50%, and so we might choose model 2. But if we get to choose Model 1's parameterization at deployment, after observing p, we can pick the better of the two modes, and its expected error is only 25%, so we should choose model 1. Each mode has error that is a linear function of p, so the error of the optimal parameterization is the minimum of a bunch of affine functions. So the paper proposes choosing the model that minimizes this lower piecewise affine concave envelope as averaged over deployment conditions instead of something more naive.
The rest of the paper is just an elaboration of this simple idea, along with some experiments. The only additional theoretical contribution I can identify is a set of results about what can be missed when an adaptive system is evaluated at only finitely many deployment conditions.
Critical strengths
The basic idea underlying the proposal is correct.
Critical weaknesses
The main weakness of this paper is that it is written very badly, seemingly entirely by AI. I quote the first few lines of the abstract (!) here as an example --- the rest of the paper is similar.
"Two serving stacks can agree on every number a leaderboard publishes and still be the right and the wrong choice for the same deployment. Of Open LLM models, are the best available choice at some task mixture while topping none of the seven numbers published. A system's \emph{risk profile}, its best achievable risk at each operating condition, is the minimal object an evaluation can depend on; practice puts point masses on it, and we characterize the linear and convex rules completely. "
Unsurprisingly it registers as 100% on pangram, but the sin is not the AI usage itself, it is the seeming lack of any human attention at all --- the paper is unreadable.
The main idea in the paper is very simple.
The paper also appears to have a number of serious technical issues that I list here (the following list of errors is generated via an LLM audit but verified) :
Throughout, denotes model 's lowest achievable loss at deployment condition . Lower is better.
-
The reported mean deployment prevalence is incorrect, with a possible implementation error. Appendix I.2 (p. 42) says the log-odds-uniform distribution on has mean . For the stated density , the mean is instead
This undermines the explanation that the default's mean is far from the median hospital prevalence of . Notably, taking the stated 60-point logit-spaced grid and then weighting its points again by gives . That would count the change-of-variable factor twice.
-
The implemented robust score can select a model that is worse everywhere. Equation (5) and Appendix D.5 specify the approximation \hat{\Phi}\epsilon(r)=E{\mu_0}[r]+\epsilon,Lip(r) when has full support. On , take uniform, , and . These are permitted losses: both are minima of finitely many affine functions.
Model A is strictly better everywhere, but its mean loss is and its maximum absolute slope is , whereas B's are and . At , the scores are therefore for A and for B, so the implemented method chooses B. The exact robust objective cannot do this, since everywhere. The approximation therefore does not preserve the basic requirement that improving a model's loss everywhere cannot worsen its evaluation.
-
Theorem 5 is false for the general reference distributions it allows. On , let and . Equation (7) gives , because every slope from to another point is . But the exact robust value is for every : the original distribution is feasible and already puts all its mass at the maximum of .
Theorem 5(i) consequently asserts , which is false for every ; its claimed first-order expansion also fails.
-
The central theorem characterizes equality of loss differences, not agreement of rankings. Proposition 1 correctly proves that for every exactly when is affine. The abstract and introduction describe this as an exact characterization of when rankings agree. Rankings require only the same sign, not equal differences.
For example, on , let and . Their difference is not affine, but A wins everywhere, so averaging and evaluating at the average condition rank them identically for every distribution. Nonlinear differences can permit a ranking change; they do not guarantee one.
-
Forbidding retuning does not imply linear dependence on deployment conditions. The introduction, discussion, and Remark 11 claim that non-reconfigurable systems have affine losses. This needs an additional assumption about how the conditions enter the loss. Even the paper's fixed-threshold classification loss,
contains a term and is generally not affine in the joint condition . The claim is valid when only prevalence varies at fixed cost, or for the stated linear task-mixture model, but not for the general framework.
-
The leaderboard example does not establish the claimed information gap. Section 3 and Appendix G.1 claim that the published numbers cannot distinguish the constructed model stacks. But the leaderboard publishes every component model's per-task scores. Given those scores and stack membership, the loss of the stack used in their construction is computable at every task mixture:
where is model 's loss on task . Thus the published component scores determine the entire function. The example shows that reporting only a few optimized stack-level scores can discard useful information; it does not show that the published component-model scores lack that information.
Questions for authors
This paper really needs a careful, human rewrite. The potential errors listed above are just further examples of lack of care; fixing these technical errors would not make the submitted writeup appropriate for publication. The main issue is lack of substantial technical contribution and unreadability.
We thank the reviewer for their detailed review. Please see our Common Response and specific rebuttal below. Language models assisted with drafting and editing this rebuttal. The manuscript’s AI use statement describes assistance with manuscript prose and experiment and analysis scripts.
We have incorporated your two-mode example in Proposition 1(iv) characterises exactly when that reversal can occur. Theorems 3 and 4 answer the associated benchmark-design question: can the reported stack-level scores reveal the configurations that make the difference? Figure 2 gives a concrete case where seven stack-level scores miss a 6.54-point gap at a specified mixture.
Mean of the log-odds-uniform measure. You identified the cause. We applied the density to a grid already spaced uniformly in log-odds, counting the change of variables twice. The correct mean on [0.02, 0.20] is 0.081. We corrected the quadrature and reran the affected experiments. On eICU, averaging reduced regret by 4.6% [1.7, 7.4] against evaluation at the mean with the full source sample. Table 9 records the outcomes and the timing of the fix.
Monotonicity of the robust score. The first-order approximation can select a profile that is worse everywhere. Remark 5 now gives an explicit example. The prevalence and routing experiments use the exact Wasserstein dual on the evaluation grid. In the RLHF experiment, the surrogate was used at radius 0.05, where Appendix C.5 checks that it gives the same choices as the exact computation on the lattice.
Theorem 5 at a maximiser. We take the positive part of the essential supremum in the definition of sensitivity. When it is zero, the reference distribution is feasible, and the upper and lower bounds coincide at the reference risk. The proof now treats that case explicitly.
Equality versus ranking. The affinity condition characterises equality of pairwise risk differences for every measure. Strict reversal requires the value at some mean to have the opposite sign from an attainable integrated difference at that mean. Proposition 1(iv) states this using the convex and concave envelopes. A non-affine difference with a constant sign can preserve the ranking for every measure, as in your example.
Fixed configurations. Assumption 1 states the missing condition directly. Risk is affine in the operating condition for each fixed configuration. It applies to linear task mixtures and to varying prevalence at a fixed cost ratio. Varying the cost ratio and prevalence jointly gives a bilinear term.
The leaderboard example. Given the component scores and stack membership, both routed risk profiles can be computed under the model used in the example. Figure 2 now describes a stack-level evaluation that reports only seven aggregate scores. Those seven stack scores coincide, even though the stacks differ by 6.54 points at the illustrated task mixture. The caption says explicitly that the published component scores determine both profiles.
We rewrote the opening around the comparison at a fixed deployment mean. The revised introduction credits the profile and expected-risk rule to prior work and states the finite-suite and ranking results separately. The experiments now use the same matched comparison when discussing whether averaging changes the selection.
If these address your concerns, we would be grateful if you would consider revising your score.