acceptodds
Under review as a conference paper at ICLR 2027

Deployment-aware model selection

Abstract

Two serving stacks can agree on every number a leaderboard publishes and still be the right and the wrong choice for the same deployment. Of Open LLM models, are the best available choice at some task mixture while topping none of the seven numbers published. A system's risk profile, its best achievable risk at each operating condition, is the minimal object an evaluation can depend on; practice puts point masses on it, and we characterize the linear and convex rules completely. What decides when that choice matters is carried not by a profile but by the differences between candidates'. Aggregating over a range ranks a pair exactly as scoring it at the range's average does, precisely when that difference profile is affine. Shared curvature therefore cancels, and profiles may bend arbitrarily yet reorder nothing. No regular linear rule reading finitely many conditions is deployment-complete, and we localize what a suite of covering radius can miss. The boundary is also a noise-calibrated test of whether a full-profile criterion can change a selection. It is correct on all eight systems we try, and bending predicts divergence, not benefit: four bend and two pay, by – on re-steerable RLHF pipelines and on moderation. A replication across hospitals shows that advantage is largely a hedge against estimation error, fading as profiles improve and the rules converge ( agreement to ). What pays throughout is evaluating at the deployment's own condition, – away in total variation from the headline's, using scores already published.

Primary area:
interpretability and explainable AI
Keywords:
model evaluation, model selection, risk profile

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pj8DOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is written extremely badly, so it is difficult to decode its contribution. As best I can tell, it is an extended elaboration on the following simple idea:

Suppose an ML model has a finite number of parameter settings that affect its performance on different kinds of examples. In the simplest case, there are two kinds of examples (A and B), and a model (call it model 1) has two modes. The first mode gets 0% error on A examples but 100% error on B examples, and the second mode is the opposite. Model 2 gets 40% error on all examples. We need to choose a model, but don't know the fraction of type A examples at deployment, which will be p. Suppose we know that p is uniform in [0,1]. For any -fixed- parameterization of Model 1, its expected error is 50%, and so we might choose model 2. But if we get to choose Model 1's parameterization at deployment, after observing p, we can pick the better of the two modes, and its expected error is only 25%, so we should choose model 1. Each mode has error that is a linear function of p, so the error of the optimal parameterization is the minimum of a bunch of affine functions. So the paper proposes choosing the model that minimizes this lower piecewise affine concave envelope as averaged over deployment conditions instead of something more naive.

The rest of the paper is just an elaboration of this simple idea, along with some experiments. The only additional theoretical contribution I can identify is a set of results about what can be missed when an adaptive system is evaluated at only finitely many deployment conditions.

Critical strengths

The basic idea underlying the proposal is correct.

Critical weaknesses

The main weakness of this paper is that it is written very badly, seemingly entirely by AI. I quote the first few lines of the abstract (!) here as an example --- the rest of the paper is similar.

"Two serving stacks can agree on every number a leaderboard publishes and still be the right and the wrong choice for the same deployment. Of Open LLM models, are the best available choice at some task mixture while topping none of the seven numbers published. A system's \emph{risk profile}, its best achievable risk at each operating condition, is the minimal object an evaluation can depend on; practice puts point masses on it, and we characterize the linear and convex rules completely. "

Unsurprisingly it registers as 100% on pangram, but the sin is not the AI usage itself, it is the seeming lack of any human attention at all --- the paper is unreadable.

The main idea in the paper is very simple.

The paper also appears to have a number of serious technical issues that I list here (the following list of errors is generated via an LLM audit but verified) :

Throughout, rs(x)r_s(x) denotes model ss's lowest achievable loss at deployment condition xx. Lower is better.

  1. The reported mean deployment prevalence is incorrect, with a possible implementation error. Appendix I.2 (p. 42) says the log-odds-uniform distribution on [0.02,0.20][0.02,0.20] has mean 0.0540.054. For the stated density dμ(π)∝dπ/[π(1−π)]d\mu(\pi)\propto d\pi/[\pi(1-\pi)], the mean is instead

    Eμ[π]=log⁡((1−a)/(1−b))log⁡(b(1−a)/(a(1−b)))=0.0809973.\mathbb{E}_\mu[\pi]=\frac{\log((1-a)/(1-b))}{\log(b(1-a)/(a(1-b)))}=0.0809973.

    This undermines the explanation that the default's mean is far from the median hospital prevalence of 0.0850.085. Notably, taking the stated 60-point logit-spaced grid and then weighting its points again by 1/[π(1−π)]1/[\pi(1-\pi)] gives 0.0540840.054084. That would count the change-of-variable factor twice.

  2. The implemented robust score can select a model that is worse everywhere. Equation (5) and Appendix D.5 specify the approximation \hat{\Phi}\epsilon(r)=E{\mu_0}[r]+\epsilon,Lip(r) when μ0\mu_0 has full support. On [0,1][0,1], take μ0\mu_0 uniform, rA(x)=min⁡{x,0.1}r_A(x)=\min\{x,0.1\}, and rB(x)=0.101r_B(x)=0.101. These are permitted losses: both are minima of finitely many affine functions.

    Model A is strictly better everywhere, but its mean loss is 0.0950.095 and its maximum absolute slope is 11, whereas B's are 0.1010.101 and 00. At ϵ=0.01\epsilon=0.01, the scores are therefore 0.1050.105 for A and 0.1010.101 for B, so the implemented method chooses B. The exact robust objective cannot do this, since rA≤0.1<rBr_A\le0.1<r_B everywhere. The approximation therefore does not preserve the basic requirement that improving a model's loss everywhere cannot worsen its evaluation.

  3. Theorem 5 is false for the general reference distributions it allows. On [0,1][0,1], let r(x)=1−xr(x)=1-x and μ0=δ0\mu_0=\delta_0. Equation (7) gives Λ(r;μ0)=−1\Lambda(r;\mu_0)=-1, because every slope from 00 to another point is −1-1. But the exact robust value is 11 for every ϵ≥0\epsilon\ge0: the original distribution is feasible and already puts all its mass at the maximum of rr.

    Theorem 5(i) consequently asserts 1≤1−ϵ1\le1-\epsilon, which is false for every ϵ>0\epsilon>0; its claimed first-order expansion also fails.

  4. The central theorem characterizes equality of loss differences, not agreement of rankings. Proposition 1 correctly proves that Eμ[g]=g(Eμ[x])E_\mu[g]=g(E_\mu[x]) for every μ\mu exactly when g=rA−rBg=r_A-r_B is affine. The abstract and introduction describe this as an exact characterization of when rankings agree. Rankings require only the same sign, not equal differences.

    For example, on [0,1][0,1], let rA(x)=0.1r_A(x)=0.1 and rB(x)=0.3+0.2min⁡{x,1−x}r_B(x)=0.3+0.2\min\{x,1-x\}. Their difference is not affine, but A wins everywhere, so averaging and evaluating at the average condition rank them identically for every distribution. Nonlinear differences can permit a ranking change; they do not guarantee one.

  5. Forbidding retuning does not imply linear dependence on deployment conditions. The introduction, discussion, and Remark 11 claim that non-reconfigurable systems have affine losses. This needs an additional assumption about how the conditions enter the loss. Even the paper's fixed-threshold classification loss,

    R(c,π)=c(1−π)FPR⁡+(1−c)πFNR⁡,R(c,\pi)=c(1-\pi)\operatorname{FPR}+(1-c)\pi\operatorname{FNR},

    contains a cπc\pi term and is generally not affine in the joint condition (c,π)(c,\pi). The claim is valid when only prevalence varies at fixed cost, or for the stated linear task-mixture model, but not for the general framework.

  6. The leaderboard example does not establish the claimed information gap. Section 3 and Appendix G.1 claim that the published numbers cannot distinguish the constructed model stacks. But the leaderboard publishes every component model's per-task scores. Given those scores and stack membership, the loss of the stack used in their construction is computable at every task mixture:

    rS(w)=min⁡m∈S∑twtamt,r_S(w)=\min_{m\in S}\sum_t w_t a_{mt},

    where amta_{mt} is model mm's loss on task tt. Thus the published component scores determine the entire function. The example shows that reporting only a few optimized stack-level scores can discard useful information; it does not show that the published component-model scores lack that information.

Questions for authors

This paper really needs a careful, human rewrite. The potential errors listed above are just further examples of lack of care; fixing these technical errors would not make the submitted writeup appropriate for publication. The main issue is lack of substantial technical contribution and unreadability.

1 comment

We thank the reviewer for their detailed review. Please see our Common Response and specific rebuttal below. Language models assisted with drafting and editing this rebuttal. The manuscript’s AI use statement describes assistance with manuscript prose and experiment and analysis scripts.

We have incorporated your two-mode example in Proposition 1(iv) characterises exactly when that reversal can occur. Theorems 3 and 4 answer the associated benchmark-design question: can the reported stack-level scores reveal the configurations that make the difference? Figure 2 gives a concrete case where seven stack-level scores miss a 6.54-point gap at a specified mixture.

  1. Mean of the log-odds-uniform measure. You identified the cause. We applied the density to a grid already spaced uniformly in log-odds, counting the change of variables twice. The correct mean on [0.02, 0.20] is 0.081. We corrected the quadrature and reran the affected experiments. On eICU, averaging reduced regret by 4.6% [1.7, 7.4] against evaluation at the mean with the full source sample. Table 9 records the outcomes and the timing of the fix.

  2. Monotonicity of the robust score. The first-order approximation can select a profile that is worse everywhere. Remark 5 now gives an explicit example. The prevalence and routing experiments use the exact Wasserstein dual on the evaluation grid. In the RLHF experiment, the surrogate was used at radius 0.05, where Appendix C.5 checks that it gives the same choices as the exact computation on the lattice.

  3. Theorem 5 at a maximiser. We take the positive part of the essential supremum in the definition of sensitivity. When it is zero, the reference distribution is feasible, and the upper and lower bounds coincide at the reference risk. The proof now treats that case explicitly.

  4. Equality versus ranking. The affinity condition characterises equality of pairwise risk differences for every measure. Strict reversal requires the value at some mean to have the opposite sign from an attainable integrated difference at that mean. Proposition 1(iv) states this using the convex and concave envelopes. A non-affine difference with a constant sign can preserve the ranking for every measure, as in your example.

  5. Fixed configurations. Assumption 1 states the missing condition directly. Risk is affine in the operating condition for each fixed configuration. It applies to linear task mixtures and to varying prevalence at a fixed cost ratio. Varying the cost ratio and prevalence jointly gives a bilinear term.

  6. The leaderboard example. Given the component scores and stack membership, both routed risk profiles can be computed under the model used in the example. Figure 2 now describes a stack-level evaluation that reports only seven aggregate scores. Those seven stack scores coincide, even though the stacks differ by 6.54 points at the illustrated task mixture. The caption says explicitly that the published component scores determine both profiles.

We rewrote the opening around the comparison at a fixed deployment mean. The revised introduction credits the profile and expected-risk rule to prior work and states the finite-suite and ranking results separately. The experiments now use the same matched comparison when discussing whether averaging changes the selection.

If these address your concerns, we would be grateful if you would consider revising your score.

Reviewer_Ep4tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines why aggregate benchmark scores may not always lead to the best model choice in real-world deployments. The authors use Risk Profiles to capture the minimum achievable risk under different deployment conditions, considering adjustments such as threshold tuning and model routing. They establish the Collapse Boundary, showing that evaluating risk across deployment conditions is equivalent to evaluating at their mean when the Difference Profile is affine. They also show how finite benchmarks can overlook the benefits of reconfigurable systems.

Based on these findings, the authors propose OPERA and R-OPERA and evaluate them across several applications, including LLM leaderboards, RLHF pipelines, content moderation, LLM routing, and clinical prediction. The results show 13–28% lower regret in adaptive RLHF pipelines and 72% lower regret in content moderation compared to evaluation at the mean deployment condition.

Critical strengths
  • The paper provides a useful theoretical explanation for the limitations of conventional benchmark evaluation. In particular, the Collapse Boundary and Theorems 3 and 4 help explain when aggregation matters and why finite benchmarks may overlook the advantages of reconfigurable systems.

  • The experiments cover a wide range of applications, including LLM leaderboards, RLHF pipelines, content moderation, and clinical prediction. I also appreciate that the authors report cases where their methods do not improve performance, rather than focusing only on positive results.

Critical weaknesses
  • Limited evaluation under realistic deployment conditions. While the paper focuses on real-world model selection, many experiments rely on hypothetical deployment mixtures or offline evaluations. Section 4.1 mainly evaluates hypothesized task distributions, while the analysis of observed Arena traffic shows that the headline model is already mixture-optimal in 98% of sampled shortlists. Although the authors acknowledge the limitations of this analysis, it remains unclear how often the proposed methods would provide meaningful benefits in real-world deployments.

  • Dependence on accurate deployment-condition estimates. The benefits of the proposed methods seem to rely heavily on knowing the deployment conditions. In Appendix H.3, OPERA achieves substantial improvements when preference weights are known, but the gains are not statistically significant when those weights are estimated, even with 300 preference labels. Since deployment preferences are often uncertain or change over time.

  • Unclear benefits of R-OPERA. Although R-OPERA is intended to handle deployment uncertainty, its advantages over simpler methods are not clear. In content moderation (Appendix I.3), R-OPERA does not improve over the development-condition point-mass baseline, while OPERA and uniform weighting perform substantially better. It would be helpful to clarify when the additional robustness is actually beneficial.

  • Some labels in Figure 2(a) overlap, making the figure difficult to read.

Questions for authors
  • How well do OPERA and R-OPERA perform when deployment conditions are only approximately known? For example, in an LLM service where the task distribution is uncertain or changes over time, would the proposed methods still provide meaningful improvements?

  • Under uncertain deployment conditions, when does R-OPERA provide meaningful improvements over OPERA or simpler deployment-aware evaluation methods?

3 comments

We thank the reviewer for their detailed comments. Please see our common response and specific reponse below.

How realistic are the deployments? The four-leaderboard study uses hypothesised workload mixtures. We report those results separately from the logged Arena analysis and the clinical evaluations. In Arena, the observed language markets remained close to the pooled mixture. In the temporal evaluation, using earlier traffic to inform the next month, the headline pick was optimal in 97.8% of single-model shortlists. The larger changes on constructed, concentrated workloads show what can happen if a service has a different task mix. The clinical analyses evaluate realised outcomes across care units and hospitals. The deployment ranges used by the rules are specified, and the actual prevalence of an individual destination is not supplied to the rule. These are held-out evaluations, and the results establish gains on those settings. They leave prospective performance in a changing service open.

What happens when the conditions are estimated? The preference experiment uses newly generated language-model responses and simulates preference comparisons from a known weighting to measure the effect of having only a few labels. At 10 labels, estimated preferences increased regret by 3.9% relative to the matched point [0.2, 8.0]. The intervals at 30, 100, and 300 labels included zero. At 300 labels the point estimate favoured averaging by 22.8%, with an interval of [−1.3, 34.8]. For a fixed-model LLM service whose current workload can be measured, published per-task scores provide a direct way to update the evaluation weights.

When is R-OPERA worth using? Exact R-OPERA selects the model with the smallest worst-case expected risk over a specified Wasserstein ball. That objective is useful when adverse deployment mixtures matter to the decision and the ball describes plausible shifts. The reported experiments evaluate regret at particular deployments. In moderation, exact R-OPERA improved on the development-prevalence point by 70.2% [11.3, 90.8], while the integral and several simpler deployment-range rules gained 86.0% [71.5, 92.3]. Under MIMIC-IV temporal shift, exact R-OPERA gave a regret reduction of −50.4% [−91.6, −18.2] relative to the matched point. We now present it as a worst-case option with an empirical cost in some settings.

Figure 2. The panel titles and legend are now outside the plotted data area.

If these address your concerns, we would be grateful if you would consider revising your score.

Thank you for the rebuttal. The clinical evaluations and the clarification regarding the exact R-OPERA address some of my concerns, but several issues remain.

Even with 300 preference labels, the improvement is not statistically significant. For R-OPERA, simpler methods such as OPERA and uniform weighting sometimes achieve better results. Although the authors mention revising Figure 2, the overlapping labels in Figure 2(a) are still present in the current manuscript. However, this issue about the Figure 2 did not affect my overall assessment.

Overall, while the response addresses some concerns, the practical benefits under realistic deployment uncertainty remain insufficiently demonstrated. I therefore maintain my original rating.

Thank you for engaging with the rebuttal. Your rating rests on one concern: that the practical benefit under realistic deployment uncertainty was not demonstrated, since the gain with estimated preferences was not significant at 300 labels. We show that this is indeed the case through an experiment.

The benefit survives estimated preferences. We extended the estimated-preference experiment to 1,000 and 3,000 labels. Both budgets were fixed in a hashed note before anything ran (included in the supplement), and every other setting was left unchanged: the same generated responses, judge ratings, 240 replications and comparator.

Preference labels Regret reduction vs. evaluation at the mean 300 22.8% (−1.3, 34.8) 1,000 37.3% (5.2, 51.1) 3,000 45.1% (13.2, 58.5) weights known exactly 46.8% (9.4, 61.2) At 1,000 labels the gain is significant, and at 3,000 the deployer that estimates its preferences recovers nearly all of the gain of one that knows them. A second run with a different random stream gives the same result: 37.6% (5.4, 51.4) and 44.7% (9.5, 58.5). As in the original experiment, the labels are simulated from a Bradley–Terry model of the true weighting. A few thousand pairwise comparisons is a small annotation budget for a deployed system.

The paper now shows benefit under deployment uncertainty in three settings, each with the deployment condition unknown to the selector.

Estimated preferences (RLHF): significant gains from 1,000 labels, as above. Unknown hospital mortality (eICU, 96 hospitals): one selection is applied to every hospital, and averaging over the plausible range reduces regret by 4.6% (1.7, 7.4) relative to evaluating at the range's mean. This was the primary prediction of a protocol hashed before we touched the data. Unknown harmful-content rate (moderation): reading the deployment range instead of the development prevalence reduces regret by 86% (71.5, 92.3). R-OPERA optimizes worst-case risk over a set of plausible deployment distributions; none of the paper's practical conclusions rests on it.

On Figure 2: the uploaded PDF was built from an outdated figure file. The corrected figure is now in the revision, and every substantive change is marked in blue.

What the paper contributes. It gives an exact condition for when averaging over deployment conditions can change which model is selected. It proves that, when risk is linear in the deployment condition, averaging never changes the selection among fixed models, which covers every model on a public leaderboard. It supplies a calibrated test that tells a practitioner in advance whether averaging can matter. And it validates all of this on eight systems, including a pre-registered replication across 96 hospitals. The new experiment closes the one gap you identified: the benefit holds when the deployment's preferences must be estimated from a realistic amount of data.

We believe this resolves the concern your rating rests on, and we would be grateful if you would reconsider your score.

Reviewer_PNRWOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies model selection when deployment conditions differ from those used for evaluation. It represents each system by its performance across different conditions and examines when averaging over these conditions changes the selected model. It proposes selection rules based on average risk and robust variants, supported by theoretical analysis and experiments on language models, moderation, routing, and clinical prediction.

Critical strengths
  1. Relevant problem and useful perspective. The paper highlights that a model ranked highest on a benchmark may not be the best choice for a particular application. Its distinction between matching evaluation to deployment and changing the evaluation rule is useful.
  2. Broad experimental investigation. The evaluation covers diverse settings and reports both positive and negative results. This helps identify limitations rather than presenting the proposed approach as universally beneficial.
Critical weaknesses
  1. The paper is not clearly written and is hard to follow. The presentation is dense, and the relationship between the theoretical results, proposed selection rules, and experiments is difficult to understand. The paper should state its main contribution more directly and clearly distinguish established ideas from new findings.
  2. The added practical value is not sufficiently established. Evaluating models according to deployment conditions is an established principle, and much of the framework builds on existing theory. The experiments show mixed benefits from the proposed aggregation rules. Gains depend on assumptions such as knowing deployment preferences, and simpler alternatives sometimes perform better. Consequently, it remains unclear when the proposed approach offers a meaningful advantage in practice.
Questions for authors
  1. What is the main new contribution beyond existing approaches that evaluate expected risk across operating conditions?
  2. Under what realistic conditions should the proposed selection rules outperform evaluation using deployment-specific weights, particularly when deployment conditions must be estimated?
  3. Minor - subcaption (a) covers the illustration in Fig. 2.
3 comments

We thank the reviewer for their review. Please see our Common Response and specific reply below.

What is new? Cost curves already tell us how risk varies with the operating condition. Integrating those curves is also established. Our comparison holds the deployment mean fixed and asks whether changing the evaluation from a single point to an average changes which system wins. Proposition 1 separates equality of risk differences from strict ranking reversals. The finite-suite results address a second question: what can be inferred about a reconfigurable system when the evaluator reports its scores at only a few conditions?

When is averaging useful in practice? If the deployment is a single known condition and the risk profiles are accurate there, selecting the model with the lowest risk at that condition is optimal. If deployments span a range, the reference comparison is averaging over the range versus evaluating at its mean. Under the affine-risk assumption, both rules make the same selection among fixed models. With reconfiguration, the rules can disagree. Whether a disagreement is useful depends on how well the profiles and the deployment distribution are estimated. The curvature test checks whether a difference from affinity exceeds estimation noise. It does not predict the size or direction of a regret change.

The simpler baseline is competitive in some settings. In MIMIC-IV, averaging reduced regret by 14.2% [4.6, 23.8] relative to the mean-condition point. Selecting at the estimated unit prevalence gave −10.3% [−23.8, 0.9] relative to that development baseline. Even using the true unit prevalence gave −6.5% [−16.4, 2.3]. Those intervals include zero. The point estimates illustrate the difficulty of using risk profiles across care units when class-conditional distributions can change.

Figure 2. We moved the panel titles above the plots and put the legend below the panels. The caption now specifies which information a stack-level evaluation uses.

If these address your concerns, we would be grateful if you would consider revising your score.

Reviewer_PNRWOpenReview ↗

Thank you for your response

Thank you for your response. I appreciate your clarifications. I am still concerned by the paper writing and its clarity. The authors also acknowledge that the simpler baseline is competitive in some settings. I therefore maintain my score.

Thank you for your response. We would like to clarify that "the simpler rule is competitive in some settings" is a result of the paper, not a limitation. Proposition 1 and Assumption 1 show that, for fixed models, evaluating at the deployment’s mean condition is exactly as good as averaging. Averaging can do better only for systems that reconfigure, where the risk differences are non-affine and change sign. The calibrated test tells a practitioner in advance which case applies. The paper therefore says that when the simple rule is enough, which the work that recommends averaging does not. On clarity and presentation, if there is a section you found hardest to follow, we would be glad to revise it.

Comments on the paper

1 comment

Thank you for the reviews. Our main contribution is an exact answer to a model-selection question: when can averaging over deployment conditions change the winner after we have already matched evaluation to the deployment’s mean? This comparison separates the value of using the right task weights from the value of evaluating across a range. A team choosing among fixed models with affine risk can stop after reweighting the published scores. A team choosing among systems that retune at deployment can gain from averaging, and the paper shows when a reversal is possible and how to test whether it matters in practice. Proposition 1 distinguishes equality of risk differences from agreement in ranking. Affinity gives equal differences for every deployment distribution. Part (iv) says exactly when the ranking can reverse. Theorems 3 and 4 extend the decision question to the design of a benchmark: they show what finite stack-level evaluations can miss and how the spacing of the evaluated conditions limits the regions that can be missed.

The experiments help show the size of these effects. On the four leaderboards, reweighting published scores changed the top pick in 17–32% of sampled shortlists under hypothesised deployment mixtures. In the observed Arena language markets, the headline pick was already optimal in about 98% of shortlists. Among reconfigurable systems, averaging reduced regret by 13–28% across five offline RLHF pipeline variants, 14.2% [4.6, 23.8] across MIMIC-IV care units, and 4.6% [1.7, 7.4] across 96 eICU hospitals, relative to evaluation at the mean condition.

ReviewerConcernResponseManuscript
PNRW, pj8DClarity and novelty beyond expected-risk evaluationWe explain the comparison at a fixed deployment mean, credit the established risk-profile and averaging framework, and state separately what the paper adds about strict ranking reversals and what finite evaluations can miss.Abstract, §1, Proposition 1, Theorems 3–4
PNRWWhen averaging improves on simpler deployment-aware rulesWe distinguish a known operating condition from a distribution over conditions. We give the matched-mean and development-prevalence comparisons separately and explain that the curvature test flags possible disagreement between rules. Held-out regret measures whether that disagreement helps.§4.3, Appendix H.1
Ep4tEvidence from realistic deploymentsWe separate hypothesised leaderboard workloads from logged Arena traffic and held-out clinical evaluations. The headline pick was optimal in 97.8% of Arena temporal-replay shortlists. Averaging reduced regret by 4.6% [1.7, 7.4] across 96 eICU hospitals relative to the matched mean.§§4.1 and 4.4, Appendices F.6 and H.2
PNRW, Ep4tEstimated or changing deployment conditionsWe report the preference estimates at each label budget, the true- and estimated-prevalence baselines, and the temporal-shift results. The intervals at 30, 100, and 300 preference labels include zero. At 10 labels, estimating the weights increased regret relative to the matched point.§4.2, Appendices G.3 and H.1
Ep4tWhen R-OPERA helpsWe explain when minimising worst-case risk over a Wasserstein ball is relevant and report its realised-regret results. The experiments did not establish an advantage over plain averaging.Appendix C.1, Appendices H.1 and H.3–H.4
PNRW, Ep4tFigure 2 labels overlap the panelsWe moved the panel titles above the plots and placed the legend below them.Figure 2
R3Incorrect mean of the log-odds-uniform measureWe corrected the quadrature and reran the affected studies. The mean on [0.02, 0.20] is 0.081. Table 9 documents how the correction affected the preregistered eICU result.Appendices D.4 and E.7, Table 9
pj8DFirst-order robust score can select a dominated modelWe give a counterexample and use the exact Wasserstein objective for prevalence and routing. In RLHF, the surrogate is used at radius 0.05, where its selections agree with the exact objective on the tested lattice.Remark 5, Appendix C.5
pj8DTheorem 5 at a maximiserWe take the positive part of the sensitivity and handle the zero-sensitivity case in the proof.Theorem 5
pj8DEquality of risk differences versus agreement in rankingWe state the equality result for every measure and give a separate necessary and sufficient condition for a strict reversal.Proposition 1(i–ii, iv)
pj8DFixed configuration does not imply affine riskWe state affinity at each fixed configuration as an assumption and identify joint variation in prevalence and cost ratio as a case where it can fail.§2.2, Assumption 1
pj8DPublished component scores determine the routed profilesWe limit the blind-set example to an evaluator with only the seven stack-level scores. The caption states that the component scores and stack membership determine both profiles.§3, Figure 2

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.