acceptodds
Under review as a conference paper at ICLR 2027

From Benchmark to Deployment: Efficient Evaluation of LLMs for Health under Task and Evaluator Shift

Abstract

Large language model (LLM) benchmarks evaluate models under a fixed task distribution and scoring protocol, both of which may differ from those relevant to deployment. We study how to reuse benchmark scores for deployment-specific model selection and performance estimation when scoring model responses on target tasks is resource-intensive. First, without target scores, we transport benchmark evaluations by reweighting tasks using semantic representations of prompts and rubrics, without requiring annotated shift attributes explicitly. On HealthBench, this approach can match or outperform reweighting based on known shift attributes in target-ranking recovery and mean-score estimation. Second, when a small number of target tasks are evaluated by the original benchmark evaluator, we combine their scores with benchmark scores to improve target-mean estimation and model-ranking recovery. Third, we address evaluator shift, a practical challenge when deployment evaluation relies on a human expert or a newer or more scalable LLM judge instead of the benchmark evaluator. Paired scores on a small target sample allow us to exploit the association between two evaluators' scores to improve target-mean estimation. We derive efficient influence functions for the target mean score in both the same- and different-evaluator settings. Building on these, we develop the Covariate-Adaptive Fusion Estimator (CAFE). Rather than separately estimating complex nuisance functions and plugging them into the efficient influence function, CAFE directly learns how to combine benchmark and target scores through variance minimization. Under suitable conditions, CAFE achieves semiparametric efficiency and supports asymptotically valid inference. Our experiments show that it improves target-mean estimation over existing transport and calibration approaches. Together, these results provide a principled route from static benchmark scores toward deployment-specific model selection and performance assessment under limited evaluation budgets.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
benchmark adaptation, causal inference, distribution shift, LLM evaluation, semiparametric efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FHr6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is well organized and well written. It addresses an important problem of predicting model performance under distribution shift with a low budget/resources to evaluate on the target distribution. They introduce a new method that combines reweighting and source benchmark scores to adjust the ranking of models on the target. The separation of evaluator shift from task shift is also an interesting formulation.

Critical strengths

The separation of evaluator shift from task shift is also an interesting formulation. The authors have done extensive experiments and have considered enough variation and benchmakr under the given assumption to support the claims.

Critical weaknesses

The main weakness of the paper is that it only considers HealthBench, and the generalizability of the results is questionable. In particular, the source and target distributions are constructed from the same benchmark.

Another issue is the assumption that the mean score is invariant between source and target, conditioned on the prompt representation. The authors try different representation dimensions and prompt vs. prompt+rubric representations, but they use the same BGE-M3 encoder for all experiments. It would be helpful to evaluate whether the results and the assumption are sensitive to different representation models.

Questions for authors

NA

1 comment
AuthorsOpenReview ↗

Response to Reviewer FHr6

We thank the reviewer for asking whether findings from one health benchmark carry over to another. Our discussion helped us see how the statistical analysis can guide the choice of which responses to score in a new domain. When the target evaluator is a human expert or an advanced LLM judge, its cost can limit how many tasks it scores.

In transfer from HealthBench to MentalHealthBench, we reserved the more expensive target judge, GPT-5.6 Sol, for a small random sample. Only the inexpensive HealthBench evaluator, GPT-4.1, scored the additional random sample of target tasks. Using GPT-4.1 in both domains lets us estimate the source–target gap for each response model and adjust its HealthBench scores before transfer. With 30 costly target evaluations, this design reduced RMSE from 6.49 to 4.83 and increased full-rank recovery from 8.6% to 12.0% relative to the target-only mean (Table 2).

1. Generalizability beyond a single benchmark

We thank the reviewer for suggesting broader validation. We have expanded our experiments to include both a second benchmark and direct transfer between distinct benchmarks.

On MentalHealthBench, which contains 1,215 synthetic mental-health conversations with evaluation rubrics, we varied user-profile composition and evaluated five response models over 500 splits. Without target evaluations, representation-based reweighting improves mean-score estimation and model-ranking recovery over the unweighted source mean. Among methods using target evaluations, CAFE achieves the lowest MAE and RMSE at all three budgets of 50, 100, and 200 evaluations in both the same- and different-evaluator settings. These results are reported in Tables 27–28 and Figure 20 in Appendix E.5.

We evaluated transfer across domains from HealthBench to MentalHealthBench, using GPT-4.1 as the benchmark evaluator and GPT-5.6 Sol as the target evaluator (Section 6.3, Table 2). For all five response models, the projected covariance measure (PCM) test found evidence that mean GPT-4.1 scores differed between benchmarks after conditioning on the task representation (Bonferroni-adjusted pp-values from 1.29×10−51.29\times10^{-5} to 0.040.04).

We address this shift through an evaluation design that collects inexpensive scores from the benchmark evaluator (GPT-4.1) on 50% of target tasks not selected for target evaluation and uses these scores to calibrate CAFE. Corrected CAFE achieves the best or tied-best performance across all four metrics at all three target-evaluation budgets of 30, 50, and 100. With only 30 target evaluations, it reduces MAE/RMSE from 5.20/6.49 to 3.84/4.83 relative to the target-only sample mean and increases full-rank recovery from 8.6% to 12.0%. This experiment demonstrates improved performance estimation and model-ranking recovery across distinct benchmarks while keeping the costly target-evaluation budget fixed.

2. Sensitivity to the representation encoder

The mean-score estimation gains persist when Qwen3-Embedding-4B replaces BGE-M3. We repeated Case 2 in both evaluator settings, keeping the other settings and hyperparameters fixed. With Qwen representations, CAFE has the lowest MAE and RMSE at all three budgets in both settings (Table 25 in Appendix E.4.4).

At 200 target evaluations, MAE decreases from 1.66 to 0.98 in the same-evaluator setting and from 1.66 to 1.38 in the different-evaluator setting relative to the target-only sample mean, on the table's ×102\times 10^2 scale. These are reductions of 41.0% and 16.9%. Exact ranking does not uniformly improve: under evaluator shift at 500 evaluations, full-rank recovery is 52% for CAFE versus 56% for the sample mean. We used 1,024-dimensional embeddings and retained 50% of the variance separately for prompts and rubrics, yielding 47 and 20 principal components.

The assumption concerns mean scores conditional on the representation, so changing the encoder also changes what is held fixed. For instance, an encoder that omits a task attribute related to the score can leave residual differences if that attribute is more common in one population. Table 25 assesses the estimator's performance with both scoring protocols. Its results alone cannot tell us whether conditional-mean invariance holds for either encoder or is equally plausible for both. The additional benchmark scores in our cross-benchmark design let us estimate and adjust for a model-specific source–target gap.

Reviewer_1RUKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to use existing benchmark results to more accurately estimate LLM deployment performance and select models under a limited target-domain budget, when task shift and evaluator shift exist between real deployment and static benchmarks. It considers three progressive scenarios: (1) without target-domain scores, density-ratio reweighting via prompt and rubric semantics transfers benchmark results to the target distribution; (2) with only a few target tasks scored by the original evaluator, source and target scores are fused; (3) with a different target evaluator, residual correlation between evaluators is exploited via a few paired evaluations. The paper derives efficient influence functions for the target mean under same- and different-evaluator settings and proposes CAFE, which learns the optimal fusion of benchmark and target information via variance minimization, avoiding direct estimation of complex conditional variances/covariances and achieving semiparametric efficiency under certain conditions. Experiments on 5,000 multilingual HealthBench samples across three task-shift scenarios and five LLMs show that prompt embeddings already notably improve ranking recovery and target mean estimation, with rubrics usually yielding further gains.

Critical strengths

The research problem is highly practical. Rather than simply improving performance on a benchmark, the paper addresses the task-distribution shift and evaluator shift that often exist between benchmarks and real deployment, and further considers the realistic constraint of a limited target evaluation budget. The three scenarios—from "no target labels" to "limited target labels" to "evaluator change"—form a complete and logically clear problem setting.

The theoretical contribution is solid and closely integrated with algorithm design. The paper derives efficient influence functions for the same-evaluator and different-evaluator settings, and clarifies that the efficiency gain in the different-evaluator case stems from the residual association between the two evaluator scores given the task representation. This gives CAFE a clear statistical interpretation rather than being mere empirical score fusion.

CAFE is methodologically innovative. Directly implementing the EIF requires estimating multiple nuisance functions such as conditional covariance, conditional variance, and density ratio, which is unstable when target evaluations are scarce. CAFE converts this into directly minimizing influence-function variance to learn how much source information to borrow—a natural and practical design.

Experimental results are strong, especially in the same-evaluator setting. In Case 2, CAFE with only 200 target evaluations outperforms the target-only sample mean with 500 target evaluations; compared with methods using the same 200 target evaluations, full-rank recovery rises from 28% to 43%, with MAE/RMSE reduced by over 40%. This well demonstrates the value of benchmark information under a limited evaluation budget.

Experiments and supplementary analyses are thorough. The paper compares not only KMM but also domain classifier, uLSIF, RuLSIF, KLIEP, predicted-theme weighting, Hybrid KMM, and other reweighting methods in the appendix, and tests PC30/50/80/95 as well as full embeddings. The results also analyze the relationship between representation dimension and effective sample size, showing that the authors systematically validate the stability of embedding-based reweighting.

Critical weaknesses

The biggest limitation is that the "deployment shift" still mainly comes from manual resampling on the same HealthBench dataset, limiting external validity. All three source-target shifts are constructed on the same 5,000 HealthBench examples via theme/language allocation, rather than natural distribution changes from real hospitals, different time periods, user groups, or benchmarks. Thus the experiments well demonstrate handling of controlled covariate shift, but still fall short of the "From Benchmark to Deployment" emphasized in the title; adding another benchmark or naturally occurring shift would significantly strengthen the paper.

CAFE's effectiveness relies on strong conditional mean invariance/overlap assumptions, but the paper lacks systematic experiments on assumption violation. The core assumption requires E_S[Y|X]=E_T[Y|X] and sufficient source-target overlap; limited target evaluation also assumes target samples are randomly selected for scoring. However, the semi-synthetic experiments explicitly construct data to satisfy Assumption 1, and the real HealthBench experiments do not systematically control the degree of violation. This is not a theoretical error, but it matters for deployment: once X fails to capture hidden factors affecting model performance, source information may introduce bias. The authors themselves acknowledge the method cannot handle arbitrary unmeasured shifts.

In the evaluator-shift setting, CAFE's advantage in ranking recovery is unstable. CAFE clearly improves MAE/RMSE in the different-evaluator setting, but in Cases 1 and 2 its full-rank recovery is usually lower than the target-only sample mean; only in Case 3 does it match or exceed the target-only baseline. Thus the paper strongly supports "performance estimation" but should be more cautious about the stronger claim of "model selection" under evaluator shift.

Evaluator-shift validation is narrow. Real experiments use only GPT-4.1 as benchmark evaluator and Gemini Flash-Lite as target evaluator. Since CAFE's gains theoretically depend heavily on the residual association between the two evaluators, validating only one evaluator pair makes it hard to judge whether conclusions generalize to human experts, different judge families, or weakly correlated evaluators.

Questions for authors

On robustness to the core assumption: CAFE's theory relies on E_S[Y|X]=E_T[Y|X]. Can the authors add controlled experiments that progressively violate conditional mean invariance and show how CAFE's bias/variance changes? In practice, are there diagnostic methods to assess how severely this assumption is violated?

On real-deployment generalization: all task shifts are currently obtained by resampling the same HealthBench dataset. Can the authors validate the method on another benchmark or more natural domain/temporal/language shifts? Otherwise, how should the scope of the paper's conclusions on "deployment-specific evaluation" be understood?

On the universality of evaluator shift: real experiments only cover one evaluator pair, GPT-4.1 → Gemini Flash-Lite. Since CAFE's gains depend on residual correlation, what happens when the residual association between two evaluators is weak or even negative? Can the authors report multiple evaluator pairs or a human–LLM evaluator setting?

On ranking performance under evaluator shift: CAFE reduces MAE/RMSE in Cases 1–2, but full-rank recovery is sometimes lower than the target-only sample mean. Can the authors explain this? Is it because CAFE's objective optimizes single-model target-mean estimation variance rather than explicitly optimizing ranking uncertainty across models?

On non-random target evaluation: the paper assumes target examples are randomly selected for evaluation, i.e., a constant evaluation probability α. In real deployment, experts tend to label difficult, uncertain, or high-risk cases. Can CAFE be extended to non-constant P(R=1|X)? What happens if evaluation selection is related to unobserved outcomes?

On uncertainty estimation: since the paper proves consistency of the variance estimator and asymptotically valid inference, can the authors add empirical coverage and interval width of 90%/95% confidence intervals, especially under the small-sample condition of 200 target evaluations?

On the design of Case 3: Case 3's allocation uses existing model-score summaries to construct a rank-shift stress test. If model scores were not used at all and a pre-specified theme-language shift were adopted instead, would CAFE's advantage still hold? The authors are suggested to clarify whether this design affects the main conclusions.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1RUK

Thank you for asking how researchers can tell when benchmark scores will transfer. Our discussion brought out a practical implication of the analysis for study design. The transport assumption concerns how the benchmark evaluator scores responses in the two domains. In a deployment study, a human expert or an advanced LLM judge may be too costly to use on every target task. We therefore score an additional random sample of target tasks using only the inexpensive benchmark evaluator, such as GPT-4.1 on HealthBench. Comparing its scores across domains lets us assess conditional-mean stability, estimate the source–target gap for each response model, and adjust its source scores without increasing the number of costly target evaluations.

Our response to Comment 5 quantifies sensitivity to such gaps. In transfer from HealthBench to MentalHealthBench, corrected CAFE achieved the lowest MAE and RMSE, the highest Kendall's τb\tau_b, and the highest or tied-highest full-rank recovery at all three target-evaluation budgets (Table 2 and our response to Comment 6).

Weaknesses

1. External validity beyond resampled HealthBench shifts.

The revised manuscript includes validation on MentalHealthBench (Appendix E.5) and direct transfer from HealthBench to MentalHealthBench (new Section 6.3), extending our evaluation beyond controlled resampling within a single benchmark. In the cross-benchmark experiment, the design described in our opening response improves target-mean estimation and model-ranking recovery at a fixed target-evaluation budget. Our response to Comment 6 gives more details.

2. Robustness under assumption violations.

Comparing the same evaluator across domains gives a transfer study a direct way to assess potential bias from borrowing source scores. We use inexpensive benchmark-evaluator scores on additional target tasks to estimate the gap for each response model and adjust its source scores. In our sensitivity experiment, uncorrected CAFE improves on the target-only mean for small shifts, and corrected CAFE achieves lower MAE and RMSE than all four baselines across all tested shifts and label budgets (Table 12). This experiment varies the mean shift at a fixed covariate distribution; covariate overlap is a separate requirement, stated in Section 2, with weight concentration for HealthBench reported in Appendices A.5–A.6 (see also our response to Comment 5).

3. Stability of ranking recovery under evaluator shift.

We agree that the evaluator-shift results support improved mean-score estimation more strongly than improved exact ranking. We have revised the Abstract and Introduction to state that CAFE improves ranking recovery in the main same-evaluator HealthBench experiments, whereas its ranking performance under evaluator shift varies across cases. We now frame model selection as an application supported by deployment-specific performance estimates.

We also observed better ranking recovery with our new evaluation design in transfer from HealthBench to MentalHealthBench (Section 6.3, Table 2). With 30 target evaluations, corrected CAFE increased full-rank recovery from 8.6% to 12.0% and Kendall's τb\tau_b from 0.52 to 0.58 relative to the target-only sample mean. Both ranking measures were also higher at 50 and 100 target evaluations.

4. Generalization across evaluator pairs.

We evaluated two additional target evaluators, Claude Opus 5 and Jev 1.13, with GPT-4.1 providing benchmark scores. In Case 2, PPI++ has the lowest MAE and RMSE among the baselines for both evaluator pairs at all three budgets. Under the new design for cross-benchmark transfer, corrected CAFE ranks first or ties for first on both ranking metrics at each budget (Table 2). Our response to Comment 7 gives more detail on evaluator pairs and the experiment with negative correlation.

Questions

5. On robustness to the core assumption.

We progressively violate conditional mean invariance by subtracting δ∈0.01,0.02,0.03,0.04,0.05,0.10\delta\in\\{0.01,0.02,0.03,0.04,0.05,0.10\\} from each source benchmark-evaluator score, yielding conditional mean gaps of 1 to 10 points on a 100-point scale, while leaving the target distribution unchanged. Without correction, CAFE achieves lower MAE and RMSE than the target-only mean for gaps of up to 4 points with 30 expected target labels, 3 with 50, and 2 with 100 or 200 (Table 12 in Appendix E.1.2). The degradation is driven mainly by bias: at 200 expected target evaluations, increasing the gap from 0 to 10 points shifts the bias from +0.05+0.05 to −6.35-6.35 points, while the standard deviation, computed from the rounded bias and RMSE, changes only from about 1.58 to 1.82.

We assess conditional-mean stability using the projected covariance measure (PCM) test of https://doi.org/10.1214/24-AOS2447. Specifically, let DD denote source–target membership. The model-specific null hypothesis is H0:E[Y∣X,D]=E[Y∣X].H_{0}:\mathbb{E}[Y\mid X,D]=\mathbb{E}[Y\mid X].

Questions (continued)

5. On robustness to the core assumption (continued).

Across 500 replications with 200 expected target evaluations, the rejection rates at nominal 5% were 7.8% under the null, 10.0% at δ=0.02\delta=0.02, and 73.0% at δ=0.10\delta=0.10. The test had limited power for small shifts and rejected somewhat more often than its nominal level under the null. Non-rejection therefore cannot establish that borrowing source information is reliable.

We then applied the correction described in our opening response, collecting benchmark-evaluator scores YY on a random 50% of target tasks outside the original labeled sample. Corrected CAFE achieves lower MAE and RMSE than the target-only mean across all tested simulation settings. At 50 expected target labels, its RMSE is 3.23 points, compared with 4.22 for the target-only mean and 3.23 using the (infeasible) true shift (Table 12, panels A and C, in Appendix E.1.2). In the HealthBench-to-MentalHealthBench transfer with 30 target evaluations, it reduces MAE and RMSE by 26.1% and 25.5% relative to the target-only mean, the strongest comparator (Table 2 in Section 6.3). Extending this correction to covariate-dependent shifts remains future work.

6. On real-deployment generalization.

We thank the reviewer for suggesting broader validation. We have expanded our experiments to include both a second benchmark and direct transfer between distinct benchmarks. Within MentalHealthBench, which contains 1,215 synthetic mental-health conversations with clinician-written evaluation rubrics, we varied user-profile composition and evaluated five response models over 500 splits. Without target evaluations, representation-based reweighting improves mean-score estimation and model-ranking recovery over the unweighted source mean. Among methods using target evaluations, CAFE achieves the lowest MAE and RMSE at all three budgets of 50, 100, and 200 evaluations in both the same- and different-evaluator settings. These results are reported in Tables 27–28 and Figure 20 in Appendix E.5.

Transfer from HealthBench to MentalHealthBench introduces a domain shift from general health questions to mental-health conversations. It approximates the change in task mix one might encounter when using a general health benchmark to select a model for a mental-health clinic. We used GPT-4.1 as the benchmark evaluator and GPT-5.6 Sol as the target evaluator (Section 6.3, Table 2). For all five response models, the projected covariance measure (PCM) test found evidence that mean GPT-4.1 scores differed between benchmarks after conditioning on the task representation (Bonferroni-adjusted pp-values from 1.29×10−51.29\times10^{-5} to 0.040.04). Applying the design described in our opening response, we obtained inexpensive GPT-4.1 scores on a random 50% of the remaining target tasks and used them to correct the source scores. Corrected CAFE achieves the best or tied-best performance across all four metrics at all three target-evaluation budgets of 30, 50, and 100. With only 30 target evaluations, it reduces MAE/RMSE from 5.20/6.49 to 3.84/4.83 relative to the target-only sample mean and increases full-rank recovery from 8.6% to 12.0%. This experiment demonstrates improved performance estimation and model-ranking recovery across distinct benchmarks while keeping the costly target-evaluation budget fixed.

7. On the universality of evaluator shift.

First, the residual association need not be positive. The optimal fusion coefficient a∗(X)a^*(X) takes the sign of the conditional covariance γT(X)\gamma_{\mathcal T}(X), so a negative association can also improve efficiency through a negative coefficient. Weaker residual association yields smaller gains, and when it vanishes, the EIF reduces to that of target-only AIPW. In our semi-synthetic experiment with marginal correlation ρY,Y~=−0.4\rho_{Y,\widetilde Y}=-0.4, CAFE has lower MAE and RMSE than target-only AIPW at 200 and 300 target evaluations (Table 10, panel B, in Appendix E.1.2). At 200 evaluations, MAE decreases from 1.76 to 1.47 and RMSE from 2.20 to 1.88, with errors multiplied by 10210^2; the RMSE reduction is 14.5%. We include the full correlation grid and confidence-interval results in Tables 8–11 in Appendix E.1.2.

Second, we now evaluate two additional evaluator pairs, GPT-4.1 →\rightarrow Claude Opus 5 and GPT-4.1 →\rightarrow Jev 1.13. In Case 2 at 200 evaluations, CAFE reduces MAE/RMSE relative to PPI++ by 14.2%/13.5% for Opus 5 and 14.3%/13.2% for Jev (Table 20, Appendix E.4.2). Both metrics also improve at 300 and 500 evaluations. At 500, full-rank recovery for CAFE versus the target-only mean is 50% versus 51% for Opus 5 and 43% versus 44% for Jev. Tables 19–21 cover the original design; Table 2 assesses the new design with GPT-4.1 and GPT-5.6 Sol in HealthBench-to-MentalHealthBench transfer. These comparisons use automated judges; generalization to human-expert scores remains open.

Questions (continued)

8. On ranking performance under evaluator shift.

CAFE targets efficient estimation of each model's mean score. Even when MAE and RMSE decrease for individual models, errors in the differences between their estimated means can still affect the ranking. For two models, ranking depends on the estimated mean difference relative to their true performance gap, with

Var⁡(μ^j−μ^k)=Var⁡(μ^j)+Var⁡(μ^k)−2Cov⁡(μ^j,μ^k).\operatorname{Var}(\widehat\mu_j-\widehat\mu_k) = \operatorname{Var}(\widehat\mu_j) +\operatorname{Var}(\widehat\mu_k) -2\operatorname{Cov}(\widehat\mu_j,\widehat\mu_k).

Because models are evaluated on shared tasks, their estimation errors can move together and partially cancel in pairwise differences. Model-specific corrections can change this dependence, so reducing individual estimation errors does not necessarily reduce ordering errors. Full-rank recovery is particularly sensitive to closely performing models: reversing even one pair makes the entire ranking count as incorrect.

Empirically, the ranking differences in Cases 1–2 are modest: Kendall's τ\tau differs from the target-only mean by at most 0.02. Across 500 paired splits, full-rank recovery differences are not statistically significant at 200 or 300 target labels (all p≥0.30p\geq0.30). At 500 labels, recovery is lower by 4.6 and 3.8 percentage points in Cases 1 and 2, respectively (unadjusted p=0.026p=0.026 and 0.0490.049). This variation in exact recovery also motivated us to examine ranking under the new design described in our opening response. In HealthBench-to-MentalHealthBench transfer, corrected CAFE achieved higher full-rank recovery than the target-only mean at each budget, reaching 12.0% versus 8.6% with 30 target evaluations, 13.2% versus 11.8% with 50, and 20.4% versus 20.0% with 100. Kendall's τb\tau_b was also higher at each budget (Table 2).

9. On non-random target evaluation.

Thank you for this suggestion. We have updated Section 2 and Proposition 1 to accommodate covariate-dependent evaluation probabilities e(X)>0e(X)>0, assuming evaluation is independent of both scores given XX in the target population. The revised EIF includes the constant-probability design as the special case e(X)≡αe(X)\equiv\alpha.

If the probability that a target task is evaluated still depends on its unobserved target-evaluator score after conditioning on XX, weighting by 1/e(X)1/e(X) cannot generally identify the mean score of a distinct target evaluator. Identification would require additional assumptions or data, such as a random sample scored by the target evaluator.

10. On uncertainty estimation.

The added experiment shows near-nominal coverage with narrower intervals, including at 200 target evaluations. We evaluated 90% and 95% confidence intervals over 500 independent semi-synthetic replications, resampling covariates and generating new scores in each replication and assessing coverage against the fixed target-population mean. In the different-evaluator setting with marginal correlation ρY,Y~=0.8\rho_{Y,\widetilde Y}=0.8, CAFE's coverage at 200 evaluations is 89.4% and 94.0% for the nominal 90% and 95% intervals. Mean interval widths (×102\times 10^2) are 5.32 and 6.33, compared with 7.32 and 8.72 for the target-only sample mean, reductions of approximately 27%. CAFE has the smallest mean widths at all three budgets; its 95% coverage is 94.6% and 96.6% at 300 and 500 evaluations. Table 9 in Appendix E.1.2 reports coverage and width for the original estimator under conditional-mean invariance.

11. On the design of Case 3.

The added Case 4 experiment shows that the mean-score estimation gains persist when the shift is specified without model scores. Case 3 uses model-score summaries to set theme/language sampling proportions, with random sampling within each stratum. Case 4 instead uses pre-specified theme–language quotas for cross-region multilingual health consultations, oversampling global-health and non-English prompts without using model scores. CAFE has the lowest or tied-lowest MAE and RMSE at 200 and 300 target evaluations in both evaluator settings (Table 18 in Appendix E.4.1). At 200 evaluations, RMSE decreases from 1.87 to 0.80 in the same-evaluator setting and from 1.97 to 1.55 in the different-evaluator setting relative to the target-only sample mean, on the table's ×102\times 10^2 scale. These reductions of 57.2% and 21.3% show that the estimation gains also occur under a shift constructed independently of model scores.

Reviewer_f66YOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The author study the deployment of benchmark scores on resource-intensive loads. The underlying assumption is that many real-world applications lack data (which is quite common) and thus lack reliable, robust, and accurate benchmarking. The authors address this through CAFE, which learns how to combine benchmark and target scores through variance minimization.

Critical strengths
  • Elegant work and method design
  • Clearly motivated and clearly relevant
  • Use of prior methods in an efficient manner to provide robustness and clarity in the framing
Critical weaknesses
  • Limited evaluation using only GPT-4.1 and Gemini Flash Lite. Is it possible to use stronger models in your evaluation, it hard to understand if these numbers are generalizable (especially considering that the numbers between sota are quite close)
  • The lack of comparison with decision models (typesafe-jev etc.), the have been show to be quite strong in the field of evaluation
  • How does the model perform under low-dimensional data distribution (or different models?)
  • In the limitations you mention that this work does not imply robustness under arbitrary shift. As such, from your experience what is the maximum shift in data that is reasonably detectable/workable? The evaluation is limited thus this is hard to understand and unclear if that means that the benchmark is weak or what the core issue is?
  • The work strongly works around assumption 1, but is the statement generalizable for LLMs? Considering the randomness of the underlying model this claim might be a little too strong.

I currently lean to an accept due to the relevance of the topic, but I am generally on the edge and require my points to be addressed.

Questions for authors

Have been asked under weakness section.

3 comments
AuthorsOpenReview ↗

Response to Reviewer f66Y

Thank you for your questions. Our discussion showed how the sensitivity analysis can guide score collection before a transfer study begins. When target scoring requires a human expert or an advanced LLM judge, we reserve those costly evaluations for a small random sample. We score an additional random sample of target tasks using only the inexpensive benchmark evaluator, such as GPT-4.1 on HealthBench. We use those scores to estimate the source–target gap for each response model and adjust its source scores, keeping the number of costly target evaluations fixed.

Corrected CAFE achieved lower MAE and RMSE than the four baselines in the controlled experiments and in transfer from HealthBench to MentalHealthBench (Tables 12 and 2). Our response to Comment 3 reports how often the diagnostic detected shifts of different sizes.

1. We now add Claude Opus 5 and Jev 1.13 as target evaluators, keeping GPT-4.1 as the benchmark evaluator. For direct comparison with the manuscript’s main table, we use Case 2 (theme shift) and hold the task allocation fixed while changing only the target evaluator. In both settings, CAFE improves mean-score estimation over PPI++ (Table 20 in Appendix E.4.2; errors multiplied by 10210^2).

At 200 target evaluations, it reduces MAE from 1.55 to 1.33 and RMSE from 1.78 to 1.54 with Claude Opus 5, and MAE from 1.47 to 1.26 and RMSE from 1.67 to 1.45 with Jev 1.13, corresponding to reductions of about 14% in MAE and 13% in RMSE in both cases. Full-rank recovery and Kendall’s τ\tau are broadly similar across methods in Case 2. Across Cases 1–3 and budgets of 200, 300, and 500, CAFE achieves the lowest MAE and RMSE in all settings except Case 3 with 500 evaluations (Tables 19–21 in Appendix E.4.2).

2. In HealthBench Case 2 under evaluator shift, a lower-dimensional representation retained mean-score estimation gains at 200 and 300 target evaluations. Here, retaining 30% rather than 50% of the variance in each prompt and rubric embedding block reduces the number of covariates from 80 to 28.

Relative to PPI++, the best-performing comparator, CAFE reduces MAE/RMSE from 1.62/1.85 to 1.47/1.69 at 200 target evaluations (reductions of 9.3% and 8.6%) and from 1.32/1.50 to 1.25/1.44 at 300 (errors multiplied by 10210^2). At 500 evaluations, PPI++ had slightly lower MAE/RMSE (0.99/1.13 versus 1.00/1.14). Tables 22–24 in Appendix E.4.3 cover all three cases.

3. How much shift could CAFE tolerate? We lowered each source benchmark score by 1, 2, 3, 4, 5, or 10 points on a 100-point scale, keeping the target distribution fixed. Uncorrected CAFE had lower empirical MAE and RMSE than the target-only mean through gaps of 4 points with 30 expected target labels, 3 points with 50, and 2 points with 100 or 200 (Table 12 in Appendix E.1.2). This tolerance narrows as the target sample grows, because the target-only mean becomes more precise while the shift-induced bias persists.

Could we detect those gaps? Across 500 replications with 200 expected target evaluations, the projected covariance measure test of doi.org/10.1214/24-AOS2447 rejected at nominal 5% in 7.8% of runs under the null, 10.0% for a 2-point gap, and 73.0% for a 10-point gap. A small shift can therefore matter before this test reliably detects it.

Did the design help? We obtained benchmark scores on a random 50% of the remaining target tasks. Corrected CAFE had lower MAE and RMSE than all four baselines across the tested shifts and budgets, with performance close to that of an infeasible oracle that is given the true shift imposed in the simulation. At 50 expected target labels, RMSE is 3.23 points for corrected CAFE, 4.22 for the target-only sample mean, and 3.23 for the (infeasible) oracle (Table 12, panels A and C, in Appendix E.1.2).

In HealthBench-to-MentalHealthBench transfer, the same design with only 30 target evaluations reduces MAE/RMSE from 5.20/6.49 to 3.84/4.83 relative to the target-only sample mean, the best-performing comparator (Table 2 in Section 6.3). Extending the correction to shifts that depend on XX remains future work.

4. LLM randomness enters through sampling generated responses and, for an LLM judge, through stochastic scoring. For a fixed task, using the same response model, benchmark evaluator, and generation/scoring protocols gives an expected score after averaging over both sources of randomness.

A sufficient condition for Assumption 1 is that this expected score depends on the full task only through its representation XX. Then ES[Y∣X]=ET[Y∣X]=m(X)E_{\mathcal S}[Y\mid X]=E_{\mathcal T}[Y\mid X]=m(X) even though individual responses and scores vary. Our assumption restricts only these conditional means and permits different conditional variances across populations. Stochastic generation is therefore compatible with Assumption 1. Whether the representation captures enough to align the means across populations is an empirical question, which motivates the design above.

Dear Authors,

Thank you for the detailed response and the extensive evaluations added, I will keep my rating as is, but I am not really convinced with these number for different models, mainly the numbers are still fairly close between sota and your approach and the distinction is minor. Additionally, your method often looses to other sota works in many metrics (lower dim, new results for some metrics etc.).

We appreciate your positive assessment of the motivation and design of our method. Your follow-up raises two points worth clarifying: the size of the gains over competing methods and the results with a lower-dimensional task representation.

Magnitude of the gains. (1) Same evaluator. In Case 2 on HealthBench, with GPT-4.1 scoring both populations and 200 target evaluations, CAFE reduced MAE by about 6% and RMSE by about 5% relative to pooled-label DR, the strongest baseline at that budget. With those same 200 evaluations, CAFE also achieved lower MAE and RMSE than the target-only mean with 500 evaluations (Table 1A). (2) Different evaluators. When Gemini Flash-Lite, Jev 1.13, or Claude Opus 5 served as the target evaluator, we compared methods at the same budget of 200 target evaluations. Relative to PPI++ (the strongest competing method on both metrics for all three evaluators), CAFE reduced MAE by 13.9–15.2% and RMSE by 12.8–13.9% (Tables 1B and 20A–B). (3) Transfer across datasets. For the task of transferring from HealthBench to MentalHealthBench, with inexpensive benchmark scores on additional target tasks and only 30 costly target evaluations, corrected CAFE reduced MAE by 26.1% and RMSE by 25.5% relative to the strongest baseline, the target-only mean. Even at 100 costly evaluations, both errors were about 16% lower than with the strongest baseline at that budget, target-only AIPW (Section 6.3, Table 2). Overall, we find that our method and evaluation design yield practically meaningful gains in accuracy, particularly when an existing benchmark is used to evaluate models on a new task population. We consider the last setting to be the most practically relevant one.

Lower-dimensional representations. A low-dimensional representation can discard task differences that matter for model scores. This is precisely the advantage of reweighting on prompt and rubric embeddings: even when a lower-dimensional set of attributes driving the covariate shift is explicitly annotated, the higher-dimensional embeddings can capture additional score-relevant differences expressed in the text. Our primary analysis used PC50. In the PC30 sensitivity analysis, we deliberately retained fewer components to mimic reweighting on a low-dimensional set of annotated attributes. With PC30 in Case 2 under evaluator shift, CAFE had lower MAE and RMSE than PPI++ at 200 and 300 target evaluations, but slightly higher errors on both metrics at 500 (Tables 22–24). In our experiments, PC50 worked well, retaining enough principal components to explain at least 50% of the variance separately in the prompt and rubric embeddings. This shows that reweighting on a richer representation of the prompts and rubrics can account for score-relevant task differences even when those differences are absent from lower-dimensional features.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.