- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper is well organized and well written. It addresses an important problem of predicting model performance under distribution shift with a low budget/resources to evaluate on the target distribution. They introduce a new method that combines reweighting and source benchmark scores to adjust the ranking of models on the target. The separation of evaluator shift from task shift is also an interesting formulation.
Critical strengths
The separation of evaluator shift from task shift is also an interesting formulation. The authors have done extensive experiments and have considered enough variation and benchmakr under the given assumption to support the claims.
Critical weaknesses
The main weakness of the paper is that it only considers HealthBench, and the generalizability of the results is questionable. In particular, the source and target distributions are constructed from the same benchmark.
Another issue is the assumption that the mean score is invariant between source and target, conditioned on the prompt representation. The authors try different representation dimensions and prompt vs. prompt+rubric representations, but they use the same BGE-M3 encoder for all experiments. It would be helpful to evaluate whether the results and the assumption are sensitive to different representation models.
Questions for authors
NA
Response to Reviewer FHr6
We thank the reviewer for asking whether findings from one health benchmark carry over to another. Our discussion helped us see how the statistical analysis can guide the choice of which responses to score in a new domain. When the target evaluator is a human expert or an advanced LLM judge, its cost can limit how many tasks it scores.
In transfer from HealthBench to MentalHealthBench, we reserved the more expensive target judge, GPT-5.6 Sol, for a small random sample. Only the inexpensive HealthBench evaluator, GPT-4.1, scored the additional random sample of target tasks. Using GPT-4.1 in both domains lets us estimate the source–target gap for each response model and adjust its HealthBench scores before transfer. With 30 costly target evaluations, this design reduced RMSE from 6.49 to 4.83 and increased full-rank recovery from 8.6% to 12.0% relative to the target-only mean (Table 2).
1. Generalizability beyond a single benchmark
We thank the reviewer for suggesting broader validation. We have expanded our experiments to include both a second benchmark and direct transfer between distinct benchmarks.
On MentalHealthBench, which contains 1,215 synthetic mental-health conversations with evaluation rubrics, we varied user-profile composition and evaluated five response models over 500 splits. Without target evaluations, representation-based reweighting improves mean-score estimation and model-ranking recovery over the unweighted source mean. Among methods using target evaluations, CAFE achieves the lowest MAE and RMSE at all three budgets of 50, 100, and 200 evaluations in both the same- and different-evaluator settings. These results are reported in Tables 27–28 and Figure 20 in Appendix E.5.
We evaluated transfer across domains from HealthBench to MentalHealthBench, using GPT-4.1 as the benchmark evaluator and GPT-5.6 Sol as the target evaluator (Section 6.3, Table 2). For all five response models, the projected covariance measure (PCM) test found evidence that mean GPT-4.1 scores differed between benchmarks after conditioning on the task representation (Bonferroni-adjusted p-values from 1.29×10−5 to 0.04).
We address this shift through an evaluation design that collects inexpensive scores from the benchmark evaluator (GPT-4.1) on 50% of target tasks not selected for target evaluation and uses these scores to calibrate CAFE. Corrected CAFE achieves the best or tied-best performance across all four metrics at all three target-evaluation budgets of 30, 50, and 100. With only 30 target evaluations, it reduces MAE/RMSE from 5.20/6.49 to 3.84/4.83 relative to the target-only sample mean and increases full-rank recovery from 8.6% to 12.0%. This experiment demonstrates improved performance estimation and model-ranking recovery across distinct benchmarks while keeping the costly target-evaluation budget fixed.
2. Sensitivity to the representation encoder
The mean-score estimation gains persist when Qwen3-Embedding-4B replaces BGE-M3. We repeated Case 2 in both evaluator settings, keeping the other settings and hyperparameters fixed. With Qwen representations, CAFE has the lowest MAE and RMSE at all three budgets in both settings (Table 25 in Appendix E.4.4).
At 200 target evaluations, MAE decreases from 1.66 to 0.98 in the same-evaluator setting and from 1.66 to 1.38 in the different-evaluator setting relative to the target-only sample mean, on the table's ×102 scale. These are reductions of 41.0% and 16.9%. Exact ranking does not uniformly improve: under evaluator shift at 500 evaluations, full-rank recovery is 52% for CAFE versus 56% for the sample mean. We used 1,024-dimensional embeddings and retained 50% of the variance separately for prompts and rubrics, yielding 47 and 20 principal components.
The assumption concerns mean scores conditional on the representation, so changing the encoder also changes what is held fixed. For instance, an encoder that omits a task attribute related to the score can leave residual differences if that attribute is more common in one population. Table 25 assesses the estimator's performance with both scoring protocols. Its results alone cannot tell us whether conditional-mean invariance holds for either encoder or is equally plausible for both. The additional benchmark scores in our cross-benchmark design let us estimate and adjust for a model-specific source–target gap.