- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MULTI-FW, a task-aware active evaluation algorithm for identifying the best-performing LLM under heterogeneous task distributions. It models winner-only preference feedback with a contextual multinomial logit model, uses Fisher information to measure uncertainty along decision-relevant directions, and applies a Frank-Wolfe allocation strategy to select informative model comparisons across tasks. Experiments in an Arena-derived semi-real environment show that MULTI-FW identifies the task-averaged best model more accurately and with fewer comparisons than several baselines. However, the evaluation relies on aggregated MiniLM response embeddings and a shared MNL parameter fitted from the same Arena data used to construct the simulated environment, so the results mainly demonstrate effectiveness under a well-specified synthetic model and provide limited evidence for robustness to real query-level and user-level preference heterogeneity.
Critical strengths
-
The paper formulates an important and practically relevant problem. Efficiently allocating expensive human preference comparisons across heterogeneous tasks is valuable for LLM evaluation, and the fixed-confidence best-model identification objective is clearly defined.
-
The algorithm and theoretical analysis are well structured. The Fisher-geometric separation criterion and the task-aware Frank-Wolfe allocation strategy provide a principled way to focus comparisons on decision-relevant uncertainty rather than using uniform sampling. The paper also includes consistency and sample-complexity guarantees under its modeling assumptions.
Critical weaknesses
- The experimental environment is constructed from a strong closed-loop modeling assumption. The task-dependent features are obtained from aggregated MiniLM embeddings, while the shared preference parameter is fitted using a contextual MNL model on Arena data. New feedback is then sampled from this same fitted MNL model. Consequently, the experiments are largely well specified by construction and do not establish that the proposed representation or shared parameter adequately captures real query-level and user-level preference heterogeneity.
- The response features are aggregated at the task-model level, so each incoming query effectively contributes only a sample from a fixed task-level preference distribution. This setting removes much of the query-level variability and evaluation noise that make real LLM comparison expensive. As a result, the reported gains may primarily reflect efficient identification in the constructed MNL simulator rather than robust sample efficiency in real-world human preference evaluation.
Questions for authors
- Can the authors evaluate MULTI-FW in a more realistic evaluation setting with fresh human judgments or at least held-out, query-level preference data? The current experiments generate feedback from a contextual MNL model fitted on the same Arena data used to construct the task features. It would be important to test whether the method remains effective when each query has its own response features and preference noise, rather than being sampled from a fixed task-level distribution.
- How sensitive is MULTI-FW to the construction of and the shared preference parameter ? In particular, can the authors compare the proposed aggregated MiniLM features against alternative representations, such as query-level embeddings, reward-model scores, or learned response-quality features, and evaluate whether the allocation and identification results remain consistent under different feature constructions and preference models?
3 comments
I thank the authors for their detailed response. It addresses my concern about the task-model-level aggregation of response features and the resulting lack of query-level variability. I will keep my score unchanged.
Thank you again for your careful reading and constructive feedback. Your comments helped us strengthen the evaluation. We are glad that our response and additional experiments address your concern about query-level variability.
Robustness to feedback misspecification, query-level variation, and task heterogeneity
1. Robustness and multitask evaluation of Multi-FW
Thank you for highlighting the limitations of task-level aggregation and fitted-model feedback. We acknowledge that the original semi-real experiments do not establish that aggregated features and a shared parameter fully capture real query- and user-level preferences. We have added Appendix F, which evaluates Multi-FW under empirical feedback, latent variation in response features and preference weights, and richer task structures. Multi-FW retains its identification advantage across these settings, each evaluated over 100 paired replications at T=1000.
Empirical preference feedback (F.1). We replace the fitted MNL feedback generator with empirical preference frequencies to assess Multi-FW's robustness to departures from the fitted contextual MNL model. The mean absolute win-probability change is 6.92 percentage points. The empirical Borda-optimal model agrees with the fitted-utility target. Multi-FW achieves a checkpoint AUC of 0.865, exceeding the strongest baseline by 15.7 percentage points. Thus, its advantage persists when feedback no longer comes from the fitted MNL generator.
Latent query-level response variation (F.2). Aggregating response embeddings within each task–model cell leaves variation across individual queries unrepresented. We introduce Gaussian perturbations around the aggregated features, independently across rounds and models, and generate feedback from these latent features. The perturbations remain hidden, while Multi-FW continues to use the task-level averages. At σ=0.05, marginal win probabilities change by 4.56 percentage points on average in absolute value, and the reference-best model's task-weighted win probability against the reference runner-up falls from 58.97% to 55.72%. Despite weaker identification signals, Multi-FW's checkpoint AUC exceeds the strongest baseline by approximately 13.0 percentage points; its terminal accuracy is 90%, versus a baseline maximum of 74%.
Latent query-level preference heterogeneity (F.3). The shared parameter leaves variation in preference weights unrepresented. We introduce independent Gaussian perturbations around the fitted parameter across rounds, with all models sharing the same latent parameter within a round. The perturbations remain hidden, and Multi-FW continues to estimate a single shared parameter. At a per-coordinate noise standard deviation of τ=25, the mean absolute change in marginal win probabilities is 6.61 percentage points, and the same reference top-two win probability falls to 55.06%. Multi-FW retains the highest checkpoint AUC and terminal accuracy, exceeding the strongest baseline in AUC by approximately 16.1 percentage points. Its terminal accuracy is 77%, versus a baseline maximum of 62%. Both perturbations weaken comparison signals and slow identification; Multi-FW's absolute performance declines, but its advantage over the baselines remains substantial.
Multitask evaluation (F.4–F.5). We expand Arena to five and ten tasks by splitting its original tasks into synthetic children whose perturbed features are observed and fixed across replications. Preference reversals increase from 10 of 105 model pairs to 70 and 54; arrival-weighted mean absolute child–parent win-probability changes are 8.61 and 6.18 percentage points. Multi-FW's checkpoint AUC exceeds the strongest baseline by 9.5 and 10.6 percentage points. In a separate fully synthetic instance with five tasks and ten models, 42 of 45 pairs reverse their preference direction across tasks, and the tasks have four distinct optimal models; Multi-FW's AUC advantage is approximately 13.3 percentage points. These results show that Multi-FW's identification advantage persists across distinct task constructions with substantially increased preference heterogeneity.
2. Held-out validation and alternative representations
Held-out evaluation. We agree that fresh human judgments or held-out query-level data provide important complementary validation. The empirical-feedback experiment reuses Arena data, and the Gaussian experiments model hidden variation synthetically. These controlled benchmarks let us compare allocation efficiency against a specified identification target while varying feedback and task structure; they do not establish out-of-sample validity on real queries.
Alternative representations. Our framework accepts task-dependent features and does not require MiniLM specifically; Section 3 allows response embeddings, rubric-based features, or evaluator-model features. However, the present perturbation tests do not compare query-level embeddings, reward-model scores, or learned response-quality features. Such comparisons remain an important next step.