- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Average baseline accuracy remains a primary metric for LLM evaluation. However, it obscures changes in model capability, task difficulty, and the dependence of the model and task on cueing. Furthermore, characterizing cue sensitivity can be costly to evaluate across thousands of configurations. To address these challenges, we introduce TriEval, a three-sided framework that jointly estimates baseline model capability, task difficulty, and the cue's impact on the model and task under a reference cue. We develop a regularized Gaussian variational expectation-maximization (RGVEM) algorithm with closed-loop updates for scalable estimation. We also introduce cue sampling to reduce the number of configurations in the evaluation while maintaining estimation quality. We evaluate TriEval through simulations and empirical analyses on five open-weight LLMs with cueing configurations and six benchmarks. Simulation results demonstrate that TriEval recovers model rankings under the reference cue more accurately compared to traditional methods. Empirical results show that: (i) a small subset of tasks exhibit anomalous difficulty variations under different cues; (ii) model rankings change with cues; (iii) immediate effects exhibit systematic differences across different models; and (iv) estimates derived from the sampled configurations are very close to those derived from the complete configuration set.
Critical strengths
- The RGVEM algorithm has a solid derivation, combining efficient closed-form solution with sparse interpretability.
Critical weaknesses
-
The authors explicitly mentioned PromptEval, a recent core method for multi-prompt evaluation based on IRT, in their related work. However, in empirical and simulation experiments, they only compared the most basic "average accuracy" with the "traditional one-sided Rasch model," completely omitting PromptEval from the experimental comparison, thus lacking persuasiveness.
-
In formulas (2) and (3), the impact of each dimension of the prompt on the model is simplified to a linear additive main effect (). As for the combined interactions between dimensions (e.g., the coupling between a few-shot and a specific question template) and the specific interactions between the task and the prompt, all are vaguely crammed into the undirected residual variances and , failing to specifically explain which two prompt dimensions' collisions led to the performance crash.
-
While the paper claims that sampling a 1,000-word configuration reduces overhead by 83.3%, running 1,000 words per task remains extremely expensive in real-world large-scale model evaluations. In a more realistic configuration of 200 words, simulations show that the correlation of the word sensitivity parameter drops to around 0.80, and the detection rate of task-level interaction features (TPR, see Figure 10 at J=6, S=200) is even less than 50%. Furthermore, on MMLU Human Sexuality, increasing the sampling from 600 to 1,000 words actually reduces the accurate ranking recovery rate from 54% to 48%.
Questions for authors
See the Critical Weaknesses
Clarification on the baseline comparison and new evaluation experiments at small budgets
Thank you for your insightful comments. In the revised paper, we will clarify the comparison with PromptEval and present new experiments evaluating TriEval at small calibration budgets.
Comparison with PromptEval (W1).
We agree that PromptEval is a relevant baseline. However, it models each LLM separately and thus, cannot directly produce model rankings. More specifically, each fit has its own reference point of the latent capability, so a larger capability estimate does not necessarily indicate a stronger model, a phenomenon well known as scale indeterminacy in latent variable modeling. TriEval jointly calibrates LLMs using shared task parameters, placing their capabilities on a common scale for direct ranking.
Interactions (W2).
We agree that the variance term in TriEval allows for potential interactions but cannot attribute a performance decrease to a specific interaction between prompt dimensions. This design balances model parsimony and flexibility. Explicit interaction features are a possible extension but require adequate configuration coverage. We will explicitly include this limitation.
Evaluation at small budgets (W3).
The sampling unit is a prompt configuration. We agree that using 1,000 instead of 6,005 configurations, despite reducing response collection by 83.35%, can still be expensive.
We also acknowledge the limitations at 200 configurations. The empirical sampling study assesses agreement with estimates from all 6,005 configurations, but note that these full-data estimates are not ground truth. To assess practical use at realistic budgets against observed performance, we added two experiments using held-out performance as the evaluation reference.
Mean-performance prediction.
We compared TriEval and direct average in performance prediction using held-out performance as the reference. Both methods were calibrated on the same 100 or 200 configurations to predict mean accuracy across 1,201 held-out configurations. Direct average uses mean calibration accuracy, and TriEval estimates model parameters from calibration samples and predicts accuracy across the held-out configurations.
Prompt-level selection.
We also compared TriEval with raw accuracy for prompt selection using observed held-out performance as the reference. When selecting a prompt level for each dimension, we grouped configurations differing only in that dimension. For instance, each enumerator group contained six configurations, one per enumerator choice, with identical settings on the other four dimensions. 20% of configurations were used for evaluation, and both methods were calibrated on the same 100 or 200 configurations sampled from the rest 80%.
For each LLM and prompt dimension, raw accuracy selected the level with the highest calibration accuracy, and TriEval selected the level with the highest predicted accuracy across matched held-out configurations. The two methods were compared in terms of effect root mean square error (RMSE) and selection regret. RMSE indicates how closely estimated prompt effects across levels match observed accuracy differences in the held-out sample, and selection regret measures the observed accuracy lost by selecting certain prompts instead of the best-performing level in the held-out sample.
Table R3 summarizes the results. At calibration budgets of 100 and 200 configurations, TriEval reduces the MAE by 15.8% and 19.2%, respectively, and selection regret by 27.6% and 34.2%. These results demonstrate TriEval’s benefits compared to the accuracy-based baseline at these small budgets.
Table R3. Practical evaluation with limited calibration budgets.
\text{Outcome} & \text{Calibration size} & \text{Accuracy baseline} & \text{TriEval} \\\\ \text{Mean-accuracy MAE} & 100 & 0.739 & 0.622 \\\\ \text{Mean-accuracy MAE} & 200 & 0.506 & 0.409 \\\\ \text{Selection regret} & 100 & 0.646 & 0.468 \\\\ \text{Selection regret} & 200 & 0.437 & 0.288 \end{array}$$ Note. All values are in percentage points. Lower is better. Results are averaged across benchmarks, LLMs, and replications. Selection regret is also averaged across prompt dimensions.