acceptodds
Under review as a conference paper at ICLR 2027

TriEval: A Three-Facet Framework for Modeling Capability, Task Difficulty, and Prompt Sensitivity in LLM Evaluation

Abstract

Average benchmark accuracy remains the dominant metric in LLM evaluation. However, it conflates model capability, task difficulty, and prompt-dependent variation across models and tasks. In addition, characterizing prompt sensitivity can require costly evaluation across thousands of configurations. To address these challenges, we introduce TriEval, a three-facet framework that jointly estimates baseline model capability under a reference prompt, task difficulty, and prompt effects on models and tasks. A regularized Gaussian variational expectation-maximization (RGVEM) algorithm with closed-form updates is developed for scalable estimation. We also introduce prompt sampling to reduce the number of configurations in the evaluation while preserving estimation quality. We evaluate TriEval through simulations and an empirical analysis of five open-weight LLMs across prompt configurations and six benchmarks. Simulations show that TriEval more accurately recovers reference-prompt model rankings than conventional methods. Empirical results show that (i) a small set of tasks exhibits anomalous difficulty shifts across prompts; (ii) model rankings vary across prompts; (iii) prompt effects vary systematically across models; and (iv) estimates from sampled configurations closely match those from the full configuration set.

Primary area:
datasets and benchmarks
Keywords:
LLM Evaluation, Prompt Sensitivity, Benchmark Reliability, Latent Variable Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eHU8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Average baseline accuracy remains a primary metric for LLM evaluation. However, it obscures changes in model capability, task difficulty, and the dependence of the model and task on cueing. Furthermore, characterizing cue sensitivity can be costly to evaluate across thousands of configurations. To address these challenges, we introduce TriEval, a three-sided framework that jointly estimates baseline model capability, task difficulty, and the cue's impact on the model and task under a reference cue. We develop a regularized Gaussian variational expectation-maximization (RGVEM) algorithm with closed-loop updates for scalable estimation. We also introduce cue sampling to reduce the number of configurations in the evaluation while maintaining estimation quality. We evaluate TriEval through simulations and empirical analyses on five open-weight LLMs with cueing configurations and six benchmarks. Simulation results demonstrate that TriEval recovers model rankings under the reference cue more accurately compared to traditional methods. Empirical results show that: (i) a small subset of tasks exhibit anomalous difficulty variations under different cues; (ii) model rankings change with cues; (iii) immediate effects exhibit systematic differences across different models; and (iv) estimates derived from the sampled configurations are very close to those derived from the complete configuration set.

Critical strengths
  1. The RGVEM algorithm has a solid derivation, combining efficient closed-form solution with sparse interpretability.
Critical weaknesses
  1. The authors explicitly mentioned PromptEval, a recent core method for multi-prompt evaluation based on IRT, in their related work. However, in empirical and simulation experiments, they only compared the most basic "average accuracy" with the "traditional one-sided Rasch model," completely omitting PromptEval from the experimental comparison, thus lacking persuasiveness.

  2. In formulas (2) and (3), the impact of each dimension of the prompt on the model is simplified to a linear additive main effect (αj⊤xs\alpha_j^\top x_s). As for the combined interactions between dimensions (e.g., the coupling between a few-shot and a specific question template) and the specific interactions between the task and the prompt, all are vaguely crammed into the undirected residual variances σθj2\sigma_{\theta_j}^2 and σbi2\sigma_{b_i}^2, failing to specifically explain which two prompt dimensions' collisions led to the performance crash.

  3. While the paper claims that sampling a 1,000-word configuration reduces overhead by 83.3%, running 1,000 words per task remains extremely expensive in real-world large-scale model evaluations. In a more realistic configuration of 200 words, simulations show that the correlation of the word sensitivity parameter drops to around 0.80, and the detection rate of task-level interaction features (TPR, see Figure 10 at J=6, S=200) is even less than 50%. Furthermore, on MMLU Human Sexuality, increasing the sampling from 600 to 1,000 words actually reduces the accurate ranking recovery rate from 54% to 48%.

Questions for authors

See the Critical Weaknesses

1 comment
AuthorsOpenReview ↗

Clarification on the baseline comparison and new evaluation experiments at small budgets

Thank you for your insightful comments. In the revised paper, we will clarify the comparison with PromptEval and present new experiments evaluating TriEval at small calibration budgets.

Comparison with PromptEval (W1).

We agree that PromptEval is a relevant baseline. However, it models each LLM separately and thus, cannot directly produce model rankings. More specifically, each fit has its own reference point of the latent capability, so a larger capability estimate does not necessarily indicate a stronger model, a phenomenon well known as scale indeterminacy in latent variable modeling. TriEval jointly calibrates LLMs using shared task parameters, placing their capabilities on a common scale for direct ranking.

Interactions (W2).

We agree that the variance term in TriEval allows for potential interactions but cannot attribute a performance decrease to a specific interaction between prompt dimensions. This design balances model parsimony and flexibility. Explicit interaction features are a possible extension but require adequate configuration coverage. We will explicitly include this limitation.

Evaluation at small budgets (W3).

The sampling unit is a prompt configuration. We agree that using 1,000 instead of 6,005 configurations, despite reducing response collection by 83.35%, can still be expensive.

We also acknowledge the limitations at 200 configurations. The empirical sampling study assesses agreement with estimates from all 6,005 configurations, but note that these full-data estimates are not ground truth. To assess practical use at realistic budgets against observed performance, we added two experiments using held-out performance as the evaluation reference.

Mean-performance prediction.

We compared TriEval and direct average in performance prediction using held-out performance as the reference. Both methods were calibrated on the same 100 or 200 configurations to predict mean accuracy across 1,201 held-out configurations. Direct average uses mean calibration accuracy, and TriEval estimates model parameters from calibration samples and predicts accuracy across the held-out configurations.

Prompt-level selection.

We also compared TriEval with raw accuracy for prompt selection using observed held-out performance as the reference. When selecting a prompt level for each dimension, we grouped configurations differing only in that dimension. For instance, each enumerator group contained six configurations, one per enumerator choice, with identical settings on the other four dimensions. 20% of configurations were used for evaluation, and both methods were calibrated on the same 100 or 200 configurations sampled from the rest 80%.

For each LLM and prompt dimension, raw accuracy selected the level with the highest calibration accuracy, and TriEval selected the level with the highest predicted accuracy across matched held-out configurations. The two methods were compared in terms of effect root mean square error (RMSE) and selection regret. RMSE indicates how closely estimated prompt effects across levels match observed accuracy differences in the held-out sample, and selection regret measures the observed accuracy lost by selecting certain prompts instead of the best-performing level in the held-out sample.

Table R3 summarizes the results. At calibration budgets of 100 and 200 configurations, TriEval reduces the MAE by 15.8% and 19.2%, respectively, and selection regret by 27.6% and 34.2%. These results demonstrate TriEval’s benefits compared to the accuracy-based baseline at these small budgets.

Table R3. Practical evaluation with limited calibration budgets.

\text{Outcome} & \text{Calibration size} & \text{Accuracy baseline} & \text{TriEval} \\\\ \text{Mean-accuracy MAE} & 100 & 0.739 & 0.622 \\\\ \text{Mean-accuracy MAE} & 200 & 0.506 & 0.409 \\\\ \text{Selection regret} & 100 & 0.646 & 0.468 \\\\ \text{Selection regret} & 200 & 0.437 & 0.288 \end{array}$$ Note. All values are in percentage points. Lower is better. Results are averaged across benchmarks, LLMs, and replications. Selection regret is also averaged across prompt dimensions.
Reviewer_yyWBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TriEval, a framework to jointly evaluate three interacting facets in LLM evaluation, including model capability, task difficulty and prompt sensitivity. To model these sources of variation, TriEval extends standard IRT with an explicit prompt dependency (s)(s) on model capability (θjs)(\theta_{js}) and task difficulty (bis)(b_{is}), both modeled as latent variables with Gaussian priors. The paper then develops a regularized Gaussian variational EM (RGVEM) algorithm to estimate the parameters. Empirical analysis spans five LLMs, over 6k prompt configurations and six benchmarks. Results reveal sensitivity of model rankings to prompts, anomalous task-difficulty shifts across prompts and performance of sampling configurations compared to the full set, among others.

Critical strengths
  • The paper addresses an important problem of systematically evaluating the three facets of LLM benchmarking. The model developed to characterize the three interacting facets is simple yet principled, allowing subsequent analysis and interpretation.

  • The analysis provides several valuable insights into model rankings and task difficulty jointly against prompt variations. While certain aspects of the modeling and evaluation can be further strengthened (as commented in the next section), the current study provides a novel foundation for the community to build upon.

Critical weaknesses
  • The models examined in this study are slightly old given the fast pace at which the model capabilities have evolved. Furthermore, model performance is closely tied to its scale and larger models may be less sensitive to prompt variations compared to the smaller ones. Thus the study should consider not only newer models (Qwen3, DeepSeek-V4, Llama4, etc.) but also larger scales.

  • On the modeling front, certain aspect could be strengthened with more intuition/analysis: a) why is the sensitivity to prompt variations modeled additively (α~jTx~s\boldsymbol{\tilde\alpha}_j^T \boldsymbol{\tilde{x}}_s)? and b) why is task difficulty not modeled similarly to model capability with explicit dependency on prompt features?

  • Finally, the paper does not clearly state how the three facets should be ultimately used for model comparison. While it argues that a single number is insufficient, can the framework's insights be used to derive a more robust aggregate score? Effectively, the end users have to rely on some global ranking to chose a model.

Questions for authors
  1. How is the reference for prompt features chosen? Is the reference something standard?
2 comments
AuthorsOpenReview ↗

Clarification on TriEval’s modeling choices and global ranking

Thank you for your valuable comments. We will clarify the model specification and use of TriEval for global model ranking. These updates will be added to the paper.

Model coverage (W1). We agree that newer and larger models are needed for more generalizable conclusions. This paper focuses on introducing TriEval and illustrating its usefulness using publicly available data. We will acknowledge the need to evaluate additional LLMs for meaningful empirical findings.

Additive prompt effects (W2). We want to highlight that while the capability mean is modeled by additive prompt effect due to its interpretability, interactions are also allowed through the variance term. This is designed intentionally to ensure model interpretability while retaining the flexibility in capturing interactions when supported by the data. In this study, almost all estimated variances σθj2\sigma^2_{\theta_j} are below 0.1, suggesting limited interaction effects.

For task difficulty, we prioritize model-prompt effects for model parsimony, treating difficulty primarily as an item characteristic while allowing prompt-related variation. We will add main prompt effects on difficulty as a possible extension.

Global ranking and reference prompt (W3 and Q1).

The dummy-coded reference is a convention, not a standard or an optimal prompt. It combines capital-letter enumerators, a vertical-bar separator, alphabetical choice ordering, the first phrasing template in Appendix B, and zero-shot demonstrations.

We agree that the global model ranking might be of greater interest. This can be easily obtained by replacing the dummy coding prompt indicator with the sum-to-zero effect coding. Under this coding strategy, the estimated capability mean can be interpreted as the global model capability across prompts. With effect-coding, the global model ranking is Llama-8B (1.2851.285), Mistral-7B (0.8590.859), Llama-3B (0.3340.334), OLMoE-7B (0.1890.189), and Llama-1B (−0.989-0.989). Absolute parameter estimates change, but relative relationships remain unchanged. We will include the main results under effect coding in the appendix.

Thank you for your response. I appreciate the authors' acknowledgment that a global model ranking would be of greater interest to the community and that evaluations on newer and larger models are needed to support more generalizable conclusions.

The limitations of modeling prompt sensitivity as an additive linear term were raised by all reviewers. While the authors argue that this choice strikes a balance between parsimony and flexibility, the approach still has limitations. Nevertheless, I find the authors' justification reasonable and do not consider it a major obstacle to my overall assessment.

I also appreciate the authors' additional illustration of global model ranking under effect coding. However, my primary concern remains the lack of evaluations on newer and larger models, which limits the generalizability of the findings. Thus, I am unable to raise my score.

Reviewer_w5bwOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

TriEval proposes a three-facet framework for LLM evaluation, considering model capability, task difficulty, and prompt sensitivity. (Basically it looks at 3 aspects of the task when evaluating the model) It extends Item Response Theory (IRT) (which already tells us that task difficulty is impt for evaluation) using a regularized Gaussian variational expectation-maximization (RGVEM) algorithm. The authors evaluate five LLMs across six benchmarks and 6,005 prompt configurations from DOVE. They also use synthetic models to demonstrate improved ranking estimation (Table 4). Prompt sampling is used to reduce evaluation costs.

Critical strengths
  1. Paper addresses the importance of prompt sensitivity in LLM evaluation.
  2. Considers model capability, task difficulty, and prompt variation together.
  3. Identifies prompt-sensitive tasks that may help improve benchmark design.
  4. Reduces evaluation costs through prompt sampling.
Critical weaknesses
  1. Table 4 relies on synthetic models to demonstrate improved ranking recovery. The benefit on real-world LLM rankings remains unclear.
  2. The framework does not show how to select the best prompt for a particular task or improve downstream accuracy.
  3. Prompt effects are mainly additive, without explicit interactions between prompt dimensions.
  4. The empirical changes in Appendix C.1 are small, raising questions about practical significance.
Questions for authors
  1. Can the authors explain why R was used instead of Python? Were there significantly different results between the two languages?
  2. Can the authors discuss or explain how the benefits of their TriEval can benefit real-world evaluations?
2 comments
AuthorsOpenReview ↗

TriEval’s practical value: New held-out experiments on performance prediction and prompt selection, with further clarifications

Thank you very much for highlighting the need for clearer demonstration of TriEval’s practical value. We added two experiments using observed accuracy on held-out configurations as an independent real-world reference. These results will be added to the paper in the highlighted sections.

Real-data evidence and evaluation efficiency (W1, W4, Q2). Table 4 evaluates ranking recovery in simulations because true model rankings are unavailable in real world. To directly demonstrate TriEval’s practical value, we compared TriEval with direct average in predicting mean accuracy under 1,201 held-out prompts (20% of all configurations). Both methods were calibrated under the same samples of 100 and 200 configurations across six benchmarks, five LLMs, and 50 replications.

Direct average uses the mean calibration accuracy to predict the performance under held-out configurations, while TriEval estimates its parameters in calibration samples and predicts mean accuracy in held-out prompts. Observed held-out accuracy provides the evaluation reference and is not used in calibration.

As shown in Table R1, TriEval reduces mean absolute error (MAE) by 15.8% and 19.2% compared to direct average, providing more accurate performance prediction at the same calibration budget.

Table R1. Mean-accuracy prediction MAE on held-out prompts.

\text{Calibration size} & \text{Direct average} & \text{TriEval} \\\\ 100 & 0.739 & 0.622 \\\\ 200 & 0.506 & 0.409 \end{array}$$ Note. MAE is in percentage points, averaged across benchmarks, LLMs, and replications. Lower is better. **Prompt design (W2, Q2).** We also added an experiment demonstrating how TriEval can inform prompt selection. We compared TriEval with raw accuracy in prompt selection using held-out performance as the reference. When selecting a prompt level for each dimension, we grouped configurations differing only in that dimension. For instance, each group in \textit{Enumerator} dimension contained six configurations, one per enumerator level, with the same settings on the other four dimensions. Each unique combination across those four dimensions formed another group. We held out 20\% of groups for evaluation and calibrated both methods on the same 100 or 200 configurations from the rest 80\%. For each LLM and prompt dimension, raw accuracy selects the prompt level with the highest accuracy in the calibration sample, while TriEval selects the level with the highest predicted accuracy across matched held-out configurations. Evaluation metrics include effect root mean square error (RMSE), which measures how closely estimated prompt effects across levels match observed accuracy differences in held-out sample, and selection regret, which measures the observed accuracy lost by selecting certain prompts instead of the optimal level, defined as the level with highest observed accuracy in the held-out sample. Table R2 summarizes results averaged across all dimensions, benchmarks, LLMs, and replications. TriEval reduces RMSE by 27.2 - 29.4\% and selection regret by 27.6 - 34.2\% and thus, supports better prompt design. **Table R2. Prompt-effect estimation and level selection.** $$\begin{array}{lrcc} \text{Outcome} & \text{Calibration size} & \text{Raw accuracy} & \text{TriEval} \\\\ \text{Effect RMSE} & 100 & 2.806 & 2.044 \\\\ \text{Effect RMSE} & 200 & 1.977 & 1.395 \\\\ \text{Selection regret} & 100 & 0.646 & 0.468 \\\\ \text{Selection regret} & 200 & 0.437 & 0.288 \end{array}$$ Note. Errors are in percentage points, averaged across benchmarks, LLMs, prompt dimensions, and 50 replications. Lower is better. **Benchmark diagnosis and practical significance (W4, Q2).** The task-level diagnosis identifies questions for review. Figure 2 and Table 2 show flagged tasks with ambiguous answers or typographical errors, motivating correction or exclusion when defects are verified. Appendix C.1 illustrates one potential consequence: changes in prompt-effect estimate after filtering fewer than 10\% of tasks. While these changes are small in absolute values, the fundamental concern is that the defective tasks compromise benchmark validity and waste evaluation resources. TriEval’s practical value is to help identify such tasks for review and improve benchmark quality. **Interactions between prompt dimensions (W3).** We are sorry it was not clearly elaborated. While LLM’s capability mean is only modeled by additive main effects, all interactions are allowed and captured by the additional variance term. This design preserves model parsimony while allowing interactions when supported by data. **Choice of implementation language (Q1).** We used R due to our familiarity with its statistical computation environment. Our RGVEM algorithm is language independent. Given the closed-form updates, our estimation algorithm can also be implemented in Python, without relying on any language-specific optimizer.

Thank you for the response. I have some concerns regarding generalization of this work. In light of other reviewers' sentiment as well, I will maintain my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall response to reviewers

Thank you all very much for your constructive comments. Following our earlier detailed replies to each reviewer, we have now updated the manuscript and summarized our responses to the common suggestions below. New experiments and clarifications are highlighted in blue in the manuscript. We hope these revisions address your concerns and would be grateful if you would consider updating your score accordingly.

Practical value of TriEval (Reviewers w5bw and eHU8).

We have added two experiments using observed performance on held-out configurations as an real-world, independent evaluation reference. Both cover six benchmarks, five LLMs, and 50 replications. Within each experiment, methods use the same calibration samples of 100 or 200 configurations, and held-out responses are excluded from fitting.

The first experiment examines TriEval's value in predicting LLM performance under unseen prompt configurations. TriEval reduces mean-accuracy prediction error on the 1,201 held-out configurations (20% of all prompts) by 15.8–19.2% compared to the direct average. The second experiment examines its usefulness in selecting prompt for each LLM. TriEval reduces prompt-effect RMSE by 27.2–29.4% and selection regret by 27.6–34.2% compared to raw accuracy. These results demonstrate more accurate performance estimation and better prompt selection at the same calibration budget (Section 4.7, Tables 4–5, and Appendix C.5).

In addition, TriEval also identifies prompt-sensitive tasks for review. Figure 2 and Table 2 illustrate flagged tasks with ambiguous answers or typographical errors, motivating correction or exclusion when defects are confirmed. This supports benchmark quality (Section 4.3).

Additive prompt effects and interactions (all reviewers).

The capability mean uses additive prompt effects for interpretability, while residual variances accommodate departures from this structure, including prompt interactions and task-specific difficulty shifts. This design balances parsimony and flexibility but does not identify the specific interactions underlying a performance shift. Nevertheless, it highlights LLMs with substantial residual variation, informing targeted post-hoc analyses of potential interaction effects. We have made this limitation explicit. In addition, task-specific prompt effects could be added to the difficulty mean with additional identification constraints (Sections 3.2 and 6.1).

Global model ranking (Reviewer yyWB).

The choice of reference prompt is a coding convention. Replacing dummy-coded prompt features with reference-free sum-to-zero effect coding yields the overall capability across the full factorial prompt design for each LLM, which may be of greater interest for model ranking. Appendix C.4 reports the results and clarifies the comparison targets discussed in Sections 3.1 and 4.4.

Comparison with PromptEval (Reviewer eHU8).

PromptEval fits each LLM separately, so its latent capability estimates lie on separate scales and cannot be directly ranked, reflecting scale indeterminacy in latent variable modeling. TriEval jointly calibrates LLMs using shared task parameters, placing their capabilities on a common scale for direct comparison (Section 2.2).

Sampling cost and empirical scope (Reviewers yyWB and eHU8).

We acknowledge that evaluating 1,000 configurations can still be expensive and that smaller budgets limit recovery of prompt effects and task-level interactions. Agreement with full-data estimates measures only stability rather than accuracy against the ground truth (Section 4.6 and Appendix C.3). The new held-out experiments evaluate practical benefits of TriEval against the empirical truth at budgets of 100 and 200 configurations. We also acknowledge the need for newer and larger LLMs to support more generalizable empirical conclusions (Section 6.1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.