acceptodds
Under review as a conference paper at ICLR 2027

What Makes a Question Hard(er)? Explaining Difficulty with Natural Language

Abstract

Difficulty is one of the most fundamental properties of a question: it determines whether the question can meaningfully discriminate between models of differing ability. Although a variety of methods can now estimate or predict difficulty automatically, they yield only a single descriptive number, with no account of the underlying factors that make a question difficult in the first place. In this work, we propose a data-driven approach that automatically generates and validates natural-language hypotheses explaining what makes one question harder than another. We first estimate each item's difficulty from the responses of a large pool of LLMs using Item Response Theory. We then sample contrasting sets of easy and hard questions and prompt an LLM to propose candidate explanations of the difference, which are subsequently validated and selected on held-out questions. Experimental results across three datasets spanning mathematical, logical, and commonsense reasoning show that our method produces interpretable and predictive hypotheses. On their own, they predict the difficulty of unseen questions competitively with, or better than, advanced black-box difficulty regressors; used as additional features, they further improve those regressors, implying that they discover difficulty signals that existing models fail to capture. Moreover, we demonstrate that editing questions according to a hypothesis can shift their measured difficulty in the expected direction, indicating that the discovered hypotheses are causally valid difficulty factors rather than post-hoc descriptions. Our approach thus turns a purely descriptive difficulty score into actionable statements.

Primary area:
interpretability and explainable AI
Keywords:
Item difficulty explanation, Hypothesis generation, Explainability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iJmCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work focuses on the difficulty estimation of questions. Unlike prior works that only estimate a score, this work also tries to reveal the underlying factors (i.e., what makes something harder?).

Thus, this work has constructed a data‑driven pipeline combining Item Response Theory (IRT), contrastive sampling and LLM-based hypothesis generation.

  • First, it computes objective question difficulty via the IRT model.
  • Then it samples contrasting easy‑hard question groups or pairs to prompt an LLM to produce natural‑language hypotheses to explain difficulty gaps.
  • Finally, it uses a new LLM function to select the hypothesis.

The contribution can be summarized as: using LLMs to automatically find a hypothesis to explain the difficulty of a question.

Critical strengths
  1. This work can automatically generate interpretable natural‑language difficulty hypotheses. This can reduce the cost and broaden the downstream tasks.

  2. This work has designed some novel evaluations. For example, counterfactual question editing is interesting, as it verifies causal effects, proving hypotheses are not merely post‑hoc correlational summaries.

  3. The pipeline shows enough generalizability, achieving promising results across math, logical deduction, and commonsense‑reasoning datasets.

Critical weaknesses
  1. The survey of related work is limited. A lot of recent works have not been well discussed:
  • Question Difficulty Estimation for Large Language Models via Answer Plausibility Scoring. ACL (1) 2026: 11124-11151
  • Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Prediction. ACL (Findings) 2026: 25414-25441
  • Estimating difficulty for the open-ended reading comprehension questions via Large Language Models. Eng. Appl. Artif. Intell. 180: 115268 (2026)
  • .....
  1. This work only adopts the naive IRT to estimate the difficulty. Thus, on the one hand, there is no contribution in this part. Meanwhile, IRT is too naive to provide a robust difficulty signal, making the convincingness of this work limited.

  2. The hypothesis (reason) is fully generated by LLMs. Although there is a quality-control pipeline, AI-human alignment is a critical challenge in this work.

Questions for authors
  1. I would like to see more evaluations on the downstream tasks to validate whether this hypothesis is useful.

  2. What is the best application scenario for this work?

2 comments

Thanks for your constructive feedback. Please find below our response.


  • W1: missing references

We thank the reviewer for pointing out these works. All three are quite recent (2026), and none is closely tied to our paper in either problem or method. They focus on predicting or estimating question difficulty, for LLMs or for students. Our goal is to explain an already-measured difficulty through human-readable, testable hypotheses. Even so, we will discuss these works in the related-work section of the revised version, to place our work more clearly within the broader difficulty-estimation literature.

  • W2: No contribution in measuring difficulty and IRT is “naive”

We would like to emphasize that our contribution is not a new way to measure difficulty but a method to explain difficulty. In addition, we respectfully disagree that IRT is a naive method for difficulty estimation. It is a principled model and a standard tool for measuring item difficulty, and is widely used both in psychometrics and education and in recent LLM evaluation research. For example, this paper [1] suggested by you also adopts the 1PL IRT model to estimate item difficulty.

Furthermore, the simplicity of the model does not mean it is unreliable. We verified the robustness of this method through 50 rounds of bootstrap sampling: each time, we sampled 50% of the LLMs and refitted the IRT model. The results show that the IRT-estimated difficulties are highly robust and stable, with correlations between different samplings all above 0.99.

Finally, our method is not tied to IRT. As stated before, the contribution of this paper is a method for explaining difficulty. We use IRT as one module because it is a standard measurement tool, but our method is agnostic to the measurement method: one can replace IRT with a better difficulty estimator if needed.

  • W3: Are the generated hypotheses aligned between AI and humans?

Our goal is to explain item difficulty for LLMs. That difficulty need not match the difficulty humans perceive. Prior studies have shown that the factors driving difficulty differ between humans and LLMs [1,2], since their knowledge and skills are not aligned to begin with. Our core objective is therefore to find explanations that are valid for LLMs and, at the same time, readable by humans. Our causal experiments directly support this: when items are edited along the discovered difficulty factors, LLM performance changes significantly in the predicted direction.


  • Question: application scenario for this work

We believe our method has many applications, for example,

  1. Benchmark evolution. As models become increasingly capable, harder benchmarks are needed, as discussed in the introduction. Our method discovers difficulty factors, which can then be used to curate new test questions, keeping benchmarks discriminative and up to date. This application is demonstrated in the paper: in the causality study, eidting questions according to our generated difficulty explanations substantially increases their difficulty, showing that our method succeeds for this purpose.

  2. Understanding model capabilities, especially weaknesses. The difficulty factors reveal the types of questions that LLMs struggle with, which is important in human–AI interaction and collaboration: knowing where a model is likely to fail helps users decide when to rely on it and when to verify its outputs. Moreover, in high-stakes or safety-critical settings, such knowledge allows error-prone inputs to be identified in advance and handled with additional caution

  3. Targeted training. Knowing a model’s weaknesses, one can generate training data that specifically targets them or reweight existing training data accordingly, making training more effective. We leave this for future work.


Reference

[1] Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Prediction

[2] Do great minds think alike? Investigating Human-AI Complementarity in Question Answering with CAIMIRA.

Reviewer_iJmCOpenReview ↗

Keep my recommendation

Thank you for the authors’ response, which addresses several of my concerns. Nevertheless, I remain unable to increase my recommendation score.

Reviewer_DrmaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a method to explain question difficulty in natural language. It estimates difficulty with IRT, samples contrastive sets of easy and hard questions, and prompts an LLM to generate hypotheses explaining the gap. Experiments on GSM8K, BBH-structured, and Winogrande show that the proposed method achieves strong results.

Critical strengths
  1. This paper is well written and easy to follow.

  2. The proposed method is simple and effective.

  3. The proposed method achieves strong results on three datasets, outperforming the evaluated baselines.

Critical weaknesses
  1. The evaluated dataset GSM8K contains easy math problems. Harder math datasets such as MATH Lvl 5 are also included in ROUTEREVAL. Will the proposed method still be effective for MATH Lvl 5?

  2. Many datasets may contain problems of different difficulty levels in practice. The selected hypotheses may differ across difficulty levels. For example, when the dataset mixes GSM8K and MATH Lvl 5, will the proposed method still be effective?

  3. The authors take continuous benchmark iteration as an example in L35-L38 to motivate the problem. Could the proposed method be used in such a dynamic process? For example, if we need to iteratively construct harder datasets over multiple rounds, how would the proposed method be applied?

    • In the first round, the method relies on IRT difficulty estimated from thousands of LLMs. For a newly constructed benchmark, is the cost of collecting such responses affordable?
    • Do the IRT estimates and hypotheses need to be recalculated in each round, and is the cost affordable?
Questions for authors

See the Weaknesses.

7 comments
AuthorsOpenReview ↗

Additional experiments on MATH level 5

Thank you for your constructive feedback. Please see our response below.


  • W1: Is our method also effective for MATH lv5?

We evaluated our method on MATH Level 5, using the same 200 held-out items for all comparisons. Our method achieves an R2R^2 of 0.5095 compared with 0.4416 for the fine-tuned RoBERTa-base (the best baseline according to Table2). We further incorporate generated hypotheses as features into the Roberta model (as in Figure 3 in the paper). Augmented by our hypotheses, the R2R^2 of RoBERTa increased to 0.5375. This confirms our method is effective for MATH lv5 and can discover difficulty factors that cannot be captured by a dense encoder.

MethodR2R^2 ↑RMSE ↓
RoBERTa-base (fine-tuned)0.44161.5003
Ours (60 hypotheses + OLS)0.50951.4063
RoBERTa + Ours combined0.53751.3652
AuthorsOpenReview ↗

Difficulty-level distribution

  • W2. Is our method effective for datasets with mixed difficulty levels?

MATH Level 5 and GSM8K were answered by different sets of LLMs, so the two datasets cannot be calibrated within a single IRT model, and their difficulty estimates are not on a common scale. For this reason, we do not merge them.

We would also like to clarify that, although GSM8K is easy for humans, it is not uniformly easy for the LLMs we evaluate. The table below shows the distribution of the LLMs’ error rates on each dataset:

Error rateGSM8KMATH Level 5BBH-structuredWinogrande
0–<10%0 (0.00%)0 (0.00%)0 (0.00%)1 (0.02%)
10–<20%15 (0.30%)0 (0.00%)55 (1.44%)1,553 (31.06%)
20–<30%464 (9.28%)0 (0.00%)429 (11.26%)2,345 (46.90%)
30–<40%1,077 (21.54%)0 (0.00%)315 (8.27%)548 (10.96%)
40–<50%495 (9.90%)7 (0.18%)668 (17.53%)467 (9.34%)
50–<60%398 (7.96%)63 (1.65%)928 (24.35%)86 (1.72%)
60–<70%428 (8.56%)248 (6.51%)487 (12.78%)0 (0.00%)
70–<80%298 (5.96%)294 (7.71%)839 (22.02%)0 (0.00%)
80–<90%443 (8.86%)886 (23.25%)90 (2.36%)0 (0.00%)
90–100%1,382 (27.64%)2,313 (60.69%)0 (0.00%)0 (0.00%)
Total5,000 (100%)3,811 (100%)3,811 (100%)5,000 (100%)

GSM8K covers a fairly balanced range of difficulty levels. In contrast, MATH Level 5 is concentrated at the high end, BBH in the middle, and WinoGrande at the low end. Our method performs well on all four datasets, which shows that it is robust to datasets with very different difficulty distributions.

WinoGrande is also informative in another respect. Although most of its items are easy, its difficulty is the hardest to predict and explain among the four datasets. This suggests that the challenge of explaining difficulty does not depend on how hard the items are. It depends on how subtle the factors driving that difficulty are.

For further analysis, we computed Δd, defined as the mean IRT difficulty of matching items minus that of nonmatching items, on the original test set within BBH-Structured and MATH Level 5. The final selected hypotheses for BBH-structured include direct lookup (Δd = −1.278), multi-entity tracking (Δd = +1.297), and arithmetic after filtering (Δd = +0.521). On MATH Level 5, selected properties include single-step formula application (Δd = −2.223), constrained case analysis (Δd = +1.173), and geometric auxiliary constructions (Δd = +1.113). These descriptive associations support our claim that our discovery pipeline can identify and retain properties corresponding to diverse reasoning requirements and difficulty levels.

Together, our results on these datasets support the effectiveness of the proposed framework throughout diverse question-difficulty distributions within the evaluated model populations.

AuthorsOpenReview ↗

Benchmark iteration and the cost of our method

  • W3. Benchmark iteration and the cost.

Continuous benchmark iteration is a natural downstream application of our framework. One iteration consists of four steps:

  1. estimating question difficulty;
  2. discovering hypotheses that explain difficulty;
  3. editing questions based on these hypotheses;
  4. re-evaluating models on the edited questions.

An iteration succeeds if the edited questions are indeed harder than the originals. Our experiments cover all four steps of a single round, and the causal intervention study confirms its success. The same procedure can be repeated over multiple rounds to further increase question difficulty.


Regarding your two sub-questions:

1. The cost of collecting responses for a newly constructed benchmark.

In this paper, we use responses from several thousand LLMs in the RouterEval dataset. However, estimating difficulty parameters requires far fewer LLMs. Efficient test-item calibration is a well-established problem in psychometrics [1]. For the 1PL model used in our paper, Linacre [2] shows theoretically that 16–36 respondents, depending on how well they are targeted, are sufficient to calibrate item difficulties to within ±1 logit at 95% confidence.

We also tested a simple baseline: randomly sampling a fixed set of LLMs to answer all target questions, with their abilities assumed known. Over 20 random repetitions, the following numbers of LLMs were enough to estimate the difficulty of at least 95% of items to within ±1 logit, in every repetition:

DatasetNumber of modelsSuccessful repetitions
GSM8K5420/20
BBH-structured2820/20
Winogrande5620/20

If only the average fraction of qualifying items across repetitions is required to reach 95%, the first crossings after integer-budget refinement are 39, 12, and 33 models, respectively. Thus, even a simple random-panel baseline demonstrates the feasibility of calibration with substantially fewer respondents.

Finally, our explanatory framework is conceptually measurement-agnostic: its discovery mechanism requires meaningful scalar difficulty targets rather than a particular IRT estimator. IRT is the measurement layer used in this study; alternative difficulty measurements that do not require LLM responses can also be used in our framework. We will clarify this and distinguish the demonstrated editing results from the prospective multi-round application.

2. regarding the cost of refitting IRT and regenerating hypotheses after each iteration.

The computational cost of IRT fitting is negligible. Our CPU timing runs required 23.52, 19.11, and 23.01 seconds for GSM8K, BBH-structured, and Winogrande, respectively, using 500 training epochs.

Hypothesis generation requires LLM calls, but our budget ablation in Figure 4 already shows useful predictive performance with fewer than 200 question presentations during generation.


References

[1] Optimal Bayesian Adaptive Design for Test-Item Calibration.

[2] Sample Size and Item Calibration [or Person Measure] Stability.

Reviewer_b8xWOpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework that combines Item Response Theory (IRT) with LLM-generated explanations to discover natural language factors underlying question difficulty. The authors estimate question difficulty based on responses from multiple language models, identify pairs or groups of easy and difficult questions, and use LLMs to generate candidate explanations. After hypothesis verification and feature selection, the discovered hypotheses are used to predict the difficulty of unseen questions and guide question editing experiments. The framework is evaluated on GSM8K, BBH-structured, and WinoGrande, covering mathematical reasoning, logical reasoning, and commonsense reasoning tasks. The main contribution is the attempt to build an interpretable difficulty analysis framework by connecting latent difficulty estimation with human-readable explanations. However, several methodological issues remain regarding the validity of the difficulty estimates, uncertainty propagation, and the stability of the discovered factors across different model populations.

Critical strengths

1.The paper provides a principled formulation of difficulty discovery. Instead of treating difficulty as a simple accuracy statistic, the authors use IRT to estimate latent item difficulty from a population of models. This is a more principled approach and provides a stronger basis for analyzing why different questions challenge models differently. 2.The framework explicitly connects measurement and interpretation. The paper does not stop at estimating difficulty values. It attempts to identify interpretable natural language factors and evaluate whether these factors can explain and predict difficulty. This combination of statistical measurement and semantic explanation is an important contribution.

Critical weaknesses

1.I think the paper has fundamental problems in its underlying logic. The robustness of IRT-based difficulty estimation is insufficiently validated. The entire framework depends on the assumption that the estimated difficulty values accurately reflect intrinsic question difficulty. However, LLMs are not independent respondents in the traditional psychometric sense. Different models may share training data, architectures, alignment procedures, and systematic weaknesses. Therefore, the estimated difficulty may partially reflect the selected model population rather than a general property of the questions. The authors should evaluate whether difficulty rankings remain stable when changing the set of responding models, removing model families, or performing bootstrap sampling. Without such analysis, it is unclear whether the discovered hypotheses represent general difficulty factors or characteristics of a specific collection of models. 2.The uncertainty of difficulty estimation is not propagated through the pipeline. IRT difficulty estimates are treated as fixed values in subsequent stages, including easy/hard partitioning, hypothesis generation, and regression analysis. However, these estimates contain uncertainty, especially for questions that are answered correctly or incorrectly by almost all models. The paper should analyze how uncertainty in difficulty estimation affects hypothesis discovery and prediction results. For example, bootstrap-based sensitivity analysis or perturbation experiments would help determine whether the main conclusions are robust.

Questions for authors

1.How stable are the estimated difficulty values when the responding model population changes? For example, have the authors evaluated difficulty ranking consistency after removing certain model families or resampling models? Please report correlation metrics or confidence intervals. This result would significantly affect my confidence in the foundation of the proposed framework. 2.Have the authors analyzed how uncertainty in IRT difficulty estimates influences the downstream hypothesis discovery and prediction results? For example, could bootstrap experiments demonstrate that the discovered hypotheses and reported performance improvements remain stable under perturbations of difficulty estimates? 3.How much overlap exists among the selected natural language hypotheses? Multiple hypotheses may describe the same underlying factor using different wording. How do the authors distinguish genuinely different factors from alternative descriptions of the same phenomenon?

7 comments
AuthorsOpenReview ↗

Regarding the robustness of IRT-estimated diffculty

Thanks for your feedback. We first address your question regarding the stability and robustness of IRT-estimated difficulty from the following aspects.


  • How does item difficulty vary across different sets of responding LLMs? — Near perfectly stable.

We performed bootstrap sampling over the LLMs 50 times. In each round, we randomly sampled 50% of the LLMs and re-fitted the IRT difficulty parameters on this subset, and then computed the Spearman’s rank correlation between the difficulties from the subset and those from the full set. The results are shown below.

Dataset#Items#Models (50%)Mean ρ\rho vs. fullMin ρ\rho vs. fullMean ρ\rho pairwiseMin ρ\rho pairwise
GSM8K1,3195,000 →\to 2,5000.99940.99930.99910.9988
BBH5,7613,811 →\to 1,9060.99910.99880.99870.9980
BBH-structured1,3963,811 →\to 1,9060.99880.99850.99830.9973
WinoGrande1,2675,000 →\to 2,5000.99900.99880.99840.9976

Across 50 times bootstrap sampling, even the min correlation exceeds 0.99, showing near-perfect stability.


  • Are IRT difficulties estimated on different LLM families correlated? Highly correlated.

We refitted three IRT models separately on three common LLM families (Mistral, Qwen, and Llama) identified from our LLM pool and computed the Spearman correlations between the difficulties learned from each pair of families. The results are shown below. In most cases, the difficulties learned from different families show high rank correlation and stability (average Spearman’s rank ρ\rho = 0.826).

Family pairGSM8KWinoGrandeBBH-structured
Mistral – Llama0.8980.8850.801
Mistral – Qwen0.8530.7610.800
Llama – Qwen0.8520.8620.718

  • How do cross-family differences in difficulty affect our pipeline? These differences are filtered out by our pipeline.

We further analyzed how these cross-family differences affect our pipeline. As described in Section 2.2, when sampling easy–hard questions, we require their difficulty gap to exceed a threshold: 4.0 logits for pair sampling, and at least 3 buckets apart for group sampling. We therefore measured the proportion of sampled easy–hard pairs whose ordering would reverse if the full-population difficulty were replaced by family-specific difficulty. The results are shown below.

DatasetFamily (#models)Reversed (pair sampling)Reversed (group sampling)
GSM8KMistral (1,987)0.00%0.00%
Llama (1,548)0.00%0.48%
Qwen (121)0.07%2.06%
WinoGrandeMistral (1,987)0.00%0.25%
Llama (1,548)0.00%0.29%
Qwen (121)0.01%2.82%
BBH-structuredMistral (497)0.00%0.84%
Llama (1,214)0.00%0.57%
Qwen (944)0.00%1.40%

The results show that the ordering of the easy–hard questions we sample based on the average difficulty is almost always preserved across LLM families. This indicates that the questions whose difficulty rankings disagree across families are mostly those with similar difficulty, which are already filtered out by our pipeline. As further evidence, our causality experiment covers 33 model–dataset pairs spanning the LLaMA, Mistral, Gemma, Qwen, Yi, and Phi families. In 30 of the 33 pairs, performance on questions rewritten according to our generated hypotheses changes in the expected direction, showing that these hypotheses are indeed effective across different families.


  • Our method can also explain family-specific difficulty Finally, our method is not tied to any single definition of difficulty. In the main experiments of the paper, we explain the average difficulty factor learned from all LLMs but it can also be used to explain a family-specific difficulty factor, simply by replacing the target with the family-specific difficulty instead of the average difficulty. The results are shown below (compared to the best baseline) show our method is effective.
MethodGSM8K MistralGSM8K LlamaGSM8K QwenBBH MistralBBH LlamaBBH QwenWinoGrande MistralWinoGrande LlamaWinoGrande Qwen
RoBERTa-base0.3220.2770.4150.5240.4900.5250.007-0.045-0.016
Ours0.3610.3440.3520.5260.4920.5210.0960.1400.070
AuthorsOpenReview ↗

On the uncertainty of difficulty estimates

  • Bootstrap-based uncertainty analysis

We quantify the uncertainty of difficulty estimates directly. For each item, we compute the standard deviation of its estimated difficulty across 50 bootstrap resamples, where each resample refits the 1PL model on a random 50% of the LLMs, as in our response to the previous point. The per-item SD is small on all three datasets (see the table below). The Spearman’s rank correlations above 0.99 in our previous response support the stability of the rankings, and the small SDs further confirm the consistency of the numerical values.

DatasetMedian SD (logits)95th-percentile SD (logits)
GSM8K0.1000.122
WinoGrande0.1560.171
BBH-structured0.0950.111

As we discussed in the last point, when sampling easy and hard items for hypothesis generation, we impose a minimum difficulty gap, which is 4.0 logits for pair sampling (more than 20 times the SD) and around 1.2-1.8 logits (varying on the datasets) for group sampling (still 6-13 times the SD). Uncertainty of this magnitude will also be filtered out and therefore cannot reverse the ordering of the items we sample.


  • Bootstrap-based sensitivity analysis of prediction results

We further test whether the uncertainty affects the regression results. We keep our generated hypotheses fixed and refit them to the difficulty estimates from each of the 50 bootstrap resamples. The results are highly consistent and robust across bootstrap samplings (See the table below).

DatasetFull-population R²Resampled R² (mean ± SD)Resampled R² [min, max]
GSM8K0.3730.375 ± 0.003[0.367, 0.383]
WinoGrande0.0900.091 ± 0.002[0.088, 0.096]
BBH-structured0.5800.579 ± 0.003[0.572, 0.584]
AuthorsOpenReview ↗

Response to your questions

For your first and second question, please see our previous responses regarding robustness and uncertainty of difficulty estimates.

For your third question on the overlap between selected natural language hypotheses, our pipeline explicitly identify and remove overlapping hypotheses during the refinement step (Section 2.3). Therefore, the final set should not contain highly redundant ones.

To verify this, we quantified the empirical overlap between hypotheses. We computed the similarity between hypotheses as the Jaccard similarity between their matching vectors (i.e., binary vectors whose length equals the number of questions, where an entry of 1 indicates that the hypothesis matches the corresponding question and 0 otherwise). The results are shown below. The similarities are very low, meaning that each hypothesis matches a largely different set of questions, which indicates that they describe substantively different content.

DatasetOursHypoGeniCNLParamEdisa
GSM8K0.0350.2060.0940.323
BBH-structured0.0980.3210.2690.049
Winogrande0.0520.0730.1710.972

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.