- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work focuses on the difficulty estimation of questions. Unlike prior works that only estimate a score, this work also tries to reveal the underlying factors (i.e., what makes something harder?).
Thus, this work has constructed a data‑driven pipeline combining Item Response Theory (IRT), contrastive sampling and LLM-based hypothesis generation.
- First, it computes objective question difficulty via the IRT model.
- Then it samples contrasting easy‑hard question groups or pairs to prompt an LLM to produce natural‑language hypotheses to explain difficulty gaps.
- Finally, it uses a new LLM function to select the hypothesis.
The contribution can be summarized as: using LLMs to automatically find a hypothesis to explain the difficulty of a question.
Critical strengths
-
This work can automatically generate interpretable natural‑language difficulty hypotheses. This can reduce the cost and broaden the downstream tasks.
-
This work has designed some novel evaluations. For example, counterfactual question editing is interesting, as it verifies causal effects, proving hypotheses are not merely post‑hoc correlational summaries.
-
The pipeline shows enough generalizability, achieving promising results across math, logical deduction, and commonsense‑reasoning datasets.
Critical weaknesses
- The survey of related work is limited. A lot of recent works have not been well discussed:
- Question Difficulty Estimation for Large Language Models via Answer Plausibility Scoring. ACL (1) 2026: 11124-11151
- Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Prediction. ACL (Findings) 2026: 25414-25441
- Estimating difficulty for the open-ended reading comprehension questions via Large Language Models. Eng. Appl. Artif. Intell. 180: 115268 (2026)
- .....
-
This work only adopts the naive IRT to estimate the difficulty. Thus, on the one hand, there is no contribution in this part. Meanwhile, IRT is too naive to provide a robust difficulty signal, making the convincingness of this work limited.
-
The hypothesis (reason) is fully generated by LLMs. Although there is a quality-control pipeline, AI-human alignment is a critical challenge in this work.
Questions for authors
-
I would like to see more evaluations on the downstream tasks to validate whether this hypothesis is useful.
-
What is the best application scenario for this work?
2 comments
Keep my recommendation
Thank you for the authors’ response, which addresses several of my concerns. Nevertheless, I remain unable to increase my recommendation score.
Thanks for your constructive feedback. Please find below our response.
We thank the reviewer for pointing out these works. All three are quite recent (2026), and none is closely tied to our paper in either problem or method. They focus on predicting or estimating question difficulty, for LLMs or for students. Our goal is to explain an already-measured difficulty through human-readable, testable hypotheses. Even so, we will discuss these works in the related-work section of the revised version, to place our work more clearly within the broader difficulty-estimation literature.
We would like to emphasize that our contribution is not a new way to measure difficulty but a method to explain difficulty. In addition, we respectfully disagree that IRT is a naive method for difficulty estimation. It is a principled model and a standard tool for measuring item difficulty, and is widely used both in psychometrics and education and in recent LLM evaluation research. For example, this paper [1] suggested by you also adopts the 1PL IRT model to estimate item difficulty.
Furthermore, the simplicity of the model does not mean it is unreliable. We verified the robustness of this method through 50 rounds of bootstrap sampling: each time, we sampled 50% of the LLMs and refitted the IRT model. The results show that the IRT-estimated difficulties are highly robust and stable, with correlations between different samplings all above 0.99.
Finally, our method is not tied to IRT. As stated before, the contribution of this paper is a method for explaining difficulty. We use IRT as one module because it is a standard measurement tool, but our method is agnostic to the measurement method: one can replace IRT with a better difficulty estimator if needed.
Our goal is to explain item difficulty for LLMs. That difficulty need not match the difficulty humans perceive. Prior studies have shown that the factors driving difficulty differ between humans and LLMs [1,2], since their knowledge and skills are not aligned to begin with. Our core objective is therefore to find explanations that are valid for LLMs and, at the same time, readable by humans. Our causal experiments directly support this: when items are edited along the discovered difficulty factors, LLM performance changes significantly in the predicted direction.
We believe our method has many applications, for example,
Benchmark evolution. As models become increasingly capable, harder benchmarks are needed, as discussed in the introduction. Our method discovers difficulty factors, which can then be used to curate new test questions, keeping benchmarks discriminative and up to date. This application is demonstrated in the paper: in the causality study, eidting questions according to our generated difficulty explanations substantially increases their difficulty, showing that our method succeeds for this purpose.
Understanding model capabilities, especially weaknesses. The difficulty factors reveal the types of questions that LLMs struggle with, which is important in human–AI interaction and collaboration: knowing where a model is likely to fail helps users decide when to rely on it and when to verify its outputs. Moreover, in high-stakes or safety-critical settings, such knowledge allows error-prone inputs to be identified in advance and handled with additional caution
Targeted training. Knowing a model’s weaknesses, one can generate training data that specifically targets them or reweight existing training data accordingly, making training more effective. We leave this for future work.
Reference
[1] Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Prediction
[2] Do great minds think alike? Investigating Human-AI Complementarity in Question Answering with CAIMIRA.