acceptodds
Under review as a conference paper at ICLR 2027

Hard, Not Rare: Human Item Difficulty, Not Concept Frequency, Orders Model Errors Across Professional Examinations

Abstract

A common explanation for language model errors on specialised tasks is that the relevant concepts are rare in pretraining data. We test this hypothesis on individual items of the Japanese National Medical Licensing Examination, pairing exact n-gram counts from the evaluated models' own pretraining corpus with per-item human examinee accuracy, and extend the analysis to Korean licensing examinations in law, psychology, management, and chemistry, for over 4,800 items in total. The hypothesis does not hold at the item level. On 1,213 single-answer medical items, concept frequency contributes 0.30 percentage points of cross-validated R² at 32B, while human item difficulty contributes 8.71 points. In a positive control, the same counts predict per-token log-probability at r ≈ 0.76 within fixed-length strata, so the negligible contribution on examination items cannot be attributed to measurement error. Difficulty and frequency are near-orthogonal, with a rank correlation of 0.058, and models fail hard items regardless of whether the concepts are rare. Scaling from 7B to 32B adds 25.5 percentage points on the easiest items and 6.2 on the hardest. Human item accuracy also predicts model confidence in five models from four additional families, with r between 0.223 and 0.338, and in all twenty domain–model combinations on the Korean examinations, where every 95% confidence interval excludes zero and concept frequency contributes under 2 R² percentage points per domain. On professional examinations, the performance gap on hard items reflects task demands that pretraining frequency does not explain.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language models, pretraining data, concept frequency, item difficulty, professional licensing examinations, scaling, model evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vCFsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether language model performance on professional examination items is better predicted by human item difficulty or by the marginal frequency of the concepts involved. The main study combines official per-item human accuracy for Japanese medical licensing examinations with exact concept-span counts from the indexed OLMo-2 pretraining mixture. On 1,213 medical items, human item accuracy provides substantially greater incremental cross-validated predictive power than the proposed concept-frequency measure: 8.71 versus 0.30 R² percentage points for OLMo-2 32B. The paper also finds that scaling gains are larger on human-easy items. Token-level positive controls, alternative frequency aggregations, answer-side analyses, and alternative scoring metrics complement the main results. Additional experiments show that the human–model difficulty association extends to other model families and English translations of Korean examinations in four professional domains.

Critical strengths

The paper investigates the relationship between concept frequency, human item difficulty, and language model performance on professional examinations, addressing a relevant research question. One strength is the use of exact n-gram counts from OLMo's pretraining corpus alongside item-level human accuracy statistics, allowing a direct comparison of the predictive contributions of the two factors. The experiments cover multiple model scales, model families, and professional domains, providing evidence beyond a single evaluation setting. The authors also conduct several robustness analyses, including cross-validated regressions, alternative frequency aggregation methods, and answer-side replication. In addition, the analysis of performance gains across difficulty levels offers an informative perspective on model scaling behavior. Overall, the paper presents a structured empirical investigation with several useful observations, although the interpretation of the findings depends on the validity of the proposed measurements.

Critical weaknesses

The paper presents an interesting investigation into the relationship between concept frequency, human item difficulty, and model performance. However, several concerns weaken its central conclusions. First, the proposed concept frequency measure may not adequately capture knowledge scarcity. The authors use the minimum n-gram frequency of extracted concepts, but isolated concept occurrences do not reflect the availability of task-relevant relationships or contextual knowledge. Therefore, the weak predictive power of this measure does not necessarily rule out knowledge scarcity as a source of model errors. Second, human item accuracy is treated as intrinsic task difficulty, although it may also reflect knowledge familiarity, educational background, and examination strategies. Thus, its correlation with model confidence does not establish that model errors are primarily driven by reasoning complexity. Third, the token-level positive control primarily validates lexical familiarity rather than actual knowledge acquisition, and therefore cannot fully establish the validity of the proposed frequency measure. Furthermore, the analysis does not sufficiently examine nonlinear frequency effects or interactions with difficulty, while the inconsistent results between stem-side and answer-side extraction suggest sensitivity to the measurement strategy. The cross-model experiments also use OLMo's corpus as a frequency proxy for other models, and English translations of Japanese and Korean examinations may introduce additional biases. Finally, although human difficulty consistently outperforms concept frequency, its absolute correlation with model confidence remains relatively modest (e.g., 0.327 at 32B). Overall, the findings support the narrower conclusion that marginal concept frequency has limited predictive power under the proposed measurement framework, but do not convincingly establish that task difficulty, rather than knowledge scarcity, primarily determines model errors.

Questions for authors
  1. Scope of the central claim: Would the authors agree that the evidence establishes limited predictive value for the tested marginal span-frequency measures, rather than ruling out insufficient task-relevant training coverage more generally? Please clarify how the token-level positive control supports the abstract’s statement that the result cannot be attributed to measurement error.

  2. Translation validity: Is there any semantic validation of the Korean-to-English translations beyond the automatic checks in Appendix H? In particular, evidence from a sample stratified by human item difficulty would help assess whether translation quality could systematically affect the reported associations.

  3. Uncertainty in the main comparisons: Can the authors report uncertainty or split sensitivity for the incremental cross-validated R² estimates, and directly test the difference in 7B-to-32B accuracy gains between the easy and hard strata? These would help establish the stability of the predictive gap and the scaling result.

Reviewer_1VvCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

ost prior frequency‑related experiments focus on general‑domain scenarios and fail to disentangle and compare concept frequency against the intrinsic difficulty of test items. Rigorous tests at the granularity of individual examination questions remain absent. Such a test must satisfy three requirements: ① extract the specific concepts required for each question; ② compute n‑gram frequencies using the exact pretraining corpus of the model under evaluation; ③ incorporate both frequency and human‑measured item difficulty within a single regression model.

Accordingly, this paper makes the following contributions:

  • An item‑level test of the marginal‑frequency hypothesis: we leverage exact n‑gram counts derived from the model’s native pretraining corpus together with per‑item human examinee accuracy, and perform regression with both difficulty and frequency as predictors. A token‑level positive control and an answer‑side replication are designed to validate the reliability of frequency counts.
  • Empirical refutation of the hypothesis: human‑measured item difficulty explains model performance far better than concept frequency. This pattern holds stably across five model families, five professional domains, and two examination languages.
  • A novel finding regarding scaling dynamics: performance gains from increasing model parameter size are largely concentrated on items that human examinees find easy, while gains on hard items are minimal.
Critical strengths

Concept proposing : idea of Difficulty, not rareness, which is reasonable

Critical weaknesses

It's staightforward that human tackle difficult tasks shoudl be smart, not just witness cases as many as possible. The paper proposes only some evidence with no solution or further directions.

Questions for authors

How do you consider solving such problem

Reviewer_pdoNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether human item difficulty or concept frequency in pretraining data better predicts LLM performance on professional examinations. The authors find that human difficulty is a stronger predictor, while scaling gains are concentrated on easier items.

Critical strengths
  1. Investigating the relationship between human item difficulty, pretraining data frequency, and LLM performance is an interesting and meaningful research question.

  2. The use of OLMo-2 allows direct measurement of concept frequency from the models' actual pretraining corpus. The experiments also cover multiple models and domains, with extensive robustness checks.

Critical weaknesses
  1. While human difficulty outperforms concept frequency, the absolute correlations remain relatively weak (0.153–0.327 for OLMo-2 and 0.068–0.217 for KMMLU). The emphasized 8.71-point R² improvement is only a relative predictive gain and does not establish strong alignment. The title and conclusions appear to overstate what these results actually demonstrate.

  2. The frequency-based hypothesis is primarily motivated by findings on factual recall and long-tail knowledge, but its applicability to reasoning-intensive tasks is unclear. Moreover, marginal concept frequency does not capture reasoning patterns or concept combinations in pretraining data, making the broader framing of the hypothesis questionable.

  3. The paper largely overlooks Educational Assessment and Psychometrics literature, as well as prior studies on human–LLM difficulty alignment. In particular, observed human item accuracy is population-dependent, yet the implications of different examinee populations and cross-cohort comparability are not sufficiently discussed.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.