- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether language model performance on professional examination items is better predicted by human item difficulty or by the marginal frequency of the concepts involved. The main study combines official per-item human accuracy for Japanese medical licensing examinations with exact concept-span counts from the indexed OLMo-2 pretraining mixture. On 1,213 medical items, human item accuracy provides substantially greater incremental cross-validated predictive power than the proposed concept-frequency measure: 8.71 versus 0.30 R² percentage points for OLMo-2 32B. The paper also finds that scaling gains are larger on human-easy items. Token-level positive controls, alternative frequency aggregations, answer-side analyses, and alternative scoring metrics complement the main results. Additional experiments show that the human–model difficulty association extends to other model families and English translations of Korean examinations in four professional domains.
Critical strengths
The paper investigates the relationship between concept frequency, human item difficulty, and language model performance on professional examinations, addressing a relevant research question. One strength is the use of exact n-gram counts from OLMo's pretraining corpus alongside item-level human accuracy statistics, allowing a direct comparison of the predictive contributions of the two factors. The experiments cover multiple model scales, model families, and professional domains, providing evidence beyond a single evaluation setting. The authors also conduct several robustness analyses, including cross-validated regressions, alternative frequency aggregation methods, and answer-side replication. In addition, the analysis of performance gains across difficulty levels offers an informative perspective on model scaling behavior. Overall, the paper presents a structured empirical investigation with several useful observations, although the interpretation of the findings depends on the validity of the proposed measurements.
Critical weaknesses
The paper presents an interesting investigation into the relationship between concept frequency, human item difficulty, and model performance. However, several concerns weaken its central conclusions. First, the proposed concept frequency measure may not adequately capture knowledge scarcity. The authors use the minimum n-gram frequency of extracted concepts, but isolated concept occurrences do not reflect the availability of task-relevant relationships or contextual knowledge. Therefore, the weak predictive power of this measure does not necessarily rule out knowledge scarcity as a source of model errors. Second, human item accuracy is treated as intrinsic task difficulty, although it may also reflect knowledge familiarity, educational background, and examination strategies. Thus, its correlation with model confidence does not establish that model errors are primarily driven by reasoning complexity. Third, the token-level positive control primarily validates lexical familiarity rather than actual knowledge acquisition, and therefore cannot fully establish the validity of the proposed frequency measure. Furthermore, the analysis does not sufficiently examine nonlinear frequency effects or interactions with difficulty, while the inconsistent results between stem-side and answer-side extraction suggest sensitivity to the measurement strategy. The cross-model experiments also use OLMo's corpus as a frequency proxy for other models, and English translations of Japanese and Korean examinations may introduce additional biases. Finally, although human difficulty consistently outperforms concept frequency, its absolute correlation with model confidence remains relatively modest (e.g., 0.327 at 32B). Overall, the findings support the narrower conclusion that marginal concept frequency has limited predictive power under the proposed measurement framework, but do not convincingly establish that task difficulty, rather than knowledge scarcity, primarily determines model errors.
Questions for authors
-
Scope of the central claim: Would the authors agree that the evidence establishes limited predictive value for the tested marginal span-frequency measures, rather than ruling out insufficient task-relevant training coverage more generally? Please clarify how the token-level positive control supports the abstract’s statement that the result cannot be attributed to measurement error.
-
Translation validity: Is there any semantic validation of the Korean-to-English translations beyond the automatic checks in Appendix H? In particular, evidence from a sample stratified by human item difficulty would help assess whether translation quality could systematically affect the reported associations.
-
Uncertainty in the main comparisons: Can the authors report uncertainty or split sensitivity for the incremental cross-validated R² estimates, and directly test the difference in 7B-to-32B accuracy gains between the easy and hard strata? These would help establish the stability of the predictive gap and the scaling result.