- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work introduces UlumBench, which has over 6000 questions from 19 courses at one university with 500 constructed diagnostic problems, each a chain of two to eight dependent operations with intermediate reference answers. By evaluating five frontier models, the work find that the best academic accuracy is 72.04% and that scores are sensitive to answer format. Correct intermediate answers help more on longer chains, the controls do not establish a growing dependency penalty. A verify-first instruction reduces the penalty from injected errors for four of the five models, while higher reasoning effort does not consistently improve accuracy.
Critical strengths
The over 600 questions were written by the professors who taught the 19 courses and cover multiple disciplines, response types and image-supported inputs, a design intended to reduce training-data contamination.
Critical weaknesses
Beyond the limited novelty of the benchmark itself, the reported scores are heavily confounded by output format and scoring choices, so it is unclear how much they reflect problem-solving ability. Under the label-only condition, 16.8–41.9% of MCQ responses are invalid, and the same items score 8.7–28.8 pp higher under the earlier permissive parsing (Table 4). So the main conclusions are affected.
Questions for authors
What causes the 16.8–41.9% invalid MCQ responses under the label-only condition (e.g., truncation, extra text, refusals)? Do the accuracies and the model ranking hold when only valid responses are scored? (Table 4)
Thank you for your careful review and for considering the additional experiments and analysis below. We conducted a supplementary label-only evaluation of all 4,158 MCQs across five models, using one fixed checkpoint per model. We applied both parsers to the same saved responses, investigated invalid outputs, compared valid-only accuracy, and completed a two-human extraction audit. Questions and reference keys were held fixed. These are new diagnostic results, not a reclassification of the original Table 4 generations.
Causes of invalid responses. Valid means a completed output yields an allowed option; correct means that option matches the reference key. Strict parsing accepts supported whole-response formats, such as
DorAnswer: D. Permissive parsing also recognizes selections expressed with additional text or formatting. Neither parser consults the key to extract a choice.Of 5,723 strict-invalid responses, 5,514 were completed outputs rejected by strict parsing and 209 hit the token limit:
d) 16.13, a label followed by option text, or an explanation with an identifiable answerPermissive parsing recovered 5,435/5,723 (94.97%) strict-invalid responses: 4,228 selections matched the key and 1,207 did not. Recovery therefore includes both correct and incorrect answers. Truncations remain failures. These counts concern responses across models, not distinct questions.
Human validation. Two human reviewers assessed the same subset of 679 completed responses from the 20,790 outputs. We randomly selected 100 recovered outputs and 20 strict-valid controls per model (500 + 100), and included all 79 completed outputs unresolved by permissive parsing. Selection did not use answer correctness. Both reviewers agreed with the parser's selection on 600/600 accepted sampled responses. This validates extraction on the audited sample; it does not establish zero error across all outputs. Final adjudication identified 78/79 unresolved selections (61 key-matching, 17 non-matching); one remained ambiguous. Including those human selections raises full-set accuracy by 0-1.25 percentage points. This separate analysis is excluded from the automatic results below. The audit checks expressed choices, not academic key correctness.
Accuracy and ranking on valid responses. We compare all 4,158 questions, each model's own strict-valid outputs, and common questions where all five models were valid. The latter contains 1,704 questions under strict parsing and 3,873 under permissive parsing. All scores are percentages; full-set scores retain invalid/truncated responses as failures.
Own strict-valid denominators, in row order, are 3,583, 3,385, 3,143, 2,542 and 2,414. They contain different questions and should not directly determine a cross-model ranking.
Accuracy increases, and the complete ranking changes. Gemini has the highest point estimate on the full set and both shared subsets; Qwen rises to second on shared permissive-valid questions. Grok's full-set permissive advantage over Claude/GPT is only one correct answer. Paired 95% bootstrap intervals, resampling source/duplicate groups, include zero for those two differences. We will avoid interpreting such small differences as reliable rank separations.
Shared-valid accuracy is conditional on selection. The shared strict subset contains only 43/1,704 image questions (2.52%), versus 587/4,158 (14.12%) in the full set. Its 95.83-97.36% accuracy cannot be generalized to all MCQs. The two shared subsets also contain different questions, so their scores are not a fixed-subset parser comparison.
We will report parser acceptance separately from answer correctness: full-set permissive accuracy is 84.80-89.71%. UlumBench's contribution includes course-derived academic coverage and a separate constructed suite with explicit dependencies and controlled operation lengths. We will revise our MCQ difficulty claims to reflect these scores.
We will add the failure breakdown, parser examples, human-audit protocol, full-set and conditional scores, and ranking uncertainty to the manuscript, and revise Table 4's interpretation accordingly.
We hope these additional experiments, analyses, and clarifications address all your concerns, and we thank you again for your thoughtful feedback.