acceptodds
Under review as a conference paper at ICLR 2027

UlumBench: A Benchmark for Evaluating Academic and Multi-Step Problem Solving in LLMs

Abstract

We present UlumBench, a benchmark for academic problem solving and controlled multi-step execution. Its 6,167 questions from 19 university courses include multiple-choice and open-ended tasks with text-only and image-supported inputs. A separate diagnostic collection provides 500 source-linked constructed problems with two to eight specified operations and intermediate references. Across five model families, the strongest academic baseline reaches 72.04% accuracy. On constructed problems, the terminal advantage of supplied correct states grows by 14–26 percentage points between two and eight steps. However, independent-task and whole-sequence controls do not establish a model-general growing dependency penalty. Verification reduces the measured penalty from supplied errors for four of five models under the recorded scorer, while cross-model plurality raises multiple-choice accuracy from 77.03% to 83.19%. These results motivate evaluating academic answers alongside intermediate-state use, sequence completeness, and error recovery, rather than interpreting a single final-answer score as a sufficient account of reasoning reliability.

Primary area:
datasets and benchmarks
Keywords:
Large language models, Benchmarks, Academic reasoning, Multi-step reasoning, Compositional reasoning, Reasoning reliability, Error recovery, Self-correction, Multimodal question answering, LLM evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ghABOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces UlumBench, which has over 6000 questions from 19 courses at one university with 500 constructed diagnostic problems, each a chain of two to eight dependent operations with intermediate reference answers. By evaluating five frontier models, the work find that the best academic accuracy is 72.04% and that scores are sensitive to answer format. Correct intermediate answers help more on longer chains, the controls do not establish a growing dependency penalty. A verify-first instruction reduces the penalty from injected errors for four of the five models, while higher reasoning effort does not consistently improve accuracy.

Critical strengths

The over 600 questions were written by the professors who taught the 19 courses and cover multiple disciplines, response types and image-supported inputs, a design intended to reduce training-data contamination.

Critical weaknesses

Beyond the limited novelty of the benchmark itself, the reported scores are heavily confounded by output format and scoring choices, so it is unclear how much they reflect problem-solving ability. Under the label-only condition, 16.8–41.9% of MCQ responses are invalid, and the same items score 8.7–28.8 pp higher under the earlier permissive parsing (Table 4). So the main conclusions are affected.

Questions for authors

What causes the 16.8–41.9% invalid MCQ responses under the label-only condition (e.g., truncation, extra text, refusals)? Do the accuracies and the model ranking hold when only valid responses are scored? (Table 4)

1 comment

Thank you for your careful review and for considering the additional experiments and analysis below. We conducted a supplementary label-only evaluation of all 4,158 MCQs across five models, using one fixed checkpoint per model. We applied both parsers to the same saved responses, investigated invalid outputs, compared valid-only accuracy, and completed a two-human extraction audit. Questions and reference keys were held fixed. These are new diagnostic results, not a reclassification of the original Table 4 generations.

Causes of invalid responses. Valid means a completed output yields an allowed option; correct means that option matches the reference key. Strict parsing accepts supported whole-response formats, such as D or Answer: D. Permissive parsing also recognizes selections expressed with additional text or formatting. Neither parser consults the key to extract a choice.

Of 5,723 strict-invalid responses, 5,514 were completed outputs rejected by strict parsing and 209 hit the token limit:

Cause/outcomeResponses
Alternative answer forms recovered automatically: e.g., d) 16.13, a label followed by option text, or an explanation with an identifiable answer5,435
Other completed outputs: humans identified 78 selections; one remained ambiguous79
Token-limit truncations209
Detected refusals0

Permissive parsing recovered 5,435/5,723 (94.97%) strict-invalid responses: 4,228 selections matched the key and 1,207 did not. Recovery therefore includes both correct and incorrect answers. Truncations remain failures. These counts concern responses across models, not distinct questions.

Human validation. Two human reviewers assessed the same subset of 679 completed responses from the 20,790 outputs. We randomly selected 100 recovered outputs and 20 strict-valid controls per model (500 + 100), and included all 79 completed outputs unresolved by permissive parsing. Selection did not use answer correctness. Both reviewers agreed with the parser's selection on 600/600 accepted sampled responses. This validates extraction on the audited sample; it does not establish zero error across all outputs. Final adjudication identified 78/79 unresolved selections (61 key-matching, 17 non-matching); one remained ambiguous. Including those human selections raises full-set accuracy by 0-1.25 percentage points. This separate analysis is excluded from the automatic results below. The audit checks expressed choices, not academic key correctness.

Accuracy and ranking on valid responses. We compare all 4,158 questions, each model's own strict-valid outputs, and common questions where all five models were valid. The latter contains 1,704 questions under strict parsing and 3,873 under permissive parsing. All scores are percentages; full-set scores retain invalid/truncated responses as failures.

ModelStrict/allPermissive/allStrict/own validShared strictShared permissive
Gemini78.7689.7191.4097.3692.12
Claude75.3287.9592.5396.5490.76
GPT69.7787.9592.3095.8390.40
Grok57.1287.9793.4396.9590.76
Qwen55.7284.8095.9896.2490.86

Own strict-valid denominators, in row order, are 3,583, 3,385, 3,143, 2,542 and 2,414. They contain different questions and should not directly determine a cross-model ranking.

Accuracy increases, and the complete ranking changes. Gemini has the highest point estimate on the full set and both shared subsets; Qwen rises to second on shared permissive-valid questions. Grok's full-set permissive advantage over Claude/GPT is only one correct answer. Paired 95% bootstrap intervals, resampling source/duplicate groups, include zero for those two differences. We will avoid interpreting such small differences as reliable rank separations.

Shared-valid accuracy is conditional on selection. The shared strict subset contains only 43/1,704 image questions (2.52%), versus 587/4,158 (14.12%) in the full set. Its 95.83-97.36% accuracy cannot be generalized to all MCQs. The two shared subsets also contain different questions, so their scores are not a fixed-subset parser comparison.

We will report parser acceptance separately from answer correctness: full-set permissive accuracy is 84.80-89.71%. UlumBench's contribution includes course-derived academic coverage and a separate constructed suite with explicit dependencies and controlled operation lengths. We will revise our MCQ difficulty claims to reflect these scores.

We will add the failure breakdown, parser examples, human-audit protocol, full-set and conditional scores, and ranking uncertainty to the manuscript, and revise Table 4's interpretation accordingly.


We hope these additional experiments, analyses, and clarifications address all your concerns, and we thank you again for your thoughtful feedback.

Reviewer_JgAKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces UlumBench, a benchmark designed to evaluate the capabilities of LLMs in handling university-level academic questions and multi-step problem-solving tasks. The benchmark includes academic questions from different subjects and a separate set of LLM-generated multi-step problems, where later steps depend on answers from earlier steps. The authors evaluate several LLMs under different experimental settings, such as providing correct or incorrect intermediate answers, asking models to verify their answers, and changing the reasoning effort. The paper also examines how LLMs use intermediate results, handle dependencies between steps, and recover from errors during multi-step problem solving.

Critical strengths

none

Critical weaknesses

1.The evaluation results are highly affected by the answer format and scoring methods. For example, the MCQ accuracy changes a lot between label-only and permissive scoring. The post-hoc normalization also changes some results in the multi-step experiments, and the original outputs of GPT and Gemini are not available for re-evaluation. This makes it difficult to know how reliable some of the reported findings are.

2.The independent-task controls have differences in context length and topic switching, as the authors mentioned in Appendix C. These differences may also affect the model performance. Since the experiment does not show a significant growing dependency penalty, it is difficult to know whether this is because the dependency effect is small or because the experimental controls cannot clearly measure it

Questions for authors

1.The dataset contains much more multiple-choice questions than other question types. Have the authors considered reporting a balanced accuracy across different question types? Would this change the overall model ranking?

2.The post-hoc normalization changes some experimental results, but the original GPT and Gemini outputs are not available for re-evaluation. Can the authors explain why these outputs are missing and whether they can reproduce these experiments?

3.The verification experiments use artificially corrupted intermediate answers. Have the authors considered testing verification on errors generated by the models themselves? Would the results still be similar?

4.Most evaluation settings only use one generation for each question. Have the authors tested whether the results are consistent across multiple runs? How much variation can be expected from the same model?

1 comment

Thank you for your careful review. We conducted supplementary experiments and audited the saved results to address your concerns about scoring, controls and repeatability.

1. Scoring reliability and reproducibility. Our supplementary MCQ experiment applies strict and permissive extraction to the same 20,790 responses (4,158 questions × five models). Strict extraction requires supported whole-response formats; permissive extraction also recognizes identifiable selections accompanied by text. Neither uses the answer key to choose an option. Full-set accuracy changes from 55.72–78.76% to 84.80–89.71%; rankings also change. We therefore separate extraction compliance from answer correctness.

Two human reviewers audited 679 completed responses: 600 sampled parser-accepted outputs and all 79 unresolved outputs. Both agreed with extraction in all 600 accepted cases. This checks expressed choices, not answer-key correctness.

For multistep tasks, we retain original scores alongside documented post-hoc normalization of equivalent representations, preserving references, tolerances and completion/schema requirements. Invalid and truncated outputs remain failures.

The original GPT/Gemini provider outputs are absent from the available historical archive. We instead completed a five-model, three-generation oracle/composed experiment with 12,000 retained responses and requests. Its original scores replay exactly. For the independent control, we can re-score saved answer fields, but cannot replay extraction from missing raw outputs. This new analysis does not reconstruct the unavailable historical normalization.

2. Balance across question types. MCQs constitute 67.4% of the academic panel. We computed type-macro accuracy: the mean of the ten response-type accuracies, giving each type 10% weight. On the original headline baseline, changing only the weights gives:

ModelQuestion-weightedType-macro
Claude72.04%65.11%
Gemini69.50%60.84%
GPT64.86%57.65%
Grok56.72%51.07%
Qwen56.02%55.29%

The leading three point ranks remain; Qwen and Grok exchange fourth/fifth. A separate historical permissive analysis retains this macro ordering but changes the question-weighted ordering. Thus rankings depend on weighting and scoring; lower macro accuracy is not evidence of greater intrinsic difficulty. Unlike the same-response experiment above, the two historical panels use different MCQ generations.

3. What the independent control establishes. You are right that matching operation pools and answer counts does not match context length or topic switching. In our new normalized analysis, the increase from two to eight operations in the independent-minus-composed final-answer accuracy gap is 0.7–5.7 percentage points across models; every 95% interval includes zero. This cannot distinguish a small dependency effect from an insensitive or confounded control. We therefore do not claim a generally increasing dependency penalty. The oracle comparison separately measures the benefit of supplying correct prerequisites; it does not isolate dependency alone.

4. Errors produced by models. We conducted a follow-up using erroneous intermediate values from saved model generations: 20 selected problem families × five models × six conditions × three repetitions = 1,800 responses. Conditions pair model-generated and handcrafted errors with ordinary or verify-first continuation, plus clean-input controls. All models receive the same donor errors and correct earlier prerequisites. This tests recovery from supplied model-originated errors, not spontaneous correction of their own uninterrupted reasoning.

A secondary finding is Grok's intermediate correction rising from 36/60 to 56/60, surviving Holm correction across ten model/endpoint comparisons. However, no model's primary final-answer effect—verification's benefit on erroneous inputs minus its benefit on clean inputs—passes correction across five models on the full 20-family panel. These results support model-dependent intermediate correction, but do not establish that model-generated errors reproduce the earlier handcrafted-error benefit.

5. Variation across generations. The repeated oracle experiment covers five models, 100 families, two conditions and four horizons, each with three generations. Across its 40 model–condition–horizon settings, the largest-minus-smallest accuracy has a median of 2 and maximum of 6 percentage points. Correctness varies in 247/4,000 model–family–condition–horizon combinations (6.2%). We average repetitions within families for inference; academic and reasoning-effort experiments remain single-generation evaluations.


UlumBench's contribution is the academic resource and the separate diagnostic protocols for studying intermediate-information use. We hope these experiments and analyses address your concerns, and thank you again for helping us improve the evaluation.

Reviewer_BMt2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a new benchmark for evaluating LLMs on academic questions and controlled diagnostic task. It includes university-level questions across multiple subjects and modalities, alongside constructed problems. The benchmark is promising

Critical strengths
  • The paper provides a potentially useful resource for evaluating LLMs on university-level academic tasks
  • Humans (professors) confirmed authorship of submitted questions, helping reduce the risk of benchmark contamination from publicly available questions
  • The benchmark covers multiple university courses and question formats
  • The dataset construction methodology is well described, and easy to follow
Critical weaknesses
  • The motivation, limitations of existing benchmarks, research gap, and evaluation are not clearly presented. Particularly, the abstract and the introduction should be rewrite
  • The third claimed contribution is not a contribution, rather, it report an experimental result
  • Unsupported claim in the introduction (e.g., "A model may solve an operation ...ts itself".) are provided without examples or references
  • The paper does not clearly describe the evaluation of the benchmark, what is evaluated, how the metric is calculated, etc.
  • Concrete examples of evaluation tasks and model outputs are missing
  • Neither the dataset subset nor an anonymous dataset link is provided as supplementary material
Questions for authors
  • What exactly is evaluated during experiments? How the accuracy is calculated?
  • The paper reports 6,167 questions in one place and 6,295 records in another. Which number is correct?
2 comments

Thank you for your careful review and for recognizing the resource's value. We have revised the motivation, contributions, and evaluation, and added concrete task and scoring examples.

1. Motivation, research gap, and contributions (Abstract; Sections 1–2). The revised paper connects academic question answering with controlled comparisons of intermediate-information use. MMLU-Pro assesses academic answer correctness, while ProcessBench evaluates error identification in supplied solutions. Our constructed collection compares generating prerequisite answers, receiving correct predecessors, and continuing after a supplied error. Horizons of two to eight count requested operations, not internal reasoning steps.

We now state two contributions: (i) the academic resource spanning 19 courses, ten response types, and text/image inputs; and (ii) the constructed diagnostic collection and its evaluation protocols. Numerical findings are presented as results rather than a separate contribution. The unsupported introductory assertion is replaced by questions motivating these comparisons, without assuming a universal dependency penalty. Section 4 now introduces the RQ1–RQ4 roadmap.

2. Evaluation and scoring (Section 4.1; Table 3; Appendices A–E). The paper separates four evaluations:

  • Academic answering: university-question accuracy, answer extraction, and supplied multipart context.
  • Constructed execution: composed versus oracle-assisted sequences, with a separate matched independent-task control.
  • Error recovery: correct versus corrupted supplied states, under ordinary continuation or explicit verification.
  • Requested effort: low, medium, and high settings on the same 658 computer-science questions.

For each model, academic accuracy is 100 × correctly scored questions / evaluated questions. Each of the 6,167 questions contributes once to the original baseline: 4,158 MCQs use fixed option targets; responses to 2,009 open-ended questions receive binary GPT-4o-mini judgments against references. Appendix A reports a separate two-reviewer check of 500 judged responses.

We distinguish extraction coverage, full-set accuracy, and accuracy conditional on successful extraction, with explicit denominators. The supplementary MCQ analysis applies both parsers to the same saved responses, separating extraction effects from differences in generation.

Constructed tasks use deterministic, answer-type-specific checks. In the oracle comparison, terminal accuracy scores the separately returned final answer. Whole-sequence accuracy requires every requested step answer, including the last, to pass; requiring both the step list and the separate final answer is a distinct all-output endpoint. Invalid responses and completion failures remain failures in full-set scores. Original and representation-normalized scores are reported separately, retaining the same tolerances and completion requirements. Appendices A–E document settings, prompts, panels, scoring, and statistical procedures.

3. Concrete tasks, outputs, and scoring (Appendix G). Appendix G.1 now contains ten academic examples, one per response type, including image-based MCQs, calculations, proofs, code, and reductions. Each identifies the task, necessary context, reference, saved model output, recorded score, and scoring explanation. Supplied preceding answers and any excerpts are explicitly identified.

For example, Claude's circuit answer “c) 10” and Grok's “10” select the same correct value, but the original extractor rejects the former. A separate calculation example shows an actual arithmetic error. Together, they make the distinction between extraction failure and incorrect content concrete.

Appendix G.2 provides the full archived input and response for a four-operation set problem involving union, intersection, symmetric difference, and set difference. Every returned set matches its reference, and the separate final answer is also correct, so both terminal and whole-sequence scores are 1.

4. Dataset accounting (Section 3.1; Table 2). The 6,295 records are the source archive. Disjoint exclusions remove 44 incomplete inputs, 73 unresolved reference-quality cases, and 11 malformed or non-unique MCQs: 6,295 − 44 − 73 − 11 = 6,167 evaluated questions. This panel contains 4,158 MCQs and 2,009 open-ended questions. The 500 constructed diagnostic problems are a separate collection and are not included in either academic count.

5. Material available for review. We have uploaded the revised manuscript to OpenReview and attached an anonymized dataset subset as supplementary material, so you can inspect the revisions and sample tasks directly.


We hope these revisions address your concerns and make the benchmark's purpose, evaluation, and scope clear. Thank you again for helping us improve the paper.

Thank you for the rebuttal. The paper would benefit from substantial revision and improvement, taking into account the reviewers feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.