acceptodds
Under review as a conference paper at ICLR 2027

Exam2Teach: Can LLM Agents Turn Exam Results into Teaching Materials?

Abstract

High-quality teaching materials empower teachers to bridge learning gaps and provide highly targeted student support. Preparing such materials from exam results requires accurately interpreting student answers, translating identified needs into targeted instruction, and producing clear, coherent multimodal documents. In this work, we introduce Exam2Teach, a benchmark for evaluating whether autonomous LLM agents can generate high-quality, personalized teaching materials. Exam2Teach spans five core subjects shared by university-entrance curricula worldwide, using real exam papers, grading rubrics, de-identified student answers and scores. We design an open-source multi-agent framework for composing multi-step agent workflows. It powers our entire benchmark: agents generate documents, and evaluation workflows combine rule-based checks with agentic inspection of source evidence and rendered documents to assess task completion, layout quality, factual correctness, and personalization. The framework is general-purpose and extends readily to new tasks. An extensive evaluation of 10 agent configurations across 150 runs reveals uneven performance across four dimensions: the task completion threshold was met in 45.3% of runs, layout quality in 62.0%, factual correctness in 12.7%, and personalization in 26.7%. Only 7.3% passed all four dimensions, with no passing runs for review slides, and no single model consistently produced high-quality teaching materials of all categories. Even the strongest model overall, GPT-5.5, passed all four dimensions in only 40.0% of its runs. These results indicate a clear gap for frontier LLM agents in teaching material generation for AI-assisted education. Code and data are available at https://anonymous.4open.science/r/Exam2TeachAnonymous-8E75/.

Primary area:
datasets and benchmarks
Keywords:
LLM agent benchmark, AI for education, teaching material generation, pedagogical personalization, multimodal document generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_uoVoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a benchmark to evaluate an LLM's ability to generate learning materials from exam results in five core subjects, along with a multi-agent framework to assess task completion, layout quality, factual correctness, and personalization, enabling teachers to tailor materials to students’ performance. Evaluations of 10 agent configurations across 150 runs show mixed results; only 7.3% of configurations passed all evaluation metrics. The low benchmarking scores establish a visible gap in the domain.

Critical strengths

The problem discussed is important in the AIED domain and addresses a key challenge in adaptive learning: teachers struggle to create learning materials based on exam results. The approach covers five core subjects (Biology, English, Mathematics, Physics, and Chemistry). The approach is being evaluated on open-source and closed-source models. It shows that even SOTA models such as GPT-5.5 and Opus 5 have 40% and 26.7% pass rates, highlighting a clear gap in the domain. Each subject includes several questions and student-written responses, which then pass through the agent harness to generate three documents (report, slides, and worksheets) based on the teacher's needs. The benchmarking data comes from real university entrance examinations and is labeled by human experts using the set grading rubrics.

Critical weaknesses

The paper's core contribution is the multi-agent framework, yet the main text does not explain it clearly; Figure 4 reads more like a complex engineering effort, but none of the components show novelty, literature grounding, or logical/empirical evidence. The paper attempts to address a core challenge in education with a multi-agentic harness but fails to establish either theoretical or empirical evidence for the proposed framework; the evaluated dataset was small, language-specific, and lacked human expert evaluation of the generated reports. The paper cannot be accepted in its current form, as it appears to attempt to solve a big problem with a complicated agentic framework but fails to establish any ground for its soundness, practicality, or generalizability. The figures are verbose, and the captions lack the context needed for understanding, along with other writing-related issues. Honest advice: narrow the scope; focus on one subject, or even a few core topics; identify core issues in student results; hire a subject expert to devise the evaluation framework based on subject-related pedagogies; and then create documentation. Later, devise a framework; it could be any agentic framework, not necessarily complicated, but simple, theoretically sound, and empirically supported. Report human-AI agreement on the generated documents, then publish that dataset for the research community for further benchmarking. That would be more valuable and useful than a purely synthetic, complex agentic framework with no educational/pedagogical/literature/theoretical grounding.

It's always appreciated to have the code available for review and to help the research community grow. However, the released code does not match the evaluation described in the paper. The factual-correctness score blends a 25% block-count term into Eq. 5, and the personalization score used in Eq. 1 is computed from issue-severity penalties rather than the 0-2 judge scores the paper describes, so the reported numbers cannot be derived from the stated formulas. The pass thresholds and cross-task total are not implemented at all. Several components the paper says are released are missing: the exam images agents received, the failure-pattern classification pipelines, and the personalization claim-consolidation procedure. Together with unpinned harness versions, this prevents rerunning the main experiment as described. The authors should correct the formulas in the text, release or clearly scope the missing components, and add the threshold and aggregation code.

Questions for authors

What is the range of N in the multi-agent framework? What factors determine it?

How does subject specialization embed into the agentic harness?

Where is the cost analysis of this evaluation?

What does subject-wise variation look like?

Where do the core evaluation metrics come from? Task completion, layer quality, factual correctness, and personalization vary across domains; how does the system define and adapt criteria for each subject?

3 comments
AuthorsOpenReview ↗

Reviewer 3 (uoVo) | Reply 1 of 3

W1 & Q1 Benchmark contribution and framework

We would like to clarify that our contribution lies in benchmarking agents' ability to turn real examination evidence into class reports, review slides, and personalized worksheets. The evaluation jointly assesses the generated materials' completeness, layout, factual reliability, and fit to learners' needs. Section 6 explains how this focus complements existing benchmarks.

The multi-agent framework provides the execution infrastructure for experiments in Exam2Teach and for other researchers and practitioners to reproduce our results and use the benchmark to evaluate their own agents, while the multi-agent framework can also be generalized to other custom workflows beyond this benchmark to evaluate other agent capabilities (Section 3). Users define agents and dependencies in YAML. The framework orchestrates those workflows, supports revision requests, and records model calls. Appendix K connects the design to prior work on coordination, tool use, and feedback-based revision. Therefore, the multi-agent framework serves the benchmark and is a part of the benchmark, but we do not claim it as a standalone contribution separated from the benchmark.

The number of agents, N, depends on the configured workflow and its inputs. Completion and personalization each use three judges. Factual verification and review expand with text and visual batches, and worksheet evaluation also scales with the student count.

Ablations on components support the necessity of different components in the evaluation workflow design.

For factual correctness, on a 30-page deck with 41 annotated factual errors, mean recall rises from 71.5% for direct GPT-5.5 verification to 88.6% with block batching and 95.1% with visual verification and independent review (Appendix H).

The relevant rows from Tables 26 and 27 show how the changes build on one another (Appendix H).

Verification setupErrors found in three runs, out of 41 eachMean recall
Direct GPT-5.5 verification30, 28, 3071.5%
Text-block batching36, 37, 3688.6%
+ Dedicated visual verification40, 38, 3793.5%
+ Independent review39, 39, 3995.1%

For personalization, the three-judge agentic panel's pooled findings cover 31/33 required issues, versus 27/33 for the API panel using the same criterion-by-criterion prompt. The panels report two and four invalid criticisms, respectively. This comparison uses human-reviewed claims drawn from judge reports on 18 diagnostic documents (Appendix I).

The table gives the two per-criterion rows from Table 37 (Appendix I.2).

Three-judge panelRequired issues foundInvalid criticisms reportedRequired recall
Agentic31/332/1393.9%
Direct API27/334/1381.8%

W2 Benchmark coverage and scope

Dataset size. We would like to clarify the scale of the evaluation. Each configuration is evaluated on 15 subject–task instances that produce 70 documents: 5 class reports, 5 review decks, and 60 personalized worksheets. Task completion alone checks these against 3,160 checklist items per run. Factual correctness checks each claim in the documents against 3,450 rubric-marked responses to 304 scored items, drawn from 120 real examination questions and 60 candidates (Appendix D). The main experiment comprises 270 generation runs, and across it and the exploratory study we evaluated 1,540 generated documents (Appendices A.1 and A.2).

Language. The source materials are in two languages. Four subjects use English questions and answers, and Biology uses non-English questions and answers that agents must explain in English.

Scope. We kept five subjects because they place different demands on agents, including mathematical derivations, scientific diagrams, and language assessment, and the results differ across them. Mathematics has the lowest mean factual score, and English Language has the lowest mean personalization score (Q4).

AuthorsOpenReview ↗

Reviewer 3 (uoVo) | Reply 2 of 3

W3 Human evaluation

We expanded human review of generated materials across all five subjects and three document formats. Higher overall scores correspond to less human-assessed revision effort (ρ = 0.65), and all 44 reviewed documents needing major changes fail the benchmark.

  • Personalization: five human experts reviewed 66 complete documents—22 reports, 22 slide decks, and 22 worksheets—across all five subjects and ten configurations. Each document received one review using assessment records, with model labels and automated scores hidden. Human and automated total scores correlate at Spearman's ρ = 0.69; automated total scores average 0.83 points higher on the 0–8 scale.
  • Factual correctness: human subject experts reviewed 150 regions across all five subjects and three formats, sampled independently of evaluator findings and scores. The evaluator flagged an error in 20 of the 21 regions where humans found an error (95.2% region coverage).
  • Completion: review of all 2,030 checklist items in 30 Biology documents found no clearly absent or incomplete requirement receiving full credit.
  • Layout: comparison with final human labels for 150 pages from five slide decks yields 94.9% recall and 97.4% precision.

Appendices E–I give the detailed human-validation procedures and results.

Higher overall benchmark scores were associated with less revision judged necessary by human reviewers (Spearman’s ρ = 0.65). Mean scores were 60.8 for documents needing Major changes, 79.4 for those needing Small changes, and 88.1 for those rated Ready. Only two documents were rated Ready, so this sample provides limited evidence about how reliably the pass thresholds identify classroom-ready materials.

Benchmark and human pass decisions. The table pools all three tasks from Table 20 (Appendix E.5). Ready counts as a human pass, and Revision needed combines Small changes and Major changes. Each worksheet uses its batch’s benchmark score and decision.

Benchmark decisionReadySmall changesMajor changesRevision neededTotal reviewed
Pass12023
Fail118446263
Total220446466

Real classroom validation: Teachers used materials meeting the four thresholds in 27 classes across 15 schools and recorded minor or no required edits. The classroom evidence concerns document usability and the edits teachers required before use.

W4 Scoring and reproducibility

Thank you for checking the released code against the paper. We rechecked the experiment results, corrected the factual-scoring description in the paper, and fixed the released personalization code. We also uploaded the pinned harness versions and the missing components needed for reproduction.

  • Factual correctness. Section 4 and Appendix H now document both the 75% text-length and 25% block-count terms. Erroneous blocks receive full deductions, and unverifiable-only blocks receive half deductions.
  • Personalization. The corrected code averages each direct 0–2 criterion score across three valid judges, then sums the four means to produce a 0–8 total. We have double checked and confirmed that the error is only in the released code and the experiment results in the paper are all correct.
  • Pass decisions and aggregation. The released scripts now implement the four thresholds, averaging across repetitions, task scores, and the cross-task total.
  • Other components. We have uploaded the missing components, including the pinned harness versions, the failure-pattern classification pipelines, and the personalization claim-consolidation procedure.
  • Exam images. Following the Area Chair's guidance, the images cannot be released now in order to comply with the double-blind submission policy. The complete dataset will be disclosed publicly after the double-blind period.

W5 Figures and captions

We revised the captions to add more detailed explanations, including the framework execution (Figure 4) and the evaluation pipeline (Figure 5). Sections 3 and 4 provide the accompanying detailed descriptions.

Q2 Subject adaptation

Agents adapt to each subject through its questions, rubrics, answers, and grading records. We evaluate existing model–harness configurations in Codex, Claude Code, and OpenCode with shared skills and task prompts across subjects (Appendix J).

AuthorsOpenReview ↗

Reviewer 3 (uoVo) | Reply 3 of 3

Q3 Cost analysis

Generation and evaluation together cost an estimated US$49.74 per run in the main benchmark, with evaluation accounting for 87.4%. The averages below cover six model configurations, five subjects, and three repetitions. We remark that the cost is for benchmarking the performance of different agents, not the cost of a solution we propose as this paper focuses on agent performance benchmarking instead of a solution to more efficiently solving the benchmarked tasks.

TaskGeneration (USD/run)Evaluation (USD/run)
Class report2.6812.58
Review slides12.4734.23
Worksheet batch3.6683.61
Across all tasks6.2743.48

Generation counts only the producing model, excluding generation helpers. Evaluation includes preparation and all four evaluation dimensions. Each worksheet run covers the complete student batch. Appendix C provides token usage, unit prices, cost breakdowns, and accounting assumptions and exclusions.

Q4 Subject-wise variation

Among the five selected assessments, Mathematics has the lowest mean factual correctness score, and English Language has the lowest mean personalization score. We have added the subject-wise results in Table 7 (Appendix A.1.2).

The table reports mean dimension scores and all-four pass counts in main benchmark results. Personalization uses a 0–8 scale; the other scores use 0–100.

SubjectTask completionLayout qualityFactual correctnessPersonalizationAll-four passes
Biology94.5488.4868.026.781/18
Chemistry93.6586.6967.536.532/18
English Language94.0489.1967.355.951/18
Mathematics92.8087.6562.616.190/18
Physics93.6888.1867.616.521/18

In addition, examples from the exploratory experiments illustrate challenges in these tasks. In the English Language assessment, B1 and B2 are examination routes with different questions. Some class reports overlook the needs of learners on one route or apply the same diagnostic assumptions to both groups (Appendix I.4). Mathematics outputs include incorrect numerical answers and mutually inconsistent problem conditions (Appendix H.4).

Q5 Metrics

Metrics choice. The four dimensions were informed jointly by feedback from collaborating teachers and failures observed during benchmark development (Appendix E.1). In our first trials with frontier models, the most visible problems were overlapping or clipped slide text and fluent claims unsupported by the exam records. As we expanded testing to more models, we found omissions, such as decks providing required statistics for only some questions. We also encountered worksheets that placed a student’s answer beside the correct answer and added a generic tip without explaining the actual error. Together with teachers’ expectations for usable materials, these observations motivated evaluating task completion, layout quality, factual correctness, and personalization separately.

Across subjects, we retain common quality standards while adapting the evaluation to the task requirements, subject evidence, document format, and intended learners. Teacher feedback supports broadly shared requirements for readable layouts and guidance aligned with learners’ needs. Specifically:

  • Task completion: Each task has a checklist derived from its requirements. Reports, slides, and worksheets therefore use different checklists, while the same checklist applies across all five subjects within each task (Appendix F).
  • Layout quality: The defect categories and scoring rules are shared across subjects, while the automated checks vary by document format. One script checks fixed-canvas PPTX slides; another checks paginated DOCX reports and worksheets. Both support visual inspection of rendered pages, with only confirmed defects contributing to the benchmark score (Appendix G).
  • Factual correctness: Exam-related textual and visual claims are checked against the corresponding subject’s input data, including questions, marking rubrics, student answers, and scores. Subject-specific content is therefore assessed against its own reference evidence (Appendix H).
  • Personalization: The same rubric applies across subjects, but the evaluation target differs by task. Reports and slides are assessed against class-level performance records to determine whether teaching priorities and proposed actions address the class’s needs. Worksheets are assessed against the intended student’s answers, marks, and marking rationales to determine whether explanations, feedback, and practice address that individual’s difficulties (Appendix I).
Reviewer_RvZnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a new benchmark, Exam2Teach, for evaluating the quality of generation by LLM-based agents in education. Exam2Teach is built on real-world exam papers and covers 5 core subjects with four tasks. The authors focus on a practical problem; however, the main text omits several details, making it difficult to assess the quality and the contribution of this manuscript. As a benchmark paper, the exact grading procedure, the data statistics, and the full evaluation protocol were missing, preventing me from recommending acceptance.

Critical strengths

This paper introduces a new benchmark, Exam2Teach, for evaluating LLM-based agents in education. Exam2Teach is of practical significance and covers a broad range of both subjects and tasks.

Critical weaknesses

Although this paper is easy to follow, the main text is informative. The paper reads more like an overview than a self-contained paper. Specifically, it is unclear from the benchmark how many questions, scored items, and responses each subject contains, how score levels are distributed, and what the inputs are. Additionally, the human expert validation is unclear.

Questions for authors

Q1. Most critically, I suggest moving more benchmark details into the main text.

Q2. What are the differences and connections to prior benchmarks? Analysing related benchmark work would strengthen this paper.

4 comments
AuthorsOpenReview ↗

Reviewer 2 (RvZn) | Reply 1 of 4

Q1 Moving more benchmark details into the main text

Following your suggestion, we added the dataset details, exam-marking procedure, pass rules, and human-validation results to the main text. With these specific results, we believe the main text is now self-contained with important insights backed by concrete evidence. Thank you for your constructive suggestion and if you have further suggestion on how we may improve the paper further, we would appreciate it if you would let us know.

  • Section 2, Tasks and Data. We added Table 1 to the main text. It reports the numbers of candidates, questions, separately scored items, and scored responses for each of the five subjects. In total, 60 candidates spanning five attainment levels contributed 3,450 scored responses across 120 questions and 304 scored items. Six human expert markers applied the official rubrics. Another human marker cross-checked each set of scripts, and high-school subject teachers reviewed the marks. Agents use the exam evidence to produce class reports, review slides, and personalized worksheets.
  • Section 4, Evaluation Method. The teacher-informed pass thresholds are 95% for task completion and 90/100 each for layout and factual correctness. Personalization requires a mean of at least 1.67/2, with each criterion at least 1.33/2. A pass requires meeting every cutoff.
  • Section 4, Human Validation. We added a factual audit of 150 regions across all five subjects and three document formats. The evaluator flagged an error in 20 of the 21 regions where reviewers found one. On 66 complete documents, automated personalization scores tracked human ratings (Spearman’s ρ = 0.69), and higher overall scores corresponded to less revision (ρ = 0.65).
  • Section 5, Evaluation and Analysis. The main experiment runs six configurations on five subjects and three tasks, with three repetitions per combination. We average the scores across repetitions before applying the pass thresholds. Section 5 also reports mean overall scores by human readiness rating. Scores averaged 60.8 for Major changes, 79.4 for Small changes, and 88.1 for Ready.

W1.1 How many questions, scored items, and responses does each subject contain, and how were they marked?

We use official examination papers and grading rubrics from a globally recognized pre-university public examination, using anonymized candidate scripts provided by the governing examination authority. We constructed one cohort per subject, each spanning all five reported attainment levels. The 60 scripts provide 3,450 scored responses across 304 items and 120 questions.

The table reproduces the subject-level counts and mark distributions from Table 17, Appendix D.

SubjectStudentsQuestionsItemsResponsesZeroPartialFull
Mathematics10194242014145234
Physics10124848016657257
Chemistry10145353020172257
Biology10114040013770193
English Language20641211,62066713940
Total601203043,4501,3122571,881

An item is the smallest separately scored part of a question. Each candidate's answer record for an item counts as one response, including a blank awarded zero. Partial marks fall between zero and the item's maximum. In English Language, ten candidates took each of two alternative routes alongside the common section. The item inventory counts the common section once, and the response counts include it for every candidate.

  1. Marking. Six subject experts scored individual responses using the official rubrics. Each had scored highly in the subjects they marked in the same or comparable examinations.
  2. Cross-check. Another human marker checked the assigned marks for each set of scripts.
  3. Final review. High-school teachers who taught the relevant subject reviewed the marked scripts.

These checks helped ensure consistent rubric application. The resulting item marks serve as the reference for checking agents' calculations and diagnoses.

AuthorsOpenReview ↗

Reviewer 2 (RvZn) | Reply 2 of 4

W1.2 What inputs do agents receive and what do they produce?

For each subject, all three generation tasks use the same exam evidence (Section 2 and Appendix J).

  • Questions and grading references. Exam questions, expected answers, and grading rubrics.
  • Student work. De-identified answers, item marks, and scoring rationales.
  • Performance summaries. Class- and question-level statistics, including rates and mark distributions.
  • Visual evidence. Images such as charts and diagrams associated with the exam and candidate work.

Each task prompt is shared across subjects, giving 15 subject–task instances. Agents deliver an editable class report (DOCX) that prioritizes what to reteach, a review slide deck (PPTX) that explains common mistakes and worked corrections, or one worksheet per student (DOCX) with individual feedback and matched practice. Appendix J provides the input specification and full generation prompts.

W1.3 How are generated materials scored?

Section 4 describes what each evaluator checks and gives the teacher-informed pass thresholds.

DimensionScoring procedurePass threshold
Task completionThree judges vote on checklist items, with a three-way split counting as incomplete. Complete, incomplete, and absent items receive full, half, and zero credit, respectively. The score is the percentage of weighted credit earned.≥95%
Layout qualityA visual judge checks rendered pages, reviews script flags, and looks for missed defects. A fixed rule converts confirmed defects into deductions.≥90/100
Factual correctnessVerifiers check text and visuals against the exam evidence, with a separate review pass. Deductions reflect affected blocks' shares of text length (75% weight) and block count (25%). Blocks with only unverifiable findings incur half the deduction.≥90/100
PersonalizationThree judges score target alignment, explanatory grounding, focus, and tailored actionability from 0 to 2. We average the judges' scores within each criterion, then sum the four means to obtain a score from 0 to 8.Mean ≥1.67/2 and each criterion ≥1.33/2

The revised main experiment in Section 5 uses three runs per model–subject–task combination. We average dimension and personalization-criterion scores over those runs before applying the cutoffs. An all-four pass requires meeting every cutoff.

For the overall score, we average each dimension equally across the five subjects within each task. Let CtC_t, LtL_t, FtF_t, and PtP_t denote these averages for completion, layout, factual correctness, and personalization. Equation 1 gives the task score StS_t and the average across the three tasks (Appendix E.3).

St=Ct100(0.30,Lt+0.40,Ft+0.30×12.5,Pt),Total=13∑tSt.S_t=\frac{C_t}{100}\left(0.30\\,L_t+0.40\\,F_t+0.30\times12.5\\,P_t\right),\qquad \mathrm{Total}=\frac{1}{3}\sum_{t}S_t.

The factor 12.5 rescales personalization from 0–8 to 0–100. Appendices F–I give the full scoring procedures, and the anonymous repository provides the evaluator prompts and checklists.

AuthorsOpenReview ↗

Reviewer 2 (RvZn) | Reply 3 of 4

W1.4 How was the automated evaluation validated against human judgments?

We expanded factual and personalization validation across all five subjects and all three formats using outputs from the ten-configuration exploratory study. Sections 4 and 5 now report the human comparisons.

  • Factual correctness. Human subject experts checked one randomly sampled evidence-linked region from each of 150 documents against the exam records. Sampling was independent of evaluator findings and scores. Experts agreed on 26 of 27 regions checked twice. The evaluator flagged an error in 20 of the 21 regions where experts found one, with detections in every subject and format (Appendix H.2.1). In a separate precision audit, humans confirmed 439 of 445 findings flagged by the evaluator in ten Biology slide decks (98.65%, Table 28).
  • Personalization. Five experts rated 66 complete documents against the assessment records, with 22 reports, 22 decks, and 22 worksheets. Each document received one review, with automated scores and generating configurations hidden. Human and automated totals aligned across documents (Spearman's ρ = 0.69). Automated ratings averaged 5.83/8 and human ratings averaged 5.00/8. Both gave their lowest ratings to tailored actionability, which captures whether the materials offer usable next steps (Appendix I.2.1).
  • Task completion. Human experts checked all 2,030 checklist items in 30 exploratory Biology documents across the three formats. No clearly absent or incomplete requirement received full credit (Appendix F).
  • Layout quality. Human experts annotated 150 pages from five complete exploratory slide decks, one per subject. With majority voting across three validation runs, the evaluator achieved 94.9% recall and 97.4% precision for pages needing correction. The benchmark runs the layout evaluator once per generated document (Appendix G.2).

Factual validation by subject. The subject rows from Table 24 show where the evaluator located an error in a region that human experts labeled as containing an error. Coverage is Found divided by Error (Appendix H.2.1).

Review groupRegionsCorrectErrorUnverifiableUnresolvedFoundMissedCoverage
Biology302450150100.0%
Chemistry302523020100.0%
English302820020100.0%
Mathematics302370070100.0%
Physics30235204180.0%
Total150123215120195.2%

Personalization by criterion. The selected columns from Table 35 report means on the 0–2 scale and Spearman’s ρ across the 66 documents (Appendix I.2.1).

CriterionHumanAutomatedρ
Target alignment1.411.560.75
Explanatory grounding1.411.480.67
Focus1.211.520.55
Tailored actionability0.971.270.54

Higher overall benchmark scores were associated with less revision judged necessary by human experts (Spearman’s ρ = 0.65). Mean scores were 60.8 for documents needing Major changes, 79.4 for those needing Small changes, and 88.1 for those rated Ready. Only two documents were rated Ready, so this sample provides limited evidence about how reliably the pass thresholds identify classroom-ready materials.

Human readiness judgments for all 66 reviews across the three tasks (Table 20, Appendix E.5). Ready means minor or no edits and counts as a human pass. Revision needed combines Small changes and Major changes. Each worksheet uses its batch’s benchmark score and decision.

Benchmark decisionReadySmall changesMajor changesRevision neededTotal reviewed
Pass12023
Fail118446263
Total220446466

The human reviews support the evaluator's ability to flag factual problems, distinguish more tailored materials, and identify documents that need substantial revision. In separate classroom use, we generated materials from other exam data and evaluated them against the same four thresholds. Teachers used the passing materials in 27 classes across 15 schools and reported that they needed minor or no edits (Section 4).

AuthorsOpenReview ↗

Reviewer 2 (RvZn) | Reply 4 of 4

Q2 What are the differences and connections to prior benchmarks?

We expanded Section 6 with the comparisons below; Appendix K provides educational foundations, further related work, and agent execution frameworks. Exam2Teach tests whether agents can turn the same exam evidence and human-verified student scores into editable class reports, review slides, and personalized worksheets across five subjects. Its four evaluation dimensions are task completion, layout quality, factual correctness, and personalization.

  • General-purpose and professional benchmarks. GAIA evaluates information gathering and tool use; OSWorld and OfficeBench assess desktop and cross-application task completion; and GDPval evaluates professional deliverables across occupations. Exam2Teach applies this practical-task perspective to one professional workflow: post-examination teaching preparation.

  • Document generation and editing benchmarks. MKA emphasizes integrating visual evidence across files to produce professional artifacts, including PPTX and DOCX. DocOps checks requested edits, native-file validity, and structural preservation. Exam2Teach shares the focus on usable editable files and inspects their rendered pages for clipping, overlaps, and illegible text.

  • Slide-generation benchmarks. SlidesBench evaluates slide design, while PresentBench uses instance-specific rubrics to assess source-grounded slide quality. We share PresentBench's emphasis on completeness, correctness, and fidelity to source material. For review decks, this includes checking that charts, numerical summaries, and explanations accurately represent marked student responses and score data.

  • Educational benchmarks. TeachArena evaluates evidence-based pedagogical judgment, multi-turn tutoring, and end-to-end teaching workflows. Exam2Teach examines the pedagogical fit of completed materials, from diagnoses of recurring mistakes and class-level reteaching priorities to individual feedback and practice. Following formative-assessment principles (Appendix K), feedback should explain observed errors and practice should target skills where students lost marks.

  • Educational applications. AutoEER generates exam reports, and Instructional Agents generates course materials, sharing our goal of supporting teachers' preparation work. Exam2Teach contributes a protocol for comparing complete model–harness configurations, with teacher-informed pass thresholds and human-validated evaluation methods.

Reviewer_vcDeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Exam2Teach, a benchmark evaluating whether autonomous LLM agents can transform examination results into reliable and personalized teaching materials. The benchmark covers five subjects and three tasks: class reports, review slides, and individualized student worksheets. The authors develop a configurable multi-agent framework supporting document generation and automated evaluation across four dimensions: task completion, layout quality, factual correctness, and personalization. Experiments involving 10 model-agent configurations, 150 runs, and 700 generated documents reveal substantial limitations in current LLM capabilities. Only 7.3% of runs satisfy all four evaluation thresholds, with factual correctness and slide generation emerging as particularly challenging areas. The findings suggest that current LLM agents remain unreliable for producing classroom-ready materials without substantial verification and correction.

Critical strengths

The paper addresses a practically important educational problem by connecting examination analysis with the generation of actionable teaching materials.

The benchmark covers multiple academic subjects, document formats, and evaluation dimensions, providing a reasonably comprehensive assessment of end-to-end agent capabilities.

Critical weaknesses

Weaknesses

Despite covering five subjects, the benchmark includes only one examination per subject and 60 student records assembled into five constructed classes, substantially limiting the diversity of educational contexts, assessment formats, and student populations.

Each primary configuration is evaluated only once per task-subject combination, while the principal ablation studies rely on individual Biology instances, making the reported rankings and performance differences potentially sensitive to stochastic variation.

Although human validation is provided, factual verification is primarily validated against one 30-slide reference deck, and personalization validation relies on a small selected corpus with judgments reviewed by one author, leaving uncertainty about reliability across subjects and formats.

The strict pass thresholds are motivated by teacher feedback but are not empirically calibrated against independently assessed classroom usability, making the headline 7.3% success rate sensitive to the chosen scoring criteria.

Questions for authors

How representative are the five examinations and constructed student cohorts of authentic classroom settings, and how might the conclusions change across different educational levels, curricula, and examination formats?

How were the human-assigned examination scores validated, and were multiple independent subject experts involved in marking or checking the student responses?

How sensitive are the principal findings, particularly the 7.3% overall pass rate and the relative model rankings, to alternative evaluation thresholds and scoring formulations?

Since GPT-5.5 participates extensively in the evaluation pipeline while also being one of the evaluated systems, what evidence rules out systematic evaluator preference or differential error detection across model families?

Given that some models use different agent harnesses, how can the authors separate improvements attributable to the underlying language models from those attributable to the execution environment, available tools, and context management?

To what extent do the generated materials provide instructional value beyond what could be achieved through deterministic examination analytics combined with structured document templates and simpler LLM-based explanations?

Can authors Include multiple independent examinations and authentic classroom cohorts per subject, repeat generation across several random seeds, and report confidence intervals or uncertainty estimates for model comparisons and pass rates.

Can authors compare autonomous agents against direct LLM prompting, deterministic statistical analysis with document templates, and hybrid pipelines that combine verified calculations with LLM-generated explanations to clarify the added value of agentic execution.

Evaluate a representative subset of models using common agent harnesses, identical tools, comparable computational budgets, and repeated trials to determine whether the reported performance differences primarily reflect model capabilities or system configurations.

7 comments
AuthorsOpenReview ↗

Reviewer 1 (vcDe) | Reply 1 of 7

Thank you for recognizing the practical importance of Exam2Teach and for asking how the results hold across runs, evaluators, and classrooms. We expanded the main experiment to 270 generation runs and the human validation to cover all five subjects and three document formats.

Q1.1 & Q7.1 How closely do the exams and cohorts reflect real classrooms?

Representativeness of the examination: The benchmark uses official examination papers and grading rubrics from a globally recognized pre-university public examination, using anonymized candidate scripts provided by the governing examination authority. Following the Area Chair’s guidance, the specific identity of this examination is withheld to maintain ICLR’s double-blind review process, but it will be disclosed when the paper is published. High-performing students in this public examination are consistently admitted into prestigious undergraduate programs worldwide, including the best universities across major global ranking systems.

We assembled each cohort from candidates who took the same exam. Their answers and marks give agents the kinds of evidence teachers use to decide which mistakes to revisit in class. Note that schools use extensively past papers of the same examination as mock exams for their upper-secondary school students. Therefore, the exams and cohorts evaluated in this paper very closely reflect real classroom practices.

Similar examinations such as Cambridge International AS & A Level includes all the five benchmark subjects, and schools in 138 countries offer the programme. The benchmark’s subject mix is therefore relevant to a widely used international upper-secondary curriculum. Cambridge subject list · Cambridge programme figures

Real classroom validation: Feedback from over 50 teachers across 15 schools shaped the benchmarked tasks around what teachers prepare after exams. For classroom use, we generated materials from exam data outside the benchmark and evaluated them against the same four quality thresholds. Teachers used materials that passed all four thresholds in 27 classes across 15 schools, spanning multiple grade levels, and reported that the materials needed minor or no edits. This supports using the benchmark’s thresholds as a practical quality target for materials generated from other exam data (Sections 1 and 4).

Q1.2 How might the results change at other educational levels or with different curricula and exam formats?

We started with upper-secondary school exams because demanding questions and detailed scoring rules make error diagnosis testable. On a multi-step mathematics problem, for example, the student’s written work and the rubric let us check whether an agent identifies where the student lost marks and suggests practice that addresses the error (Appendix D).

We compared the same six model–harness configurations on the same three tasks for each subject, averaging scores across three repetitions per combination. At most two of the 18 combinations in any subject passed all four criteria. English Language had the highest mean layout score and the lowest mean personalization score of the five subjects. Mathematics had the lowest mean factual correctness (Table 7, Appendix A.1.2).

SubjectCompletion (0–100)Layout (0–100)Factual correctness (0–100)Personalization (0–8)All four
Biology94.5488.4868.026.781/18 (5.6%)
Chemistry93.6586.6967.536.532/18 (11.1%)
English Language94.0489.1967.355.951/18 (5.6%)
Mathematics92.8087.6562.616.190/18 (0.0%)
Physics93.6888.1867.616.521/18 (5.6%)
All subjects93.7488.0466.636.405/90 (5.6%)

Some English Language reports overlooked one of the exam’s two reading routes and missed the review needs of students who took it (Appendices A.1.2 and I.4). A different assessment can expose a different weakness. An exam that records only final answers gives agents less evidence about why a student made an error. Different grade levels and curricula can also change which explanations and exercises are appropriate. These differences could shift which criteria limit pass rates and which configurations perform best (Appendix L).

AuthorsOpenReview ↗

Reviewer 1 (vcDe) | Reply 2 of 7

W2.1 Single runs in the main experiment

We added two runs for each of the six model–harness configurations on every subject–task pair. The revised benchmark uses three runs per combination, 270 runs in total, and averages the scores across repetitions. We include the main experiment’s per-repetition scores and rankings in the table under Q7.2 & Q7.3 below.

Q7.2 & Q7.3 Can you repeat generation across several random seeds and report confidence intervals or uncertainty estimates for model comparisons and pass rates?

With the published scoring rules held fixed, GPT-5.5 with Codex ranked first in all three repetitions. We evaluated the same 90 combinations in each repetition, with generation seed values unspecified in the run protocol. Table 6 gives overall scores out of 100 and ranks (Appendix A.1.1).

ModelRepetition 1Repetition 2Repetition 3
GPT-5.585.12 (1)86.57 (1)86.60 (1)
Claude Opus 582.28 (2)79.68 (2)79.09 (3)
Grok 4.580.57 (3)78.48 (3)81.19 (2)
GLM 5.274.75 (4)67.68 (4)67.18 (4)
Qwen 3.7 Max60.82 (5)61.65 (5)62.99 (5)
DeepSeek V4 Pro60.24 (6)53.17 (6)57.69 (6)

Score variation. Table 5(a) reports the mean, sample standard deviation, and observed range of the three repetition scores on the 0–100 scale (Appendix A.1.1).

ModelMean ± SDMin–max
Claude Opus 580.35 ± 1.6979.09–82.28
DeepSeek V4 Pro57.03 ± 3.5853.17–60.24
GLM 5.269.87 ± 4.2367.18–74.75
GPT-5.586.09 ± 0.8585.12–86.60
Grok 4.580.08 ± 1.4278.48–81.19
Qwen 3.7 Max61.82 ± 1.1060.82–62.99
All models72.54 ± 1.3871.20–73.96
  • Model comparisons. The lead over the runner-up ranged from 2.8 to 6.9 points. Ranks four through six also held. Claude Opus 5 and Grok 4.5 exchanged second and third place. Four of six configurations had overall-score SDs below two points.
  • Pass-rate variation. Applying the published thresholds to individual runs gave 11/90 (12.2%), 6/90 (6.7%), and 7/90 (7.8%) passing all four criteria in the three repetitions. Factual correctness had the fewest passes in each repetition (16/90, 7/90, and 11/90). The main benchmark rate of 5/90 (5.6%) applies the same thresholds to scores averaged across the three repetitions for each combination.

W2.2 Biology-only exploratory ablations

We used one Biology exam as a common test case so the three ablations could compare workflow choices against the same questions, student answers, and scoring rules. We chose Biology because we knew its exam and marking rubric best, which helped us check factual claims in the generated materials against the source evidence.

AuthorsOpenReview ↗

Reviewer 1 (vcDe) | Reply 3 of 7

W3 Human validation across subjects and formats

We expanded human validation of factual correctness and personalization to all five subjects and all three formats using outputs from the ten-configuration exploratory study.

  • Factual correctness. To complement the original 30-slide reference test, we added a human audit of 150 regions. We randomly sampled one evidence-linked region per configuration–subject–task combination, independently of the evaluator’s findings and scores. Human experts checked each region against the assessment records. The evaluator flagged an error in 20 of the 21 regions where humans found one (95.2%), with detections in every subject and format. Human experts agreed on 26 of the 27 regions reviewed twice (96.3%) (Appendix H).
  • Personalization. In the original diagnostic study, one human expert reviewed 72 individual claims from 18 selected documents. In the new review, five human experts rated 66 complete documents, with 22 class reports, 22 review slide decks, and 22 student worksheets. Each document received one review. Human experts used the assessment records, with the generating configuration and automated scores hidden. Documents rated higher by humans tended to receive higher automated scores (Spearman’s ρ = 0.69). Automated totals averaged 5.83/8 compared with 5.00/8 from humans. Both groups gave their lowest ratings to tailored actionability, which assesses whether the materials give the class or student practical next steps (0.97/2 human, 1.27/2 automated) (Appendix I).

The table shows the factual audit by format (Table 24, Appendix H.2.1). Coverage is the share of regions with a human-identified error where the evaluator flagged an error.

Review groupRegionsCorrectErrorUnverifiableUnresolvedFoundMissedCoverage
Class reports50371030100100.0%
Review slides50389218188.9%
Student worksheets504820020100.0%
Total150123215120195.2%

Manual precision audit. In a separate audit, we checked every factual error the evaluator flagged in ten Biology slide decks against the slides and exam records. We confirmed 439 of 445 findings (98.65% precision). The original row for the manually reviewed slides is below (Table 28, Appendix H.2.5).

SubjectReported itemsFalse positivesFP share
Biology (manual)44561.35%

Personalization by criterion. The selected columns from Table 35 compare human and automated means on the 0–2 scale and report Spearman’s ρ across the same 66 documents (Appendix I.2.1).

CriterionHumanAutomatedρ
Target alignment1.411.560.75
Explanatory grounding1.411.480.67
Focus1.211.520.55
Tailored actionability0.971.270.54

Our evaluation aligned with human judgments, with factual flags covering most regions where human experts found errors, personalization scores tracking their document ratings, and overall scores tending to be higher for materials they judged to need less revision (see the overall-score table in W4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.