- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces a benchmark to evaluate an LLM's ability to generate learning materials from exam results in five core subjects, along with a multi-agent framework to assess task completion, layout quality, factual correctness, and personalization, enabling teachers to tailor materials to students’ performance. Evaluations of 10 agent configurations across 150 runs show mixed results; only 7.3% of configurations passed all evaluation metrics. The low benchmarking scores establish a visible gap in the domain.
Critical strengths
The problem discussed is important in the AIED domain and addresses a key challenge in adaptive learning: teachers struggle to create learning materials based on exam results. The approach covers five core subjects (Biology, English, Mathematics, Physics, and Chemistry). The approach is being evaluated on open-source and closed-source models. It shows that even SOTA models such as GPT-5.5 and Opus 5 have 40% and 26.7% pass rates, highlighting a clear gap in the domain. Each subject includes several questions and student-written responses, which then pass through the agent harness to generate three documents (report, slides, and worksheets) based on the teacher's needs. The benchmarking data comes from real university entrance examinations and is labeled by human experts using the set grading rubrics.
Critical weaknesses
The paper's core contribution is the multi-agent framework, yet the main text does not explain it clearly; Figure 4 reads more like a complex engineering effort, but none of the components show novelty, literature grounding, or logical/empirical evidence. The paper attempts to address a core challenge in education with a multi-agentic harness but fails to establish either theoretical or empirical evidence for the proposed framework; the evaluated dataset was small, language-specific, and lacked human expert evaluation of the generated reports. The paper cannot be accepted in its current form, as it appears to attempt to solve a big problem with a complicated agentic framework but fails to establish any ground for its soundness, practicality, or generalizability. The figures are verbose, and the captions lack the context needed for understanding, along with other writing-related issues. Honest advice: narrow the scope; focus on one subject, or even a few core topics; identify core issues in student results; hire a subject expert to devise the evaluation framework based on subject-related pedagogies; and then create documentation. Later, devise a framework; it could be any agentic framework, not necessarily complicated, but simple, theoretically sound, and empirically supported. Report human-AI agreement on the generated documents, then publish that dataset for the research community for further benchmarking. That would be more valuable and useful than a purely synthetic, complex agentic framework with no educational/pedagogical/literature/theoretical grounding.
It's always appreciated to have the code available for review and to help the research community grow. However, the released code does not match the evaluation described in the paper. The factual-correctness score blends a 25% block-count term into Eq. 5, and the personalization score used in Eq. 1 is computed from issue-severity penalties rather than the 0-2 judge scores the paper describes, so the reported numbers cannot be derived from the stated formulas. The pass thresholds and cross-task total are not implemented at all. Several components the paper says are released are missing: the exam images agents received, the failure-pattern classification pipelines, and the personalization claim-consolidation procedure. Together with unpinned harness versions, this prevents rerunning the main experiment as described. The authors should correct the formulas in the text, release or clearly scope the missing components, and add the threshold and aggregation code.
Questions for authors
What is the range of N in the multi-agent framework? What factors determine it?
How does subject specialization embed into the agentic harness?
Where is the cost analysis of this evaluation?
What does subject-wise variation look like?
Where do the core evaluation metrics come from? Task completion, layer quality, factual correctness, and personalization vary across domains; how does the system define and adapt criteria for each subject?
3 comments
Reviewer 3 (uoVo) | Reply 2 of 3
W3 Human evaluation
We expanded human review of generated materials across all five subjects and three document formats. Higher overall scores correspond to less human-assessed revision effort (ρ = 0.65), and all 44 reviewed documents needing major changes fail the benchmark.
- Personalization: five human experts reviewed 66 complete documents—22 reports, 22 slide decks, and 22 worksheets—across all five subjects and ten configurations. Each document received one review using assessment records, with model labels and automated scores hidden. Human and automated total scores correlate at Spearman's ρ = 0.69; automated total scores average 0.83 points higher on the 0–8 scale.
- Factual correctness: human subject experts reviewed 150 regions across all five subjects and three formats, sampled independently of evaluator findings and scores. The evaluator flagged an error in 20 of the 21 regions where humans found an error (95.2% region coverage).
- Completion: review of all 2,030 checklist items in 30 Biology documents found no clearly absent or incomplete requirement receiving full credit.
- Layout: comparison with final human labels for 150 pages from five slide decks yields 94.9% recall and 97.4% precision.
Appendices E–I give the detailed human-validation procedures and results.
Higher overall benchmark scores were associated with less revision judged necessary by human reviewers (Spearman’s ρ = 0.65). Mean scores were 60.8 for documents needing Major changes, 79.4 for those needing Small changes, and 88.1 for those rated Ready. Only two documents were rated Ready, so this sample provides limited evidence about how reliably the pass thresholds identify classroom-ready materials.
Benchmark and human pass decisions. The table pools all three tasks from Table 20 (Appendix E.5). Ready counts as a human pass, and Revision needed combines Small changes and Major changes. Each worksheet uses its batch’s benchmark score and decision.
| Benchmark decision | Ready | Small changes | Major changes | Revision needed | Total reviewed |
|---|---|---|---|---|---|
| Pass | 1 | 2 | 0 | 2 | 3 |
| Fail | 1 | 18 | 44 | 62 | 63 |
| Total | 2 | 20 | 44 | 64 | 66 |
Real classroom validation: Teachers used materials meeting the four thresholds in 27 classes across 15 schools and recorded minor or no required edits. The classroom evidence concerns document usability and the edits teachers required before use.
W4 Scoring and reproducibility
Thank you for checking the released code against the paper. We rechecked the experiment results, corrected the factual-scoring description in the paper, and fixed the released personalization code. We also uploaded the pinned harness versions and the missing components needed for reproduction.
- Factual correctness. Section 4 and Appendix H now document both the 75% text-length and 25% block-count terms. Erroneous blocks receive full deductions, and unverifiable-only blocks receive half deductions.
- Personalization. The corrected code averages each direct 0–2 criterion score across three valid judges, then sums the four means to produce a 0–8 total. We have double checked and confirmed that the error is only in the released code and the experiment results in the paper are all correct.
- Pass decisions and aggregation. The released scripts now implement the four thresholds, averaging across repetitions, task scores, and the cross-task total.
- Other components. We have uploaded the missing components, including the pinned harness versions, the failure-pattern classification pipelines, and the personalization claim-consolidation procedure.
- Exam images. Following the Area Chair's guidance, the images cannot be released now in order to comply with the double-blind submission policy. The complete dataset will be disclosed publicly after the double-blind period.
W5 Figures and captions
We revised the captions to add more detailed explanations, including the framework execution (Figure 4) and the evaluation pipeline (Figure 5). Sections 3 and 4 provide the accompanying detailed descriptions.
Q2 Subject adaptation
Agents adapt to each subject through its questions, rubrics, answers, and grading records. We evaluate existing model–harness configurations in Codex, Claude Code, and OpenCode with shared skills and task prompts across subjects (Appendix J).
Reviewer 3 (uoVo) | Reply 3 of 3
Q3 Cost analysis
Generation and evaluation together cost an estimated US$49.74 per run in the main benchmark, with evaluation accounting for 87.4%. The averages below cover six model configurations, five subjects, and three repetitions. We remark that the cost is for benchmarking the performance of different agents, not the cost of a solution we propose as this paper focuses on agent performance benchmarking instead of a solution to more efficiently solving the benchmarked tasks.
| Task | Generation (USD/run) | Evaluation (USD/run) |
|---|---|---|
| Class report | 2.68 | 12.58 |
| Review slides | 12.47 | 34.23 |
| Worksheet batch | 3.66 | 83.61 |
| Across all tasks | 6.27 | 43.48 |
Generation counts only the producing model, excluding generation helpers. Evaluation includes preparation and all four evaluation dimensions. Each worksheet run covers the complete student batch. Appendix C provides token usage, unit prices, cost breakdowns, and accounting assumptions and exclusions.
Q4 Subject-wise variation
Among the five selected assessments, Mathematics has the lowest mean factual correctness score, and English Language has the lowest mean personalization score. We have added the subject-wise results in Table 7 (Appendix A.1.2).
The table reports mean dimension scores and all-four pass counts in main benchmark results. Personalization uses a 0–8 scale; the other scores use 0–100.
| Subject | Task completion | Layout quality | Factual correctness | Personalization | All-four passes |
|---|---|---|---|---|---|
| Biology | 94.54 | 88.48 | 68.02 | 6.78 | 1/18 |
| Chemistry | 93.65 | 86.69 | 67.53 | 6.53 | 2/18 |
| English Language | 94.04 | 89.19 | 67.35 | 5.95 | 1/18 |
| Mathematics | 92.80 | 87.65 | 62.61 | 6.19 | 0/18 |
| Physics | 93.68 | 88.18 | 67.61 | 6.52 | 1/18 |
In addition, examples from the exploratory experiments illustrate challenges in these tasks. In the English Language assessment, B1 and B2 are examination routes with different questions. Some class reports overlook the needs of learners on one route or apply the same diagnostic assumptions to both groups (Appendix I.4). Mathematics outputs include incorrect numerical answers and mutually inconsistent problem conditions (Appendix H.4).
Q5 Metrics
Metrics choice. The four dimensions were informed jointly by feedback from collaborating teachers and failures observed during benchmark development (Appendix E.1). In our first trials with frontier models, the most visible problems were overlapping or clipped slide text and fluent claims unsupported by the exam records. As we expanded testing to more models, we found omissions, such as decks providing required statistics for only some questions. We also encountered worksheets that placed a student’s answer beside the correct answer and added a generic tip without explaining the actual error. Together with teachers’ expectations for usable materials, these observations motivated evaluating task completion, layout quality, factual correctness, and personalization separately.
Across subjects, we retain common quality standards while adapting the evaluation to the task requirements, subject evidence, document format, and intended learners. Teacher feedback supports broadly shared requirements for readable layouts and guidance aligned with learners’ needs. Specifically:
- Task completion: Each task has a checklist derived from its requirements. Reports, slides, and worksheets therefore use different checklists, while the same checklist applies across all five subjects within each task (Appendix F).
- Layout quality: The defect categories and scoring rules are shared across subjects, while the automated checks vary by document format. One script checks fixed-canvas PPTX slides; another checks paginated DOCX reports and worksheets. Both support visual inspection of rendered pages, with only confirmed defects contributing to the benchmark score (Appendix G).
- Factual correctness: Exam-related textual and visual claims are checked against the corresponding subject’s input data, including questions, marking rubrics, student answers, and scores. Subject-specific content is therefore assessed against its own reference evidence (Appendix H).
- Personalization: The same rubric applies across subjects, but the evaluation target differs by task. Reports and slides are assessed against class-level performance records to determine whether teaching priorities and proposed actions address the class’s needs. Worksheets are assessed against the intended student’s answers, marks, and marking rationales to determine whether explanations, feedback, and practice address that individual’s difficulties (Appendix I).
Reviewer 3 (uoVo) | Reply 1 of 3
We would like to clarify that our contribution lies in benchmarking agents' ability to turn real examination evidence into class reports, review slides, and personalized worksheets. The evaluation jointly assesses the generated materials' completeness, layout, factual reliability, and fit to learners' needs. Section 6 explains how this focus complements existing benchmarks.
The multi-agent framework provides the execution infrastructure for experiments in Exam2Teach and for other researchers and practitioners to reproduce our results and use the benchmark to evaluate their own agents, while the multi-agent framework can also be generalized to other custom workflows beyond this benchmark to evaluate other agent capabilities (Section 3). Users define agents and dependencies in YAML. The framework orchestrates those workflows, supports revision requests, and records model calls. Appendix K connects the design to prior work on coordination, tool use, and feedback-based revision. Therefore, the multi-agent framework serves the benchmark and is a part of the benchmark, but we do not claim it as a standalone contribution separated from the benchmark.
The number of agents, N, depends on the configured workflow and its inputs. Completion and personalization each use three judges. Factual verification and review expand with text and visual batches, and worksheet evaluation also scales with the student count.
Ablations on components support the necessity of different components in the evaluation workflow design.
For factual correctness, on a 30-page deck with 41 annotated factual errors, mean recall rises from 71.5% for direct GPT-5.5 verification to 88.6% with block batching and 95.1% with visual verification and independent review (Appendix H).
The relevant rows from Tables 26 and 27 show how the changes build on one another (Appendix H).
For personalization, the three-judge agentic panel's pooled findings cover 31/33 required issues, versus 27/33 for the API panel using the same criterion-by-criterion prompt. The panels report two and four invalid criticisms, respectively. This comparison uses human-reviewed claims drawn from judge reports on 18 diagnostic documents (Appendix I).
The table gives the two per-criterion rows from Table 37 (Appendix I.2).
Dataset size. We would like to clarify the scale of the evaluation. Each configuration is evaluated on 15 subject–task instances that produce 70 documents: 5 class reports, 5 review decks, and 60 personalized worksheets. Task completion alone checks these against 3,160 checklist items per run. Factual correctness checks each claim in the documents against 3,450 rubric-marked responses to 304 scored items, drawn from 120 real examination questions and 60 candidates (Appendix D). The main experiment comprises 270 generation runs, and across it and the exploratory study we evaluated 1,540 generated documents (Appendices A.1 and A.2).
Language. The source materials are in two languages. Four subjects use English questions and answers, and Biology uses non-English questions and answers that agents must explain in English.
Scope. We kept five subjects because they place different demands on agents, including mathematical derivations, scientific diagrams, and language assessment, and the results differ across them. Mathematics has the lowest mean factual score, and English Language has the lowest mean personalization score (Q4).