acceptodds
Under review as a conference paper at ICLR 2027

Beyond Problem-Solving: A Multi-Agent Framework and Pedagogically-Aligned Benchmark for Evaluating LLMs’ Educational Question Generation

Abstract

Large Language Models (LLMs) have achieved impressive performance in educational applications, particularly in solving mathematical problems. However, existing evaluation benchmarks overwhelmingly emphasize solution correctness while neglecting the equally important capability of generating high-quality educational questions. This results in a fundamental gap in assessing LLMs’ pedagogical competence and prevents a complete teaching–learning evaluation loop. We address this gap by introducing a multi-agent mathematics Question Generation (QG) framework that explicitly decomposes teacher-like behaviors—such as intent interpretation, curriculum planning, difficulty calibration, and solution verification—into collaborative LLM agents, and the first pedagogically aligned, multi-dimensional benchmark specifically designed to evaluate question generation quality. The benchmark assesses generated questions along five major dimensions—Accuracy, Difficulty, Novelty, Educational Value, and Human Preference Alignment—covering 19 fine-grained indicators grounded in educational psychology and assessment theory. Experimental results show that our multi-agent framework consistently outperforms strong LLM baselines across all five dimensions, with particularly large gains in higher-order pedagogical metrics such as diagnostic value, reasoning depth, and instructional clarity. Overall, this work takes an important step toward systematic and pedagogically grounded evaluation of LLM-driven educational content generation.

Primary area:
datasets and benchmarks
Keywords:
educational question generation, multi-agent LLM framework, pedagogical evaluation benchmark, large language models for education, mathematics question generation, LLM-as-a-judge evaluation, AI for education

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_KEHhOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents a dataset intended to evaluate models' ability to generate math questions. They also describe what appears to be a standard LangChain framework with shared state for question generation. Overall, the utility of the agent harness is unclear, and the benchmark is not sufficiently deep (only 100 questions have human review) to be practically useful.

Critical strengths

Nothing to report.

Critical weaknesses

This is a straightforward agentic question-generation approach that provides no new insight into the mechanism or quality of question generation. The benchmark is weak and does not represent the standards of comparable AQG benchmarks in the literature (some of which are missing in the paper).

Further, the so-called improvements are all within noise.

Questions for authors

Did you try with frontier models? In this reviewer's experience, modern frontier models perform well out of the box on these topics.

Reviewer_LqUeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an agentic framework for mathematical question generation. The agents correspond to actions a teacher would take while creating questions. The questions are evaluated with a judge on 5 general rubrics, which include a total of 19 fine-grained rubrics. They propose a benchmark with 1000 problems called PedQG-Bench which they use to evaluate several closed and open source models on the proposed rubric. However, the baselines are weak - the same LLM but in a zero-shot setting. A sample of 100 questions was evaluated by humans to validate the judge. They also present an ablation study where they test the pipeline without the RAG component, or without the iterative reflection and refinement part. I think the overall architecture is sound, but it isn't really new.

Critical strengths
  • I think the most useful contribution of the paper is evaluation rubric, which seems to cover all aspects of a question. This could be a base for future experiments if the dataset is made public, but not many details are included regarding the dataset.
Critical weaknesses
  • The biggest issue with the article is the lack of details, some in critical parts. No details about the dataset used are provided, besides a number (1000) of entries in a table. The prompts for each rubric and agent are very important for replication. There are no details about the knowledge base used for RAG.
  • The proposed architecture is compared only with week baselines - the same models in zero-shot. Nevertheless, the improvements are small, well bellow the provided confidence intervals. How these intervals are computed, is not specified in any place. Furthermore, there are inconsistencies in the main table. The average column is not the average of the other 5 columns. This questions the credibility of the study.
Questions for authors
  • Section 3.1 defines Difficulty as whether the challenge matches the intended level, and Figure 2 calls it “Difficulty Fitness”. Yet the results interpret higher Difficulty scores as better (“encourages more challenging … questions”). Figure 3b shows total scores rising monotonically from Easy to Olympiad, which suggests the judge rewards harder items per se rather than fitness to the requested level.
  • A compute/price / number of tokens report should be given, in order to observe how higher costs contribute to better results.
  • The Quality Assessor and Reflection agents use an LLM judge and a novelty gate (similarity threshold 0.3) during generation. The final evaluation is also LLM-judged on overlapping criteria, including Novelty, where the largest gains appear. The paper does not state whether the generation-time assessor sees the evaluation rubric or rubric-like criteria. If it does, part of the improvement may reflect rubric fitting rather than better items.
Reviewer_zL7nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a multi-agent framework for educational mathematics question generation and a benchmark with 19 indicators across Accuracy, Difficulty Alignment, Novelty, Educational Value, and Human Preference Alignment. The framework combines intent analysis, curriculum planning, retrieval, item design, quality assessment, and reflective revision. Experiments compare direct generation with the full framework across nine backbone models and report higher average scores for eight models, with one tie at the reported precision. Evaluation uses Seed2.0 Pro, complemented by component ablations and a small teacher-rating study.

Critical strengths
  • Evaluating the ability to design instructional questions addresses a relevant capability beyond answering existing problems. The five dimensions and detailed indicator-level rubrics make the intended evaluation criteria inspectable and encourage attention to instructional intent and pedagogical utility. This is a useful starting point for systematic evaluation, provided that the measures are validated.
  • The evaluation covers nine backbone models and includes retrieval and refinement ablations, together with a teacher-rating comparison. These experiments provide initial evidence that the full workflow can improve the reported rubric scores relative to direct generation and give some insight into the roles of retrieval and revision.
Critical weaknesses
  • The independent contribution of the multi-agent architecture is not established, and its implementation is underspecified

Sections 4.3–4.4 give a teacher-inspired rationale for six roles but do not establish an advantage over simpler organizations. The main experiments compare only the proposed agent system with direct generation by the same backbones. They omit existing agent-based systems for the same educational or mathematical question-generation task, leaving its advantage over existing agents unestablished.

The full system also adds retrieved evidence, planning, assessment, revision opportunities, and additional model calls. Its gains therefore cannot be attributed specifically to multi-agent organization. The ablations in Section 5.5 show the effects of removing retrieval or refinement within the proposed system, but do not test whether multiple agents outperform a single agent given the same evidence and revision opportunities. Comparisons with existing same-task agents and with a single-agent iterative baseline under comparable resource budgets, together with role-merging or role-removal ablations, are needed to identify the architectural contribution. Calls, tokens, or latency should be reported alongside quality.

The description is also insufficient to reproduce all agents. Appendix B provides a generation-and-revision example, but not a complete specification of every role. Model assignments, full prompts, input/output structures, state updates, quality gates, and conditional routing rules are needed. These omissions limit both reproducibility and the interpretation of the claimed architectural contribution.

  • The benchmark’s validity and aggregation require stronger support

First, Discrimination Power in Appendix Table 12 concerns whether an item separates students of different proficiency levels, whereas the reported score is assigned by an LLM from the item and a rubric. This measures a judgment about potential discrimination, not observed discrimination in a target student population. Agreement with teacher ratings supports alignment with expert judgments, but does not replace validation using student responses. The authors should establish the relationship between the score and empirical item discrimination, or explicitly limit the interpretation to predicted or perceived discrimination.

Second, some indicators overlap in definition. Linguistic Complexity and Readability both concern clarity and suitability for the target learners (Appendix Tables 7 and 18); Knowledge Point Alignment and Instructional Intent Alignment partly cover the same instructional requirements (Tables 4 and 19). If several indicators measure the same underlying property, averaging them may give that property unintended extra weight. Equal weighting of the five dimensions does not eliminate overlap across dimensions. This is a risk to investigate, not proof that bias has occurred, and correlation alone does not establish redundancy. Item-level correlations and sensitivity analyses after removing, merging, or reweighting overlapping indicators would clarify the stability of the rankings and gains.

Third, Equation (2) specifies equal weighting of the five dimensions. On the 0–5 scale used in the tables, this gives: S_5 = (A + D + N + E + H)/5

However, the Qwen3-Max Base row in Table 3 gives: (4.92 + 3.91 + 1.99 + 3.60 + 4.56)/5 = 3.796 ≠ 3.89

The difference cannot be explained by rounding to two decimal places. The reported overall scores are instead broadly consistent with averaging all 19 indicators directly: S_19 = (4A + 4D + 3N + 4E + 4H)/19

These rules assign different weights: Novelty receives 20% under equal dimension weighting but approximately 15.79% under equal indicator weighting. The implemented rule, equations, and all affected tables must be reconciled, and any changes to rankings or improvement magnitudes should be reported. Per-item scores and aggregation code would make this check possible. These validity and consistency issues directly affect what the benchmark scores mean and how much confidence can be placed in the main results.

Questions for authors
  • What evidence separates the benefit of the six-role organization from the additional retrieval and inference budget? Are comparisons with existing same-task agents or a single-agent iterative baseline available under comparable conditions? Please also specify the per-agent configuration, prompts, interfaces, and routing rules.
  • Is Discrimination Power based solely on rubric judgments, or are there student-response data validating its relationship with empirical discrimination? If such data are unavailable, can the claims be restricted to predicted or perceived discrimination?
  • Which aggregation rule was actually implemented? Please reconcile Equation (2), Table 3, and Appendix Table 20 and report whether corrected scores change model rankings or Base/Agent gains.
  • Are rankings and gains stable when potentially overlapping indicators are removed, merged, or reweighted? Item-level scores or the corresponding correlation and sensitivity analyses would materially affect my assessment of the benchmark.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.