acceptodds
Under review as a conference paper at ICLR 2027

HG-MAQG: Heterogeneous Graph Constrained Multi-Agent System For Personalized Question Generation

24 views

Abstract

High-quality educational questions are essential for diagnosing students’ knowledge states and facilitating personalized instruction, making question generation a crucial task in intelligent education systems and pedagogical agents. However, existing methods primarily emphasize linguistic fluency and template reproduction, limiting their ability to jointly support question novelty, learner-state adaptation, and cognitive progression. Recent multi-agent approaches designed to address these limitations lack a shared structured evidence base, resulting in inconsistent reasoning and error propagation across agents. To address these challenges, we propose HG-MAQG, a heterogeneous graph constrained multi-agent system for personalized question generation. HG-MAQG encodes learner states, concept prerequisites and question relations in a heterogeneous graph to ground agent reasoning in structured, shared and traceable evidence. Building on this representation, a diagnosis-driven cognitive progression mechanism uses concept-specific mastery estimates from knowledge tracing to adjust target Bloom levels, lowering cognitive demands to address mastery gaps, maintaining them to consolidate current-level competence, or raising them to introduce the next level of reasoning. The resulting cognitive targets and retrieved graph evidence jointly constrain question generation, while validation of concept coverage, cognitive alignment and novelty guides iterative refinement, with validated questions incorporated into the graph to provide additional references for subsequent generation. To evaluate cognitive alignment and learner-specific differentiation, we introduce Bloom Accuracy and Personalization Gap beyond conventional question generation metrics. Evaluation across six metrics on Eedi and XES3G5M demonstrates proposed HG-MAQG effectiveness in personalized question generation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Personalized Question Generation, Multi-Agent Systems, Heterogeneous Graph, Knowledge Tracing, Adaptive Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_n3R2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes HG-MAQG, a heterogeneous-graph-constrained multi-agent system for personalized question generation. Given a learner’s response history, the plug-and-play knowledge tracing model MIKT estimates concept-level mastery, and a fixed rule lowers, keeps, or raises the target Bloom level by one step. A heterogeneous graph stores concept prerequisites, question relations, and learner states. The Diagnostician, Composer, Validator, and Meta agents leverage this graph as shared evidence for generation, validation, and iterative refinement, and validated questions are then integrated into the graph. Experiments on Eedi and XES3G5M, using GPT-5.5 and DeepSeek-V4-Flash as backbone LLMs, report Readability, Solvability, BLEU, ROUGE-L, Bloom Accuracy, and Personalization Gap.

Critical strengths
  1. HG-MAQG formulates personalized question generation as a graph-constrained multi-agent system, which is different from previous work.
  2. A heterogeneous graph provides shared, structured evidence for question generation, including knowledge dependency graph, question graph and learner state graph. The graph construction approaches are reasonable.
  3. The experimental results show that HG-MAQG outperforms previous methods from distinct perspectives of QG.
Critical weaknesses
  1. The heterogeneous graph construction relies on high‑quality manual annotations; human annotators are required to assign Bloom’s taxonomy cognitive‑level labels to all questions in the dataset. In the paper, an annotation platform was constrcuted, with several students performing the annotation work. In real‑world deployment scenarios, large‑scale Bloom‑level annotation incurs extremely high costs. Without such annotations, the learner‑state graph cannot be constructed, which constrains practical deployment.

  2. The proposed method incurs high computational costs. The heterogeneous‑graph construction is time‑consuming; the multi‑agent system requires multiple large language model calls per question (for diagnosis, generation, validation, and iterative refinement), resulting in substantial inference overhead.

  3. HG-MAQG adds no new knowledge tracing model or collaboration mechanism beyond existing educational question generation methods. It combines plug-and-play knowledge tracing (MIKT), target Bloom levels, and retrieved graph evidence.

  4. The collaboration mechanism among four agents should be strengthened.

Questions for authors
  1. The fixed ±1 Bloom update and its mastery thresholds have no ablation or sensitivity analysis. The linear interpolation of unobserved Bloom levels is not validated.

  2. Section 4.3 takes HG-SAQG’s lower performance as confirmation of multi-agent collaboration, without stating whether HG-SAQG keeps the heterogeneous graph, the mastery rule, or the four agents. The conclusion claims that HG-MAQG outperforms the baselines across six metrics, but Table 1 shows otherwise (Vanilla outperforms it on Eedi ROUGE-L; Socratic outperforms it on XES3G5M solvability).

1 comment
AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer n3R2,

Thank you for your constructive feedback on our paper. With the ICLR public discussion phase ending shortly, we wanted to confirm whether our responses have sufficiently addressed your concerns. If there are any remaining issues, we would be happy to provide additional clarifications. We sincerely hope that you will consider reevaluating our manuscript.

Thank you very much for your time!

Reviewer_ndzKOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on question generation and argues that recent methods lack a shared, structured evidence base, resulting in inconsistent reasoning and error propagation across agents. The authors propose HG-MAQG, a heterogeneous graph-constrained multi-agent system. Experiments demonstrate its effectiveness.

Critical strengths
  • The task is clearly defined, and the paper is well structured.

  • Personalization is an important problem in education.

Critical weaknesses
  • The method uses a heterogeneous graph to ground agent reasoning in structured, shared, and traceable evidence. However, several design choices lack sufficient justification. How reliably does the graph represent learner states and support cognitive progression? Cognitive progression remains a challenging scientific problem, and it is unclear how the proposed graph addresses it.

  • The method maps mastery probabilities to Bloom levels and introduces Bloom Accuracy for evaluation. This raises concerns about potential circularity, as matching the method’s own targets does not establish their effectiveness.

  • PG accounts for target-concept coverage, it measures inter-student semantic diversity rather than alignment with individual learner states.

  • What theoretical or empirical evidence supports the proposed mapping from mastery probabilities to Bloom levels?

  • The provided code appears incomplete and lacks the core implementation needed to reproduce the results.

  • Minor: punctuation is missing around several equations

Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer ndzK,

Thank you for your constructive feedback on our paper. With the ICLR public discussion phase ending shortly, we wanted to confirm whether our responses have sufficiently addressed your concerns. If there are any remaining issues, we would be happy to provide additional clarifications. We sincerely hope that you will consider reevaluating our manuscript.

Thank you very much for your time!

Thanks for the response. However, explaining how learner states are estimated and represented in the heterogeneous graph does not establish the validity of those estimates or the cognitive modeling assumptions.

My concerns about the Bloom-based design also remain. The cited studies do not provide sufficiently strong empirical support for the specific Bloom-based mapping. Direct validation of its reliability and effectiveness is needed, including evidence showing why Bloom-level guidance works and justifying the parameter settings used in the paper. Therefore, I cannot support the acceptance.

Reviewer_AUxzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HG-MAQG, which organizes students' answer histories, knowledge concepts, item relations, and knowledge-tracing estimates into a heterogeneous graph. Based on each student's mastery of the target concept, the system adjusts the target Bloom level, after which diagnostic, generation, verification, and meta-control agents collaborate to produce, filter, and iteratively refine items. On Eedi and XES3G5M, the authors compare against 7 baselines using Readability, Solvability, BLEU, ROUGE-L, Bloom Accuracy, and Personalization Gap, and also provide ablations, case studies, and a 200-item human evaluation.

Critical strengths

1.The paper clearly separates concept selection, cognitive level, and item novelty, and tries to carry students' historical evidence through the diagnostic, generation, and verification stages; this is more actionable than simply asking an LLM to "generate personalized items."

2.The authors provide initial evidence at several levels. Two educational datasets, several baselines, component ablations, and a human evaluation give a starting point for further testing of the system; the gains in Bloom Accuracy and Personalization Gap in Table 1 are worth pursuing in detail.

Critical weaknesses

1.The metrics do not clearly establish that the generated questions match students’ mastery. Bloom Accuracy in Appendix C.2 measures whether an item matches the system’s chosen Bloom level, but does not check whether that level is appropriate for the student. Personalization Gap can likewise increase because students receive questions about different concepts, in different formats, or with different wording. The per-item concept-coverage filter does not ensure that the compared item sets cover the same concepts. Table 1 therefore shows alignment with chosen targets and differences between item sets, but does not directly demonstrate diagnosis-driven cognitive progression.

2.The graph ablation and human evaluation leave the main contribution difficult to assess. Table 1 shows a sharp drop in Bloom Accuracy without HG, but it is unclear whether this variant retains the same knowledge-tracing state, Bloom targets, and supporting item evidence. If these inputs also change, the comparison does not isolate the graph’s contribution. Section 3.1 and Appendix C.4 also need to explain how MIKT’s concept-level outputs are mapped and calibrated to Bloom-level mastery. Finally, the 200-item human study in Appendix D.4 assesses agreement between automatic and human scores, without reporting per-method results or uncertainty. This supports evaluator validation, but does not substantiate the comparative effectiveness claimed in contribution (3) and the Conclusion.

Questions for authors

1.For students on the same target concept but with different known mastery states, can teachers who do not know the method names judge whether the generated items fit each student's current level? Please report per-method grouped results with error intervals. This test could directly change my judgment on the personalization claim.

2.How do MIKT's outputs form a "student × concept × Bloom level" mastery estimate? Can the graph relations and the multi-agent collaboration each be removed separately under the same mastery estimates, Bloom targets, retrieved materials, and generation budget, so as to isolate each component's contribution?

2 comments
AuthorsOpenReview ↗

Response to Reviewer AUxz’s Updated Weaknesses on November 15, 2026

Dear Reviewer AUxz

Thank you for your reply on November 15 through revisions to the weaknesses in your original comment.

Response for W1

  • For the concern of Bloom Accuracy, students' Bloom levels for different concepts are derived from their responses to questions associated with the corresponding concepts, while their mastery of each question is estimated by the knowledge tracing model MIKT. As you mentioned in the first point of Question 2, we have already provided a detailed response in Responses to Question 2, item 1, of our November 10 AUxz 2/3 reply.

  • For the concern of Personalization Gap, different students master different concepts, rather than all students receiving the same set of questions generated from the entire concept set, which supports our motivation to address "one-size-fits-all massed practice." Moreover, we explicitly state on page 20, lines 1070–1074, that to avoid confusing superficial semantic diversity with personalization, we calculate the personalization gap only for questions that satisfy the target concept coverage constraint.

Therefore, HG-MAQG generates personalized questions with different Bloom levels for different concepts according to students' concept mastery states, improving both Bloom Accuracy and Personalization Gap. Furthermore, the teacher-blind evaluation presented in our response to your Question 1 further confirms that the generated questions align with students' cognitive progression.

Response for W2

  • For the concern that the graph ablation and human evaluation leave the main contribution difficult to assess, we have already explicitly stated in the first line of Details of the Evaluation Setup in our AUxz 2/3 reply that "all variants follow the same pipeline and configuration."

  • For the concern about further explaining "how MIKT’s concept-level outputs are mapped and calibrated to Bloom-level mastery," we have incorporated the content from our November 10 AUxz 2/3 reply into Appendix C.4 of the paper.

  • For the concern about the human study in Appendix D.4, we explicitly clarify in Appendix D.4 that the human evaluation aims to assess whether the automated metrics for Readability, Solvability, Bloom-level Alignment, and Personalization align with human evaluations, thereby validating the effectiveness of these evaluation metrics, rather than evaluating and comparing different methods, as suggested by your concern about "without reporting per-method results or uncertainty."

We hope that these responses have adequately addressed your remaining concerns and sincerely hope that you will consider reevaluating our manuscript.

Thank you for the detailed response. I’ll keep my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.