- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes HG-MAQG, a heterogeneous-graph-constrained multi-agent system for personalized question generation. Given a learner’s response history, the plug-and-play knowledge tracing model MIKT estimates concept-level mastery, and a fixed rule lowers, keeps, or raises the target Bloom level by one step. A heterogeneous graph stores concept prerequisites, question relations, and learner states. The Diagnostician, Composer, Validator, and Meta agents leverage this graph as shared evidence for generation, validation, and iterative refinement, and validated questions are then integrated into the graph. Experiments on Eedi and XES3G5M, using GPT-5.5 and DeepSeek-V4-Flash as backbone LLMs, report Readability, Solvability, BLEU, ROUGE-L, Bloom Accuracy, and Personalization Gap.
Critical strengths
- HG-MAQG formulates personalized question generation as a graph-constrained multi-agent system, which is different from previous work.
- A heterogeneous graph provides shared, structured evidence for question generation, including knowledge dependency graph, question graph and learner state graph. The graph construction approaches are reasonable.
- The experimental results show that HG-MAQG outperforms previous methods from distinct perspectives of QG.
Critical weaknesses
-
The heterogeneous graph construction relies on high‑quality manual annotations; human annotators are required to assign Bloom’s taxonomy cognitive‑level labels to all questions in the dataset. In the paper, an annotation platform was constrcuted, with several students performing the annotation work. In real‑world deployment scenarios, large‑scale Bloom‑level annotation incurs extremely high costs. Without such annotations, the learner‑state graph cannot be constructed, which constrains practical deployment.
-
The proposed method incurs high computational costs. The heterogeneous‑graph construction is time‑consuming; the multi‑agent system requires multiple large language model calls per question (for diagnosis, generation, validation, and iterative refinement), resulting in substantial inference overhead.
-
HG-MAQG adds no new knowledge tracing model or collaboration mechanism beyond existing educational question generation methods. It combines plug-and-play knowledge tracing (MIKT), target Bloom levels, and retrieved graph evidence.
-
The collaboration mechanism among four agents should be strengthened.
Questions for authors
-
The fixed ±1 Bloom update and its mastery thresholds have no ablation or sensitivity analysis. The linear interpolation of unobserved Bloom levels is not validated.
-
Section 4.3 takes HG-SAQG’s lower performance as confirmation of multi-agent collaboration, without stating whether HG-SAQG keeps the heterogeneous graph, the mastery rule, or the four agents. The conclusion claims that HG-MAQG outperforms the baselines across six metrics, but Table 1 shows otherwise (Vanilla outperforms it on Eedi ROUGE-L; Socratic outperforms it on XES3G5M solvability).
Gentle Reminder
Dear Reviewer n3R2,
Thank you for your constructive feedback on our paper. With the ICLR public discussion phase ending shortly, we wanted to confirm whether our responses have sufficiently addressed your concerns. If there are any remaining issues, we would be happy to provide additional clarifications. We sincerely hope that you will consider reevaluating our manuscript.
Thank you very much for your time!