- The independent contribution of the multi-agent architecture is not established, and its implementation is underspecified
Sections 4.3–4.4 give a teacher-inspired rationale for six roles but do not establish an advantage over simpler organizations. The main experiments compare only the proposed agent system with direct generation by the same backbones. They omit existing agent-based systems for the same educational or mathematical question-generation task, leaving its advantage over existing agents unestablished.
The full system also adds retrieved evidence, planning, assessment, revision opportunities, and additional model calls. Its gains therefore cannot be attributed specifically to multi-agent organization. The ablations in Section 5.5 show the effects of removing retrieval or refinement within the proposed system, but do not test whether multiple agents outperform a single agent given the same evidence and revision opportunities. Comparisons with existing same-task agents and with a single-agent iterative baseline under comparable resource budgets, together with role-merging or role-removal ablations, are needed to identify the architectural contribution. Calls, tokens, or latency should be reported alongside quality.
The description is also insufficient to reproduce all agents. Appendix B provides a generation-and-revision example, but not a complete specification of every role. Model assignments, full prompts, input/output structures, state updates, quality gates, and conditional routing rules are needed. These omissions limit both reproducibility and the interpretation of the claimed architectural contribution.
- The benchmark’s validity and aggregation require stronger support
First, Discrimination Power in Appendix Table 12 concerns whether an item separates students of different proficiency levels, whereas the reported score is assigned by an LLM from the item and a rubric. This measures a judgment about potential discrimination, not observed discrimination in a target student population. Agreement with teacher ratings supports alignment with expert judgments, but does not replace validation using student responses. The authors should establish the relationship between the score and empirical item discrimination, or explicitly limit the interpretation to predicted or perceived discrimination.
Second, some indicators overlap in definition. Linguistic Complexity and Readability both concern clarity and suitability for the target learners (Appendix Tables 7 and 18); Knowledge Point Alignment and Instructional Intent Alignment partly cover the same instructional requirements (Tables 4 and 19). If several indicators measure the same underlying property, averaging them may give that property unintended extra weight. Equal weighting of the five dimensions does not eliminate overlap across dimensions. This is a risk to investigate, not proof that bias has occurred, and correlation alone does not establish redundancy. Item-level correlations and sensitivity analyses after removing, merging, or reweighting overlapping indicators would clarify the stability of the rankings and gains.
Third, Equation (2) specifies equal weighting of the five dimensions. On the 0–5 scale used in the tables, this gives:
S_5 = (A + D + N + E + H)/5
However, the Qwen3-Max Base row in Table 3 gives:
(4.92 + 3.91 + 1.99 + 3.60 + 4.56)/5 = 3.796 ≠ 3.89
The difference cannot be explained by rounding to two decimal places. The reported overall scores are instead broadly consistent with averaging all 19 indicators directly:
S_19 = (4A + 4D + 3N + 4E + 4H)/19
These rules assign different weights: Novelty receives 20% under equal dimension weighting but approximately 15.79% under equal indicator weighting. The implemented rule, equations, and all affected tables must be reconciled, and any changes to rankings or improvement magnitudes should be reported. Per-item scores and aggregation code would make this check possible. These validity and consistency issues directly affect what the benchmark scores mean and how much confidence can be placed in the main results.