- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper describes the process for finetuning 4B, 9B, and 27B Qwen family models for education-related tasks, including curriculum grounding, K-12 problem solving, and pedagogical tutoring.
Critical strengths
Open models are usually a helpful resource for the community, and the authors show that their model outperforms other open models in this domain on relevant benchmarks (e.g. Table 1, Table 2).
Critical weaknesses
The data curation and filtering process could have been more strongly justified by some experiments motivating the choices being made or illustrating the choices’ empirical benefits. Some parts of the process are pretty sensible on a surface-level, e.g. sampling for diversity. However, the LM-assisted step of auditing and rewriting training data raises questions about the reliability of the LM in actually substantively improving the data rather than potentially harming quality and performance.
The “benchmarks” paragraph of the paper is a bit dense and could be revised/rewritten so that it doesn’t resemble a laundry list of benchmarks too much.
The paper shows solid performance numbers and producing a strong open model can facilitate future research, but I would have been interested in seeing more scientific contributions from a paper like this. Essentially this paper describes domain-specific model finetuning, and that finetuning yields domain-specific gains, which is a phenomenon that is already known by the research community. By scientific contributions, I’m looking for findings that target the “why” and link model design and training decisions to improvements in performance, so that future model developers in this domain have concrete findings or recommendations to learn from. In particular, it would have been good to emphasize more the unique nature of developing a model for education and potential takeaways or challenges of this specific area. Right now, the paper reads in a way where the domain itself is kind of a secondary concern (e.g. the paper narrative focuses more on generic model development concerns like, “data is varied and messy” and “our performance numbers on benchmarks are high!”). Currently, the main potential recommendation of this paper for other researchers seems to be “curate your data like we do” which is a bit vague. However, I have not carefully looked at the paper appendix, so if there are some concrete nuggets in there, it could be good to reference those during discussion/rebuttal.
Questions for authors
None
4 comments
Authors' Response [2/n]
(3) Human Validation of Fine-Grained Quality Scoring.
As described in Section 3.4, we employ GPT-5.6-Terra to score training examples using capability-specific quality rubrics and predefined acceptance thresholds. To validate its filtering decisions, we sample 2,000 scored examples, stratified by capability and filtering outcome. Two annotators independently assess each example against the corresponding quality criteria without access to the LLM scores. Disagreements are resolved through adjudication (inter-annotator agreement: Cohen's κ = 0.84).
Table 5. Human validation of LLM quality-filtering decisions (%).
| Capability | Valid among accepted (95% CI) | Invalid among rejected (95% CI) |
|---|---|---|
| Subject Competence | 96.4 (93.3, 98.1) | 88.4 (83.8, 91.8) |
| Pedagogical Action & Scaffolding | 96.8 (93.8, 98.4) | 94.4 (90.8, 96.6) |
| Curriculum Grounding | 95.6 (92.3, 97.5) | 90.8 (86.6, 93.8) |
| Diagnostic Reasoning | 94.8 (91.3, 96.9) | 92.0 (88.0, 94.8) |
| Overall | 95.9 (94.5, 97.0) | 91.4 (89.5, 93.0) |
Human evaluation confirms that 95.9% of LLM-accepted examples meet the required quality standards, while 91.4% of rejected examples are independently judged unsuitable, supporting the reliability of our fine-grained quality filtering.
Together, the controlled ablations and human evaluations demonstrate the downstream benefits of our data curation strategy and the reliability of its two LLM-assisted quality-control stages. We will incorporate these analyses into the revised appendix.
Response to Weakness 2
Thank you for this suggestion. Our intention was to explain what each benchmark evaluates, helping readers understand the results beyond the reported scores. We will revise this section to organize the evaluation around three educational capability groups—curriculum grounding, K–12 problem solving, and pedagogical tutoring—highlighting their complementary evaluation objectives. Detailed descriptions of individual benchmarks will be moved to the appendix for reference.
Response to Weakness 3
Thank you for this thoughtful suggestion. We would like to clarify that education-specific considerations are central to our framework. We also provide additional controlled experiments to establish more explicit connections between our design choices and model performance.
(1) Education-Specific Data Design.
Our framework organizes educational supervision around four core capabilities: Subject Competence, Pedagogical Action & Scaffolding, Curriculum Grounding, and Diagnostic Reasoning. These capabilities guide data collection, task-specific quality assessment, semantic auditing and rewriting, and training-budget allocation. Moreover, as discussed in Section 3.6, educational tasks may require conflicting response behaviors (e.g., providing answers versus guiding learners without revealing them), which we address through 19 task-specific pedagogical system instructions. This solution highlights a practical lesson for educational model development.
(2) Dataset, Pipeline, and Model Contributions.
We contribute a 69,999-example capability-oriented dataset, a reproducible education-specific data construction pipeline, and 4B/9B/27B open-weight educational models achieving leading or competitive performance across diverse educational benchmarks. These resources provide a practical foundation for further research and development in educational AI.
(3) Empirical Insights and Generalizable Lessons.
As detailed in our Response to Weakness 1, controlled experiments show that our full pipeline outperforms conventional educational SFT, and that capability-aware allocation brings additional gains under matched training budgets.
These findings highlight the importance of balancing training data across educational capabilities. Moreover, our task-specific system instructions address another challenge of adapting model responses to different teaching tasks, which often require distinct pedagogical behaviors.
We will revise the paper to emphasize these education-specific challenges, empirical findings, and actionable design insights.
Follow-up on Our Rebuttal
Dear Reviewer UGRE,
Thank you again for your insightful and constructive suggestions. In response to your concerns, we have added controlled ablation experiments and human evaluations of LLM-assisted auditing, rewriting, and quality filtering to provide stronger evidence for our data curation decisions. We have also clarified the education-specific design considerations and empirical insights of our framework and outlined revisions to improve the presentation of our evaluation benchmarks.
We would greatly appreciate it if you could review our rebuttal and share whether these additions address your concerns. If so, we would also be grateful if you could consider updating your overall rating. We would be happy to address any remaining questions.
Best regards, The Authors
Dear Reviewer UGRE,
Thank you for asking what other educational model developers can learn from this work. Table 1 of our response tests a concrete design choice. With the candidate pool, selection procedure, and token budget held fixed, explicitly allocating tokens across educational capabilities improved K12-Bench EM by 5.97 points and LongTutor-E by 4.78 points. In a separate human assessment of 1,000 pairs, annotators preferred the rewritten supervision in 88.9% of cases. Does the allocation result address your concern about what the paper contributes beyond domain-specific fine-tuning?
Authors' Response [1/n]
Thank you very much for your recognition and further valuable suggestions. We will clarify the following points regarding these issues and will eventually include them in the main text or appendix.
Response to Weakness 1
Thank you for this valuable suggestion. We provide additional controlled experiments and human evaluations to validate our data curation decisions, particularly the effectiveness of LLM-assisted auditing, rewriting, and quality filtering.
(1) Controlled Evaluation of Data Curation. We conduct controlled SFT experiments using Qwen3.5-4B-Base with matched supervised token budgets, training hyperparameters, and evaluation protocols:
Table 1. Controlled ablation study results
As shown in Tables 1, A outperforms both B and C across all 12 metrics. Compared with B, A improves K12-Bench EM by 12.22 points, GAOKAO-Bench by 12.58 points, and MathTutor-S by 20.21 points. Compared with C, A further gains 5.97 points on K12-Bench EM, 4.79 on EXAMS-V, and 4.78 on LongTutor-E. These results demonstrate benefits beyond conventional educational SFT and provide direct evidence for the effectiveness of capability-aware data allocation.
(2) Human Evaluation of Semantic Auditing and Rewriting.
As described in Section 3.3, we employ Qwen3.5-122B-A10B-FP8 for capability-specific semantic auditing and targeted rewriting. To evaluate rewriting quality, we randomly sample 1,000 original–rewritten pairs, stratified across four educational capabilities. Two annotators independently examine each pair in randomized order and select the better version, or indicate that both are comparable, based on task-specific quality criteria. Disagreements are resolved through adjudication (inter-annotator agreement: Cohen's κ = 0.92).
Table 4. Human preference between original and rewritten supervision (%).
Human evaluation shows that rewriting improves supervision quality in 88.9% of cases, with only 1.0% degraded. The consistent improvements across all four educational capabilities support the effectiveness of targeted rewriting while indicating a low risk of introducing errors.