- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper targets LLM alignment for education along three axes: helpfulness, personalization, and creativity (H/P/C). It builds a GPT-5 pipeline that refines scenario-specific rubrics and generation prompts over 330 scenarios, yielding 12,390 labeled QA pairs. These train a generative reward model, Edu-RM, to output H/P/C scores. The authors then propose IdealPO, a GRPO-style method. Its Ideal-point Marginal Credit (IMC) is the leave-one-out change in the group’s squared deficit from a score ceiling. Its Continuous Conflict Weighting (CCW) shrinks credits whose per-dimension contributions have opposing signs. On Qwen2.5-7B and 32B, EduAlign gets the top expert and Claude Sonnet 4.6 ratings on every dimension, plus the top aggregate scores on EduBench, ELMES, and EduValues. Edu-RM correlations with experts improve after iterative synthesis, and 7B ablations remove IMC or CCW.
Critical strengths
The reporting is candid. Sec. 4.3 and Appendix C.3 show per-criterion results and state that the method does not Pareto-dominate. The reward-model check also compares against expert labels at matched data volume (Table 3), which is more than most reward-model papers do. IdealPO is cheap and easy to inspect. It needs no extra rollouts, and Eq. 4 gives a clear closed form for how deficit affects credit. Dividing the model into two parameter group is nice, it would be easy if wrote full name on experiments.
Critical weaknesses
(a) The evidence for “educational quality” is mostly in-loop.
The policy optimizes Edu-RM, which is distilled from GPT-5 rubric scores. Experts and Claude then score with “the same 0–5 rubric” (Sec. 4.2). Edu-RM is validated only on 500 pairs from the synthesized pool, not on policy rollouts (Sec. 4.4). That pool is also the “final” distribution the “after” model was trained on. The creativity anchor rewards multiple approaches, structured steps, and examples, which plausibly rewards length and structure. No response lengths, length-controlled comparisons, or over-optimization analysis are given (cf. Gao et al., 2023, “Scaling Laws for Reward Model Overoptimization”, ICML). Some side effects fit this reading. ELMES teaching-condition feasibility falls below Instruct at both scales (3.6→3.1 at 7B, 4.1→3.5 at 32B), and KPE role appropriateness drops (3.3→2.1, 4.3→2.7). Margins over GRPO/GDPO are small and come from a single run, for example Claude H +0.0175 at 7B and EduValues +0.035 at 32B. ELMES per-criterion scores move in steps of 0.1, which suggests about ten items per task. Gaps such as CQG 4.000 vs 3.420 are then hard to interpret. The H rubric in A.3 reads as a values and civility criterion and is already about 4.6–4.8 at baseline. The introduction’s motivating claim (guiding versus answering) is never measured. Fix: report lengths and length-matched comparisons, seeds or bootstrap intervals, and RM accuracy on policy outputs. Add at least one rubric-independent or pedagogy-specific evaluation. (b) IdealPO’s gain is under-isolated relative to the claims.
Eq. 4 shows IMC is roughly a centered reward scaled by the group deficit d_k, plus a small quadratic term. CCW’s conflict ratio depends only on the sign pattern of centered integer rewards with K=3 and n=8. Appendix B.3 concedes that attenuation can make a conflicting bad response less penalized and gives no guarantee, so CCW’s benefit is argued only empirically. The closest cited methods, Focal Reward (Huang et al., 2026), SA-MRPO (Wang et al., 2026), and GD2PO (Liu et al., 2026a), are not compared. Neither is the cheap baseline of GRPO/GDPO with static deficit-based dimension weights. That baseline would show whether the gains come simply from upweighting the low-mean creativity and personalization channels. “w/o IMC” replaces the base credit rather than removing it. λ is tested only at 0 and 0.5, and the ablations are single-seed. Fix: add a deficit-weighted GRPO/GDPO baseline, a λ sweep, and seeds.
Questions for authors
Sec. 4.2 uses five annotators, two per question, while Sec. 4.4 says “all ten experts” rate each pair. Are these different panels? What is the inter-rater agreement for each, and are the raters teachers? Please clarify, Is the rubric given to experts and Claude identical to the unified Edu-RM rubric in A.3? What are the mean response lengths per method? How are the three Edu-RM scores aggregated for GRPO, PPO, and DAPO? Do PPO and DAPO share the policy-SFT initialization and the rollout budget? Sec. 4.5 cites ELMES “creativity” and “personalization” scores (4.075 vs 3.600; 4.175 vs 4.150) that appear in no table. How are they defined? How many items does each ELMES task contain? (Clarification) How often does CCW materially change a group’s advantages? Please give the distribution of R_i and the fraction of groups with σ_q ≤ ε. Does a deficit-weighted raw-mean GRPO recover most of IMC’s gain? How does IdealPO compare with Focal Reward or SA-MRPO?
8 comments
Response to Reviewer RDJB (2/5)
4. Declines in teaching-condition feasibility and role appropriateness
Our goal is to improve overall educational task performance through joint optimization of Helpfulness, Personalization, and Creativity. We train with Edu-RM on our constructed training data and evaluate transfer to public educational tasks using each benchmark's own criteria. Table 2 of the original manuscript shows that the ELMES aggregate score increases from 3.218 to 3.901 at 7B and from 3.611 to 4.362 at 32B, corresponding to relative improvements of 21.22% and 20.80%. EduBench and EduValues aggregate scores also improve at both scales.
The two criterion-level declines you identify are reported in Appendix C.3, and Section 4.3 of the original manuscript discusses task trade-offs. They show that the overall gains coexist with weaker performance on teaching-condition feasibility and KPE role appropriateness. The results therefore support our intended improvement in overall educational task performance, with uneven effects across individual criteria.
5. Small margins over strong baselines and single-run results
We have added three training seeds for 7B GRPO, GDPO, and IdealPO to examine the gains across repeated training runs. All runs use the same public evaluation items. The table reports the means across three seeds.
Table R3. Public-task results averaged over three training seeds (7B).
| Method | EduBench | ELMES | ||||||
|---|---|---|---|---|---|---|---|---|
| IP | PLS | TMG | AVG | ILPG | CQG | KPE | AVG | |
| GRPO | 7.934 | 9.024 | 9.164 | 8.707 | 4.047 | 3.400 | 3.289 | 3.579 |
| GDPO | 7.815 | 9.012 | 9.164 | 8.664 | 4.158 | 3.500 | 3.394 | 3.684 |
| IdealPO (IMC+CCW) | 8.385 | 9.109 | 9.258 | 8.917 | 4.276 | 3.780 | 3.511 | 3.856 |
IdealPO exceeds both GRPO and GDPO in the three-seed mean of every task. Its aggregate gains over GDPO are 0.253 on EduBench and 0.172 on ELMES; the corresponding gains over GRPO are 0.210 and 0.277.
6. ELMES item counts, score increments, and task-level aggregation
Section 4.1 of the original manuscript states that all items in the selected ELMES tasks are evaluated, and Appendix C.3 lists the fine-grained criteria. Specifically, ILPG, CQG, and KPE each contain 10 items, totaling 30, and use 15, 5, and 6 criteria, respectively. Per-criterion means have increments of 0.1, whereas task scores average multiple criteria and therefore have finer numerical resolution. For example, IdealPO's CQG criterion scores in the original appendix are 3.6, 4.3, 3.0, 4.2, and 4.9, averaging to 4.000. Thus, 4.000 versus 3.420 compares task-level aggregate scores.
7. Helpfulness definition and the rubric version in Appendix A.3
Appendix A.3 of the original manuscript displayed an earlier rubric version. Edu-RM training, rollout scoring, expert evaluation, and Claude evaluation all used the same final H/P/C rubric. In that version, Helpfulness measures how well a response helps the learner address their question or advance toward their goal through accurate, clear, and safe responses, with actionable guidance where appropriate and responsible handling of risks.
We will update Appendix A.3 in the revised manuscript with the final prompts actually used in the experiments.
8. Evaluation using other criteria or pedagogy-specific tasks
The original manuscript already evaluates generalization to public educational benchmarks whose evaluation data are separate from our training data and whose scoring criteria are defined by the benchmarks themselves. These evaluations test transfer beyond our constructed training distribution and the unified H/P/C rubric used for reward supervision. The aggregate results in Table 2 of the original manuscript show improvements on all three benchmarks at both model scales. Appendix C.3 further reports fine-grained results across educational tasks. As an illustrative example, Table R4 presents selected criteria for the 7B model on EduBench's Idea Provision task.
Table R4. Selected EduBench IP criterion scores from Appendix C.3 of the original manuscript (7B).
| EduBench IP criterion | Instruct | IdealPO |
|---|---|---|
| Instruction Following & Task Completion | 7.540 | 9.000 |
| Domain Knowledge Accuracy | 7.180 | 8.720 |
| Reasoning Process Rigor | 6.870 | 8.600 |
| Higher-Order Thinking & Skill Development | 5.540 | 7.360 |
These examples show improvements in task completion, knowledge accuracy, reasoning rigor, and support for higher-order thinking. Together with the other task-specific results reported in Appendix C.3, they provide evidence that complements the HPC ratings.
Response to Reviewer RDJB (3/5)
W(b). Isolation of IdealPO’s contribution
IdealPO’s gain is under-isolated relative to the claims.
Response:
1. IMC and deficit-weighted centered rewards
Eq. 4 and Appendix B.3 of the original manuscript already provide this decomposition. IMC uses the current question group's target deficits, with a finite-group quadratic correction from the exact derivation. The added static-weighting baseline falls below IdealPO on all three benchmark aggregates, while the three-seed ablation supports IMC's contribution (W(b), points 4 and 7).
2. CCW's dependence on credit magnitudes and attenuation of negative credit
CCW uses the magnitudes of positive and negative credits, so identical sign patterns can yield different weights. Appendix B.3 of the original manuscript explains that attenuation can reduce negative penalties and that final advantages depend on group normalization. The three-seed ablation shows higher mean scores with CCW on all six tasks; the λ sweep and offline statistics provide further evidence (W(b), points 6–7; Q8).
3. Comparison with a closely related method
We have added , one of the related methods you identified, at both 7B and 32B. The table combines the new baseline results with IdealPO results from Table 2 of the original manuscript. Each result is from one run.
Table R5. Comparison with at two model scales.
| Scale | Method | EduBench | ELMES | EduValues |
|---|---|---|---|---|
| 7B | 8.707 | 3.737 | 3.929 | |
| 7B | IdealPO | 8.851 | 3.901 | 4.097 |
| 32B | 8.830 | 4.177 | 4.142 | |
| 32B | IdealPO | 9.000 | 4.362 | 4.177 |
IdealPO exceeds on all three aggregate scores at both scales. We will add this method to the revised baseline comparison and report the results and implementation settings in the appendix.
4. Can static deficit weighting recover the gains?
We have added static deficit-weighted GRPO to test whether fixed upweighting of low-scoring dimensions recovers the gains. GRPO and IdealPO scores below are from Table 2 of the original manuscript; static weighting is a new comparison. All results are from single 7B runs.
Table R6. Comparison with static deficit-weighted GRPO (7B).
| Method | EduBench | ELMES | EduValues |
|---|---|---|---|
| GRPO | 8.690 | 3.733 | 4.027 |
| Static deficit-weighted GRPO | 8.764 | 3.733 | 4.018 |
| IdealPO | 8.851 | 3.901 | 4.097 |
Static weighting improves EduBench, leaves ELMES unchanged, and slightly reduces EduValues. IdealPO improves further on all three aggregate scores. Fixed weighting therefore recovers part of the gain on one benchmark, but does not reproduce the full method's performance across benchmarks.
5. The “w/o IMC” operation
Section 4.5 of the original manuscript specifies that w/o IMC removes IMC and falls back to the raw mean of the three rewards, retaining CCW and group normalization. Conflicts are estimated using centered raw rewards. The new three-seed results support IMC's contribution (W(b), point 7).
6. The λ sweep
Section 4.5 of the original manuscript compares λ=0.5 with λ=0. We have added λ=0.25 and 0.75 while holding the seed, other settings, and evaluation items fixed.
Table R7. Sensitivity to λ on EduBench and ELMES (7B).
| λ | EduBench | ELMES |
|---|---|---|
| 0 | 8.788 | 3.692 |
| 0.25 | 8.751 | 3.697 |
| 0.5 | 8.851 | 3.901 |
| 0.75 | 8.794 | 3.730 |
The default λ=0.5 achieves the highest aggregate scores on both benchmarks among the tested settings, supporting this parameter choice in our experiments.
7. Component ablations across training seeds
We have extended the original single-run ablations to three training seeds for IdealPO, w/o CCW, and w/o IMC. The table reports means across three seeds.
Table R8. Component ablations averaged over three training seeds (7B).
| Method | EduBench | ELMES | ||||||
|---|---|---|---|---|---|---|---|---|
| IP | PLS | TMG | AVG | ILPG | CQG | KPE | AVG | |
| IdealPO (IMC+CCW) | 8.385 | 9.109 | 9.258 | 8.917 | 4.276 | 3.780 | 3.511 | 3.856 |
| w/o CCW | 8.138 | 9.061 | 9.223 | 8.807 | 4.111 | 3.380 | 3.039 | 3.510 |
| w/o IMC | 7.617 | 8.970 | 9.157 | 8.581 | 3.922 | 3.600 | 3.122 | 3.548 |
The full method has higher means than both ablations on all six tasks. Removing IMC reduces the EduBench and ELMES aggregate scores by 0.336 and 0.308; removing CCW reduces them by 0.110 and 0.346.
Response to Reviewer RDJB (1/5)
Thank you for your detailed comments on evaluation validity and the contribution of IdealPO. We have added evidence addressing both concerns:
We first address your two main concerns, then respond to your specific questions.
W(a). Evaluation validity and result stability
Response:
1. Evaluation beyond the training reward and the shared rubric
The original manuscript evaluates policy responses using experts and Claude, as well as public benchmarks with their own task-specific criteria. Section 4.2 and Table 1 of the original manuscript report evaluations of the same questions and responses with model identities hidden. Section 4.1 specifies that the three benchmarks use their original scoring prompts, with results reported in Table 2 of the original manuscript.
The 200 questions extracted from student interviews were excluded from training. Experts and Claude share the HPC rubric to evaluate the same target qualities. Both evaluators give EduAlign the highest mean scores in all three dimensions at both model scales. The public evaluations additionally cover Idea Provision, Personalized Learning Support, Teaching Material Generation, Interdisciplinary Lesson Plan Generation, Contextual Question Generation, Knowledge Point Explanation, and Subjective Analysis. EduAlign achieves the highest aggregate score on each of the three benchmarks at both scales among the methods compared in the original manuscript. Appendix C.3 of the original manuscript also reports task-specific criteria. These evaluations support improvements in HPC response quality and performance on public educational tasks.
2. The 500 QA pairs, training separation, and RM validation on policy outputs
Section 4.4 of the original manuscript specifies that the 500 QA pairs come from the final synthesized pool, with their questions excluded from RM training. With data volume, negative examples, and training settings held constant, refinement improves both Pearson and Spearman correlations with expert ratings across H/P/C (Table 3).
Section 4.6 (Training Dynamics) and Fig. 3 of the original manuscript further track the three reward dimensions using Edu-RM scores throughout training. IdealPO achieves higher final rewards than GRPO and GDPO in all three dimensions. This direction of improvement is consistent with the final expert evaluations in Table 1 of the original manuscript, where EduAlign also receives higher mean scores in each H/P/C dimension. Together, the training curves and expert evaluations show that the higher RM rewards are accompanied by improved expert-rated response quality.
3. Response length, structure preference, and length-matched comparisons
We have added response-length statistics and comparisons of responses to the same question with similar lengths. The following statistics concern the 7B responses to the 200 questions evaluated in Table 1 of the original manuscript.
Table R1. Mean response lengths on the 200-question evaluation set (7B).
For each question, we compute the ratio of the longer response's token count to the shorter response's token count. We retain response pairs with ratios at most 1.10 or 1.20 and compare their existing expert and Claude scores.
Table R2. IdealPO minus GDPO scores in subsets matched by response length (7B).
All three differences are positive in both matched subsets, with consistent directions across experts and Claude. IdealPO therefore retains a quality advantage among these responses of similar length.