acceptodds
Under review as a conference paper at ICLR 2027

EduAlign: Aligning Educational LLMs for Helpfulness, Personalization, and Creativity

Abstract

Educational language models should pursue three complementary objectives: helpfulness through responsible guidance, personalization to learner needs, and creativity that supports exploration. General-purpose models do not consistently satisfy these educational requirements, motivating explicit alignment with all three objectives. We present EduAlign, a framework that combines educational reward modeling with multi-objective reinforcement learning. To obtain training signals for these goals across educational scenarios, we develop a rubric-guided data synthesis pipeline that iteratively refines scoring criteria and generation prompts for each subject, grade band, and task type. The resulting supervision trains a multidimensional reward model, Edu-RM, to predict helpfulness, personalization, and creativity scores. To jointly optimize these educational objectives, we introduce Ideal-point Policy Optimization (IdealPO), a multi-objective reinforcement learning algorithm. IdealPO combines ideal-point marginal credit with continuous conflict weighting to guide policy updates that account for uneven progress and conflicting contributions across objectives. Experiments with 7B and 32B models show that EduAlign achieves the highest mean ratings on all three dimensions from both education experts and a model judge, alongside the highest aggregate scores on EduBench, ELMES, and EduValues among the compared methods. Reward-model validation and component ablations further support the effectiveness of the framework. Code is available at https://anonymous.4open.science/r/EduAlign-3BD1.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM alignment, pedagogical alignment, multi-objective reinforcement learning, reward modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RDJBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets LLM alignment for education along three axes: helpfulness, personalization, and creativity (H/P/C). It builds a GPT-5 pipeline that refines scenario-specific rubrics and generation prompts over 330 scenarios, yielding 12,390 labeled QA pairs. These train a generative reward model, Edu-RM, to output H/P/C scores. The authors then propose IdealPO, a GRPO-style method. Its Ideal-point Marginal Credit (IMC) is the leave-one-out change in the group’s squared deficit from a score ceiling. Its Continuous Conflict Weighting (CCW) shrinks credits whose per-dimension contributions have opposing signs. On Qwen2.5-7B and 32B, EduAlign gets the top expert and Claude Sonnet 4.6 ratings on every dimension, plus the top aggregate scores on EduBench, ELMES, and EduValues. Edu-RM correlations with experts improve after iterative synthesis, and 7B ablations remove IMC or CCW.

Critical strengths

The reporting is candid. Sec. 4.3 and Appendix C.3 show per-criterion results and state that the method does not Pareto-dominate. The reward-model check also compares against expert labels at matched data volume (Table 3), which is more than most reward-model papers do. IdealPO is cheap and easy to inspect. It needs no extra rollouts, and Eq. 4 gives a clear closed form for how deficit affects credit. Dividing the model into two parameter group is nice, it would be easy if wrote full name on experiments.

Critical weaknesses

(a) The evidence for “educational quality” is mostly in-loop.

The policy optimizes Edu-RM, which is distilled from GPT-5 rubric scores. Experts and Claude then score with “the same 0–5 rubric” (Sec. 4.2). Edu-RM is validated only on 500 pairs from the synthesized pool, not on policy rollouts (Sec. 4.4). That pool is also the “final” distribution the “after” model was trained on. The creativity anchor rewards multiple approaches, structured steps, and examples, which plausibly rewards length and structure. No response lengths, length-controlled comparisons, or over-optimization analysis are given (cf. Gao et al., 2023, “Scaling Laws for Reward Model Overoptimization”, ICML). Some side effects fit this reading. ELMES teaching-condition feasibility falls below Instruct at both scales (3.6→3.1 at 7B, 4.1→3.5 at 32B), and KPE role appropriateness drops (3.3→2.1, 4.3→2.7). Margins over GRPO/GDPO are small and come from a single run, for example Claude H +0.0175 at 7B and EduValues +0.035 at 32B. ELMES per-criterion scores move in steps of 0.1, which suggests about ten items per task. Gaps such as CQG 4.000 vs 3.420 are then hard to interpret. The H rubric in A.3 reads as a values and civility criterion and is already about 4.6–4.8 at baseline. The introduction’s motivating claim (guiding versus answering) is never measured. Fix: report lengths and length-matched comparisons, seeds or bootstrap intervals, and RM accuracy on policy outputs. Add at least one rubric-independent or pedagogy-specific evaluation. (b) IdealPO’s gain is under-isolated relative to the claims.

Eq. 4 shows IMC is roughly a centered reward scaled by the group deficit d_k, plus a small quadratic term. CCW’s conflict ratio depends only on the sign pattern of centered integer rewards with K=3 and n=8. Appendix B.3 concedes that attenuation can make a conflicting bad response less penalized and gives no guarantee, so CCW’s benefit is argued only empirically. The closest cited methods, Focal Reward (Huang et al., 2026), SA-MRPO (Wang et al., 2026), and GD2PO (Liu et al., 2026a), are not compared. Neither is the cheap baseline of GRPO/GDPO with static deficit-based dimension weights. That baseline would show whether the gains come simply from upweighting the low-mean creativity and personalization channels. “w/o IMC” replaces the base credit rather than removing it. λ is tested only at 0 and 0.5, and the ablations are single-seed. Fix: add a deficit-weighted GRPO/GDPO baseline, a λ sweep, and seeds.

Questions for authors

Sec. 4.2 uses five annotators, two per question, while Sec. 4.4 says “all ten experts” rate each pair. Are these different panels? What is the inter-rater agreement for each, and are the raters teachers? Please clarify,  Is the rubric given to experts and Claude identical to the unified Edu-RM rubric in A.3? What are the mean response lengths per method? How are the three Edu-RM scores aggregated for GRPO, PPO, and DAPO? Do PPO and DAPO share the policy-SFT initialization and the rollout budget? Sec. 4.5 cites ELMES “creativity” and “personalization” scores (4.075 vs 3.600; 4.175 vs 4.150) that appear in no table. How are they defined? How many items does each ELMES task contain? (Clarification) How often does CCW materially change a group’s advantages? Please give the distribution of R_i and the fraction of groups with σ_q ≤ ε. Does a deficit-weighted raw-mean GRPO recover most of IMC’s gain? How does IdealPO compare with Focal Reward or SA-MRPO?

8 comments
AuthorsOpenReview ↗

Response to Reviewer RDJB (1/5)

Thank you for your detailed comments on evaluation validity and the contribution of IdealPO. We have added evidence addressing both concerns:

  • Evaluation validity: response-length statistics and comparisons of responses to the same question with similar lengths. IdealPO receives higher mean expert and Claude ratings than GDPO in all three dimensions in both matched subsets. Across three training seeds, IdealPO also exceeds GDPO by 0.253 on EduBench and 0.172 on ELMES.
  • Algorithmic contribution: a static deficit-weighted GRPO baseline, comparisons with GD2PO\mathrm{GD}^{2}\mathrm{PO} at 7B and 32B, a λ sweep, and component ablations across three seeds. IdealPO exceeds both added baselines on all three benchmark aggregate scores, and exceeds both ablations in the three-seed means of all six evaluated tasks.

We first address your two main concerns, then respond to your specific questions.

W(a). Evaluation validity and result stability

The evidence for “educational quality” is mostly in-loop.

Response:

1. Evaluation beyond the training reward and the shared rubric

The original manuscript evaluates policy responses using experts and Claude, as well as public benchmarks with their own task-specific criteria. Section 4.2 and Table 1 of the original manuscript report evaluations of the same questions and responses with model identities hidden. Section 4.1 specifies that the three benchmarks use their original scoring prompts, with results reported in Table 2 of the original manuscript.

The 200 questions extracted from student interviews were excluded from training. Experts and Claude share the HPC rubric to evaluate the same target qualities. Both evaluators give EduAlign the highest mean scores in all three dimensions at both model scales. The public evaluations additionally cover Idea Provision, Personalized Learning Support, Teaching Material Generation, Interdisciplinary Lesson Plan Generation, Contextual Question Generation, Knowledge Point Explanation, and Subjective Analysis. EduAlign achieves the highest aggregate score on each of the three benchmarks at both scales among the methods compared in the original manuscript. Appendix C.3 of the original manuscript also reports task-specific criteria. These evaluations support improvements in HPC response quality and performance on public educational tasks.

2. The 500 QA pairs, training separation, and RM validation on policy outputs

Section 4.4 of the original manuscript specifies that the 500 QA pairs come from the final synthesized pool, with their questions excluded from RM training. With data volume, negative examples, and training settings held constant, refinement improves both Pearson and Spearman correlations with expert ratings across H/P/C (Table 3).

Section 4.6 (Training Dynamics) and Fig. 3 of the original manuscript further track the three reward dimensions using Edu-RM scores throughout training. IdealPO achieves higher final rewards than GRPO and GDPO in all three dimensions. This direction of improvement is consistent with the final expert evaluations in Table 1 of the original manuscript, where EduAlign also receives higher mean scores in each H/P/C dimension. Together, the training curves and expert evaluations show that the higher RM rewards are accompanied by improved expert-rated response quality.

3. Response length, structure preference, and length-matched comparisons

We have added response-length statistics and comparisons of responses to the same question with similar lengths. The following statistics concern the 7B responses to the 200 questions evaluated in Table 1 of the original manuscript.

Table R1. Mean response lengths on the 200-question evaluation set (7B).

MethodResponsesMean tokens
Instruct200398.19
GRPO2001242.68
GDPO2001142.07
IdealPO2001499.57

For each question, we compute the ratio of the longer response's token count to the shorter response's token count. We retain response pairs with ratios at most 1.10 or 1.20 and compare their existing expert and Claude scores.

Table R2. IdealPO minus GDPO scores in subsets matched by response length (7B).

ComparisonLength-ratio thresholdEvaluatorΔHelpfulnessΔPersonalizationΔCreativity
IdealPO−GDPO1.10Experts+0.1429+0.0238+0.1905
IdealPO−GDPO1.10Claude+0.1429+0.0476+0.1905
IdealPO−GDPO1.20Experts+0.2245+0.1020+0.2245
IdealPO−GDPO1.20Claude+0.1224+0.0612+0.3265

All three differences are positive in both matched subsets, with consistent directions across experts and Claude. IdealPO therefore retains a quality advantage among these responses of similar length.

AuthorsOpenReview ↗

Response to Reviewer RDJB (2/5)

4. Declines in teaching-condition feasibility and role appropriateness

Our goal is to improve overall educational task performance through joint optimization of Helpfulness, Personalization, and Creativity. We train with Edu-RM on our constructed training data and evaluate transfer to public educational tasks using each benchmark's own criteria. Table 2 of the original manuscript shows that the ELMES aggregate score increases from 3.218 to 3.901 at 7B and from 3.611 to 4.362 at 32B, corresponding to relative improvements of 21.22% and 20.80%. EduBench and EduValues aggregate scores also improve at both scales.

The two criterion-level declines you identify are reported in Appendix C.3, and Section 4.3 of the original manuscript discusses task trade-offs. They show that the overall gains coexist with weaker performance on teaching-condition feasibility and KPE role appropriateness. The results therefore support our intended improvement in overall educational task performance, with uneven effects across individual criteria.

5. Small margins over strong baselines and single-run results

We have added three training seeds for 7B GRPO, GDPO, and IdealPO to examine the gains across repeated training runs. All runs use the same public evaluation items. The table reports the means across three seeds.

Table R3. Public-task results averaged over three training seeds (7B).

MethodEduBenchELMES
IPPLSTMGAVGILPGCQGKPEAVG
GRPO7.9349.0249.1648.7074.0473.4003.2893.579
GDPO7.8159.0129.1648.6644.1583.5003.3943.684
IdealPO (IMC+CCW)8.3859.1099.2588.9174.2763.7803.5113.856

IdealPO exceeds both GRPO and GDPO in the three-seed mean of every task. Its aggregate gains over GDPO are 0.253 on EduBench and 0.172 on ELMES; the corresponding gains over GRPO are 0.210 and 0.277.

6. ELMES item counts, score increments, and task-level aggregation

Section 4.1 of the original manuscript states that all items in the selected ELMES tasks are evaluated, and Appendix C.3 lists the fine-grained criteria. Specifically, ILPG, CQG, and KPE each contain 10 items, totaling 30, and use 15, 5, and 6 criteria, respectively. Per-criterion means have increments of 0.1, whereas task scores average multiple criteria and therefore have finer numerical resolution. For example, IdealPO's CQG criterion scores in the original appendix are 3.6, 4.3, 3.0, 4.2, and 4.9, averaging to 4.000. Thus, 4.000 versus 3.420 compares task-level aggregate scores.

7. Helpfulness definition and the rubric version in Appendix A.3

Appendix A.3 of the original manuscript displayed an earlier rubric version. Edu-RM training, rollout scoring, expert evaluation, and Claude evaluation all used the same final H/P/C rubric. In that version, Helpfulness measures how well a response helps the learner address their question or advance toward their goal through accurate, clear, and safe responses, with actionable guidance where appropriate and responsible handling of risks.

We will update Appendix A.3 in the revised manuscript with the final prompts actually used in the experiments.

8. Evaluation using other criteria or pedagogy-specific tasks

The original manuscript already evaluates generalization to public educational benchmarks whose evaluation data are separate from our training data and whose scoring criteria are defined by the benchmarks themselves. These evaluations test transfer beyond our constructed training distribution and the unified H/P/C rubric used for reward supervision. The aggregate results in Table 2 of the original manuscript show improvements on all three benchmarks at both model scales. Appendix C.3 further reports fine-grained results across educational tasks. As an illustrative example, Table R4 presents selected criteria for the 7B model on EduBench's Idea Provision task.

Table R4. Selected EduBench IP criterion scores from Appendix C.3 of the original manuscript (7B).

EduBench IP criterionInstructIdealPO
Instruction Following & Task Completion7.5409.000
Domain Knowledge Accuracy7.1808.720
Reasoning Process Rigor6.8708.600
Higher-Order Thinking & Skill Development5.5407.360

These examples show improvements in task completion, knowledge accuracy, reasoning rigor, and support for higher-order thinking. Together with the other task-specific results reported in Appendix C.3, they provide evidence that complements the HPC ratings.

AuthorsOpenReview ↗

Response to Reviewer RDJB (3/5)

W(b). Isolation of IdealPO’s contribution

IdealPO’s gain is under-isolated relative to the claims.

Response:

1. IMC and deficit-weighted centered rewards

Eq. 4 and Appendix B.3 of the original manuscript already provide this decomposition. IMC uses the current question group's target deficits, with a finite-group quadratic correction from the exact derivation. The added static-weighting baseline falls below IdealPO on all three benchmark aggregates, while the three-seed ablation supports IMC's contribution (W(b), points 4 and 7).

2. CCW's dependence on credit magnitudes and attenuation of negative credit

CCW uses the magnitudes of positive and negative credits, so identical sign patterns can yield different weights. Appendix B.3 of the original manuscript explains that attenuation can reduce negative penalties and that final advantages depend on group normalization. The three-seed ablation shows higher mean scores with CCW on all six tasks; the λ sweep and offline statistics provide further evidence (W(b), points 6–7; Q8).

3. Comparison with a closely related method

We have added GD2PO\mathrm{GD}^{2}\mathrm{PO}, one of the related methods you identified, at both 7B and 32B. The table combines the new baseline results with IdealPO results from Table 2 of the original manuscript. Each result is from one run.

Table R5. Comparison with GD2PO\mathrm{GD}^{2}\mathrm{PO} at two model scales.

ScaleMethodEduBenchELMESEduValues
7BGD2PO\mathrm{GD}^{2}\mathrm{PO}8.7073.7373.929
7BIdealPO8.8513.9014.097
32BGD2PO\mathrm{GD}^{2}\mathrm{PO}8.8304.1774.142
32BIdealPO9.0004.3624.177

IdealPO exceeds GD2PO\mathrm{GD}^{2}\mathrm{PO} on all three aggregate scores at both scales. We will add this method to the revised baseline comparison and report the results and implementation settings in the appendix.

4. Can static deficit weighting recover the gains?

We have added static deficit-weighted GRPO to test whether fixed upweighting of low-scoring dimensions recovers the gains. GRPO and IdealPO scores below are from Table 2 of the original manuscript; static weighting is a new comparison. All results are from single 7B runs.

Table R6. Comparison with static deficit-weighted GRPO (7B).

MethodEduBenchELMESEduValues
GRPO8.6903.7334.027
Static deficit-weighted GRPO8.7643.7334.018
IdealPO8.8513.9014.097

Static weighting improves EduBench, leaves ELMES unchanged, and slightly reduces EduValues. IdealPO improves further on all three aggregate scores. Fixed weighting therefore recovers part of the gain on one benchmark, but does not reproduce the full method's performance across benchmarks.

5. The “w/o IMC” operation

Section 4.5 of the original manuscript specifies that w/o IMC removes IMC and falls back to the raw mean of the three rewards, retaining CCW and group normalization. Conflicts are estimated using centered raw rewards. The new three-seed results support IMC's contribution (W(b), point 7).

6. The λ sweep

Section 4.5 of the original manuscript compares λ=0.5 with λ=0. We have added λ=0.25 and 0.75 while holding the seed, other settings, and evaluation items fixed.

Table R7. Sensitivity to λ on EduBench and ELMES (7B).

λEduBenchELMES
08.7883.692
0.258.7513.697
0.58.8513.901
0.758.7943.730

The default λ=0.5 achieves the highest aggregate scores on both benchmarks among the tested settings, supporting this parameter choice in our experiments.

7. Component ablations across training seeds

We have extended the original single-run ablations to three training seeds for IdealPO, w/o CCW, and w/o IMC. The table reports means across three seeds.

Table R8. Component ablations averaged over three training seeds (7B).

MethodEduBenchELMES
IPPLSTMGAVGILPGCQGKPEAVG
IdealPO (IMC+CCW)8.3859.1099.2588.9174.2763.7803.5113.856
w/o CCW8.1389.0619.2238.8074.1113.3803.0393.510
w/o IMC7.6178.9709.1578.5813.9223.6003.1223.548

The full method has higher means than both ablations on all six tasks. Removing IMC reduces the EduBench and ELMES aggregate scores by 0.336 and 0.308; removing CCW reduces them by 0.110 and 0.346.

Reviewer_dQSuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposed a framework with the goal of training a good large language model educator. They propose three criteria: helpfulness, personalization, and creativity that the training should focus on. To achieve the goal, they first train a reward model (RM) to generate a vector of rewards to score the response based on the three criteria. The supervision data for training the RM is obtained via synthetic data generation and rubric-based scoring with iterative refinement. Then using this RM, they propose a multi-objective reinforcement learning algorithm to train the educator, with two designs for better credit assignment and conflict mitigation. Through experiments, they show the RM aligns better with the expert evaluation scores, the trained educator achieved higher scores under expert and LLM evaluation, and it also achieves higher scores on three public benchmarks.

Critical strengths

None.

Critical weaknesses
  1. The main result of this work is Table 1. But I don't think it provides strong enough evidences showing that EduAlign truly gives a better educator. It is true that a higher score is achieved, but the increment is too limited and it is unclear what +0.1 in score means in terms of the quality change in real responses, for example, a score of 4.2 versus 4.1 is just one scorer increasing 1 score out of 10 scorer.

  2. The comparison with baseline algorithms seems confusing to me. The trained RM is part of the proposed framework. I don't know on which reward function, the baselines are trained, but if they are also trained using the same RM, then the advantage of the framework is just the multi-objective RL algorithm. And this algorithm should be a general multi-objective RL algorithm, but the paper only shows a limited gain over GDPO in this specific domain.

Questions for authors
  1. For the results in Table 1 and Table 2, are the baselines algorithms trained under the same reward model as EduAlign?

  2. How is the reward model trained? Are the rubrics just used to get the final vector of rewards in the supervision data or are they kept in the training of the reward model? And how large is the reward model?

5 comments
AuthorsOpenReview ↗

Response to Reviewer dQSu (1/3)

Thank you for pointing out the need to clarify the meaning of the score improvements and the baseline comparisons. We respond to the two weaknesses and two questions you raised point by point below.

Weakness 1

The main result of this work is Table 1. But I don't think it provides strong enough evidences showing that EduAlign truly gives a better educator. It is true that a higher score is achieved, but the increment is too limited and it is unclear what +0.1 in score means in terms of the quality change in real responses, for example, a score of 4.2 versus 4.1 is just one scorer increasing 1 score out of 10 scorer.

Response: EduAlign improves educational response quality across Helpfulness, Personalization, and Creativity. Experts and Claude evaluate the same questions and responses and report improvements over Instruct in all three dimensions at both model scales. Table R1 summarizes the gains of the complete framework.

Table R1. Improvements over Instruct in expert and Claude ratings, from Table 1 of the original manuscript.

EvaluatorScaleInstruct meanEduAlign meanScore increaseRelative increase
Education experts7B3.76504.4008+0.635816.89%
Education experts32B3.84334.5617+0.718418.69%
Claude Sonnet 4.67B3.67504.2475+0.572515.58%
Claude Sonnet 4.632B3.72004.3642+0.644217.32%

Appendix C.1 (Case Studies) of the original manuscript illustrates the corresponding changes in response quality. For a Grade 9 student who prefers independent thinking and explores mechanical efficiency through simulations, Instruct suggests varying parameters and comparing results. EduAlign connects the activity to the efficiency formula, asks the student to predict outcomes before simulation, and proposes testing how friction affects efficiency. This turns general advice into a concrete activity involving prediction and hypothesis testing.

Section 4.2 of the original manuscript specifies 200 questions, with each response scored by two experts and their scores averaged. A one-point increase from one expert on one response changes the overall mean for that dimension by 0.0025.

The improvements also extend to public educational tasks evaluated using each benchmark's own standards. Table 2 of the original manuscript reports results on EduBench, ELMES, and EduValues, covering tasks such as personalized learning support, teaching material generation, and knowledge explanation. Table R2 summarizes the gains over Instruct.

Table R2. Relative improvements over Instruct on public benchmarks, calculated from Table 2 of the original manuscript.

ScaleMethodEduBenchELMESEduValues
7BGRPO + Edu-RM+7.38%+16.00%+12.64%
7BGDPO + Edu-RM+7.10%+15.23%+12.14%
7BEduAlign+9.37%+21.22%+14.60%
32BGRPO + Edu-RM+4.87%+13.74%+9.11%
32BGDPO + Edu-RM+5.21%+12.85%+9.35%
32BEduAlign+6.62%+20.80%+10.27%

GRPO and GDPO also improve over Instruct when trained with our Edu-RM. Under the same reward supervision, IdealPO achieves further improvements over GDPO, as summarized in Table R3. This distinguishes the complete framework's gains from the additional benefit of IdealPO.

Table R3. Relative improvements of EduAlign over GDPO, calculated from Tables 1 and 2 of the original manuscript.

ScaleExpert meanClaude meanEduBench IPEduBenchELMESEduValues
7B+3.61%+3.39%+5.10%+2.11%+5.20%+2.20%
32B+3.22%+1.89%+3.29%+1.34%+7.04%+0.85%

Together, the response example, expert and Claude ratings, and public task results support improved educational response quality.

AuthorsOpenReview ↗

Response to Reviewer dQSu (2/3)

Weakness 2

The comparison with baseline algorithms seems confusing to me. The trained RM is part of the proposed framework. I don't know on which reward function, the baselines are trained, but if they are also trained using the same RM, then the advantage of the framework is just the multi-objective RL algorithm. And this algorithm should be a general multi-objective RL algorithm, but the paper only shows a limited gain over GDPO in this specific domain.

Response: All RL methods in Tables 1 and 2 of the original manuscript use the same post-refinement Edu-RM, scoring prompt, and RM inference settings. Giving the baselines the same educational reward supervision isolates the additional contribution of IdealPO. The reward-modeling contribution is evaluated separately, so the framework’s contribution is not inferred solely from the comparison with GDPO.

EduAlign has two components with corresponding evidence:

  • Iterative educational reward modeling. Scenario-specific rubrics translate HPC into supervision, while generation prompts and scoring criteria are jointly refined. Section 4.4 and Table 3 of the original manuscript compare RMs at matched data volume, with identical negative examples and training settings. On 500 expert-rated QA pairs whose questions are excluded from RM training, refinement improves both Pearson and Spearman correlations in all three dimensions.
  • Joint optimization using objective deficits and conflicting contributions. IMC assigns credit from the change in the group’s squared distance to the target when a response is removed. CCW adjusts credit according to conflicting dimensional contributions. This constructs a training signal using target deficits and contribution conflicts beyond GDPO’s dimension-wise normalization; the original component ablations test the two modules.

We additionally compared GDPO and IdealPO with both pre- and post-refinement Edu-RM at 7B. This directly examines the downstream effects of the two components.

Table R4. Performance of GDPO and IdealPO with pre- and post-refinement Edu-RM (7B).

Reward modelPolicy / RL methodEduBenchELMESEduValues
—7B Instruct8.0933.2183.575
Pre-refinement Edu-RMGDPO8.3653.3173.549
Pre-refinement Edu-RMIdealPO8.7003.4473.584
Post-refinement Edu-RMGDPO8.6683.7084.009
Post-refinement Edu-RMIdealPO8.8513.9014.097

Refining Edu-RM improves all three aggregate scores for both RL methods, while IdealPO outperforms GDPO under both RM conditions. Thus, the shared-RM comparison isolates the choice of RL method, and the pre-/post-refinement comparison supports the value of improved reward supervision. We will include these results in the revised manuscript.

The original manuscript also provides complementary evidence about joint optimization. In Table 1, both experts and Claude rate EduAlign higher than GDPO in every H/P/C dimension at both scales. Section 4.6 and Fig. 3 track the 7B training rewards: GDPO trails GRPO on Creativity during the middle and later stages, whereas IdealPO exceeds both. IdealPO has the highest endpoint reward and final-ten-step mean in all three dimensions. Its gains are more pronounced for Creativity and Personalization, while Helpfulness remains close and slightly higher. These trajectories characterize optimization under Edu-RM and complement the expert and public-task evaluations.

IdealPO’s mathematical construction can apply to other multidimensional reward settings. The empirical contribution established here is its effectiveness for HPC educational alignment, evaluated at two policy scales, across multiple educational tasks, and with component ablations and two reward-model conditions.

AuthorsOpenReview ↗

Response to Reviewer dQSu (3/3)

Question 1

For the results in Table 1 and Table 2, are the baselines algorithms trained under the same reward model as EduAlign?

Response: Yes. All RL baselines use the same post-refinement Edu-RM as EduAlign, with the same scoring prompt and RM inference settings. PPO, DAPO, and GRPO use the arithmetic mean of the three scores. GDPO uses dimension-wise reward normalization, and IdealPO uses IMC and CCW. We will explicitly state these shared settings and reward definitions in the revised experimental setup.

Question 2

How is the reward model trained? Are the rubrics just used to get the final vector of rewards in the supervision data or are they kept in the training of the reward model? And how large is the reward model?

Response: Edu-RM is a 32B model, and the unified H/P/C rubric is retained in its training and scoring inputs. These settings are described in Section 3.1, Appendix A.3 (Reward Model Prompts and Score Interface), and Appendix B (Training Details and Configuration) of the original manuscript.

Model and training. Edu-RM is initialized from Qwen2.5-32B-Instruct and trained by supervised fine-tuning for two epochs, with a learning rate of 1×10⁻⁵ and a global batch size of 256. The training set contains 12,390 synthesized examples and 15,000 low-quality negative examples, totaling 27,390. Training uses cross-entropy on the output score tokens to predict a serialized H/P/C score vector. The same 32B Edu-RM provides rewards for both 7B and 32B policies.

Role of the rubric. Section 3.1 and Appendix B.1 distinguish two roles: scenario-specific rubrics produce the supervision labels during synthesis, while the unified H/P/C rubric is supplied alongside each question–response pair as the RM’s scoring instruction. Appendix A.3 specifies that the rubric placeholders are replaced with the complete corresponding criteria during both RM training and policy-rollout scoring. The question and response provide the context for applying these criteria. We will replace the older rubric version displayed in Appendix A.3 with the final version actually used throughout the experiments, so that the displayed criteria match this training and evaluation interface.

Reviewer_V5TtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript proposes a framework combining educational reward modeling with multi-objective reinforcement learning, which aims to improve several qualities of educational responses jointly.

Critical strengths

Strengths:

  1. Clear motivation. The study issues a gap between general-purpose answers and educationally appropriate assistance. Its three objectives include responsible guidance, adaptation to learner context, and support for exploration give the work a clear motivation.

  2. Detailed reward-modeling pipeline. The study sets up scoring rules for different subjects, grade levels, and tasks, and refine both the prompts and the rules used to score them. The paper also explains in detail how the data were made, labeled, and used to train the reward model.

  3. Interpretable optimization design. The paper provides a clear derivation of how IdealPO assigns credit to each response and attenuates conflicting contributions, making the method easier for readers to understand.

  4. Expert validation of the reward model. When comparing reward models trained on initial and refined data, the study holds data volume, negative examples, and training settings constant. On 500 expert-rated question-answer pairs whose questions were excluded from reward-model training, both Pearson and Spearman correlations improve across all three dimensions. These results support the value of the refinement process.

  5. Evaluation across scales and measures. The study evaluates both 7B and 32B models using expert ratings, an independent model judge, and three public educational benchmarks, allowing readers to see the overall gains and remaining trade-offs.

Critical weaknesses

Weakness:

  1. The study uses H/P/C framework, but it is not clearly whether covering tutoring skills such as diagnosing misconceptions and providing graduated hints, which is essential for education. Please clarify the rubric, or delimit the framework’s claims for tutoring tasks.

  2. The evaluation of is about response quality rather than learning outcomes, which is not sufficient for students’ ability improvement in real world. Suggest do real-world experiments.

  3. IdealPO treats distance from a shared five-point ceiling as remaining improvement potential. Lower creativity scores may due to stricter calibration or prompts with limited scope for creative responses. Suggest test sensitivity to dimension-specific attainable targets and calibrated reward scales to determine whether the reported gains depend on this assumption.

  4. The study illustrates the instruct backbones and rollout settings matched with GRPO/GDPO, but the PPO and DAPO comparisons lack equally detailed training protocols, which makes it hard for reader to reproduce. Please report their reward definitions, rollout counts, optimization settings, and computational budgets.

  5. The study does not show how much of the final improvement comes from the refined reward model versus IdealPO. Please compare pre- and post-refinement Edu-RM under both GDPO and IdealPO.

  6. Rigor. The evaluation protocol collected questions through interviews with K-12 students and selected 200 for evaluation. Such 200-sample selection looks arbitrary. Why not use all questions for evaluation? Moreover, 200 samples also look too limited to demonstrate the generalization ability of the proposed model. The five raters’ annotation (including researchers themselves) also raises significant concerns regarding fairness during the annotation process and ethic concerns. Finally, from EduBench (Xu et al., 2025), the authors use proportional random sampling to select 100 questions per task. The same concern also arises regarding the random sampling fairness/rationale and limitation of question number (100) per task.

Questions for authors

Please see my weakness part.

6 comments
AuthorsOpenReview ↗

Response to Reviewer V5Tt (1/3)

Thank you for recognizing the motivation, reward-modeling pipeline, interpretability of IdealPO, and expert validation. Following your suggestions, we have compared pre- and post-refinement Edu-RM under both GDPO and IdealPO and conducted an offline target-sensitivity analysis. Below, we address each of the six weaknesses and clarify the evaluation scope and protocols.

Weakness 1

The study uses H/P/C framework, but it is not clearly whether covering tutoring skills such as diagnosing misconceptions and providing graduated hints, which is essential for education. Please clarify the rubric, or delimit the framework’s claims for tutoring tasks.

Response: HPC specifies response-quality objectives that support educational assistance across tasks, including tutoring and conceptual understanding. As motivated in the Introduction, effective educational assistance requires more than supplying information: it should address learners’ needs and support their reasoning and exploration. The three objectives operationalize these requirements:

  • Helpfulness: helping the learner address their question or advance toward their goal through accurate, clear, and safe responses, with actionable guidance where appropriate and responsible handling of risks.
  • Personalization: adaptation to the learner’s stated knowledge, context, and constraints. For tutoring, this includes matching explanations and the specificity of hints to the learner’s demonstrated needs.
  • Creativity: relevant and useful exploration through alternative approaches, comparisons, analogies, and questions that encourage the learner to examine their reasoning.

These objectives provide complementary criteria for tutoring responses. Appendix A.3 presents the H/P/C scoring rubric. The original manuscript included an earlier version, which we will replace in the revised manuscript with the final version used consistently for Edu-RM training, rollout scoring, expert evaluation, and Claude evaluation.

Table R1. Educational tasks evaluated in the original manuscript.

BenchmarkEvaluated tasks
EduBenchIdea Provision (IP), Personalized Learning Support (PLS), Teaching Material Generation (TMG)
ELMESInterdisciplinary Lesson Plan Generation (ILPG), Contextual Question Generation (CQG), Knowledge Point Explanation (KPE)
EduValuesSubjective Analysis (SA)

Table 2 in the original manuscript reports the highest aggregate scores for EduAlign among the compared methods on all three benchmarks at both 7B and 32B. These results connect the alignment objectives to performance on educational tasks.

Weakness 2

The evaluation of is about response quality rather than learning outcomes, which is not sufficient for students’ ability improvement in real world. Suggest do real-world experiments.

Response: Our evaluation asks whether training improves responses along the three educational objectives and whether those improvements transfer to public educational tasks. Helpfulness, Personalization, and Creativity target useful assistance, adaptation to learner needs, and support for reasoning and exploration.

We evaluate this at two levels. First, education experts and Claude Sonnet 4.6 assess identical responses to the same 200 student-interview questions using the same rubric, with model identities hidden. Table 1 in the original manuscript shows that EduAlign receives the highest mean score in every dimension from both evaluators at both model scales.

Second, EduBench and ELMES assess educational task performance and instructional support quality using their own task-specific standards. Their tasks include personalized learning support, instructional material generation, knowledge explanation, contextual question generation, and lesson planning. Table 2 in the original manuscript reports the highest EduBench and ELMES aggregate scores for EduAlign among the compared methods at both scales.

These results support improvements in response quality and educational task performance. We have not yet conducted a real-world teaching intervention because of the organizational and implementation requirements. The original manuscript’s Conclusion and Ethics and Broader Impacts already distinguish response evaluation from classroom learning outcomes.

AuthorsOpenReview ↗

Response to Reviewer V5Tt (2/3)

Weakness 3

IdealPO treats distance from a shared five-point ceiling as remaining improvement potential. Lower creativity scores may due to stricter calibration or prompts with limited scope for creative responses. Suggest test sensitivity to dimension-specific attainable targets and calibrated reward scales to determine whether the reported gains depend on this assumption.

Response: The five-point scale was developed through discussions with education experts, with dimension-specific quality criteria and scoring anchors. A score of five denotes meeting the rubric’s high-quality requirements; the common ceiling is therefore a rubric-defined optimization target. Creativity is evaluated in context: relevant, useful exploration is rewarded, rather than novelty detached from the user’s goal. The 200 evaluation questions were selected to allow assessment of all three objectives. In Table 1 of the original manuscript, experts also assign relatively lower creativity scores to Instruct and higher scores to EduAlign, supporting the effectiveness of training on this dimension.

We additionally tested how changing individual targets affects IdealPO’s advantage assignment. We sampled 128 prompts from the RL question pool and generated eight responses per prompt with the original 7B Instruct model, yielding 1,024 responses. Keeping these responses and their post-refinement Edu-RM scores fixed, we lowered the Personalization or Creativity target from 5 to 4.5 and recomputed the advantages without updating policy parameters.

Table R2. Sensitivity of advantage assignment to dimension-specific targets.

Target (H, P, C)Mean within-group Spearman correlationAdvantage sign-flip rateRelative change in the advantage vector
Original: (5, 5, 5)———
Lower P: (5, 4.5, 5)0.99130.6881%7.5785%
Lower C: (5, 5, 4.5)0.99250.4587%5.1659%

Advantage rankings remain highly consistent, with sign-flip rates below 0.7% under both perturbations. Magnitudes change, but the ordering and direction of the training signal are largely preserved on these sampled responses. This supports local stability of advantage assignment under the tested target changes. The raw reward scale was held fixed, so this analysis concerns target sensitivity rather than reward calibration or end-to-end training under altered rewards.

Weakness 4

The study illustrates the instruct backbones and rollout settings matched with GRPO/GDPO, but the PPO and DAPO comparisons lack equally detailed training protocols, which makes it hard for reader to reproduce. Please report their reward definitions, rollout counts, optimization settings, and computational budgets.

Response: PPO and DAPO use the same Instruct initialization at each model scale, RL question pool, post-refinement Edu-RM, scoring prompt, and RM inference settings as the other RL methods. PPO, DAPO, and GRPO use the arithmetic mean of the three reward scores, (r_H + r_P + r_C) / 3. GDPO applies dimension-wise reward normalization, while IdealPO uses IMC and CCW.

PPO and DAPO also use the same update batch size as the other RL methods. Section 4.1 and the training appendix of the original manuscript already document the IdealPO configuration and the shared rollout and optimization settings for GRPO/GDPO: 40 rollout rounds, 128 prompts per round, eight responses per prompt, a global update batch size of 512, and two updates per round. We will supplement the training appendix with the actual PPO/DAPO rollout counts, optimization configurations, and computational budgets to make their protocols equally explicit.

AuthorsOpenReview ↗

Response to Reviewer V5Tt (3/3)

Weakness 5

The study does not show how much ... pre- and post-refinement Edu-RM under both GDPO and IdealPO.

Response: We have compared GDPO and IdealPO using both pre- and post-refinement Edu-RM at the 7B scale, as requested. The table combines the new pre-refinement RM runs with the post-refinement results in Table 2 of the original manuscript.

Table R3. Performance of GDPO and IdealPO with pre- and post-refinement Edu-RM (7B).

Reward modelPolicy / RL methodEduBenchELMESEduValues
—7B Instruct8.0933.2183.575
Pre-refinement Edu-RMGDPO8.3653.3173.549
Pre-refinement Edu-RMIdealPO8.7003.4473.584
Post-refinement Edu-RMGDPO8.6683.7084.009
Post-refinement Edu-RMIdealPO8.8513.9014.097

Both RL methods perform better with the post-refinement RM on all three benchmarks. This complements the improved RM–expert agreement reported in the original manuscript and shows that refined reward supervision also improves downstream policy performance.

IdealPO outperforms GDPO under both pre- and post-refinement RMs on all three aggregate metrics. Its advantage therefore persists with the weaker, pre-refinement reward model. Combining the refined RM with IdealPO yields the best results among the four training conditions, supporting the contributions of both reward modeling and policy optimization. We will include this comparison in the revised manuscript.

Weakness 6

Rigor. The evaluation protocol collected questions ... The same concern also arises regarding the random sampling fairness/rationale and limitation of question number (100) per task.

Response: We clarify the question selection, annotator independence, ethics, and benchmark sampling separately.

Selection of the 200 questions. The original interviews were conversations organized by the school in collaboration with partner primary and secondary schools, rather than an existing collection of independent questions. Ten experts in intelligent education extracted important questions, removed semantically duplicate or highly similar content, and selected 200 questions suitable for assessing all three objectives. These questions were excluded from model training. This process focuses evaluation on distinct student needs rather than repeatedly counting similar interview content. The original appendix reports the category distribution; we will supplement the manuscript with the extraction and selection procedure. Generalization across tasks is assessed additionally through the three public benchmarks.

Annotator independence and evaluation protocol. None of the five raters was a paper author. They were teachers and doctoral students specializing in intelligent education. As described in Section 4.2 of the original manuscript, the same pair of raters evaluates all methods’ responses to a given question, sees only the question and response without model identity, and provides dimension-wise ratings that are averaged first across the two raters and then across questions. Claude Sonnet 4.6 evaluates the identical questions and responses using the same rubric.

Ethics. The original manuscript already addresses the ethical procedures in the Ethics and Broader Impacts section, including participant consent, project-supported compensation, school-level review for expert interviews and ratings, and the absence of recorded identifying information. Student interviews were organized by the school in collaboration with partner primary and secondary schools, with participant consent.

EduBench sampling and evaluation coverage. Following EduBench’s task-generation setting, we first generated candidate evaluation questions for IP, PLS, and TMG, then used stratified random sampling to select 100 questions per task, for 300 questions in total. Sampling aimed to preserve the candidate pool’s distribution across subjects, educational stages, and applicable question types. IP was stratified by subject, educational stage, and question type; PLS by subject and educational level; and TMG by subject, educational level, and question type. The IP field labeled “difficulty” corresponds to educational stage in these data. There is a comparable evaluation precedent within EduBench: its original study reports a test set of 198 questions across nine tasks, with 99 Chinese and 99 English questions and coverage of different educational sub-contexts. Our evaluation uses 100 questions for each of the three selected tasks and stratifies sampling to retain category coverage. After sampling, we fixed the question set: all methods answer the same questions and are evaluated with the same task rubric and scoring settings. We also use all original items in the selected ELMES tasks (ILPG, CQG, KPE) and all Subjective Analysis items in EduValues.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.