- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces CogRaise, a 4B LLM tutor that connects cognitive-process diagnosis with instructional guidance using the 19 processes in revised Bloom’s taxonomy. The framework combines synthetic tutoring trajectories for supervised fine-tuning, a gated multi-head reward model, and GRPO with a cognitive-advancement reward favoring responses targeting the next process in an imposed progression. The authors also introduce CogEdu-Bench, an 849-instance benchmark evaluating cognitive diagnosis, pedagogical decisions, and tutoring quality. CogRaise improves conditional fine-grained diagnosis QWK over its SFT initialization, while achieving smaller changes in instructional decisions and broader tutoring performance.
Critical strengths
-
The paper addresses an important distinction between producing generally helpful tutoring responses and selecting instruction appropriate to a learner’s demonstrated cognitive processing.
-
The work connects theory-guided data synthesis, supervised initialization, reward modeling, reinforcement learning, and evaluation.
-
The benchmark includes independent expert audits of cognitive labels and decisions, with reported Fleiss’ κ values of 0.806 and 0.817, respectively. The automatic response evaluator is also validated against human judgments.
Critical weaknesses
-
Minor QWK is evaluated only on instances where the broad cognitive category is correctly predicted. Appendix G.1 reports that CogRaise’s score uses just 191 of 849 instances, and the eligible subset differs across models. Consequently, its higher conditional QWK cannot establish superior diagnosis across the complete 19-class space. Excluding unparseable diagnosis outputs introduces further coverage differences. Full-set accuracy, macro-F1, confusion matrices, and valid-output rates are needed.
-
The framework assumes a fixed sequence across all 19 processes, although the taxonomy does not prescribe that sequence. This assumption governs both training rewards and benchmark targets, creating a risk that evaluation primarily measures conformity to the authors’ chosen ordering. Comparisons with branching prerequisites, flexible targets, or expert-selected instructional paths would strengthen the formulation.
-
A classifier identifies the process addressed by the tutor’s response; it does not establish whether the learner subsequently acquires that capability. The supporting analysis uses the same classifier employed for reward computation and shows only four additional immediate-next-level responses among 96 matched queries. Independent human assessment and closed-loop learner evaluation are needed.
-
The reward classifier’s architecture, training data, held-out performance, and confidence calibration are insufficiently documented. Beyond SFT, decision accuracy improves by approximately nine examples out of 849, while tutoring quality remains essentially unchanged. Statistical uncertainty is therefore important.
-
ACT already combines pedagogical taxonomies, a 4B model, SFT, and taxonomy-aligned GRPO. MHPO combines turn-level pedagogical quality with trajectory-level effectiveness. CogRaise should more clearly demonstrate what its specific cognitive representation and reward add beyond these approaches.
Questions for authors
-
Does CogRaise improve full-set cognitive diagnosis? Please provide 19-class accuracy, macro-F1, prediction-validity rates, and comparisons on a shared evaluation subset, given that the headline conditional score covers only 191 instances.
-
What is the independent contribution of the advancement reward? Can you provide full-minus-one ablations, multiple training seeds, and independent human evaluation of the targeted cognitive processes?
8 comments
Question 2: What is the independent contribution of the advancement reward? Can you provide full-minus-one ablations, multiple training seeds, and independent human evaluation of the targeted cognitive processes?
Response: Thank you for your meaningful concern. To address this concern in Q2, we provide minus-one ablations, multiple training seeds and independent human evaluation.
A. Minus-one ablations The results are reported newly added ablation table below. The Radv uses the frozen cognitive level classifier to firstly identify the cognitive process targeted by a tutor response, and then evaluates its offset relative to the learner's current level. Radv provides an explicit optimization signal for both the direction and the distance of cognitive progression.
| Model | RM | Aux. | Major QWK↑ | Minor QWK↑ | Decision ACC↑ | Tutoring Quality↑ | |
|---|---|---|---|---|---|---|---|
| w/o Aux. | ✓ | ✓ | – | 0.2751 | 0.3485 | 0.6890 | 0.5600 |
| w/o | ✓ | – | ✓ | 0.2969 | 0.4736 | 0.6796 | 0.5596 |
| w/o RM | – | ✓ | ✓ | 0.2736 | 0.3141 | 0.7032 | 0.5663 |
| CogRaise | ✓ | ✓ | ✓ | 0.2875 | 0.4940 | 0.6961 | 0.5686 |
Decision Accuracy measures whether the model can provide correct direction based on the learner’s current cognitive state, and this metric is mainly influence by Rdav. Compared to CogRaise, removing Radv resulted in a diagnostic accuracy decrease from 0.6961 to 0.6796. After removing RM and AUX, the decision accuracy was 0.7032 and 0.6890, respectively, which was roughly the same as the 0.6961 of the complete model (RM mainly strengthened the matching of 19 cognitive processes; AUX mainly assists in the organization and structural optimization of coaching responses.
However, although Radv can affect the specific content and corresponding cognitive processes of tutor responses, RM and Aux mainly undertake the functions of level diagnosis and organizational optimization, and influence Minor QWK. When RM and AUX were removed, Minor QWK was 0.3141 and 0.3485, respectively, lower than the full model’s 0.4940.
Overall, the additional ablation experiment directly evaluated the contribution of individual rewards to the overall goal. They support our core design philosophy: RM provides fine-grained, multidimensional judgments on tutoring responses, Radv clearly guides cognitive progress, perceives cognitive control direction and distance, and Aux supports response organization. Their combination creates a more balanced cognitive awareness coaching policy than relying on any single reward component.
B. Multiple training seeds Combine with the result of minus-one ablations, to further verify the stability of the experimental results (as shown below table), we conducted three independent trainings on CogRaise and w/o Radv using random seeds 43, 44, and 45, reported the mean and sample standard deviation of each indicator, and compared the change of Decision Accuracy (the impact of Radv on Decision Accuracy has been analyzed above).
| Model / Seed | Major QWK↑ | Minor QWK↑ | Decision ACC↑ | Tutoring Quality↑ |
|---|---|---|---|---|
| CogRaise (Full) · 43 | 0.2683 | 0.3271 | 0.6855 | 0.5628 |
| CogRaise (Full) · 44 | 0.2811 | 0.3363 | 0.7008 | 0.5678 |
| CogRaise (Full) · 45 | 0.2634 | 0.4246 | 0.6949 | 0.5633 |
| CogRaise (Full) · Mean ± SD | 0.2709 ± 0.0091 | 0.3627 ± 0.0538 | 0.6937 ± 0.0077 | 0.5646 ± 0.0028 |
| w/o Radv · 43 | 0.2865 | 0.5225 | 0.6949 | 0.5567 |
| w/o Radv · 44 | 0.2818 | 0.3601 | 0.6879 | 0.5630 |
| w/o Radv · 45 | 0.2728 | 0.4081 | 0.6879 | 0.5590 |
| w/o Radv · Mean ± SD | 0.2804 ± 0.0070 | 0.4302 ± 0.0834 | 0.6902 ± 0.0040 | 0.5596 ± 0.0032 |
According to the result, CogRaise showed consistent advantages in cognitive advancement related evaluations: under three random seeds, CogRaise’s Decision Accuracy steadily higher than w/o Radv. These results further support the positive contribution of Radv in cognitive advancement and instructional decision-making.
Although there is fluctuation between different training seeds for the minor QWK. We guess that this is mainly related to the sensitivity of fine-grained consistency evaluation under different category distribution conditions, and does not necessarily mean that the overall training of the model is unstable. Specifically, the minor QWK is only calculated based on relatively limited effective evaluation samples, and the predicted changes may have a significant impact on observation consistency and expected consistency. Therefore, different training seeds may lead to different prediction results of the model near fine-grained cognitive category boundaries, resulting in significant fluctuations in the minor QWK. However, it cannot be ignored that the fluctuations of minor QWK may also come from other factors, and we will continue to explore them in future research.
Overall, the additional ablation experiment directly evaluated the contribution of individual rewards to the overall goal. They support our core design philosophy: RM provides fine-grained, multidimensional judgments on tutoring responses, Radv clearly guides cognitive progress, perceives cognitive control direction and distance, and Aux supports response organization. Their combination creates a more balanced cognitive awareness coaching policy than relying on any single reward component.
Dear Reviewer, thank you very much for providing us with valuable feedback. Please allow us to answer each question one by one.
Question 1: Minor QWK is evaluated only on instances where the broad cognitive category is correctly predicted. Appendix G.1 reports that CogRaise’s score uses just 191 of 849 instances, and the eligible subset differs across models. Consequently, its higher conditional QWK cannot establish superior diagnosis across the complete 19-class space. Excluding unparseable diagnosis outputs introduces further coverage differences. Full-set accuracy, macro-F1, confusion matrices, and valid-output rates are needed. The supplementary results are as follows:
Response: we recognize the importance of providing accuracy and other metrics.
19-class Accuracy:CogRaise achieved the highest overall accuracy across 19 levels (12.37%) in these baselines, indicating stronger precision matching performance in the entire evaluation set and achieving the maximum combined matching degree in major and minor diagnostics.
Validity rate:CogRaise also achieved the highest prediction efficiency (88.57%), indicating its strongest ability to generate effective cognitive process predictions. In summary, these results highlight the advantages of CogRaise in cognitive diagnosis and predictive effectiveness.
Macro-F1:We also consider Macro-F1, while Qwen3-32B and Qwen3-4B-Thinking-2507 achieved higher scores in Macro-F1. We analyzed the reason is that Macro-F1 assigns equal weights to each cognitive levels, emphasizing the balance of performance among different categories. More importantly, it does not take into account the distance between cognitive processes: For instance, suppose the actual cognitive level of the learner is 2.1. Some models might predict it to be 2.7, while CogRaise predicts it to be 2.2. Although both predictions are error, the former has a significantly serious deviation in terms of cognitive development, while the latter has a much smaller gap from the actual value. Thus, Macro-F1 may not consider the ordinal distance between the predicted level and the reference level, and treats both as error, confusing two adjacent cognitive processes is punished the same as confusing a process that is farther apart. Therefore, the macro F1 may not fully reflect the degree of match between the predicted results and the target cognitive processes within the defined cognitive development path.
Overall, CogRaise achieved better performance in accurately matching diagnostic and predictive effectiveness, while Macro-F1 provides supplementary evaluation for the recognition balance of 19 cognitive categories.