acceptodds
Under review as a conference paper at ICLR 2027

CogRaise: From Cognitive-Process Diagnosis to Progression-Aware LLM Tutoring

Abstract

Effective tutoring requires diagnosing how learners process knowledge and determining how instruction should support their cognitive progression. Yet existing LLM tutors emphasize observable pedagogical behaviors, with limited explicit modeling of learner cognition. To address this gap, we introduce CogRaise, a 4B cognition-aware tutor that operationalizes fine-grained cognitive processes as explicit training and optimization targets. First, we construct a Theory-Driven Dataset grounded in the 19 cognitive processes of Anderson’s revised Bloom taxonomy, comprising multi-turn diagnosis-intervention-response trajectories for SFT-based initialization and reward learning. Second, a multi-head reward model separately scores cognition-related tutoring criteria and general response quality. Third, GRPO incorporates a cognitive-advancement reward conditioned on the learner’s current cognitive level to encourage interventions supporting cognitive advancement. Finally, we introduce CogEdu-Bench, an 849-instance multi-subject benchmark evaluating fine-grained cognitive diagnosis, pedagogical decisions, and tutoring response quality. CogRaise improves fine-grained diagnosis QWK by 37.6% relative to its SFT initialization and enhances scaffolding on MathTutorBench, while general reasoning performance remains within 0.3 percentage points of that initialization on each of GSM8K, MATH500, and MMLUPro. These results suggest that established educational theory can be operationalized into structured supervision and reward signals to align LLM tutors with cognitive progression.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Cognition-Aware Tutoring, Reward Modeling, Reinforcement Learning, Benchmark Design

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gk1vOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CogRaise, a 4B LLM tutor that connects cognitive-process diagnosis with instructional guidance using the 19 processes in revised Bloom’s taxonomy. The framework combines synthetic tutoring trajectories for supervised fine-tuning, a gated multi-head reward model, and GRPO with a cognitive-advancement reward favoring responses targeting the next process in an imposed progression. The authors also introduce CogEdu-Bench, an 849-instance benchmark evaluating cognitive diagnosis, pedagogical decisions, and tutoring quality. CogRaise improves conditional fine-grained diagnosis QWK over its SFT initialization, while achieving smaller changes in instructional decisions and broader tutoring performance.

Critical strengths
  1. The paper addresses an important distinction between producing generally helpful tutoring responses and selecting instruction appropriate to a learner’s demonstrated cognitive processing.

  2. The work connects theory-guided data synthesis, supervised initialization, reward modeling, reinforcement learning, and evaluation.

  3. The benchmark includes independent expert audits of cognitive labels and decisions, with reported Fleiss’ κ values of 0.806 and 0.817, respectively. The automatic response evaluator is also validated against human judgments.

Critical weaknesses
  1. Minor QWK is evaluated only on instances where the broad cognitive category is correctly predicted. Appendix G.1 reports that CogRaise’s score uses just 191 of 849 instances, and the eligible subset differs across models. Consequently, its higher conditional QWK cannot establish superior diagnosis across the complete 19-class space. Excluding unparseable diagnosis outputs introduces further coverage differences. Full-set accuracy, macro-F1, confusion matrices, and valid-output rates are needed.

  2. The framework assumes a fixed sequence across all 19 processes, although the taxonomy does not prescribe that sequence. This assumption governs both training rewards and benchmark targets, creating a risk that evaluation primarily measures conformity to the authors’ chosen ordering. Comparisons with branching prerequisites, flexible targets, or expert-selected instructional paths would strengthen the formulation.

  3. A classifier identifies the process addressed by the tutor’s response; it does not establish whether the learner subsequently acquires that capability. The supporting analysis uses the same classifier employed for reward computation and shows only four additional immediate-next-level responses among 96 matched queries. Independent human assessment and closed-loop learner evaluation are needed.

  4. The reward classifier’s architecture, training data, held-out performance, and confidence calibration are insufficiently documented. Beyond SFT, decision accuracy improves by approximately nine examples out of 849, while tutoring quality remains essentially unchanged. Statistical uncertainty is therefore important.

  5. ACT already combines pedagogical taxonomies, a 4B model, SFT, and taxonomy-aligned GRPO. MHPO combines turn-level pedagogical quality with trajectory-level effectiveness. CogRaise should more clearly demonstrate what its specific cognitive representation and reward add beyond these approaches.

Questions for authors
  1. Does CogRaise improve full-set cognitive diagnosis? Please provide 19-class accuracy, macro-F1, prediction-validity rates, and comparisons on a shared evaluation subset, given that the headline conditional score covers only 191 instances.

  2. What is the independent contribution of the advancement reward? Can you provide full-minus-one ablations, multiple training seeds, and independent human evaluation of the targeted cognitive processes?

8 comments

Dear Reviewer, thank you very much for providing us with valuable feedback. Please allow us to answer each question one by one.

Question 1: Minor QWK is evaluated only on instances where the broad cognitive category is correctly predicted. Appendix G.1 reports that CogRaise’s score uses just 191 of 849 instances, and the eligible subset differs across models. Consequently, its higher conditional QWK cannot establish superior diagnosis across the complete 19-class space. Excluding unparseable diagnosis outputs introduces further coverage differences. Full-set accuracy, macro-F1, confusion matrices, and valid-output rates are needed. The supplementary results are as follows:

Response: we recognize the importance of providing accuracy and other metrics.

ModelAccuracy (%)Validity (%)Macro-F1 (%)
CogRaise (Full)12.3788.575.61
Qwen3-32B10.8464.907.99
Qwen3-4B-Thinking-2507 (Base)8.1348.296.11
TutorRL-7B-think5.6528.503.35
socraticlm4.8351.121.67
  1. 19-class Accuracy:CogRaise achieved the highest overall accuracy across 19 levels (12.37%) in these baselines, indicating stronger precision matching performance in the entire evaluation set and achieving the maximum combined matching degree in major and minor diagnostics.

  2. Validity rate:CogRaise also achieved the highest prediction efficiency (88.57%), indicating its strongest ability to generate effective cognitive process predictions. In summary, these results highlight the advantages of CogRaise in cognitive diagnosis and predictive effectiveness.

  3. Macro-F1:We also consider Macro-F1, while Qwen3-32B and Qwen3-4B-Thinking-2507 achieved higher scores in Macro-F1. We analyzed the reason is that Macro-F1 assigns equal weights to each cognitive levels, emphasizing the balance of performance among different categories. More importantly, it does not take into account the distance between cognitive processes: For instance, suppose the actual cognitive level of the learner is 2.1. Some models might predict it to be 2.7, while CogRaise predicts it to be 2.2. Although both predictions are error, the former has a significantly serious deviation in terms of cognitive development, while the latter has a much smaller gap from the actual value. Thus, Macro-F1 may not consider the ordinal distance between the predicted level and the reference level, and treats both as error, confusing two adjacent cognitive processes is punished the same as confusing a process that is farther apart. Therefore, the macro F1 may not fully reflect the degree of match between the predicted results and the target cognitive processes within the defined cognitive development path.

Overall, CogRaise achieved better performance in accurately matching diagnostic and predictive effectiveness, while Macro-F1 provides supplementary evaluation for the recognition balance of 19 cognitive categories.

Question 2: What is the independent contribution of the advancement reward? Can you provide full-minus-one ablations, multiple training seeds, and independent human evaluation of the targeted cognitive processes?

Response: Thank you for your meaningful concern. To address this concern in Q2, we provide minus-one ablations, multiple training seeds and independent human evaluation.

A. Minus-one ablations The results are reported newly added ablation table below. The Radv uses the frozen cognitive level classifier to firstly identify the cognitive process targeted by a tutor response, and then evaluates its offset relative to the learner's current level. Radv provides an explicit optimization signal for both the direction and the distance of cognitive progression.

ModelRMRadvR_{\mathrm{adv}}Aux.Major QWK↑Minor QWK↑Decision ACC↑Tutoring Quality↑
w/o Aux.✓✓–0.27510.34850.68900.5600
w/o RadvR_{\mathrm{adv}}✓–✓0.29690.47360.67960.5596
w/o RM–✓✓0.27360.31410.70320.5663
CogRaise✓✓✓0.28750.49400.69610.5686

Decision Accuracy measures whether the model can provide correct direction based on the learner’s current cognitive state, and this metric is mainly influence by Rdav. Compared to CogRaise, removing Radv resulted in a diagnostic accuracy decrease from 0.6961 to 0.6796. After removing RM and AUX, the decision accuracy was 0.7032 and 0.6890, respectively, which was roughly the same as the 0.6961 of the complete model (RM mainly strengthened the matching of 19 cognitive processes; AUX mainly assists in the organization and structural optimization of coaching responses.

However, although Radv can affect the specific content and corresponding cognitive processes of tutor responses, RM and Aux mainly undertake the functions of level diagnosis and organizational optimization, and influence Minor QWK. When RM and AUX were removed, Minor QWK was 0.3141 and 0.3485, respectively, lower than the full model’s 0.4940.

Overall, the additional ablation experiment directly evaluated the contribution of individual rewards to the overall goal. They support our core design philosophy: RM provides fine-grained, multidimensional judgments on tutoring responses, Radv clearly guides cognitive progress, perceives cognitive control direction and distance, and Aux supports response organization. Their combination creates a more balanced cognitive awareness coaching policy than relying on any single reward component.

B. Multiple training seeds Combine with the result of minus-one ablations, to further verify the stability of the experimental results (as shown below table), we conducted three independent trainings on CogRaise and w/o Radv using random seeds 43, 44, and 45, reported the mean and sample standard deviation of each indicator, and compared the change of Decision Accuracy (the impact of Radv on Decision Accuracy has been analyzed above).

Model / SeedMajor QWK↑Minor QWK↑Decision ACC↑Tutoring Quality↑
CogRaise (Full) · 430.26830.32710.68550.5628
CogRaise (Full) · 440.28110.33630.70080.5678
CogRaise (Full) · 450.26340.42460.69490.5633
CogRaise (Full) · Mean ± SD0.2709 ± 0.00910.3627 ± 0.05380.6937 ± 0.00770.5646 ± 0.0028
w/o Radv · 430.28650.52250.69490.5567
w/o Radv · 440.28180.36010.68790.5630
w/o Radv · 450.27280.40810.68790.5590
w/o Radv · Mean ± SD0.2804 ± 0.00700.4302 ± 0.08340.6902 ± 0.00400.5596 ± 0.0032

According to the result, CogRaise showed consistent advantages in cognitive advancement related evaluations: under three random seeds, CogRaise’s Decision Accuracy steadily higher than w/o Radv. These results further support the positive contribution of Radv in cognitive advancement and instructional decision-making.

Although there is fluctuation between different training seeds for the minor QWK. We guess that this is mainly related to the sensitivity of fine-grained consistency evaluation under different category distribution conditions, and does not necessarily mean that the overall training of the model is unstable. Specifically, the minor QWK is only calculated based on relatively limited effective evaluation samples, and the predicted changes may have a significant impact on observation consistency and expected consistency. Therefore, different training seeds may lead to different prediction results of the model near fine-grained cognitive category boundaries, resulting in significant fluctuations in the minor QWK. However, it cannot be ignored that the fluctuations of minor QWK may also come from other factors, and we will continue to explore them in future research.

Overall, the additional ablation experiment directly evaluated the contribution of individual rewards to the overall goal. They support our core design philosophy: RM provides fine-grained, multidimensional judgments on tutoring responses, Radv clearly guides cognitive progress, perceives cognitive control direction and distance, and Aux supports response organization. Their combination creates a more balanced cognitive awareness coaching policy than relying on any single reward component.

Reviewer_5p3rOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

The paper introduces CogRaise, an educational tutoring model built on Qwen. The tutor is trained to 1) diagnose a learner’s current cognitive level (19 fine-grained cognitive processes of Anderson’s revised Bloom taxonomy) and 2) respond in a way that could support learner’s cognitive advancement. The paper also introduces CogEdu-Bench, which includes over 800 instances built from C-Eval. It measures cognitive diagnosis, pedagogical decisionsand tutoring quality.

Critical strengths

The paper has a complete pipeline with careful evaluations and checks, from Bloom thoery to training data, reward design, and to a mathcing benchmark, which is a clear and systematic research paradiam. The benchmark checks are also solid, and could be potentially useful for the entire educational field.

Critical weaknesses
  1. My main concern is on the core idea of cognitive level and progression. The method treats the 19 processes as a learner state that moves up one step at a time, while the Bloom taxonomy does not describe the learners this way. It sorts different kinds of things tasks and is a very strong assumption for the entire method. Two concerns follow this: 1a. I wonder if this true---the diagonsis mostly reduces to checking the tutor’s previous question. By the update rule, c_t=g_{t-1} if the learner demonstrates the previous target, otherwise c_t=c_{t-1}. If the diagnosis becomes “did the learner answer the last question?”, then it says little about how the learner actually thinks. 1b. The tutor is rewarded for always moving to the next process, e.g., from 2.1 interpreting to 2.2 exemplifying. Is this actually a good tutoring?
  2. I wonder how much this pipeline tells about human learners, as every part of the pipeline is build with the same rule and with LLMs, e.g., the dialogues are llm-generated, the preference scores come from Qwen-Max, tutroing quality is scored by deepseek. Even inside this closed setup, if I read correctly, the results are relatively weak. Major QWK is slightly lower than SFT and far below Bemini, the gain in Minor QWK is measured only on ~25% instances where the major categori is correct.
Questions for authors

See weaknesses.

7 comments

Dear Reviewer, it is our honor that you read our paper patiently and carefully, we apologize for causing your confusion, could you please give us a chance to explain. This may take up your valuable time, but we highly cherish your review comments and have provided detailed explanations.

Weakness1: My main concern is on the core idea of cognitive level and progression. The method treats the 19 processes as a learner state that moves up one step at a time, while the Bloom taxonomy does not describe the learners this way. It sorts different kinds of things tasks and is a very strong assumption for the entire method.

Response: You mentioned that Bloom Taxonomy sorted different kinds of things tasks and did not describe the 19 processes as a learner state that moves up one step at a time. You are right, the original Bloom’s Taxonomy is frequently used to classify the cognitive objectives or task objectives[1].

1. After Bloom, Anderson thinks that the original Taxonomy only describe cognitive objective, but does not illustrate “How learners reach the objectives”. For this reason, Anderson extracts the verbs from the original knowledge dimensions of Bloom’s Taxonomy, use these verbs to represent cognitive processes[2].

2. Because cognitive process is the mental operation methods when learners process knowledge[3], Anderson retains the major levels (cognitive objectives), and adds the mental operation method (19 minor cognitive processes) that can achieve the objectives in the Revision Taxonomy[4].

3. Therefore, as shown in figure 3 in Appendix B.1, six major categories represent the cognitive objectives from shallow to deep, 19 minor levels are cognitive process for achieving objectives. For example, the objective is “3. Apply”, the learners should execute a procedure to a familiar task (3.1), and then implement the procedure to an unfamiliar task (3.2). After that, it could be diagnosed that the learner can carry out and use the procedure which is related to the knowledge what he has learned and he has reached “3. Apply” objectives. With the deepen of cognitive process, the cognitive level could be deepened as the same time.

[1]Cheng, Y., Cai, Y., Chen, H., Cai, Z., Wu, G., & Huang, J. (2021). A cognitive level evaluation method based on a deep neural network for online learning: From a Bloom's taxonomy of cognition objectives perspective. Frontiers in Psychology, 12, Article 661235.

[2]Krathwohl, D. R. (2002). A Revision of Bloom’s Taxonomy: An Overview. Theory Into Practice, 41(4), 212–218.

[3]Tudela, P. (2004). Cognitivism. In C. D. Spielberger (Ed.), Encyclopedia of applied psychology (pp. 393–401). Elsevier.

[4]Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives (Complete ed.). Longman.

1a in Weakness 1: 1a. I wonder if this true---the diagnosis mostly reduces to checking the tutor’s previous question. By the update rule, c_t=g_{t-1} if the learner demonstrates the previous target, otherwise c_t=c_{t-1}. If the diagnosis becomes “did the learner answer the last question?”, then it says little about how the learner actually thinks.

Response: Based on the analysis in weakness 1, the key that CogRaise can guide learners cognitive progression is to guide their mental operation move up step by step (If you have any questions about progressive development step by step, please feel free to ask me further questions. We look forward to having in-depth discussions with you).

1. Because of Anderson uses verbs represent cognitive process, the main task of CogRaise is guiding learners use next level’s verb process knowledge based on his current cognitive level .

2. Therefore, in the dataset (for example SFT data), nearly each tutor’s response contains a verb which is represent minor cognitive level. For a instance, “Compare the Industrial Revolution with French Revolution”. If the leaner can answer the question correctly, it means he could processing knowledge by using the tutor’s guided cognitive process, and he has reach that cognitive level, and then the tutor will guide him to next level (target level). If the learner’s answer is incorrect, it means that he cannot use the operation method successfully, the tutor will not advance, and stay at this level.

3. For the same instance, “Compare the Industrial Revolution with French Revolution” (2.6), the tutor may make such a response in two situations, one is learner reached “2.5 Inferring” successfully, tutor guide him progress to 2.6; other one is learner did not reach “2.6 Comparing” in the last turn of dialogue, the tutor choose to stay at 2.6 and give more guidance, so in this turn the diagnosis result is 2.5→2.6.

4. Overall, as you mentioned in 1a, the reason why it looks like “diagnosing whether a learner correctly answer the tutor’s question”, is that learners’ cognitive state is latent. According to Anderson’s revision of Bloom’s Taxonomy, CogRaise could diagnose “did learner use the cognitive process which I guided him to processing knowledge” to visualize his current cognitive level. Thus, tutor’s question and guidance are based on learner’s current cognitive level, we consider that it maybe a good tutoring. In the future investigation, we will try to verify the real effectiveness of CogRaise by empirical research.

5. Therefore, tutor’s (CogRaise) response contains three parts: Diagnosis result (current level and target level); Intervention scaffold (depends on target level); Socratic response (do not reveal answer; give real and correct knowledge; around learner’s personal preferences)

1b in Weakness 1: 1b. The tutor is rewarded for always moving to the next process, e.g., from 2.1 interpreting to 2.2 exemplifying. Is this actually a good tutoring?

Response: We try to analysis maybe you are worried about that the advancement reward may push tutor constantly move to the next process in order to achieve high scores, but ignoring the real state of learners. We used RM to aviod this. We design a multi-head reward model. It is designed to enable the tutor model to have the abilities of diagnosing the correct level and providing a scaffold that conforms to the learner’s cognitive process. If the tutor model diagnose incorrectly, it cannot earn score. Radv (advancement reward) does not act on the diagnostic stage, but on the intervention stage (after diagnosis), as a reward for distance and direction. Radv is designed to guide learners to advance only one level each time, avoiding the model guiding students to advance multiple levels (may cause learners’ cognitive load) or causing erroneous rollback.

Reviewer_yidHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CogRaise, a cognition-aware LLM tutor based on the fine-grained cognitive processes of Bloom's taxonomy. It constructs theory-driven tutoring data and fine-tunes a 4B language model through supervised fine-tuning and GRPO with a multi-head reward model and a cognitive-advancement reward. The paper also introduces CogEdu-Bench to evaluate cognitive diagnosis, pedagogical decisions, and tutoring quality. Experiments on several benchmarks demonstrate improvements in fine-grained cognitive diagnosis and competitive tutoring performance.

Critical strengths

1.The motivation is clear and interesting. Incorporating cognitive processes into LLM-based tutoring is a meaningful direction for personalized education.

  1. The proposed framework is relatively complete, including theory-driven data construction, reward modeling, and reinforcement learning for tutoring.

3.The experiments are relatively comprehensive, including comparisons with different LLMs, several benchmarks, ablation studies, and evaluation reliability analysis.

Critical weaknesses
  1. The paper organizes the 19 cognitive processes into a fixed progression order, where the tutor aims to guide the learner toward the next cognitive level. However, different learners may have different learning paths, and some cognitive processes may not strictly follow this order. THus the authors may consider a more flexible cognitive progression structure, such as a graph-based or learner-adaptive structure. It would be interesting to investigate whether this could further improve the tutoring performance.

  2. In Table 4, the auxiliary-reward-only variant already achieves a Minor QWK of 0.4851, which is close to the full model's 0.4940.The authors may provide more detailed ablations, such as removing individual rewards from the full reward function, to better demonstrate the contribution of the cognitive-advancement reward.

Questions for authors
  1. Minor QWK is evaluated only on samples with correctly predicted major cognitive categories. Could the authors also report the fine-grained diagnosis performance on all evaluation samples, such as the overall 19-class accuracy or macro-F1?

  2. The proposed model is mainly evaluated on predefined tutoring tasks and generated dialogues. It would be interesting to further investigate whether the improved cognitive diagnosis and tutoring decisions can lead to better learning outcomes in real multi-turn tutoring interactions.

  3. The evaluation prompt does not provide the complete descriptions of the 19 cognitive processes to the baseline models. Could the authors further investigate whether providing these descriptions or more in-context examples to the baselines would affect the comparison results?

7 comments

Dear Reviewer, thanks for your diligent efforts in reviewing our paper and for recognizing the contributions of our work. We sincerely apologize for any confusion caused. We earnestly request that you give us an opportunity to explain.

Question 1: Minor QWK is evaluated only on samples with correctly predicted major cognitive categories. Could the authors also report the fine-grained diagnosis performance on all evaluation samples, such as the overall 19-class accuracy or macro-F1?

Response: We are really grateful to you for pointing out this concern. To address this concern, we further calculated the overall accuracy and Macro-F1 for the 19 levels on the complete 849 evaluation samples. The supplementary results are as follows:

ModelAccuracy (%)Validity (%)Macro-F1 (%)
CogRaise (Full)12.3788.575.61
Qwen3-32B10.8464.907.99
Qwen3-4B-Thinking-2507 (Base)8.1348.296.11
TutorRL-7B-think5.6528.503.35
socraticlm4.8351.121.67
  1. 19-class Accuracy:The results show that CogRaise achieves the highest overall 19-class Accuracy among these baselines (12.37%), indicating stronger exact-match performance across the full evaluation set, and achieving the highest combined mathching degree in both major and minor diagnoses.
  2. Validity rate:CogRaise also achieves the highest prediction-validity rate (88.57%), indicating its the strongest ability to produce valid cognitive-process predictions. Taken together, these results highlight CogRaise’s advantages in both cognitive diagnosis and prediction validity.
  3. Macro-F1:We also consider Macro-F1, while Qwen3-32B and Qwen3-4B-Thinking-2507 achieved higher scores in Macro-F1. We analyzed the reason is that Macro-F1 assigns equal weights to each cognitive levels, emphasizing the balance of performance among different categories. More importantly, it does not take into account the distance between cognitive processes: For instance, suppose the actual cognitive level of the learner is 2.1. Some models might predict it to be 2.7, while CogRaise predicts it to be 2.2. Although both predictions are error, the former has a significantly serious deviation in terms of cognitive development, while the latter has a much smaller gap from the actual value. Thus, Macro-F1 may not consider the ordinal distance between the predicted level and the reference level, and treats both as error, confusing two adjacent cognitive processes is punished the same as confusing a process that is farther apart. Therefore, the macro F1 may not fully reflect the degree of match between the predicted results and the target cognitive processes within the defined cognitive development path.

Overall, CogRaise achieved better performance in accurately matching diagnostic and predictive effectiveness, while Macro-F1 provides supplementary evaluation for the recognition balance of 19 cognitive categories.

Question 2:The proposed model is mainly evaluated on predefined tutoring tasks and generated dialogues. It would be interesting to further investigate whether the improved cognitive diagnosis and tutoring decisions can lead to better learning outcomes in real multi-turn tutoring interactions.

Response:Dear reviewer, we agree with your suggestion. It really necessary to further investigate whether CogRaise can lead to better learning outcomes in real multi-turn tutoring interactions. We try to guess you are concerned about two questions: a. The proposed model is mainly evaluated on predefined tutoring tasks and generated dialogues, can it be applied to real interaction between human-learner and tutor LLM? b. If CogRaise could be applied, campared with other models, to what extend it is effectiveness, can it promote learners cognitive develop?

Starting from these two speculations, we did a minor experiment. But considering that the rebuttle period is limited, please forgive us we have only two participants and only campare the effectiveness between CogRaise and GPT-6. Firstly, we give the same prompt to the two model, the content of prompt is in Figure 8 Appendix G.1, and require GPT provide diagnostic results for current level and target level when outputting answers. Second, we recruit two participants with similar exam score in the same class, but we need to admit that we did not do prior knowledge test beacuse of the limited sample size, instead of selecting similar-grades participants. Third, one participant communicates with CogRaise, the other on interacts with GPT-6. The result shows (Figure 11 and 12 in Appendix J) that a. CogRaise could be used in real human-computer interaction dialogue, and learners’ cognition presents a state of development towards higher levels; b. GPT-6 has a little cognitive guidance ability, but there are some mistakes in diagnosing process and effective intevention, the guiding cannot processing stability.

However, the result is based on only two participants, and did not strictly obey the rules of experiment, the result could only prove that CogRaise could be uesd in real human-computer interaction and can lead participant’s cognition develop towards higher level compared with GPT-6, but it is so weak to support the effectiveness of CogRaise. We need to do further comprehensive empirical research on sufficient sample size and compared with various models, and the design of experiment need to be elaborate.

Question 3: The evaluation prompt does not provide the complete descriptions of the 19 cognitive processes to the baseline models. Could the authors further investigate whether providing these descriptions or more in-context examples to the baselines would affect the comparison results?

Response:Thank you for pointing out the possible limitations in the assessment. To address this issue, we have supplemented the complete definitions of all 19 cognitive processes to the baseline and CogRaise (the content of prompt is in Figure 8 Appendix G.1), and re-evaluated .The updated results are as follows:

ModelMajor QWK↑Minor QWK↑Decision ACC↑Tutoring Quality↑
socraticlm0.05180.00170.57360.3346
TutorRL-7B-think0.75330.26490.61250.4859
Qwen3-4B-Thinking-2507 (Base)0.57120.41330.65960.4863
CogRaise (Full)0.49940.46000.69690.5529

1. Before analyzing the result, could you please give us a a little time to explain the motivation of we creating CogRaise?

Our design was inspired by two observations: A. We conducted preliminary experiments before our research to explore whether the method based on prompt engineering could enhance the cognitive diagnostic capabilities of existing large language models. we found that the larger the number of model parameters, the higher the diagnostic accuracy of the major level, especially when provide detailed-description of 19 cognitive process. However, despite giving prompt, the accuracy of diagnosing minor levels is not high, the propmt hardly improve the fine-grained diagnosis. B. The occasional accurate 19 fine-grained predictions did not stable and hardly translate into reliable tutoring. From the two reasons, our research (especially RM and RL) focus more on improving the diagnostic accuracy and intervention quality of minor levels, so in the terms of major level, we maintain the ability of base model.

2. Therefore, as shown in the figure, CogRaise achieved the highest Minor QWK (0.4600) among the compared models, indicating that it has a stronger match with the fine-grained cognition. Besides, CogRaise also achieved the highest decision accuracy rate (0.6969) and Task2 score (0.5529), indicating that while it diagnoses fine-grained cognition, its teaching decision performance is better, and the quality of tutoring responses remains stable. Furthermore, some models (like TutoringRL 7B) gain higher score on major level further proves the credibility of our previous research and affirms our research value.

Comments on the paper

2 comments

Dear Reviewers,

The authors have posted their responses to the reviews. Please go through their rebuttal at your earliest convenience, and engage in a discussion or leave any follow-up comments.

The discussion period ends on November 18.

Thanks,

Area Chair

AuthorsOpenReview ↗

Summary of Rebuttal Discussion

Dear Reviewers, Area Chairs and Program Committee Members,

We sincerely express our gratitude to you for your great efforts in reviewing our paper, as well as the detailed comments and supportive review, the below is a concise summary.

Firstly, we thank the reviewers for appreciating that a. The work is driven by a clear and interesting motivation (yidH); b. The proposed framework forms a relatively completed system (yidH, gk1v, 5p3r); c. The experimental evaluations are relatively well-rounded (yidH); d. The CogEdu-Bench is relatively valuable (5p3r, gk1v.)

Moreover, we sincerely cherish the insightful and important suggestions that the reviewers given us, we have provided detailed response to each question. We have made a number of changes to address these suggestions and concerns. A summary of the changes below.

  • We provided 19-class diagnostic results on the entire evaluation samples, including accuracy, prediction-validity rates, and macro F1. (summaried in Sec. 4.2 with details in Appendix G.1 Table 14).

  • We conducted a leave-one-out ablation (Table 5 in Sec. 4.3) and reported the results of w/o Radv and CogRaise in three training seeds to evaluate robustness ( Appendix D.3 Table 9).

  • We campared the evaluation (Table 13 in Appendix G.1) between CogRaise and baseline models when both provided complete description of 19 cognitive processes (Figure 8 in Appendix G.1)

  • We conducted a small learner closed-loop assessment to verify the true usage of CogRaise (Figure 11 in Appendix J).

  • We explained the rationale of organizing 19 cognitive processes in fixed sequence from both theorical and practical perspectives (summaried in Sec.3.1).

In summary, these changes now provided even stronger support for CogRaise's performance, further confirm that the contributions of each reward component. We also revised the manuscript (with highlighted blue) to present our method more clearly in response to reviewers' feedback.

Finally, the main contribution of CogRaise is addressing the gap between general tutoring responses and selecting instructional guidance that matches the leaners’ cognitive processes. To bridge this gap, our works are as follows:

i) We construct a Theory-Driven Dataset based on the 19 cognitive processes, which forms diagnosis-intervention-response trajectories through SFT initialization. ii) We train a multi-head reward model to enhance the accuracy of cognitive process diagnosis. iii) We combine the advancement reward into GRPO to constraint the direction and distance of target level movement in intervention stage. iv) We introduce CogEdu-Bench, an 849-instance multisubject benchmark evaluating cognitive diagnosis, pedagogical decisions, and tutoring response quality. In conclusion, we speculate that with certain cognitive guidance abilities, tutor models (CogRaise) may guide learners processing knowledge more personally and effectively.

At the end, please let us express our gratitude to you for your careful consideration and constructive discussion again.

Best,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.