- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates a limitation of training tutor LLMs with assistant LLMs as simulated students. These students are overly cooperative and remain engaged throughout the conversation, unlike real learners who may lose interest or leave. The authors show that a tutor trained in this setting is more likely to reveal solutions when its cooperative simulated student is replaced with a user model that can disengage. They propose EngagementRL, which trains tutors against existing user models capable of ending conversations. The method combines an engagement reward based on the student’s probability of leaving with a learning reward assessed by an LLM judge, while penalizing solution leakage. The authors train two tutor models, UserLM-tutor and OSIM-tutor. Experiments in simulated math tutoring show that both models leak fewer solutions and achieve higher judged learning scores than prior tutor models.
Critical strengths
-
The paper shows that a tutor trained with consistently cooperative simulated students is more likely to give away solutions when interacting with less engaged students. Figure 2 demonstrates this failure in TutorRL. Although UserLM is an imperfect proxy for real students, this result exposes a concrete weakness in maintaining pedagogical behavior when students participate less actively or may leave the conversation. This finding provides a compelling motivation for incorporating student engagement into tutor training.
-
The method combines engagement and learning rewards to encourage continued student participation alongside learning progress. The ablations show that the combined objective achieves higher judged learning than either reward alone in both student settings, while also improving retention in the UserLM setting.
-
The engagement reward is inexpensive to obtain because it uses the student simulator’s probability of ending the conversation without requiring an additional judge. This provides a practical way to incorporate student engagement into tutor training.
Critical weaknesses
-
The experiments do not sufficiently separate the contribution of the student simulator from the proposed training method. There is no matched comparison using an assistant model as the student with the same combined reward and optimization procedure. It therefore remains unclear how much additional benefit comes from using user models. This limits the evidence that replacing the student simulator is a key factor in improving tutor robustness to student disengagement.
-
The evaluation metrics that the author propose have several limitations. Quit rate does not distinguish students who leave after learning from those who leave because they lose interest. Engagement and learning are evaluated using similar procedures and some scoring criteria overlap, particularly cognitive engagement and understanding. Whether these metrics adequately distinguish engagement from learning therefore needs further validation. Besides, only the engagement-related metrics were compared with human annotations, and the correlations were weak. The evaluation could also include some other objective measures. For example, solve rate can be affected by solution leakage, but it would provide additional evidence of whether higher learning scores reflect better performance in solving problems.
Questions for authors
-
Do you have a matched comparison between user models and assistant models as students, using the same training setting and optimization procedure? What additional benefit does using a user model provide under these conditions?
-
How is misconception repair scored when the student has no initial errors or misconceptions to correct?
-
Beyond the proposed metrics, is there other objectively verifiable evidence of student learning? Does this evidence support the proposed method’s improvements over the baselines?
-
In Table 2, the combined objective achieves higher judged engagement than engagement-only training in both student settings. Which dimensions account for this improvement? Could you explain the OSIM results, where the combined objective achieves higher judged engagement but also a higher quit rate compared with engagement-only training?
4 comments
Critical Weaknesses clarifications
-
Quit rate: leaving after learning vs. giving up. (See Q4 table) As suggested, we now split every early quit by whether the student had already solved the problem. Against TutorRL-7B, far fewer students leave our tutor unfinished (UserLM: 4% vs. 25%; OSIM: 10% vs. 13%). Our extra OSIM quits are all students who had already finished, since OSIM tends to leave once it is done. (Appendix N, Table 18)
-
Overlap between engagement and learning metrics. We agree that cognitive engagement and understanding are related. They are scored in separate judge calls with separate rubrics: cognitive engagement rates depth of processing regardless of correctness, while learning credits only correct, student-produced work. Each score also has two dimensions that ask about different things (behavioral and affective engagement; solution progress and misconception repair). With UserLM, affective engagement correlates only 0.21 per turn with understanding, against 0.43 for cognitive engagement. The two metrics also rank tutors differently: with UserLM, the engagement-only tutor scores far higher on engagement than the learning-only tutor (0.483 vs. 0.398) but not on learning (0.100 vs. 0.106). With OSIM, learning-only training gives much higher learning (0.374 vs. 0.151) but also more quits (0.47 vs. 0.19) and more leaks (0.175 vs. 0.021). If the metrics measured the same thing, they would not separate like this. Table 2; Appendix B (rubrics)
-
Human evaluation, covering learning as well as engagement. We ran a blind pairwise study on Prolific. 20 raters compared our tutor with TutorRL-7B on 100 random test problems, both with the same UserLM student. Raters preferred our tutor for both engagement and learning, and found that TutorRL-7B gives away the answer more often (all p ≤ 0.02). (Section 5, Table 3a; Appendix I, Table 12)
Ours TutorRL-7B Same p Keeps student engaged 54% 26% 20% 0.002 Helps student learn 56% 27% 17% 0.002 Overall preference 64% 36% – 0.007 Gives away the answer ↓ 21% 35% – ≤ 0.02
As we approach the end of the discussion period, we would appreciate any last comments, feedback or questions. We have tried to fully address all of the reviewer's concerns and questions. We have added extensive experiments and details to the revised paper. Please let us know if our responses are satisfactory, and we are happy to provide more details or clarifications if necessary.
Thanks for your response! Your clarification has addressed most of my concerns, and the experiments are now more comprehensive. I have updated my score to 3.
Response to questions
We thank the reviewer for the detailed questions. All new results are in the updated paper, and each point below lists where to find them. Happy to discuss more details or add experiments if necessary.
Q1: What does a user model add over an assistant model playing the student? Our main baseline already answers this. TutorRL trains its tutor against an assistant LLM prompted to act as a student, which is the setup the reviewer describes. As we discuss in the paper, assistant models are built to be helpful and responsive, so as students they almost never lose interest or leave, and they answer every tutor message in good faith. A tutor trained against such a student never learns to handle a student who is bored, frustrated or ready to give up. Often its fastest route to reward is simply to give the answer away. This is why we train against user models built to behave like real people, including disengaging when the tutoring is poor.
Q2: Misconception repair when the student made no mistake. In the original rubric, such a turn effectively scores 0 on this dimension, and the case is common (35–64% of turns). We rescored all test dialogues with an added "not applicable" option and left it out of those turns' average. All scores rise, but the ranking barely changes (rank correlation 0.98 with UserLM, 1.00 with OSIM), and our tutor stays well ahead of TutorRL-7B. (Appendix L, Table 16)
Q3: Objective evidence of learning. We add a judge-free solve-rate measure: a dialogue counts if a student message contains the correct final answer before any tutor message states it. With OSIM, students reach the answer on their own far more often with our tutor (33.2% vs. 21.2%), and our tutor states the answer less often. With UserLM the gap is small, because UserLM rarely writes a final answer with any tutor, but our tutor states the answer about half as often. (Appendix M, Table 17)
Q4: Which engagement dimensions improve, and why OSIM quits more than with engagement-only training. The gain comes mainly from cognitive (+0.80) and behavioral (+0.69) engagement, that is, the student actually working on the problem. The engagement-only tutor keeps OSIM talking without making it work; the learning reward adds that work. The higher quit rate comes almost entirely from students who solved the problem and then left (25% vs. 12%), while students leaving unfinished change little (10% vs. 7%). Students with the combined tutor reach the answer on their own twice as often, and with turns after a quit counted as zero, it still has higher engagement. (Appendix O, Table 19)