acceptodds
Under review as a conference paper at ICLR 2027

Co-evolving Tutor and Student Simulator for Dialogue Tutoring

Abstract

Optimizing large language models for dialogue tutoring requires feedback about how guidance changes students' reasoning and problem-solving progress. An objective inspired by the zone of proximal development (ZPD) favors guidance associated with intermediate predicted response correctness. Our analysis identifies the largest simulation errors at intermediate reference correctness. As teaching policies evolve, student models must continue to represent the mistakes and incomplete reasoning that instructional guidance is intended to address. We propose COMET, an online reinforcement-learning framework that couples instructional utility optimization with student-response modeling through policy-induced dialogue histories. Simulator validation performance, in turn, controls the granularity of the Tutor's ZPD reward. The simulator objective combines cognitive rewards with reference-based behavioral constraints, jointly targeting reasoning quality and agreement with recorded student responses. A shared judge-anchored evaluator refines its rubric and distills rubric-based judgments into a prediction head for efficient cognitive scoring. COMET achieves 85.3% Clean Success@15. With an independent frozen student and a shared answer-disclosure guard, it improves Success@15 over a same-base prompted tutor by 20.1 percentage points. These findings demonstrate the effectiveness of tutor–student co-evolution for learning tutoring policies through self-play, with instructional gains that transfer to an independent student simulator.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Dialogue tutoring, student simulation, tutor–student co-evolution, multi-turn reinforcement learning, dialogue knowledge tracing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PhSBOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper is concerned with LLM tutoring, where an LLM is trained for tutoring via an RL loop with a student simulated by another LLM. While prior work has kept this student simulator fixed, this work trains both the tutor and student together and does not keep it fixed. The authors show favorable results in the evaluation.

Critical strengths
  • To me the overall promise makes a lot of sense and, on a methodological level, the execution also seems sound as described in Alg. 1. The results are also favorable, though there are concerns that I outline below.
Critical weaknesses
  • My main concern is about the evaluation. This concern might also stem partially from not being able to clearly understand the set-up despite thorough reading. The way I understand 5.1 is that all baselines are evaluated against a frozen student simulator while the presented method is compared against the jointly-trained student (cf. the writing in L326-330). This would mean that these numbers are not comparable; a more proper evaluation would be to use a held-out set of students that is the same for each method. Similarly, it is unclear how to relate 5.2 to 5.1 and which model was used for evaluation in 5.3: was it again the respective jointly-trained student? If so, these numbers are again not directly comparable and it is unclear if the student training actually helps. This to me is a critical flaw that needs to be addressed before this work can be accepted. If it is due to my misunderstanding, I would like to ask the authors to clarify it both to me and in the writing. Overall, it would help to describe the method and experiments more clearly.

  • A less significant concern is that the paper largely shares the methodology with Dinucu-Jianu et al. 2025, for example, in terms of rewarding the expected solution rate but does not properly attribute this. A comparison seems missing to me but might be contained in 5.3, though this needs clarification as written above.

Questions for authors
  • Please address my concerns above about the evaluation protocol, these are critical for my assessment.

  • Could you please justify how the utility in L236 models ZPD? Did you try other functions, how did they perform?

Suggestions for future versions:

  • Fig. 2 is quite hard to understand, it seems more like a collection of approaches than one unified approach. It would greatly improve the figure to show the flow of information through it and how, for example, specific rollouts are treated.

  • The notation could be improved, for example, in the factorization at the beginning of Sec. 4.1 it is not immediately clear that xmx_m is contained in hmh_m. Perhaps, hh could be used for history throughout?

  • It would be helpful to put a forward reference for RTR_T and RSR_S next to their respective definitions in Eq. 2 and Eq. 3.

  • In Sec. 4.1 the joint distribution is exchangeable (it seems?) refered to as pp and μ\mu which should be unified

2 comments

Thank you once again for your constructive feedback and assessment! As the discussion phase is drawing to a close, we are reaching out to follow up and see whether our responses and additional experiments have adequately addressed your concerns. Should you have any further questions, we would be very happy to provide additional clarification, and we look forward to your further consideration.

Thank you for your response and for clarifying the evaluation set-up. That alleviates my concern that the student model used in evaluation is different, it will be good to write this very clearly in the paper. I will raise my score accordingly.

Reviewer_ZdJxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the Assistance Dilemma in multi-turn reinforcement learning for LLM tutors, where rewarding student success may encourage tutors to provide answers rather than facilitate learning. The authors propose EDUARDO, which combines a near-transfer post-test, tutor utterance masking, and binary reward gates to encourage effective tutoring while discouraging solution handover. Experiments across different tutor model sizes and tutoring benchmarks evaluate the resulting pedagogical behaviors and performance.

Critical strengths

S1. Meaningful research problem and motivation.

The paper addresses an important challenge in LLM-based tutoring: student simulators may become inaccurate as tutoring policies evolve. Jointly adapting the tutor and student simulator is a reasonable approach to mitigating the distribution shift caused by changing tutoring strategies.

S2. Comprehensive experiments with encouraging results.

The paper evaluates tutoring performance, answer disclosure, student-response prediction, and simulator calibration. COMET achieves strong Clean Success@15 results and demonstrates improvements over same-backbone baselines when evaluated with an independent frozen student simulator. The ablation studies also provide useful evidence for the contributions of different components.

Critical weaknesses

W1. Limited transferability from simulated students to real human learners.

The paper aims to train an effective tutor model to support student learning. However, the proposed framework relies on LLM-based student simulators, whose reasoning processes, misconceptions, and learning behaviors may differ substantially from those of real students. Consequently, a tutor optimized to teach LLM-based students may not necessarily be effective for human learners. Although the authors evaluate the model using an independent LLM student simulator, this does not adequately establish its effectiveness in real educational settings.

W2. Lack of evaluation of genuine knowledge acquisition.

The paper primarily evaluates whether students can solve the original problem under the tutor's guidance, rather than whether they have actually mastered the underlying problem-solving methods. A student may successfully complete a problem by following step-by-step instructions without developing the ability to solve similar problems independently. Without post-tutoring evaluations on unseen problems, the reported improvements in tutoring success do not necessarily demonstrate genuine learning or knowledge transfer.

Questions for authors

Q1. How can the authors establish that tutoring strategies optimized using LLM-based student simulators are effective for real human learners, given the substantial differences in their reasoning and learning behaviors?

Q2. Why does the evaluation focus on successful completion of the original problem during tutoring rather than independent problem-solving after tutoring? Could the authors evaluate students on unseen but related problems to determine whether they have genuinely learned the underlying solution methods?

1 comment
  1. Transfer from simulated students to human learners

Thank you for raising this question. Methods such as TutorRL use reinforcement learning with a fixed student simulator to train effective tutors. COMET further addresses student-response modeling as the tutoring policy changes. We use distinct objectives for instructional optimization and student-response modeling, adapting the simulator to changing instructional contexts while using recorded responses to preserve student errors and incomplete reasoning.

To test whether the learned guidance transfers beyond the co-trained partner, we evaluate it with an independent, frozen student that did not participate in co-evolution. Under the same student model, information access, and answer-disclosure guard, COMET achieves 67.7%67.7\% Success@15\mathrm{Success@15}, compared with 47.5%47.5\% for TutorRL. The difference, calculated from unrounded results, is 20.120.1 percentage points (paired 95%95\% CI: [13.9, 26.6][13.9,\,26.6]). This result supports the transfer of COMET's tutoring advantage to another student model.

We have added this future research direction to the Conclusion: conducting studies with human students to evaluate COMET's effects on independent problem solving and knowledge retention.

  1. Independent problem-solving after tutoring

Following your suggestion to assess independent problem-solving after tutoring, we have added a near-transfer test on unseen related problems. A correct answer may reflect understanding or simply reproduce an answer supplied by the tutor. COMET therefore evaluates student-response correctness and instructional behavior separately. The Tutor reward encourages attention to key steps and probing of student reasoning, penalizes direct answer disclosure, and combines ZPD-inspired challenge selection with progress rewards to help students make progress toward a solution under guidance. Cognitive evaluation in the simulator objective also considers correct reuse of earlier steps and self-correction.

The ablations provide evidence for the role of instructional constraints. In the fifth-round ablation, removing the pedagogical reward reduces Clean Success@15\mathrm{Clean\ Success@15} from 85.3%85.3\% to 74.0%74.0\% and increases Telling Rate\mathrm{Telling\ Rate} from 1.56%1.56\% to 3.45%3.45\%. These results show the importance of considering both progress toward a solution and instructional behavior when optimizing for task completion with disclosure control.

The new post-test contains 7575 related problems constructed from MathDial. The 169169 initial contexts in the original independent-student evaluation cover 7575 distinct source problems. For each source problem, we constructed one new problem that preserves the core mathematical relationship while changing the numerical values and scenario wording, and independently solved it to verify the answer. All conditions use the same frozen Llama-3.1-8B student. The post-test retains the original problem, initial context, and the student's own utterances, without tutor utterances or further guidance.

Post-test conditionAccuracy on new problems
COMET71.43%71.43\%
TutorRL67.87%67.87\%
SFT67.10%67.10\%
No tutoring, same initial context71.11%71.11\%

In the primary analysis of all 7575 problems, COMET exceeds SFT by 4.324.32 percentage points (paired 95%95\% CI: [1.26, 7.27][1.26,\,7.27]) and TutorRL by 3.563.56 points (paired 95%95\% CI: [0.31, 6.90][0.31,\,6.90]). Confidence intervals are clustered by source problem. The advantage over SFT also holds in the prespecified sensitivity analysis. We will report the full auxiliary-baseline and sensitivity-analysis results together.

The pedagogical-reward ablation examines the contribution of instructional constraints, and the new post-test extends evaluation to related problems after tutoring. With the same student and post-test settings, COMET achieves the highest accuracy among the three tutoring methods, providing additional evidence of its advantage over tutoring baselines on new problems.

Reviewer_wWojOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The goal of the paper is to introduce a joint training objective for LLM-based math tutors and simulated students that claims to optimize for making the tutor provide just the right amount of scaffolding for the student to learn best (known as the Zone of Proximal Development).

The problem the paper is trying to address is that most methods to train LLM tutors train them against fixed simulated students. As the tutor is trained, the answers it produces become more and more out of the distribution for simulated students, which can affect their validity as a simulation. To the extent of my knowledge of the field, this problem is important and largely unsolved in the literature.

The paper shows results supporting their design, but I question their validity as written in the critical weaknesses section below.

Critical strengths

The main idea of the paper, which is to treat tutor optimization and simulator adaptation as two separate RL objectives, is well motivated and novel. I believe this setup would also be useful more broadly to any situation where an agent is trained using another simulated agent as both the conversation generator and training signal, which would make it valuable to the broader audience of ICLR.

Critical weaknesses

The paper is extremely hard to follow because of unclear writing and jargon not introduced (e.g. Success @ 15, knowledge tracing, GRPO without reference and expanding the acronym the first time they mention it). The paper seems to be written assuming some additional context on the project, which is likely a telltale sign of LLM writing. This is not a problem in and of itself, but makes me wonder if the AI statement is faithful. I will give the authors the benefit of the doubt, but the writing significantly affected my ability to understand and assess the paper. The figures in the paper also need polishing both for clarity and for display quality.

I believe their scope is too restricted for the paper to be broadly useful to the community at large. They are restricted to mathematics, they do not test for spillover effects on other attributes of the tutor (like its correctness or fluency).

No comparison with other RL methods is run. Their experiments also do not isolate the effect of co-evolution beyond the impact of RL (no RL tutor trained against a fixed simulator). The +20.1 result therefore only shows that using RL to train a tutor can beat a prompted baseline, which e.g. Dinucu-Jianu et al. (2025) already showed.

Questions for authors

I cannot think of questions whose answers could materially affect your assessment. I still think that the problem they are addressing is important to the community and I encourage the authors to work on the readability of the paper and expand the scope which could make their contribution truly valuable.

1 comment

Thank you once again for your constructive feedback and assessment! As the discussion phase is drawing to a close, we are reaching out to follow up and see whether our responses and additional experiments have adequately addressed your concerns. Should you have any further questions, we would be very happy to provide additional clarification, and we look forward to your further consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.