acceptodds
Under review as a conference paper at ICLR 2027

TutorSim: Evaluating the Tutoring Capabilities of Large Language Models

Abstract

We propose a novel simulation environment for evaluating the tutoring capabilities of large language models (LLMs). Our simulation is based on data from a randomized controlled trial testing two different LLM tutors on several hundred high school students. Based on this data, we design two key components: (1) an LLM prompted to mimic student queries, and (2) an LLM rubric calibrated to estimate the grade a student achieved based on their conversation with the tutor. Using our evaluation, we perform two empirical analyses on existing LLM tutors. First, we ablate different portions of a system prompt focused on tutoring. We find that portions of the prompt focusing on avoiding giving away answers are most critical for achieving good performance; surprisingly, portions suggesting tutoring strategies actually reduces performance, suggesting that the LLM is more effective at identifying good tutoring strategies on its own. Second, we ablate the size of the underlying LLM. Surprisingly, we find that the medium-sized LLM is a more effective tutor, suggesting that LLM errors might actually improve learning by forcing students to think. We also find that the ranking is inconsistent with the na\"ive LLM judge for assessing tutoring capabilities, highlighting the need to calibrate tutoring performance estimates to rigorous outcomes such as exam grades. Our simulation-based results complement rigorous causal evaluations on LLM-based tutors by enabling a more systematic evaluation of the impact of different design decisions on the tutoring capabilities of LLMs.

Primary area:
datasets and benchmarks
Keywords:
LLM tutors, AI and education

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_iUFTOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Main contribution of the paper is a simulation environment for evaluating the tutoring capabilities of LLMs, based on an existing RCT dataset of high school math conversations (Bastani et al., 2025). It consists of a) an LLM role-playing a student from an existing conversation using 5 extracted free-form attributes, and b) an LLM rubric, where an LLM scores 5 conversation features and a linear regression predicts the student’s exam grade. The authors ablate a potentially leaked ChatGPT Study Mode prompt and three GPT-5.6 model sizes. They claim that instructions against giving away answers are critical, that tutoring strategy suggestions reduce performance, and that a medium-sized model outperforms the largest one. While the paper works on an important problem, I do not think it is ready for publication. The key novelty is applying LLM-Rubric (Hashemi et al., 2024) to an existing dataset, which reads more like a data analysis than a novel contribution. Neither the reward model nor the student simulator is validated.

Critical strengths
  • Works on the important and timely problem of cognitive offloading and evaluation of LLM tutors.
  • Grounds the evaluation in actual exam grades from an RCT rather than intuition of what good tutoring looks like.
  • Clearly written, uses a public dataset, and is honest about its limitations.
Critical weaknesses
  • The reward model is not validated. It is fit on real conversations (GPT-4 tutors, real students, variable length) but applied to fixed 5-turn simulated conversations (L174), with no evidence that the calibration transfers. Even the held-out fit on real data is not reported (L269, L294).
  • The finding that misunderstanding identification hurts learning (Table 1, L376-377) is questionable. How can a tutor respond appropriately without identifying the misunderstanding first? A calculation mistake differs from a misconception, following desirable difficulties, the tutor should guide the student to see why their solution does not generalize. The regression is observational, with no controls for student ability or treatment arm (I at L153 is never used). The tutor features are also likely correlated, so the negative sign may be an artifact. Likewise, the claim that worse models can be better tutors (L380) is not supported, as the paper does not show that Terra makes errors students learn from.
  • The reward is driven mostly by the simulated student. The only significant positive features (independence, progress) describe the student, and simulator fidelity is not evaluated. GPT-5.6-Sol is also the student, judge and default tutor (L297-300), which confounds the model size ablation. Only proprietary GPT models are compared.
  • The paper does not engage well with the existing tutoring evaluation literature, and the “naive LLM judge” baseline (an average of the same 5 features) is weak. I encourage the authors to discuss TutorMoments [1] and to compare against MathTutorBench [2] and MRBench [3], which cover misunderstanding identification and hints from the proposed rubric. For example, they could correlate these judges with exam grades on the real RCT conversations.
  • The empirical evaluation is limited. The claims in the abstract (L019-024) and the “comprehensive evaluation” contribution (L089) rest on 4 prompt ablations and 3 models, a single run and no standard errors. The claim that tutoring strategies hurt rests on 0.480 vs. 0.478 (Table 2). I suggest the authors revisit their approach, for example by annotating exact parts in a conversation where the tutor should intervene and how, similarly to TutorMoments [1].

[1] (Tutormoments) When Help is Unhelpful: Evaluating AI Tutors for Productive Struggle. Preprint. [2] Mathtutorbench: A benchmark for measuring open-ended pedagogical capabilities of llm tutors. EMNLP 2025. [3] Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM-Powered AI Tutors. NAACL 2025.

Questions for authors
  • What is the test MSE or R^2 of the linear model compared to simple baselines (mean grade, student features only)?
  • Does the negative weight on misunderstanding identification hold when controlling for treatment arm and prior performance? Criterion 3 (Figure 4) asks whether the tutor "identify and address" the misunderstanding. Could the weight reflect models giving away the solution when addressing it?
  • Do simulated students keep their misconceptions, and does TutorSim reproduce the GPT Base vs. GPT Tutor ranking from the RCT?
  • Do the model size results hold with a simulator and judge from a different model family? Are the differences in Tables 2 and 3 significant?
  • Figure 1 belongs in the appendix. Examples of simulated conversations would be more useful.
  • L319- the original 1994 desirable difficulties paper should be cited.
  • L192- typo: missing ending bracket.
  • L492- citation Wooldridge 2009 has a duplicate name of the author
Reviewer_AwDeOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Existing simulation-based evaluations of LLM tutors often lack calibration to real student learning outcomes. The authors propose TutorSim, a simulation environment for evaluating the effectiveness of tutor LLMs. It simulates students using real conversation examples and extracted student attributes, and uses a linear regression model trained on actual exam scores to map five LLM-assessed dialogue features to predicted exam performance. Using TutorSim, the authors find that the medium-sized model achieves the highest predicted exam score among the evaluated models.

Critical strengths
  1. The authors use real students’ exam scores to train the grade prediction model and extract student attributes from real conversations to simulate different students. This connects the evaluation to real learning contexts and provides a basis for examining whether simulated scores reflect student learning.

  2. The final score is computed from clearly defined dialogue features using linear regression, making each feature’s contribution to the predicted score explicit. This improves the interpretability of the scoring process and helps explain differences in scores and rankings across tutors.

Critical weaknesses
  1. The evaluation environment lacks direct validation. The paper reports neither test-set prediction accuracy nor comparisons with simple baselines for the grade predictor. The reliability of the LLM feature scores and the realism of simulated student behavior are also not evaluated. Furthermore, the resulting tutor rankings are not checked against actual learning outcomes. These gaps make it difficult to establish whether TutorSim reliably measures tutoring effectiveness.

  2. The explanation for the medium-sized model’s higher predicted score is insufficiently supported. The negative coefficient for Misunderstanding identification could reflect student differences, since students with lower prior proficiency may need more help while also earning lower exam scores. The authors use this relationship to suggest that accurate misunderstanding identification may reduce learning. The analysis does not rule out alternative explanations for this relationship.

Questions for authors
  1. What are the prediction errors and correlations with actual exam scores on the 216 test conversations? Does the grade predictor outperform simple baselines, such as predicting the mean training-set exam score?
  2. What theoretical or pedagogical basis supports the selection of the five evaluation features?
  3. How stable are the LLM judge’s scores across repeated evaluations of the same conversations?
  4. How closely do simulated students match real students in their misunderstandings, help-seeking behavior, and responses to tutor hints? Is there quantitative evidence supporting this similarity?
Reviewer_zd64OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces TutorSim, a simulation environment for evaluating LLM tutors that is grounded in data from a Randomized Controlled Trial (RCT) in which high school students practiced math problems with GPT-based tutors before taking a closed-book exam. The environment has two components. The first is a student simulator: an LLM extracts attributes from a real student’s conversation and is then prompted to imitate that student. The second is a reward model: an LLM judge scores each conversation on five rubric dimensions and a linear regression trained on the real conversations maps those scores to the student’s exam grade that aimes to calibrate the judge to actual learning outcomes. The authors apply TutorSim in two example studies. First, they ablate sections of the ChatGPT Study Mode prompt and find that the instruction to not give away answers matters most, followed by explicit strategy suggestions. Second, they compared three model sizes and find that the medium-sized model reaches the highest predicted exam score even though the uncalibrated judge ranks the largest model first. The authors interpret this as evidence that tutor imperfections may encourage student thinking and that tutor evaluation should be calibrated to rigorous outcomes.

Critical strengths

The findings are consistent with the current state of the art on how LLMs affect student learning.

Their result on the positive coefficient on student independence matches results in the learning sciences literature on desirable difficulty and productive struggle.

The paper documents an important trend that runs counter to the assumed utility of LLMs for learning, which is valuable.

Critical weaknesses

Validity of the simulation. Evaluating a new tutor changes two of the three ingredients at once: the tutor (2023 GPT → GPT-5.6) and the student (real teenager → LLM). The predicted scores are meaningful only if three key assumptions hold, and the paper discusses none of them. 1) Simulated students respond as real ones would. This is not validated, even against the original tutors. 2) Rubric scores mean the same thing on cleaner, fixed 5-turn simulated text as on real text. 3) The feature-to-grade map learned with 2023 tutors still holds with new ones.

The reward model. The features are observational, and the treatment indicator I is not in the regression, so the coefficients are correlations rather than effects. Fit quality (R² or test error on the 216 held-out conversations) is not reported, meaning that the narrow range of predicted scores (0.433–0.503) might mostly reflect prediction noise.

Experiments and claims. GPT-5.6-Sol serves as a student, judge, and large tutor, and so self-preference could affect the model-size comparison. Tables 2 and 3 report no standard errors or significance tests. The 1,200 conversations reuse personas and problems, and so they are not independent. Given the stochasticity of LLMs, the observed differences may stem from sampling variance rather than the ablations. This should be ruled out. The proposed mechanism, that Terra’s mistakes force students to think, does not appear in the measured features: student independence barely changes (6.51 vs. 6.59), and student progress drops (7.78 vs. 6.51).

Novelty and related work. Predicting learning from dialogue features with regression has a long history (e.g., Litman and Forbes-Riley). Outcome-based scoring of LLM tutors has also been explored: Scarlatos et al. (AIED 2025) combine an LLM student model that predicts correctness with a GPT-4o pedagogical rubric, and Macina et al. (MathTutorBench, EMNLP 2025) train a pedagogical reward model. Scarlatos et al. (ACL 2026) further show that prompting-based student simulation performs poorly. TutorSim’s distinguishing feature appears to be its calibration to closed-book exam grades; explicitly stating this would strengthen the paper.

Minor points:

Specify which GPT model students used in 2023. We recognize RCTs cannot keep pace with model releases, but the conclusions are most directly tied to that model.

Notation is inconsistent in places in Sections 3.2–3.4; for example, R appears to denote different quantities.

A diagram of the pipeline (simulator -> judge -> linear predictor) would clarify the process.

Questions for authors

Why is the control arm (no GPT tutor) not used, for example, as a baseline?

Why were 1,000 of the 5,900 conversations used to fit the reward model rather than all of them?

What is the scale of the exam score?

Could you justify the choice of a linear model further, and report the other models you compared in the appendix?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.