- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Main contribution of the paper is a simulation environment for evaluating the tutoring capabilities of LLMs, based on an existing RCT dataset of high school math conversations (Bastani et al., 2025). It consists of a) an LLM role-playing a student from an existing conversation using 5 extracted free-form attributes, and b) an LLM rubric, where an LLM scores 5 conversation features and a linear regression predicts the student’s exam grade. The authors ablate a potentially leaked ChatGPT Study Mode prompt and three GPT-5.6 model sizes. They claim that instructions against giving away answers are critical, that tutoring strategy suggestions reduce performance, and that a medium-sized model outperforms the largest one. While the paper works on an important problem, I do not think it is ready for publication. The key novelty is applying LLM-Rubric (Hashemi et al., 2024) to an existing dataset, which reads more like a data analysis than a novel contribution. Neither the reward model nor the student simulator is validated.
Critical strengths
- Works on the important and timely problem of cognitive offloading and evaluation of LLM tutors.
- Grounds the evaluation in actual exam grades from an RCT rather than intuition of what good tutoring looks like.
- Clearly written, uses a public dataset, and is honest about its limitations.
Critical weaknesses
- The reward model is not validated. It is fit on real conversations (GPT-4 tutors, real students, variable length) but applied to fixed 5-turn simulated conversations (L174), with no evidence that the calibration transfers. Even the held-out fit on real data is not reported (L269, L294).
- The finding that misunderstanding identification hurts learning (Table 1, L376-377) is questionable. How can a tutor respond appropriately without identifying the misunderstanding first? A calculation mistake differs from a misconception, following desirable difficulties, the tutor should guide the student to see why their solution does not generalize. The regression is observational, with no controls for student ability or treatment arm (I at L153 is never used). The tutor features are also likely correlated, so the negative sign may be an artifact. Likewise, the claim that worse models can be better tutors (L380) is not supported, as the paper does not show that Terra makes errors students learn from.
- The reward is driven mostly by the simulated student. The only significant positive features (independence, progress) describe the student, and simulator fidelity is not evaluated. GPT-5.6-Sol is also the student, judge and default tutor (L297-300), which confounds the model size ablation. Only proprietary GPT models are compared.
- The paper does not engage well with the existing tutoring evaluation literature, and the “naive LLM judge” baseline (an average of the same 5 features) is weak. I encourage the authors to discuss TutorMoments [1] and to compare against MathTutorBench [2] and MRBench [3], which cover misunderstanding identification and hints from the proposed rubric. For example, they could correlate these judges with exam grades on the real RCT conversations.
- The empirical evaluation is limited. The claims in the abstract (L019-024) and the “comprehensive evaluation” contribution (L089) rest on 4 prompt ablations and 3 models, a single run and no standard errors. The claim that tutoring strategies hurt rests on 0.480 vs. 0.478 (Table 2). I suggest the authors revisit their approach, for example by annotating exact parts in a conversation where the tutor should intervene and how, similarly to TutorMoments [1].
[1] (Tutormoments) When Help is Unhelpful: Evaluating AI Tutors for Productive Struggle. Preprint. [2] Mathtutorbench: A benchmark for measuring open-ended pedagogical capabilities of llm tutors. EMNLP 2025. [3] Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM-Powered AI Tutors. NAACL 2025.
Questions for authors
- What is the test MSE or R^2 of the linear model compared to simple baselines (mean grade, student features only)?
- Does the negative weight on misunderstanding identification hold when controlling for treatment arm and prior performance? Criterion 3 (Figure 4) asks whether the tutor "identify and address" the misunderstanding. Could the weight reflect models giving away the solution when addressing it?
- Do simulated students keep their misconceptions, and does TutorSim reproduce the GPT Base vs. GPT Tutor ranking from the RCT?
- Do the model size results hold with a simulator and judge from a different model family? Are the differences in Tables 2 and 3 significant?
- Figure 1 belongs in the appendix. Examples of simulated conversations would be more useful.
- L319- the original 1994 desirable difficulties paper should be cited.
- L192- typo: missing ending bracket.
- L492- citation Wooldridge 2009 has a duplicate name of the author