- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper presents StudentSim, a framework for training individualized LLM-based student simulators through pooled domain training followed by per-student specialization. It introduces StudentSimEval, covering 60 students across chess, second-language English writing, and mathematics. Evaluation measures behavioral fidelity to recorded student responses and guidance responsiveness against canonical corrections. The paper reports improvements over prompted LLMs and domain-specific references. A chess proof of concept additionally uses the simulator within a composite reward for tutor reinforcement learning, with expert evaluations favoring the resulting tutor.
Critical strengths
- The paper addresses the challenge of simulating learner behavior while supporting natural-language instructional interactions. Evaluating both behavioral fidelity and instructional interaction provides a more comprehensive assessment of student simulation.
- The experiments span three domains, use chronological held-out records, and report results across three training runs. The inclusion of indirect guidance settings and blinded expert evaluation further strengthens the empirical evidence.
Critical weaknesses
-
The metric measures agreement with canonical corrections rather than actual student responses to guidance. High and scores may therefore reflect general problem-solving ability rather than faithful, guidance-sensitive student simulation. Controls with absent, shuffled, or misleading guidance are needed to distinguish these possibilities.
-
Appendix C.6 reports a substantial discrepancy ( vs. ) across greedy inference runs using the same mathematics checkpoint. The explanation based on bf16 nondeterminism remains unverified, and the reported stability on the full evaluation does not resolve this discrepancy. Controlled repeated inference experiments are needed to establish the reliability of these results.
Questions for authors
-
What causes the same-checkpoint discrepancy ( vs. )? Can the authors report repeated evaluations on both the subset and full mathematics set, including per-record consistency and sensitivity to numerical precision and batching?
-
How do the pooled Stage-1 and personalized Stage-2 models compare on the same held-out records? How does change when guidance is removed or shuffled?
-
In Table 25, multiplying a negative reward by a gate in makes the reward less negative as the gate decreases. Does the implementation address this issue? If not, how frequently does it occur, and does correcting it affect downstream results?
Thank you for your review. We are glad you recognize the sixty real students across three domains, each with a frozen per-student split, and the downstream application, "an RL experiment that trains a tutor with simulator feedback". These are the two ends the paper connects: a simulator learned from each student's own records, and its use as feedback for training an AI tutor. We have uploaded a revised manuscript, in which changes from the submitted version are marked in purple. Turning to the comments you raise, we list in the table below where each is addressed, in the version you reviewed or in this revision; the details follow.
Analysis of F and R
Why F and R take the values they do has been analyzed in the main text since the original submission. For example, Section 1 (p. 2) explains why the models differ in all three domains: "prompted GPT-5.4 responds to guidance but poorly reproduces individual behavior, while Maia2 models human moves but cannot consume natural-language guidance. StudentSim is strong on both axes, and the same pattern holds in L2 and mathematics." Sections 5.2 and 5.3 (pp. 6–8) state what a higher F and R mean in each domain and trace these differences on held-out chess cases (Figures 4 and 5). Appendix C.7 gives R under each guidance mode for every model that takes guidance.
W1. Guidance and R
The comparison in your comment already shows that R is not explained by reading the endpoint out of the guidance. The baselines you cite read the same guidance as StudentSim, and untrained Qwen is the same base model as StudentSim, so if reading the endpoint explained R, they would reach the same R. StudentSim is higher in every domain (Table 3), and the gap is largest under the guidance types that do not state the endpoint (Section 5.3, p. 7; Appendix C.7).