- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper deals with the issue that tutoring models are evaluated against student simulations, where the student simulation heavily influences the final ranking of the models. They show that this evaluation protocol is especially sensitive to changing the simulated student and to models being biased towards their own model family.
Critical strengths
- I do find the overall question important of how to pick student simulators to evaluate tutoring models and progress on this can have significant impact in the community. Oftentimes, these choices are made out of convenience but make results less reliable.
Critical weaknesses
-
The paper is very hard to understand due to the writing. Despite trying my best and significant prior experience in the field I was unable to understand the exact set-up under which evaluation is done. For example, I was unable to even understand which exact tutoring agent is used for the evaluation? I checked Appendix D which outlines implementation details but could also not find it there. This is a significant weakness, because it is essential for readers to precisely understand the findings of the paper but, to me, these are not obvious because the set-up is not exactly clear and writing is very hard to parse.
-
As written above, many details are left out and only reported in the Appendix D which makes it hard for readers to follow the main text, as they have to skip forward, and even there important details are left out. In addition, I would expect clear recommendations for future practice but Sec. 5 is a) very short and b) also hard to parse which makes it unclear for readers how they can improve their benchmarks in the future.
Questions for authors
-
"The fitted model that simulates the learner also scores the teaching." Is this claim really true the way it is written? Take Dinucu-Jianu et al. 2025 as a counter-example where student simulator and LLM judge are different. The judge is also changed between training and testing. It is only true that student model of training and evaluation match but these do not handle the scoring. Could you please clarify this claim? Do you mean that the simulated student is used together with another LLM that establishes the scoring (which is not the simulated student)?
-
"The resulting advantage remains unmeasured" (L49) - what exactly is this advantage? Could you please clarify?
-
Can you please clearly define what you understand under knowledge tracing?