- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
ITS systems are often evaluated with simulated human students on sampled learning episodes.
The paper proposes to modify a test suite of learning episodes to see whether a simstudent behaves naturally according to well-known student behaviors, like response to spaced repetition (and three others). It then evaluates several simstudent models that use knowledge tracing and show that they mostly do NOT behave in a human-like way.
Critical strengths
AFAIK this is a novel way of assessing knowledge-tracing simstudents, and shows that they are indeed limited in important ways.
Critical weaknesses
- The paper is not written for a general ICLR reader (ie not an ITS expert). The abstract, eg, immediately dives into way more detail than was appropriate. Some concrete suggestions: why not use "human student" where appropriate, just to remind the general reader of the context?
- The paper is of interest mainly to people building simstudents, which is at best a small subset of ICLR attendees.
Questions for authors
- Are there general lessons for learning that should be taken from this analysis, or is the work mainly aimed at researchers working on KT simstudents and/or ITS bechmarks?
- What are the practical consequences for an ITS benchmark/simstudent that fails this certification? Clearly it's not modeling human behavior perfectly, but is there a measurable negative consequence of not modeling these aspects of behavior?