acceptodds
Under review as a conference paper at ICLR 2027

Can the Simulated Student Learn? Fitted Knowledge-Tracing Models Disagree on Spacing, Recency and Transfer

Abstract

Tutoring agents are increasingly scored against a simulated student. In benchmarks that track what the learner knows, the simulated student is a knowledge-tracing model fitted to interaction logs. The score is the change in the predicted probability of a correct response after teaching. It measures learning gain only if the model responds to teaching as a learner does. This paper introduces four contrasts that test this condition on five architectures fitted to two corpora at three seeds each. The contrasts are spacing against a position-matched control, recency, transfer from a related concept and the error penalty. Each contrast appends two sequences to the same learner history. They differ in only one systematic respect: the order of practice, the concept of the items practised first or whether the first response is correct. The effect is the paired difference in predicted gain on held-out items of the target concept, averaged across histories. A contrast passes when its effect has the expected sign at in a two-sided paired -test. That sign is positive for the three schedule contrasts and negative for the error penalty. A failure is an absence if the effect is not significant and an inversion if it is significant the other way. Responses are written in rather than sampled, because they move the probe score a median of 10 times more than the concept practised does. Every simulator penalises a wrong answer at every seed. Spacing and recency each fail in six of 10 simulators at the first seed, and transfer fails in eight. Across the three seeds, some absences and one small inversion persist, but the largest inversions vanish. Only five of 10 simulators give the same verdicts at every seed, and new target concepts change verdicts at least as often as a refit. DKVMN fails spacing at every seed and target concept. Validating DKT on ASSIST2017 alone finds no failure at the first seed. When the simulator answers, as in a benchmark, a spaced tutor beats a massed tutor on 16 of 30 fitted models, ties with it on 12 and loses on two. The fitted simulator that a benchmark adopts therefore limits which tested phenomena the benchmark can credit. A four-contrast certification costs minutes and reports which of the four a fitted simulator registers.

Primary area:
datasets and benchmarks
Keywords:
knowledge tracing, simulated students, construct validity, tutoring agents, benchmark validity, spacing effect, AI for education

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Bd2ZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

ITS systems are often evaluated with simulated human students on sampled learning episodes.

The paper proposes to modify a test suite of learning episodes to see whether a simstudent behaves naturally according to well-known student behaviors, like response to spaced repetition (and three others). It then evaluates several simstudent models that use knowledge tracing and show that they mostly do NOT behave in a human-like way.

Critical strengths

AFAIK this is a novel way of assessing knowledge-tracing simstudents, and shows that they are indeed limited in important ways.

Critical weaknesses
  • The paper is not written for a general ICLR reader (ie not an ITS expert). The abstract, eg, immediately dives into way more detail than was appropriate. Some concrete suggestions: why not use "human student" where appropriate, just to remind the general reader of the context?
  • The paper is of interest mainly to people building simstudents, which is at best a small subset of ICLR attendees.
Questions for authors
  • Are there general lessons for learning that should be taken from this analysis, or is the work mainly aimed at researchers working on KT simstudents and/or ITS bechmarks?
  • What are the practical consequences for an ITS benchmark/simstudent that fails this certification? Clearly it's not modeling human behavior perfectly, but is there a measurable negative consequence of not modeling these aspects of behavior?
Reviewer_8WAsOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether a fitted knowledge-tracing model, used as a simulated student, responds to practice schedules the way human learners do. Across four paired contrasts, the error penalty passes everywhere, the three schedule effects mostly fail, and verdicts change with the seed, the item draw, and the target concept.

Critical strengths
  1. The question is genuinely prior: benchmarks score tutors against simulators whose sensitivity to teaching has not been checked.
  2. Certification is cheap, taking seconds to minutes, so benchmarks can adopt it directly.
Critical weaknesses

There is a serious problem with the writing: the paper reads more like an experiment log than a scientific study. What it lacks is analysis of the results. For example, why do these models fail? The main text does not explain this.

  1. There is no positive control at the level of schedules and no null model, so "the model does not respond" cannot be separated from "the test is not sensitive". The error penalty is the only contrast with a prior answer, and it passes 30/30, so it cannot distinguish a sensitive test from an insensitive one. None of spacing, recency, or transfer has a model that is known in advance to pass, and the two-state BKT model listed as a standard architecture is never fitted. The 22/30 rule also has no null model.

  2. Spacing and recency are not fully separated. The spacing control matches practice positions on average, but a spaced sequence and a massed block do not end at the same distance from the probe, so the spacing contrast retains a recency component. Since the models read no timestamps, the construct is practice order rather than multi-day spacing, which the paper acknowledges in Section 6.

Questions for authors
  1. Can you fit BKT or a synthetic learner with a known spacing effect as a positive control?
  2. What is the minimum detectable effect or power?
  3. Can you diagnose why DKVMN and AKT fail, for example through memory structure, concept mapping, or training?
Reviewer_6aHEOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates an important yet insufficiently examined question: whether KT models used as simulated students can faithfully reflect the learning effects of instructional interventions. Existing tutoring-agent benchmarks often employ pretrained KT models to simulate student behavior and quantify learning gains through changes in predicted response correctness. However, KT models are primarily optimized for next-response prediction from observational interaction histories, which does not necessarily guarantee realistic responses to alternative instructional strategies. To investigate this issue, the authors introduce four controlled contrasts examining spacing, recency, knowledge transfer, and error penalties. The experiments evaluate five KT architectures, including DKT, DKVMN, AKT, SAKT, and simpleKT, on ASSIST2009 and ASSIST2017. The authors further examine robustness across three random seeds, alternative item draws, and target concepts.

Critical strengths
  1. The primary novelty of this work lies in reconsidering the behavioral validity of KT models when employed as simulated students, rather than pursuing further improvements in conventional next-response prediction accuracy.
  2. The experimental evaluation considers five representative KT architectures and investigates the influence of random initialization, practice-item sampling, and target-concept selection.
  3. The paper is generally well organized, progressing logically from motivation and contrast construction to empirical findings and practical implications for tutoring-agent evaluation.
Critical weaknesses
  1. A fundamental concern is the mismatch between the operational definitions of the proposed contrasts and the psychological phenomena they are intended to represent. For instance, the spacing contrast manipulates the positions of practice items within interaction sequences without explicitly modeling elapsed time, inter-study intervals, or retention delays. However, the psychological spacing effect depends critically on the relationship between practice intervals and retention intervals. Similarly, the recency contrast changes not only the distance between target-concept practice and evaluation but also the relative positions of other-concept interactions. Consequently, these contrasts primarily measure sensitivity to interaction ordering and concept arrangements rather than directly establishing the presence or absence of human-like spacing and recency effects. Although the authors acknowledge the lack of temporal information in Section 6, this limitation directly affects the construct validity of the central claims rather than merely restricting their generalizability.
  2. The transfer contrast raises an even more substantial construct-validity concern. The authors define related and unrelated concepts according to their co-occurrence frequencies in student interaction histories. However, statistical co-occurrence does not necessarily reflect prerequisite relationships, semantic similarity, or positive knowledge transfer. Frequent co-occurrence may simply result from curriculum organization, item popularity, or multi-concept annotations, whereas infrequently co-occurring concepts may still exhibit meaningful prerequisite dependencies. Furthermore, the contrast introduces eight additional target-concept practice interactions after the related or unrelated preparation phase, potentially attenuating or obscuring the influence of the preceding concept. Consequently, when a model fails the transfer contrast, it remains unclear whether the failure reflects an inability to capture knowledge transfer or limitations in the construction of the related/unrelated concept pairs and the subsequent evaluation sequence.
  3. The paper correctly argues that predictive accuracy does not guarantee valid simulation of instructional interventions. However, its evaluation still relies on changes in the same conditional prediction mechanism.
Questions for authors
  1. In Section 3.3, the authors deliberately position massed practice around the mean position of spaced practice to reduce recency confounding. However, Section 4.3 compares a massed tutor that places all eight target-concept items at the beginning of the sequence against a spaced tutor that distributes them throughout the sequence. Consequently, the two strategies differ not only in practice distribution but also in the temporal ordering of target practice relative to the final evaluation. The observed differences in Table 2 may therefore reflect a combination of spacing and recency effects. This weakens the evidence that spacing certification specifically anticipates spacing-related tutor rankings.
  2. Despite evaluating five KT architectures across multiple random seeds, the main experiments rely on only two datasets, with relatively limited target-concept coverage under the primary protocol.
  3. Would the pass rates of the four contrasts change if each KT architecture were reasonably tuned or if models with explicit temporal and cross-concept mechanisms, such as HawkesKT, were included?
  4. Although the five selected KT models cover representative recurrent, memory-augmented, and attention-based architectures, the experimental coverage remains insufficient to support the broader conclusions. In particular, three of the five models rely on attention mechanisms, while approaches explicitly modeling learning processes, temporal dependencies, or inter-concept relationships are not adequately represented. Models such as HawkesKT, LPKT, and GKT are particularly relevant to the recency, spacing, and transfer phenomena investigated in this paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.