acceptodds
Under review as a conference paper at ICLR 2027

Investigating the Robustness of Knowledge Tracing Models in the Presence of Shifting Student Distributions

Abstract

Knowledge Tracing (KT) has been an established problem in the educational data mining field for decades, and it is commonly assumed that the underlying learning process being modeled remains static. Given the ever-changing landscape of online learning platforms (OLPs), we investigate how concept drift and changing student populations can impact student behavior within an OLP through testing model performance both within a single academic year and across multiple academic years. Four well-studied KT models were applied to five academic years of data to assess how susceptible KT models are to concept drift. Through our analysis, we find that all four families of KT models can exhibit degraded performance. Bayesian Knowledge Tracing (BKT) remains the most stable KT model when applied to newer data, while more complex, attention-based models lose predictive power significantly faster when features provided are diverse enough to allow overfitting. Finally, we apply well-studied measures of dataset similarity to assess which kinds of drifts may account for the performance degradation we document. To foster more longitudinal evaluations of KT models, the data used to conduct our analysis is available at [this link](https://osf.io/hvfn9/?view_only=b936c63dfdae4b0b987a2f0d4038f72a)

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Distributional Shifts, Knowledge Tracing, Data Drift

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oTwfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the robustness of Knowledge Tracing (KT) models in Online Learning Platforms (OLP) when student data distributions change over time. The experimental results show that the predictive performance of all tested models decreases as the time interval between training data and evaluation data increases. This paper also uses the distribution divergence index in data mining to quantify and compare changes in data distributions, such as knowledge points and answer results, across years, and reveals significant cross-year differences. In addition, the paper opens up longitudinal KT data resources containing student interaction records from multiple academic years, aiming to provide support for subsequent research on time generalization and model robustness evaluation.

Critical strengths

S1. The paper breaks the traditional convention in the fields of Educational Data Mining (EDM) and KT that the modeled learning process is static. The concept drift was introduced into the KT model evaluation to explore the impact of constantly changing student populations over a five-year span on the model's predictive ability. S2. The study implemented bidirectional time crossover evaluation and quantified the slope of performance degradation through multiple linear regression analysis controlling for the "Years Between" variable. S3. The paper’s four research questions map cleanly onto distinct sections of the analysis. In addition, the author released a dataset containing longitudinal student interaction records from multiple academic years as part of the open-source release of this paper.

Critical weaknesses

W1. The study calculates YB as the absolute time difference, treating forward prediction identically to backward prediction. Mixing backward evaluations into the YB regression confounds the true temporal degradation effect with the inherent difficulty of specific evaluation years. Real-world KT models are deployed forward in time; the lack of a forward-only (train year < evaluate year) robustness check makes it difficult to determine if the reported performance degradation is a statistical artifact of this asymmetric design. W2. The paper claims that research has demonstrated that concept drift leads to model degradation. According to theoretical concepts, drift refers to a change in the intrinsic mapping relationship P(Y|X). But the author actually only measured P(X), which is the proportion of question types that changed. It lacks persuasiveness to directly conclude that concept drift has occurred based solely on the decrease in model prediction accuracy. Because in the case where P(X) has already changed, the decrease in accuracy may simply be due to the model encountering new problems that it has not seen before. W3. Regression analysis data shows that the SAKT-KC model, which also uses an attention mechanism, has no statistically significant difference in the rate of decrease in AUC and logarithmic loss compared to the simplest BKT model. The parameter count of SAKT-KC is over 100 times that of BKT. This indicates that the real reason for SAKT-E's rapid decline is likely to be its use of fine-grained specific exercises as input features, which led to overfitting. The paper expands the conclusion to all attention-based models that lack data support. W4. Appendix B of the paper indicates that the validation set constructed by the author for hyperparameter tuning of the deep model was extracted from 100000 job logs from "all available academic years". When evaluating the robustness of a model to future time distributions, using a validation set containing data from future years in advance to select hyperparameters can lead to a more optimistic performance degradation of the evaluated model. W5: In the preprocessing stage of the paper, problem sets with fewer than 100 assignments were excluded, resulting in up to 36.7% of student interaction data being directly discarded. In the context of studying the concept drift of online learning platforms, newly added exercises or old exercises that are about to be eliminated in the long-tail distribution are often the most intense part of the platform's content evolution and data drift.

Questions for authors

Q1. Can the authors provide a robustness check that is strictly limited to "forward only" evaluations in chronological order (such as updating regression tables or drawing decay curves)? Q2. Could the authors provide a controlled experiment specifically targeting changes in P(Y|X)? Q3. SAKT-KC and SAKT-E share the same attention architecture but differ mainly in input granularity and vocabulary size; could the authors test whether the number of unique embedding IDs better predicts the rate of performance decline? Q4: If the interference of future data is excluded, will the performance degradation trend of the deep model become more apparent during cross-year testing? Q5: Can the author add a sensitivity analysis, such as retaining these low-frequency data or lowering the filtering threshold and recalculating the distribution drift index? If this large-scale data exclusion result in models having higher robustness than their actual performance on real education platforms.

Reviewer_g1dKOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates Knowledge Tracing, a core task in educational data mining, examining how concept drift and shifts in student populations affect model robustness on online learning platforms. The authors systematically evaluate four families of classical KT models, namely BKT, PFA, DKT, and SAKT with both knowledge components and concrete exercises as input, on a real-world educational dataset spanning five academic years from 2019–2020 to 2023–2024, in both within-year and cross-year settings. The experiments show that all tested models exhibit performance degradation over time: classical Bayesian Knowledge Tracing remains the most robust under temporal evaluation, whereas the heavily parameterized attention-based model SAKT-E, while achieving the best in-distribution performance, degrades most sharply as the temporal gap increases. Furthermore, the authors quantify covariate shift and label shift across academic years using KL divergence, JS divergence, and the Population Stability Index, and release this multi-year sampled dataset to facilitate future longitudinal evaluation research.

Critical strengths

The research perspective addresses a real deployment pain point, with strong empirical value and a notable resource contribution. The vast majority of prior KT work implicitly assumes the underlying learning process is static and evaluates on datasets from a single time slice. This paper introduces a dataset-shift analysis framework and conducts a longitudinal degradation analysis of mainstream KT model families over a five-year horizon. The released multi-year standardized sampled dataset fills a gap in the community's longitudinal benchmarking and has direct practical value for designing model retraining cycles in real educational systems.

The experimental controls and statistical testing are fairly rigorous. To eliminate the confounding effect of vastly different raw log volumes across years, where the largest year contains more than twenty-one times the logs of the smallest, on model fitting, the authors design an equal-sized random sampling strategy based on assignment logs, with ten sample sets per academic year and 50,000 assignment logs in each. In evaluation, beyond reporting Log Loss, AUC, and F1, the authors construct a regression equation with interaction terms between model type and training year, and use heteroskedasticity-consistent standard errors to rigorously test the statistical significance of the performance-degradation slopes.

The paper is well structured, progressing layer by layer around four core research questions RQ1 through RQ4. The ablation-style decomposition of SAKT into SAKT-KC with 262K trainable parameters and SAKT-E with 12.4M trainable parameters is intuitive and effectively reveals the negative impact of feature input granularity and overparameterization on cross-temporal generalization.

Critical weaknesses

For a paper submitted to ICLR 2027, the most advanced architecture evaluated dates back to SAKT from 2019. In recent years, many SOTA models combining cognitive forgetting mechanisms and improved attention structures have emerged, such as AKT, SAINT, and simpleKT. Without testing these contemporary architectures, the conclusion that attention mechanisms cause rapid model degradation lacks support for generalizability to modern architectures.

In Figure 4a and Table 6, DKT's in-distribution baseline AUC at zero years between training and evaluation is only about 0.75 to 0.76, which is more than 0.12 points below both BKT at roughly 0.885 and SAKT-KC at roughly 0.90, and on par even with the simple logistic-regression model PFA. This severely contradicts the broad consensus in the KT literature that DKT typically outperforms or at least matches classical BKT. The authors fail to explain why DKT with 734K parameters shows such severe underfitting or training anomalies on this dataset, which undermines the credibility of the RQ2 conclusion that model complexity is unrelated to degradation rate, drawn partly on the basis of DKT.

Section 3.1 notes that under random assignment-log sampling, one student may have multiple assignment logs. This means that across the ten sample sets within the same academic year, different assignment sequences from the same student very likely appear in both the training and test sets, causing leakage of individual-student characteristics in the within-year evaluation at zero years between.

The paper repeatedly emphasizes, in the abstract and body, evaluating models' sensitivity to concept drift, and in Section 2.2 explicitly defines concept drift as a change in the conditional distribution of Y given X. Yet in Section 4.4, the authors only compute divergences of the marginal input distribution over KC proportions and the marginal output distribution over overall accuracy, corresponding to covariate shift and label shift respectively, without any quantification of the conditional distribution, for example changes in conditional accuracy given a fixed KC or changes in learning rates.

Questions for authors

In real OLP deployment, a model can only be trained on past academic years and inferred on future ones. If you remove all data points where the evaluation year precedes the training year and re-plot Figures 3 through 5 and re-run the regressions in Tables 5 through 7 using only forward temporal gaps, how would the performance-degradation slopes, especially the comparison between BKT and SAKT-KC, and the significance of training year change?

Why does DKT, in your experiments, achieve an in-distribution AUC of roughly 0.76 at zero years between, so much lower than the KC-based BKT at roughly 0.885 and SAKT-KC at roughly 0.90? Is this anomalous gap caused by the sequence truncation length of 40 steps, the optimizer configuration, or implementation details such as the alignment of prediction timesteps?

Although Section 3.1 filters out problem sets assigned fewer than 100 times over the five years, what is the out-of-vocabulary rate of concrete exercises in the cross-year test sample sets relative to the training sample sets? If the cross-year test sets are restricted to the subset of exercises the training set has sufficiently seen, is SAKT-E's degradation slope still significantly steeper than SAKT-KC's?

When a student completes multiple assignments within a year, is the model input sequence restricted to a single assignment log, or is it concatenated with all of the student's prior assignment records? Also, in the ten-fold cross-validation within an academic year, are different assignments from the same student allowed to fall simultaneously into the training and test folds?

Table 1 shows that the overall student accuracy in the filtered data declines monotonically from 72.8 percent in 2019–2020 to 66.0 percent in 2023–2024, a drop of 6.8 points. Yet Section 4.4 concludes that the distributions of correct and incorrect answers across years are very similar, with no label shift. Please explain this apparent contradiction, and state whether directly computing the cross-year divergence of the conditional distribution of Y given a specific KC could provide direct quantitative evidence for or against the existence of concept drift.

Reviewer_VoVPOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how concept drift and changing student populations can impact student behavior, by analyzing four Knowledge Tracing models with the collected student behavior drift data. Some classic measures such as Log loss AUC, F1 Score and statistical results have been used for assessment. Although the authors focus on educational data mining field, it lacks technical novelty and provide no in-depth analysis and further findings.

Critical strengths

This paper investigates an interesting problem about how concept drift and changing student populations can impact student behavior. A real dataset spanning five academic year is collected and constructed.

Critical weaknesses
  1. The technical novelty is limited. Although this paper focus on investigation and analysis, it lacks enough technical novelty, by leveraging classic measures such as Log loss AUC, F1 Score and statistical results.
  2. As the investigate paper, this paper did not provide and conclude further important findings for the investigated problem.
Questions for authors

refer to the Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.