- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the robustness of Knowledge Tracing (KT) models in Online Learning Platforms (OLP) when student data distributions change over time. The experimental results show that the predictive performance of all tested models decreases as the time interval between training data and evaluation data increases. This paper also uses the distribution divergence index in data mining to quantify and compare changes in data distributions, such as knowledge points and answer results, across years, and reveals significant cross-year differences. In addition, the paper opens up longitudinal KT data resources containing student interaction records from multiple academic years, aiming to provide support for subsequent research on time generalization and model robustness evaluation.
Critical strengths
S1. The paper breaks the traditional convention in the fields of Educational Data Mining (EDM) and KT that the modeled learning process is static. The concept drift was introduced into the KT model evaluation to explore the impact of constantly changing student populations over a five-year span on the model's predictive ability. S2. The study implemented bidirectional time crossover evaluation and quantified the slope of performance degradation through multiple linear regression analysis controlling for the "Years Between" variable. S3. The paper’s four research questions map cleanly onto distinct sections of the analysis. In addition, the author released a dataset containing longitudinal student interaction records from multiple academic years as part of the open-source release of this paper.
Critical weaknesses
W1. The study calculates YB as the absolute time difference, treating forward prediction identically to backward prediction. Mixing backward evaluations into the YB regression confounds the true temporal degradation effect with the inherent difficulty of specific evaluation years. Real-world KT models are deployed forward in time; the lack of a forward-only (train year < evaluate year) robustness check makes it difficult to determine if the reported performance degradation is a statistical artifact of this asymmetric design. W2. The paper claims that research has demonstrated that concept drift leads to model degradation. According to theoretical concepts, drift refers to a change in the intrinsic mapping relationship P(Y|X). But the author actually only measured P(X), which is the proportion of question types that changed. It lacks persuasiveness to directly conclude that concept drift has occurred based solely on the decrease in model prediction accuracy. Because in the case where P(X) has already changed, the decrease in accuracy may simply be due to the model encountering new problems that it has not seen before. W3. Regression analysis data shows that the SAKT-KC model, which also uses an attention mechanism, has no statistically significant difference in the rate of decrease in AUC and logarithmic loss compared to the simplest BKT model. The parameter count of SAKT-KC is over 100 times that of BKT. This indicates that the real reason for SAKT-E's rapid decline is likely to be its use of fine-grained specific exercises as input features, which led to overfitting. The paper expands the conclusion to all attention-based models that lack data support. W4. Appendix B of the paper indicates that the validation set constructed by the author for hyperparameter tuning of the deep model was extracted from 100000 job logs from "all available academic years". When evaluating the robustness of a model to future time distributions, using a validation set containing data from future years in advance to select hyperparameters can lead to a more optimistic performance degradation of the evaluated model. W5: In the preprocessing stage of the paper, problem sets with fewer than 100 assignments were excluded, resulting in up to 36.7% of student interaction data being directly discarded. In the context of studying the concept drift of online learning platforms, newly added exercises or old exercises that are about to be eliminated in the long-tail distribution are often the most intense part of the platform's content evolution and data drift.
Questions for authors
Q1. Can the authors provide a robustness check that is strictly limited to "forward only" evaluations in chronological order (such as updating regression tables or drawing decay curves)? Q2. Could the authors provide a controlled experiment specifically targeting changes in P(Y|X)? Q3. SAKT-KC and SAKT-E share the same attention architecture but differ mainly in input granularity and vocabulary size; could the authors test whether the number of unique embedding IDs better predicts the rate of performance decline? Q4: If the interference of future data is excluded, will the performance degradation trend of the deep model become more apparent during cross-year testing? Q5: Can the author add a sensitivity analysis, such as retaining these low-frequency data or lowering the filtering threshold and recalculating the distribution drift index? If this large-scale data exclusion result in models having higher robustness than their actual performance on real education platforms.