- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RECAST, a Transformer-based model for knowledge tracing. RECAST predicts whether a student will answer the next question correctly based on their past interactions. The model introduces four components inspired by learning science. These components track the last response on the same skill, local response patterns, different forgetting patterns for same-skill and cross-skill interactions, and personalised transfer between skills. The authors evaluate RECAST on some public datasets and compare it with recurrent, memory-based, and Transformer-based models. RECAST achieves good AUC performance on four datasets and good accuracy performance on three. The improvements are larger when students frequently practise the same skill.
Critical strengths
The main components are simple and easy to understand. They make useful information, such as the last response on the same skill and recent response streaks, directly available to the model.
Critical weaknesses
-
First concern: Local Markov consists of previous-response, streak-length, and skill-switch features. The lag/count features and temporal biases are related to DKT-Forgetting and HawkesKT, but the paper does not clearly explain its advances over these designs. The hard-versus-soft comparison also changes multiple components, so it cannot establish which proposed contribution drives the gains or whether hard recency is necessary.
-
Second concern: Proposition 2 is invalid without constraints on the linear readout weights, since simple rescaling violates its bound. The behavioral diagnostics do not rule out ability, difficulty, or curriculum effects and therefore do not validate the claimed cognitive mechanisms. Meanwhile, Figure 1 and Eq. (3) specify different prediction inputs, concept-memory updates remain unclear, and backbone results differ across Tables 4–5 without explanation. These issues undermine both the central claims and reproducibility.
Questions for authors
-
Local Markov largely comprises handcrafted history features. Differences from DKT-Forgetting, ELAKT, and HawkesKT are insufficiently explained, while relevant session-aware (e.g., HiTSKT) and graph-based (e.g., JKT, GKT) methods are missing from discussion or evaluation.
-
Proposition 2 bounds the output of any linear readout without constraining its weights; simple rescaling invalidates the claimed bound. Proposition 1 largely restates its sufficiency assumption.
-
The hard-versus-soft comparison does not isolate the representation change. Removing the Skill Recency Bank costs only 0.0023 AUC, insufficient to establish that hard recency is “necessary.”
-
Predictive associations and learned attention biases do not establish testing, forgetting, or interference mechanisms.
-
Dense, repetitive prose obscures missing implementation details. Figure 1 conflicts with Eq. (3), backbone results differ across Tables 4–5 without explanation, and template text and formatting errors remain.
I look forward to the authors’ response and am open to further discussion.