acceptodds
Under review as a conference paper at ICLR 2027

RECAST: Learning-Science Inductive Biases for Knowledge Tracing

Abstract

Knowledge tracing predicts whether a learner will answer the next item correctly from their interaction history. Transformer KT models are strong sequence predictors, yet they typically treat tutoring logs as generic tokens and rely on Softmax attention to rediscover structure that learning science already identifies as central: last-attempt retrieval, local response persistence, spacing-dependent forgetting, and individualised cross-skill transfer. We encode these regularities as explicit inductive biases rather than incidental byproducts of attention. In this paper, we propose REcency- and Concept-lag Aware Skill Tracing (RECAST), a dual-stream causal Transformer with a Skill Recency Bank for last same-skill outcome, lag, and visit count; a Local Markov state for streaks and skill switches; concept-conditioned lag biases with distinct same- vs. cross-skill schedules; and a student factor that gates a mixture of low-rank transfer graphs. On five pyKT benchmarks with 5-fold evaluation, RECAST obtains the best AUC on four datasets and the best ACC on three against recurrent, memory, and Transformer baselines, with the largest gains when learners frequently revisit the same skill and competitive accuracy under heavily interleaved practice.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation Learning, Knowledge tracing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qrj2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RECAST, a Transformer-based model for knowledge tracing. RECAST predicts whether a student will answer the next question correctly based on their past interactions. The model introduces four components inspired by learning science. These components track the last response on the same skill, local response patterns, different forgetting patterns for same-skill and cross-skill interactions, and personalised transfer between skills. The authors evaluate RECAST on some public datasets and compare it with recurrent, memory-based, and Transformer-based models. RECAST achieves good AUC performance on four datasets and good accuracy performance on three. The improvements are larger when students frequently practise the same skill.

Critical strengths

The main components are simple and easy to understand. They make useful information, such as the last response on the same skill and recent response streaks, directly available to the model.

Critical weaknesses
  1. First concern: Local Markov consists of previous-response, streak-length, and skill-switch features. The lag/count features and temporal biases are related to DKT-Forgetting and HawkesKT, but the paper does not clearly explain its advances over these designs. The hard-versus-soft comparison also changes multiple components, so it cannot establish which proposed contribution drives the gains or whether hard recency is necessary.

  2. Second concern: Proposition 2 is invalid without constraints on the linear readout weights, since simple rescaling violates its bound. The behavioral diagnostics do not rule out ability, difficulty, or curriculum effects and therefore do not validate the claimed cognitive mechanisms. Meanwhile, Figure 1 and Eq. (3) specify different prediction inputs, concept-memory updates remain unclear, and backbone results differ across Tables 4–5 without explanation. These issues undermine both the central claims and reproducibility.

Questions for authors
  1. Local Markov largely comprises handcrafted history features. Differences from DKT-Forgetting, ELAKT, and HawkesKT are insufficiently explained, while relevant session-aware (e.g., HiTSKT) and graph-based (e.g., JKT, GKT) methods are missing from discussion or evaluation.

  2. Proposition 2 bounds the output of any linear readout without constraining its weights; simple rescaling invalidates the claimed bound. Proposition 1 largely restates its sufficiency assumption.

  3. The hard-versus-soft comparison does not isolate the representation change. Removing the Skill Recency Bank costs only 0.0023 AUC, insufficient to establish that hard recency is “necessary.”

  4. Predictive associations and learned attention biases do not establish testing, forgetting, or interference mechanisms.

  5. Dense, repetitive prose obscures missing implementation details. Figure 1 conflicts with Eq. (3), backbone results differ across Tables 4–5 without explanation, and template text and formatting errors remain.

1 comment

I look forward to the authors’ response and am open to further discussion.

Reviewer_K3WzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RECAST, a knowledge tracing model that explicitly incorporates behavioral signals motivated by learning science. Its dual-stream Transformer combines a Skill Recency Bank containing the last same-skill response, lag, and visit count; a Local Markov representation of response streaks and skill switches; separate attention biases for same-skill and cross-skill lags; and a learner-dependent mixture of low-rank transfer graphs. The paper provides theoretical arguments for hard last-attempt features and evaluates the model on five pyKT benchmarks. It also examines how performance varies across practice regimes and presents soft-versus-hard comparisons and module ablations on ASSIST2009.

Critical strengths

1-The distinction between the last response on the queried skill and persistence in the immediately preceding interaction sequence is meaningful. Combining these signals with separate same-skill and cross-skill lag functions and learner-dependent transfer provides a coherent architectural motivation. The named components make the modeling assumptions easier to inspect and ablate, and the explicit causal construction of the feature bank and learner summary is clearly described.

2-Under the reported pyKT five-fold protocol, RECAST achieves the highest AUC on four of five datasets and the highest ACC on three, with means and standard deviations provided. The improvements on ASSIST2015 and NIPS34 are particularly notable. The discussion of practice regimes, including the weaker result on OLI Statics, is also useful: it attempts to identify conditions under which the proposed inductive biases help. The existing ASSIST2009 ablations provide initial evidence that several components contribute to the full model.

Critical weaknesses

1-The proof correctly observes that, for (s=(1-\lambda)y_{\mathrm{last}}+\lambda s_{\mathrm{older}}), changing only the last response changes the EWMA feature by (1-\lambda). However, for an affine readout (f(s)=as+b), the corresponding output gap is (a(1-\lambda)). An unconstrained readout can choose (a=\Delta/(1-\lambda)) and recover the full gap (\Delta).

Thus, attenuation of the input feature does not imply the stated upper bound for “any linear readout.” The proof’s argument about amplifying older residuals would require explicit assumptions about nuisance variation, noise, regularization, or prediction risk. In the two histories considered by the proposition, the older summary is identical and cancels from the score difference.

This matters because Proposition 2 and Figure 2 are used to support the stronger claim that soft summaries cannot substitute for hard last-attempt features. The proposition and the associated interpretation in Section 5.3 should be corrected. The reported experiments can support an advantage over the tested soft-summary configurations, but they do not establish the stated general limitation.

2-Both the soft-versus-hard comparisons and the module ablations in Tables 4–5 are limited to ASSIST2009. However, the largest improvement occurs on ASSIST2015, where RECAST reaches 0.8162 AUC versus 0.7282 for the strongest reported baseline. This dataset also has a particularly high proportion of immediate same-skill revisits. Focused ablations there are important for determining whether the same mechanisms explain this substantially larger gain.

Lightweight predictors using the same explicit recency, count, streak, and switching features would also help establish how much of the improvement requires the proposed architecture. A feature-augmented conventional backbone would provide another useful control.

Furthermore, Table 4 does not clearly document a single-component replacement of hard recency with EWMA while keeping the remaining RECAST modules fixed. The full hard-recency configuration includes other components, and the backbone scores differ between Tables 4 and 5. These configurations should be clarified. The practice-regime analysis is informative, but the descriptive patterns across datasets do not resolve these attribution questions.

Questions for authors

1-What assumptions constrain the linear readout and prevent it from compensating for EWMA attenuation through rescaling? If no such constraints are intended, can the authors revise the proposition and the corresponding claims in Figure 2 and Section 5.3?

2-Can the key module ablations and a matched hard-to-soft recency substitution be evaluated on ASSIST2015, where the largest improvement occurs? The substitution should retain the other modules and training settings. Please also explain the distinction between the backbone configurations in Tables 4 and 5.

3-How do a regularized logistic regression or small MLP, and a conventional KT backbone augmented with the same explicit causal features, perform under the same evaluation protocol? These comparisons would help determine which benefits come from exposing the behavioral information and which require the full RECAST architecture.

Reviewer_ert3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RECAST, a knowledge tracing model that incorporates learning-science-inspired inductive biases into a dual-stream causal Transformer. Its Skill Recency Bank explicitly represents the most recent same-skill outcome, practice interval, and visit count, while a Local Markov module captures response streaks and skill switches. Concept-conditioned attention biases model different temporal effects for same-skill and cross-skill interactions, and a student factor controls individualized skill-transfer graphs. Experiments on five pyKT benchmarks compare RECAST with recurrent, memory-based, and Transformer baselines. RECAST achieves the highest AUC on four datasets and the highest accuracy on three, with larger gains in settings involving frequent same-skill practice.

Critical strengths
  1. Learning-science-motivated architecture. The paper incorporates learning behaviors, including last-attempt retrieval, response persistence, spacing, and individualized skill transfer, as explicit inductive biases within a unified Transformer architecture. This provides a coherent modeling rationale and allows individual components to be examined through ablation studies.

  2. Strong empirical performance across multiple benchmarks.RECAST achieves the best AUC on four of five datasets and the best accuracy on three, compared with a broad range of recurrent, memory-based, and Transformer baselines. The five-fold evaluation, module ablations, and analyses across different practice regimes provide useful empirical evidence for the proposed approach, although the contributions of individual mechanisms require further clarification.

Critical weaknesses
  1. Ambiguous evaluation protocol and insufficient explanation of performance gains. Section 5.1 reports concept-level evaluation under pyKT but does not clarify how multi-concept questions are handled during prediction. This distinction matters because pyKT identifies potential label leakage under one-by-one evaluation, and the reported ASSIST2009 baseline results are close to those obtained under that protocol. Separately, the unusually large AUC improvement on ASSIST2015 (+0.088) lacks a clear explanation and dataset-specific ablations. These issues make it difficult to fully assess the reliability and sources of the reported gains.

Insufficient theoretical justification and methodological specification. The theoretical claims in Section 4.1 require additional assumptions. 2. Proposition 1 does not establish that discretized lag features preserve all prediction-relevant information. Proposition 2 appears to require constraints on the linear readout; otherwise, unconstrained weights can compensate for EWMA scaling, leaving the stated attenuation bound unjustified. Furthermore, several architectural details, particularly the concept memory and student-factor modulation, are insufficiently specified, making reproducibility difficult. Closely related methods such as FoLiBiKT and LefoKT are also not adequately discussed, making the distinct contribution harder to assess.

Questions for authors
  1. Could the authors clarify whether the concept-level evaluation uses the leakage-free all-in-one protocol or the one-by-one protocol for multi-concept questions? Additionally, could they explain the unusually large AUC improvement on ASSIST2015 (+0.088) and provide ablations on this dataset to identify the main contributing components?

  2. In Proposition 1, under what conditions does lag discretization preserve the claimed sufficiency? For Proposition 2, how does the EWMA attenuation bound hold when the linear readout weights are unconstrained and can compensate for the scaling factor? Clarifying these assumptions would help assess the validity of the paper's theoretical contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.