- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a refreshing and well-engineered perspective on Knowledge Tracing by bridging continuous-time state-space dynamics, psychometric item response theory, and causal update decoupling. The benchmark re-evaluation is exceptionally rigorous, and the paper is well-written. However, the empirical justification for its central claim—that continuous multiscale dynamics drive the main performance leap—remains partially compromised by the dominant contribution of the episodic memory module and the lack of direct single-vs-multiscale ablations.
Critical strengths
-
The problem motivation is clear and relevant. Explicitly modeling continuous-time decay over latent mastery states is more principled than treating Δt only as an auxiliary feature.
-
The combination of ordered multi-timescale states, event-conditioned gating, and predict-observe-update separation is conceptually meaningful. In particular, separating response-driven evidence revision from practice-induced learning gain is a useful modeling idea.
-
The causal execution order is carefully described, and the appendix provides an online inference procedure. If implemented as stated, it reduces the risk of future-label leakage.
-
The experimental protocol for the first three datasets is relatively rigorous: the authors rerun 14 baselines under a pyKT-compatible protocol, use five-fold validation, and provide ablations, sequence-length slices, and time-gap slices.
-
The paper is generally well structured, with a complete symbol table, implementation details, reproducibility statement, and an explicit discussion of some limitations such as statistical identifiability.
Critical weaknesses
-
The central claim is not sufficiently supported by targeted ablations. The paper claims that multi-timescale continuous-time state evolution is the core contribution, but Table 4 does not directly ablate: single-scale vs. two-scale vs. three-scale states; continuous-time decay vs. Δt-only features; learnable gating vs. fixed mixing; the hard ordering constraints; and the separation of evidence revision from learning gain. Without these controls, the main claim remains under-supported.
-
Episodic memory and hierarchical 4PL appear to dominate the reported gains. In Table 4, removing episodic memory and hierarchical item-response causes the largest drops, while removing hierarchical state update causes only a small drop. This raises a legitimate concern that performance may be driven mainly by an external memory module and 4PL parameterization rather than by the proposed multi-timescale continuous-time state-space mechanism. A controlled experiment attaching episodic memory to strong baselines such as simpleKT or AKT is needed.
-
Direct continuous-time and multi-scale baselines are missing, and the XES3G5M comparison is not fully apples-to-apples. CT-NCM is the most direct continuous-time forgetting baseline but is absent from the main comparison; DMKT/FlucKT/TFKT are also not compared. On XES3G5M, CT-LFSSM uses timestamp and multi-KC inputs, while several published baselines may not have the same input conditions. Strong baselines should be rerun under matched timestamp/multi-KC settings.
-
Statistical significance and time-modeling evidence are insufficient. The paper fixes a single random seed (3407), and the reported five-fold standard deviations are fold-level rather than seed-level. Many gains are only 0.3–1.25 percentage points, close to baseline variance. Multi-seed runs, paired significance tests, or bootstrap confidence intervals are needed. In addition, ASSIST2009 sets Δt = 0 and cannot support the continuous-time claim; time-gap results should be added on ASSIST2012 and XES3G5M.
Questions for authors
1.Can you provide targeted ablation results comparing single-scale, two-scale and three-scale states, continuous-time decay against Δt-only features, learnable gating against fixed mixing, the effect of hard ordering constraints, and the separation of evidence revision from learning gain to validate the central claim of multi-timescale continuous-time state evolution?
2.Could you report controlled experiments that attach your episodic memory module to strong baselines such as simpleKT and AKT, to clarify whether the performance gains mainly come from episodic memory and hierarchical 4PL rather than the proposed multi-timescale continuous-time state-space mechanism?
3.Will you add comparisons against direct continuous-time and multi-scale baselines including CT-NCM, DMKT, FlucKT and TFKT, and rerun the relevant baselines on XES3G5M under matched timestamp and multi-KC input settings for a fair comparison?
4.Can you supply experimental results with multiple random seeds and paired significance tests or bootstrap confidence intervals to verify whether the reported performance improvements are statistically significant? Could you also present time-gap experimental results on ASSIST2012 and XES3G5M to support the continuous-time modeling claim, given that Δt is fixed to 0 in ASSIST2009?
6 comments
W2 — Separating the effects of episodic memory, hierarchical 4PL, and latent-state dynamics
Episodic retrieval, hierarchical item-response, and continuous-time KC states operate at different locations in the predict–observe–update computation. Episodic retrieval adds a bounded history-dependent observation-logit residual; hierarchical 4PL shares KC-level item parameters with item-specific residuals; the latent state evolves between interactions and is revised/updated after responses. Their contributions can therefore coexist rather than having to be attributed to a single module.
The submitted Table 4 establishes that episodic memory and hierarchical item-response are individually valuable. We present those effects explicitly, rather than treating the w/o hierarchical state update row as a test of the entire temporal-state model: that particular ablation replaces hierarchical evidence/consolidation parameterizations but retains three states and continuous-time decay.
Table R4.4. Submitted Table 4: actual held-out test AUC after component removal (five-fold mean ± fold-level population SD). Each w/o row denotes a model retrained without that component.
| Model | ASSIST2012 AUC ↑ | ASSIST2009 AUC ↑ | Algebra2005 AUC ↑ |
|---|---|---|---|
| Full CT-LFSSM | 0.7833 ± 0.0005 | 0.7873 ± 0.0019 | 0.8304 ± 0.0005 |
| w/o Episodic Memory | 0.7699 ± 0.0008 | 0.7511 ± 0.0006 | 0.8037 ± 0.0017 |
| w/o Hierarchical Item-Response | 0.7730 ± 0.0006 | 0.7680 ± 0.0022 | 0.8135 ± 0.0014 |
| w/o Cognition and Acquisition Gates | 0.7796 ± 0.0015 | 0.7818 ± 0.0016 | 0.8243 ± 0.0005 |
| w/o Outcome Conditioning | 0.7822 ± 0.0003 | 0.7857 ± 0.0017 | 0.8267 ± 0.0007 |
| w/o Hierarchical State Update | 0.7816 ± 0.0007 | 0.7863 ± 0.0017 | 0.8261 ± 0.0007 |
Removing episodic memory or hierarchical item-response consistently reduces AUC in all three datasets. The narrower hierarchical state-update ablation also lowers AUC, but retains the latent timescale count and the continuous-time decay mechanism; their specific effects are tested separately in Table R4.1–R4.3.
More focused attribution comes from the new experiments in Table R4.1–R4.3: episodic retrieval and hierarchical observation remain present while the active states, their mixing, or their explicit decay are
changed. Thus the performance difference attributable to these state-space operations is measured with the two strong prediction components held fixed.
We retrained ASSIST2012 controls that selectively remove the two post-response transitions, retaining episodic memory and hierarchical item-response modeling. NoRevision disables belief revision, NoLearning disables learning gain, and NoBoth disables both. Table R4.5 reports the submitted Full result and these five-fold controls.
Table R4.5. ASSIST2012 submitted Full result and five-fold post-response pathway controls (mean ± fold-level population SD; 524,251 test targets per fold).
| Model | AUC ↑ | ACC ↑ |
|---|---|---|
| Full CT-LFSSM | 0.7833 ± 0.0005 | 0.7582 ± 0.0003 |
| NoRevision | 0.7817 ± 0.0004 | 0.7568 ± 0.0004 |
| NoLearning | 0.7780 ± 0.0005 | 0.7553 ± 0.0002 |
| NoBoth | 0.7762 ± 0.0003 | 0.7547 ± 0.0001 |
Full CT-LFSSM attains the strongest reported mean AUC and ACC in Table R4.5. Its mean AUC exceeds NoRevision by +0.0016, NoLearning by +0.0053, and NoBoth by +0.0071. When belief revision is absent, retaining practice-induced learning increases AUC by +0.0055 (NoRevision − NoBoth); when practice-induced learning is absent, retaining belief revision raises AUC by +0.0018 (NoLearning −NoBoth). These two conditional effects are positive in all five folds, showing how the signed evidence correction and nonnegative practice-acquisition transitions contribute to future predictions.
W3 — Direct temporal baselines and explicit input conditions on XES3G5M
We expanded the direct temporal/forgetting-oriented comparisons with CT-NCM, DMKT, FlucKT, and LefoKT to ASSIST2012, ASSIST2009, and Algebra2005. Table R4.6 reports completed five-fold AUC and ACC (mean ± fold-level population SD), complementing the original submission’s broader baseline evaluation.
Table R4.6. Expanded direct temporal and multiscale baseline results on three datasets (five-fold mean ± fold-level population SD).
| Model | ASSIST2012 ACC ↑ | ASSIST2012 AUC ↑ | ASSIST2009 ACC ↑ | ASSIST2009 AUC ↑ | Algebra2005 ACC ↑ | Algebra2005 AUC ↑ |
|---|---|---|---|---|---|---|
| CT-NCM (2022) | 0.7569 ± 0.0025 | 0.7795 ± 0.0031 | 0.7355 ± 0.0015 | 0.7663 ± 0.0016 | 0.7994 ± 0.0006 | 0.8032 ± 0.0008 |
| DMKT (2025) | 0.7284 ± 0.0108 | 0.7527 ± 0.0144 | 0.7131 ± 0.0051 | 0.7416 ± 0.0067 | 0.7886 ± 0.0021 | 0.7958 ± 0.0024 |
| FlucKT (2025) | 0.7577 ± 0.0010 | 0.7809 ± 0.0019 | 0.7373 ± 0.0028 | 0.7731 ± 0.0033 | 0.8101 ± 0.0012 | 0.8280 ± 0.0010 |
| LefoKT (2025) | 0.7559 ± 0.0031 | 0.7792 ± 0.0035 | 0.7400 ± 0.0100 | 0.7858 ± 0.0025 | 0.8109 ± 0.0009 | 0.8272 ± 0.0010 |
| CT-LFSSM | 0.7582 ± 0.0003 | 0.7833 ± 0.0005 | 0.7419 ± 0.0014 | 0.7873 ± 0.0019 | 0.8168 ± 0.0016 | 0.8304 ± 0.0005 |
CT-LFSSM obtains the highest mean AUC and ACC among these direct baselines on all three datasets. For TFKT, our reproduction attempt did not yield a reliably validated comparison due to unresolved issues in the released implementation and its reported experimental configuration. Only completed baseline results are included.
For XES3G5M, the submitted Appendix B.3 reports CT-LFSSM under the official five-fold benchmark alongside 19 published benchmark baselines. These baseline values are not newly rerun results. To clarify the information-condition issue, we now show the benchmark’s explicit question-ID, KC-ID, and timestamp flags for representative methods.
Table R4.7. XES3G5M published official-benchmark results and reported input flags (submitted Appendix B.3).
| Model | Question ID | KC ID | Timestamp | AUC ↑ | ACC ↑ |
|---|---|---|---|---|---|
| DKT-F | No | Yes | Yes | 0.7940 | 0.8209 |
| SAINT | Yes | Yes | Yes | 0.8074 | 0.8177 |
| IEKT | Yes | Yes | No | 0.8280 | 0.8316 |
| CT-LFSSM | Yes | Yes | Yes | 0.8315 | 0.8380 |
In the published benchmark comparison, CT-LFSSM exceeds IEKT by +0.0035 AUC and +0.0064 ACC. SAINT has all three input flags marked as available in the benchmark table; CT-LFSSM exceeds its reported results by +0.0241 AUC and +0.0203 ACC. The table makes the reported question, KC, and timestamp input categories visible alongside the published benchmark metrics.
We sincerely thank the reviewer for the thorough assessment of the central contribution, component attribution, baseline fairness, and statistical reliability of CT-LFSSM. We address W1–W4 and Q1–Q4 in the order raised. Our additional evidence comprises five-fold active-timescale, fixed-mixing, unconstrained-half-life, and unified-update controls; an explicit-decay ablation on real-timestamp data; a four-model analysis of response-driven belief revision and practice-induced learning; additional direct temporal baselines; and paired, student-cluster bootstrap analysis.
W1 — Targeted evidence for multi-timescale continuous-time state evolution
CT-LFSSM maintains three KC-level latent states whose elapsed-time evolution is governed by student–KC-personalized half-lives. An event-conditioned convex gate reads these states before the current response; after observing the response, a signed evidence-based correction is followed by a nonnegative practice-induced learning transition. The corresponding mechanisms can be tested separately while retaining episodic memory and the hierarchical 4PL observation model. We evaluated the S1 Fast-only and S2 Fast + Slow active-scale controls on Algebra2005, retaining the same overall architecture and parameter budget. Inactive state slots were masked during readout and post-response updates, while episodic retrieval and hierarchical 4PL remained unchanged. Full CT-LFSSM uses all three timescales. The five-fold evaluation uses seed 3407 and 93,481 held-out test targets per fold.
Table R4.1. Algebra2005 Full three-scale result and newly trained S1/S2 active-scale controls (five-fold mean ± fold-level population SD).
Full CT-LFSSM achieves the highest reported mean AUC (0.8304) and ACC (0.8168). Relative to S1, the reported mean differences are +0.0080 AUC and +0.0071 ACC; relative to S2, they are +0.0038 AUC and +0.0042 ACC. Among the matched active-scale controls, S2 also improves AUC over S1 in all five folds (+0.0042 on average). These results support using multiple retention horizons and the complete three-scale design. The remaining controls retain three latent-state slots and isolate the readout or response-update parameterization. FixedMix replaces the event-conditioned scale gate by the constant vector (1/3, 1/3, 1/3). NoHalfLifeOrder replaces cumulative positive increments by independent trainable log half-lives, retaining the same initialization and numerical bounds. UnifiedUpdate replaces the sequential evidence-revision/learning transitions by a single signed saturating update while reusing the other model modules.
Table R4.2. Algebra2005 Full CT-LFSSM and additional mechanism controls (five-fold mean ± fold-level population SD).
Full CT-LFSSM attains the highest mean AUC and ACC in Table R4.2. Relative to FixedMix, the reported mean differences are +0.0057 AUC and +0.0074 ACC, supporting event-conditioned scale mixing. The half-life hierarchy guarantees Hu,n,k,f≤Hu,n,k,m≤Hu,n,k,s, preserving the meanings of the three retention horizons. Personalized half-life medians are 0.6183/14.8189/281.2777 days on ASSIST2012 and 0.3540/6.8189/104.2161 days on Algebra2005. NoHalfLifeOrder and UnifiedUpdate further test the half-life parameterization and the choice of separate response-update transitions.
We separately isolated explicit latent-state decay on ASSIST2012 by setting forgetting_mode=none while retaining the other elapsed-time inputs, episodic aging, dynamic scale gate, and response-update architecture. Both models were selected on validation AUC and evaluated on the same 524,251 held-out test targets (fold 0).
Table R4.3. ASSIST2012 explicit latent-decay control, fold 0 (Full S3 versus S3-NoDecay).
Because NoDecay still receives elapsed-time information outside the latent decay operator, this is a focused test of state evolution rather than a removal of timestamps from the complete model. It improves all five aggregate metrics in this fold. Together, the scale-count, gating, and NoDecay controls isolate three distinct aspects of the central state-space mechanism.