- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper discusses the reliance of existing knowledge tracing research on knowledge components (KCs), which require costly annotation work. The paper proposes "Embedding-Only Knowledge Tracing (EoKT)," a method that predicts the correctness of answers to new questions based solely on the question text and answer history. EoKT does not require KCs or question-specific ID learning. The EoKT model proposed in this paper is lightweight and embedding-based. It takes question and answer histories as input. Because various studies have been conducted on text-based KT models, the novelty of the EoKT concept itself is limited. Furthermore, based on the EoKT description in Section 2, it does not appear to offer significant technical innovation. If the EoKT had demonstrated performance surpassing the latest KC-based models, it would likely be considered a breakthrough. Unfortunately, however, it does not present such results.
Critical strengths
-
S1: The anonymous supplement contains reproduction code and aggregate results.
-
S2: The paper describes the limitations of this study.
-
S3: This study involved extensive experimentation.
Critical weaknesses
-
W1: Several text-based knowledge tracing models have been proposed in the past, so this idea is not new. What makes the proposed EoKT model interesting is that it does not use knowledge components (KC). However, it has not yet been compared to the latest models that use KC. For example,
-
Li, Z., Gan, J., Zhou, J., Wang, J., & Wei, W. (2026). ExerCAKT: A Knowledge Tracing Model Based on GRU Capturing Contextual Features of Exercises. Information, 17(6), 578. (This is not in the reference list. )
-
Ziwei Wang, Jie Zhou, Qin Chen, Bo Jiang, Qingchun Bai, Liang Dou, and Liang He (2026). LLMKT: Enhancing Large Language Models with Knowledge Tracing via Multi-Level Plug-and-Play Alignment. Findings of the Association for Computational Linguistics: ACL 2026, 35775–35792. https://aclanthology.org/2026.findings-acl.1781/ (This is included in the reference list, but is not included in baseline models. )
-
-
W2: The baseline method consisted solely of the basic knowledge tracing model. No comparison was made with state-of-the-art models that incorporate knowledge capture (KC) as input, such as LLM-KT.
-
W3: Although the report on the experimental results is objective, it lacks a discussion of the reasons behind the results. For example, see the last paragraph of Section 4.1 on page 5.
Questions for authors
- Why didn't you compare the latest text-based KT model with KC to the EoKT?
Response to Reviewer 57kS
Thank you for the review and for pointing us to ExerCAKT, which we had missed.
W1 / W2 / Question — Comparison with recent KC-based models. The submission compared EoKT only with the KC-based method selected on validation among standard baselines. We have now added ExerCAKT, reimplemented from the code in its supplementary material and trained under our protocol (5 folds × 2 seeds; hyperparameters chosen on validation only). EoKT − ExerCAKT in standard AUC is +0.0281 [+0.0264, +0.0300] on MOOCRadar, +0.0132 [+0.0126, +0.0138] on XES3G5M, and +0.0056 [+0.0050, +0.0063] on NIPS34 (intervals resample learners). On questions withheld from training (MOOCRadar; Table 5), EoKT reaches 0.6041 and ExerCAKT 0.5358: +0.0683 [+0.0015, +0.1504], resampling learners and questions. The lower bound is close to zero, and this holdout is exploratory, as for the other MOOCRadar cold comparisons in the paper. ExerCAKT uses no question text, so only its KC inputs carry over to an unseen question.
One caveat: our reimplementation does not reach the published number. ExerCAKT reports 0.8051 on NIPS34; ours gives 0.7989 under the same scoring, and we could not identify the cause (the selected hyperparameters are not published). Against the published number itself, EoKT is level: 0.8051 under All-in-One scoring, where our chunks are 200 responses and ExerCAKT's are 200 KC-expanded rows. NIPS34 informed our model configuration, so this is a check rather than independent confirmation. We therefore read the result as "on par with a recent KC-based model on NIPS34, without KC annotations", not as a win. Appendix C.4 reports both.
LLM-KT adds question- and concept-specific tokens and a sequence-model state to an LLM and runs it for every prediction; we did not train it. The language-model predictor we ran, LKT, is below EoKT in standard AUC on both datasets (Table 2) and takes 769.45 ms per prediction against 0.50 ms; nine of its ten XES3G5M runs reached the training-time limit.
W1 — What we claim. Thank you for this assessment. We agree that text-based KT is not new and that EoKT is not a new state of the art. The claim is the one you identified as its interest: it needs no KC annotations, and it still matches the validation-selected KC-based method (+0.0056 / +0.0046, Table 3).
W3 — Reasons behind the results (Sections 4.1 and 5). Thank you for this suggestion. We added the reason to Section 4.1 and rewrote the Discussion around it. Adding KC inputs to EoKT does not raise standard AUC. We interpret this as follows: on questions seen in training, the question-side vector already tells every question apart, and a KC is a coarser grouping of the same questions. A control is consistent with this: replacing the text embeddings with a learned ID table does not lower standard AUC (ID table − EoKT: 0.0000 on MOOCRadar and +0.0023 on XES3G5M). Text matters where an ID table has nothing to offer: on questions absent from training.