acceptodds
Under review as a conference paper at ICLR 2027

EMBEDDING-ONLY KNOWLEDGE TRACING: PREDICTING RESPONSES WITHOUT KNOWLEDGE COMPONENT ANNOTATIONS

Abstract

Knowledge tracing (KT) predicts a learner's next response from the response history, and adaptive systems use it to choose the next question and to find learners who need support. Standard KT methods rely on expert-defined knowledge components (KCs) with a costly concept vocabulary and on a learned question-ID table, which has no row for questions absent from training. Embedding-only Knowledge Tracing (EoKT) feeds fixed general-purpose question-text embeddings to a compact recurrent predictor, so it needs neither KC annotations nor an ID table. On MOOCRadar and XES3G5M, we change only the question-side input of this predictor and score warm questions and cold questions withheld from training. In standard AUC, which pools test responses to almost entirely seen questions, EoKT reached the validation-selected KC-based method, with paired differences of +0.0056 on MOOCRadar and +0.0046 on XES3G5M under unequal search budgets. In the same predictor, a learned ID table was at least as good on warm questions, and adding learned or pretrained KC inputs did not raise AUC. In cold AUC, which ranks each learner's responses to questions withheld from training, text embeddings reached 0.6041 on MOOCRadar and 0.6489 on XES3G5M, against 0.5047 and 0.5896 for learned IDs, which cannot distinguish withheld questions; the ordering held for three embedding models. Text embeddings from gemini-embedding-2 also exceeded permuted embeddings on both datasets by +0.0848 and +0.0833 in cold AUC, with 95% intervals that exclude zero when learners and questions are both resampled, so matching each question to its own text contributes to the gain. With question representations precomputed, EoKT returns one prediction from a retained state in about 0.50 ms on a single CPU thread. EoKT can be trained across datasets by simple concatenation: one model pooled over three datasets that differ in language and subject lost at most 0.0012 AUC against per-dataset models in a single-fold check. These results support EoKT as a practical KC-free knowledge tracing method whose text input also supports prediction on questions absent from training.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
knowledge tracing, text embeddings, knowledge components, cold-start, student response prediction, educational data mining

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_57kSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper discusses the reliance of existing knowledge tracing research on knowledge components (KCs), which require costly annotation work. The paper proposes "Embedding-Only Knowledge Tracing (EoKT)," a method that predicts the correctness of answers to new questions based solely on the question text and answer history. EoKT does not require KCs or question-specific ID learning. The EoKT model proposed in this paper is lightweight and embedding-based. It takes question and answer histories as input. Because various studies have been conducted on text-based KT models, the novelty of the EoKT concept itself is limited. Furthermore, based on the EoKT description in Section 2, it does not appear to offer significant technical innovation. If the EoKT had demonstrated performance surpassing the latest KC-based models, it would likely be considered a breakthrough. Unfortunately, however, it does not present such results.

Critical strengths
  • S1: The anonymous supplement contains reproduction code and aggregate results.

  • S2: The paper describes the limitations of this study.

  • S3: This study involved extensive experimentation.

Critical weaknesses
  • W1: Several text-based knowledge tracing models have been proposed in the past, so this idea is not new. What makes the proposed EoKT model interesting is that it does not use knowledge components (KC). However, it has not yet been compared to the latest models that use KC. For example,

    • Li, Z., Gan, J., Zhou, J., Wang, J., & Wei, W. (2026). ExerCAKT: A Knowledge Tracing Model Based on GRU Capturing Contextual Features of Exercises. Information, 17(6), 578. (This is not in the reference list. )

    • Ziwei Wang, Jie Zhou, Qin Chen, Bo Jiang, Qingchun Bai, Liang Dou, and Liang He (2026). LLMKT: Enhancing Large Language Models with Knowledge Tracing via Multi-Level Plug-and-Play Alignment. Findings of the Association for Computational Linguistics: ACL 2026, 35775–35792. https://aclanthology.org/2026.findings-acl.1781/ (This is included in the reference list, but is not included in baseline models. )

  • W2: The baseline method consisted solely of the basic knowledge tracing model. No comparison was made with state-of-the-art models that incorporate knowledge capture (KC) as input, such as LLM-KT.

  • W3: Although the report on the experimental results is objective, it lacks a discussion of the reasons behind the results. For example, see the last paragraph of Section 4.1 on page 5.

Questions for authors
  • Why didn't you compare the latest text-based KT model with KC to the EoKT?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 57kS

Thank you for the review and for pointing us to ExerCAKT, which we had missed.

W1 / W2 / Question — Comparison with recent KC-based models. The submission compared EoKT only with the KC-based method selected on validation among standard baselines. We have now added ExerCAKT, reimplemented from the code in its supplementary material and trained under our protocol (5 folds × 2 seeds; hyperparameters chosen on validation only). EoKT − ExerCAKT in standard AUC is +0.0281 [+0.0264, +0.0300] on MOOCRadar, +0.0132 [+0.0126, +0.0138] on XES3G5M, and +0.0056 [+0.0050, +0.0063] on NIPS34 (intervals resample learners). On questions withheld from training (MOOCRadar; Table 5), EoKT reaches 0.6041 and ExerCAKT 0.5358: +0.0683 [+0.0015, +0.1504], resampling learners and questions. The lower bound is close to zero, and this holdout is exploratory, as for the other MOOCRadar cold comparisons in the paper. ExerCAKT uses no question text, so only its KC inputs carry over to an unseen question.

One caveat: our reimplementation does not reach the published number. ExerCAKT reports 0.8051 on NIPS34; ours gives 0.7989 under the same scoring, and we could not identify the cause (the selected hyperparameters are not published). Against the published number itself, EoKT is level: 0.8051 under All-in-One scoring, where our chunks are 200 responses and ExerCAKT's are 200 KC-expanded rows. NIPS34 informed our model configuration, so this is a check rather than independent confirmation. We therefore read the result as "on par with a recent KC-based model on NIPS34, without KC annotations", not as a win. Appendix C.4 reports both.

LLM-KT adds question- and concept-specific tokens and a sequence-model state to an LLM and runs it for every prediction; we did not train it. The language-model predictor we ran, LKT, is below EoKT in standard AUC on both datasets (Table 2) and takes 769.45 ms per prediction against 0.50 ms; nine of its ten XES3G5M runs reached the training-time limit.

W1 — What we claim. Thank you for this assessment. We agree that text-based KT is not new and that EoKT is not a new state of the art. The claim is the one you identified as its interest: it needs no KC annotations, and it still matches the validation-selected KC-based method (+0.0056 / +0.0046, Table 3).

W3 — Reasons behind the results (Sections 4.1 and 5). Thank you for this suggestion. We added the reason to Section 4.1 and rewrote the Discussion around it. Adding KC inputs to EoKT does not raise standard AUC. We interpret this as follows: on questions seen in training, the question-side vector already tells every question apart, and a KC is a coarser grouping of the same questions. A control is consistent with this: replacing the text embeddings with a learned ID table does not lower standard AUC (ID table − EoKT: 0.0000 on MOOCRadar and +0.0023 on XES3G5M). Text matters where an ID table has nothing to offer: on questions absent from training.

Reviewer_PztsOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1(−1)
Read review
Paper summary

This paper proposes Embedding-only Knowledge Tracing (EoKT), which uses fixed question-text embeddings and a recurrent model to predict student responses without KC annotations or learned question-ID embeddings. The authors evaluate EoKT on MOOCRadar and XES3G5M, focusing on both standard prediction and questions withheld from training. The results show competitive standard AUC and better within-learner cold AUC than question-ID baselines. The paper also examines different embedding models, text permutations, inference speed, and joint training across datasets.

Critical strengths

EoKT offers a practical way to handle new questions without KC annotations or question-specific training. The comparison between text embeddings, question IDs, and KC inputs is useful, and the permutation experiment provides evidence that question content contributes to cold-question prediction.

The experiments cover several aspects of the method, including different embedding models, warm and cold questions, and inference efficiency. The authors also report uncertainty across learners and questions and discuss the limitations of their experimental settings.

Critical weaknesses

I find the novelty somewhat limited. Previous methods such as EERNN, SINKT, and Text-LENS have already explored question-text representations and unseen-question prediction. EoKT uses fixed pretrained embeddings with a recurrent predictor, but it is unclear what new insight this provides beyond simplifying existing approaches. The absence of direct cold-question comparisons with SINKT and Text-LENS makes this harder to judge.

I have concerns about the cold-question evaluation. The within-learner cold AUC compares predictions made at different points in a student's history, so changes in the learner state may also affect the ranking. The question-ID baseline cannot distinguish unseen questions by design, making part of the improvement expected. The permutation experiment is more informative, but it still does not show how well the model distinguishes unseen questions given the same learner state.

Questions for authors

What is the main difference between EoKT and existing text-aware KT methods, particularly SINKT and Text-LENS? Could the authors provide a more direct comparison on unseen questions?

How much of the cold AUC improvement comes from question representations rather than changes in learner states? Would the advantage remain when different unseen questions are evaluated using the same learner state?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Pzts

Thank you for the detailed assessment. Both of your concerns about the cold evaluation led to analyses that changed the paper.

Same learner state. Thank you for suggesting this analysis. For each learner we fixed the state just before their first withheld question and scored every withheld question they answered from that one state, so that ranking differences can only come from the question representation (Appendix C.1). You were right on two points. A question-ID table then scores exactly 0.5 by construction; its 0.5896 on XES3G5M under the original metric came from state changes alone. EoKT's cold AUC also drops: 0.6041 → 0.5820 on MOOCRadar and 0.6489 → 0.5706 on XES3G5M. What remains is the difference between text and permuted embeddings, which is as large as before: +0.0963 [+0.0079, +0.1866] and +0.0795 [+0.0400, +0.1202] (original metric: +0.0848 and +0.0833). The MOOCRadar interval is wide because all learners share the same 220 withheld questions.

SINKT and Text-LENS. Thank you for pointing to these methods. Table 1 now compares requirements in the main text. SINKT needs KC annotations and an LLM-generated concept graph, and its inductive result is on one proprietary dataset. Its public code has no license statement and includes neither preprocessing code for new datasets nor the inductive protocol, so we reimplemented it from the paper and ran it on our MOOCRadar holdout with MOOCRadar's KC annotations, an LLM-generated concept graph, and the same fixed question embeddings as EoKT (Table 5 and Appendix C.3; we do not claim identity with the original). It reaches 0.6193 against 0.6041 for EoKT: −0.0152 [−0.0639, +0.0429] on cold questions, where it is higher in 9/10 runs, and +0.0110 [+0.0057, +0.0173] for EoKT on warm questions. Text-LENS is not a sequential KT model: it predicts a query item from 19 answered items and is compared only with LENS. We are not aware of released code, so we instead added a question holdout on NIPS34, which draws on the same source data (+0.0970 [+0.0469, +0.1538] over permuted embeddings; Appendix C.2).

What is new. Thank you for raising this question. Not the use of question text. EoKT shows how little is needed: without KC annotations, an ID table, or encoder training, it is not separable on cold questions from a method that uses KC annotations and a concept graph, and it matches the validation-selected KC-based method in standard AUC. The fixed-state and history-free analyses separate what the question text contributes from what the response history contributes.

Thank you for the response. While the additional explanations address some of my concerns, my main concern remains novelty. Designing a KT model that handles newly introduced questions without relying on question IDs or KCs is interesting. However, it is still unclear what distinct advantages this approach offers over existing LLM-based and text-based KT models. Beyond performance gains, the methodological contribution relative to prior text-based KT methods is not sufficiently convincing, and direct comparative experiments supporting it are still lacking. I find the problem and direction interesting, but given the limited methodological novelty and insufficient comparisons, I will lower Soundness my score.

Thank you for the follow-up and for finding the problem and direction interesting. We agree that EoKT's architecture is deliberately simple, and we see the contribution as subtractive. Under one shared predictor, we remove one input at a time and test whether it is needed. On these datasets, KC annotations and a question-ID table are not needed for accuracy on seen questions: replacing the text with an ID table or adding KCs moves standard AUC by at most 0.0037 (Table 3). The insight is why: on seen questions any input that tells questions apart suffices, so question content matters only for unseen questions, and standard AUC cannot show what text contributes (Section 5). Because each removal is tested against a control, this is a finding about what KT needs, not only a simplification.

Direct comparisons. On the same MOOCRadar question holdout, Table 5 (Section 4.2) compares cold AUC with methods that add what EoKT removes: EERNN-A, which trains its text encoder with KT (0.5950); AKT with the same fixed embeddings (0.5481); a SINKT-like reimplementation with KC annotations and an LLM-generated concept graph (0.6193); and ExerCAKT (0.5358), against 0.6041 for EoKT. None of them is above EoKT with an interval that excludes zero; the SINKT-like model is numerically higher. For a language-model predictor, Table 2 reports LKT on our partition: 0.7505 on MOOCRadar and 0.7898 on XES3G5M in standard AUC, against 0.9015 and 0.8288. We did not run language-model predictors on the cold holdout.

What the removals buy. A new question needs only one embedding call: no KC annotation, concept graph, answer key, or encoder training (Table 1). Language-model predictors such as LKT run the language model for each prediction (Section 1); EoKT does not, and an incremental prediction takes 0.50 ms on one CPU thread (Table 6).

Evaluating the question representation. Scoring cold questions from a fixed learner state, as you suggested, separates the question representation from state changes (Section 4.2). An ID table then scores 0.5 by construction, and EoKT drops to 0.5820 on MOOCRadar and 0.5706 on XES3G5M, but its margin over permuted embeddings remains (+0.0963 and +0.0795). This protocol applies to any text-aware KT method.

Supporting details in the appendices. Appendix C.1 gives the fixed-state procedure and the intervals. Appendix G describes our LKT implementation; on XES3G5M, nine of its ten runs stopped at a 36-hour training cap and are not converged. Appendix F (Table 31) times LKT: one forward pass of its language model takes about 770 ms.

We understand if this does not change your assessment, and we appreciate the discussion.

Reviewer_E6CoOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EoKT predicts learners' responses using frozen question-text embeddings and a GRU-based learner state, without knowledge component (KC) annotations or learned question-ID embeddings. Experiments on MOOCRadar and XES3G5M evaluate standard prediction, held-out questions, and different text embedding models. The paper also reports inference latency with precomputed embeddings and explores pooled training across three datasets.

Critical strengths
  1. Removing the need for KC annotations and question-specific embeddings is practically useful. New questions can be represented directly from their text, without manually assigning KCs or learning a separate embedding for each question. The reported standard AUCs of 0.9015 on MOOCRadar and 0.8288 on XES3G5M show that this simple design can achieve reasonable predictive performance.

  2. The cold-question experiments provide evidence that the text embeddings contain useful information. Permuting the question-to-embedding mapping reduces cold AUC from 0.6041 to 0.5193 on MOOCRadar and from 0.6489 to 0.5656 on XES3G5M. Similar results with BGE-M3 and multilingual-e5-large suggest that the method is not tied to a particular embedding model. The reported 0.50 ms incremental inference latency is also promising when question embeddings are precomputed.

Critical weaknesses
  1. The contribution over existing text-based knowledge tracing methods is not established. The paper discusses ESKT and Text-LENS but does not compare them under the same held-out-question setting. The current ablations show that question-text embeddings are useful, but this alone does not establish an improvement over prior methods that also use textual information.

  2. The history-free comparison does not isolate the effect of learner history. TEXTONLY in Appendix I differs from EoKT in model capacity, IRB, and time-gap inputs. The first-encounter AUC difference therefore cannot be attributed to learner history alone. A control retaining the same question-side modeling and prediction head would be more informative.

Questions for authors
  1. Which existing text-based knowledge tracing method is the closest feasible baseline? Can EoKT and that method be evaluated under the same question holdout and scoring protocol?

  2. Can the authors provide a history-free control that retains EoKT's question encoder, IRB, and prediction head, using the same first-encounter cohort as Appendix I?

  3. How should the cold-question results be interpreted relative to existing text-based methods? What evidence supports an improvement in generalization to unseen questions beyond the permutation control?

3 comments
AuthorsOpenReview ↗

Response to Reviewer E6Co

Thank you for the specific questions. We answer them in order.

W1 / Q1 — Closest feasible baseline. Thank you for this question. ESKT encodes each learner's chosen option, which correctness-only logs do not record, and we are not aware of a public implementation of ESKT or of its ablation ESKT-AM. Two baselines can be run on our holdout. EERNN-A trains its text encoder jointly and uses no KC annotation: 0.5950 on MOOCRadar against 0.6041 for EoKT (+0.0091 [−0.0539, +0.0869]) and 0.6132 against 0.6489 on XES3G5M (+0.0357 [+0.0131, +0.0619]; its stopping rule differs there, as noted in the paper). Following ESKT-AM's design, we also replaced AKT's question-difficulty table with the same frozen text embeddings (AKT's KC-ID tables remain): 0.5481, i.e. +0.0560 [−0.0114, +0.1305] for EoKT (higher in 10/10 runs; the interval includes zero). Table 5 compares these baselines and Appendix C.3 gives details; the EERNN-A intervals are new and are in Table 4. Text-LENS is not a sequential KT model; see Table 1 and our reply to Reviewer Pzts.

W2 / Q2 — History-free control. Thank you for suggesting this control. We removed only the history path (GRU, state–question term, time-gap input) and kept the question projection, IRB, and prediction head, on the first-encounter cohort of Appendix I. On MOOCRadar the control has the same point estimate as EoKT: 0.6048 vs. 0.6048 (difference 0.0000 [−0.0437, +0.0580]). Your concern was correct there: the history-free control reproduces the gap to TEXTONLY (+0.0383), so that gap cannot be attributed to history. On XES3G5M, history adds +0.0732 [+0.0501, +0.0989]. History matters far more for AUC over all test responses, which falls from 0.8929 to 0.7547 and from 0.8201 to 0.7118 without it. We rewrote Section 4.2 and Appendix I accordingly.

W1 / Q3 — Interpreting the cold results. Thank you for asking this directly. We do not claim an improvement over existing text-based methods. A SINKT-style reimplementation that uses KC annotations and a concept graph is numerically higher on cold questions (EoKT − SINKT-style = −0.0152 [−0.0639, +0.0429]) and lower on warm questions (+0.0110 [+0.0057, +0.0173]). Beyond the permutation control, three results support that question content generalizes to unseen questions: the text–permuted difference persists when the learner state is fixed (Appendix C.1), it repeats on a third dataset (+0.0970 [+0.0469, +0.1538] on NIPS34, Appendix C.2), and the history-free control above reaches 0.6048 on MOOCRadar from the question side alone.

Thank you for your rebuttal. I will improve my score to 2.

Thank you for reading our response and for raising your score. Your Q2 led to the history-free control, which changed how we report the first-encounter results in Section 4.2. If any concerns remain, we would be glad to address them before the discussion period closes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers for their careful reading. The reviews ask how EoKT relates to text-based KT and whether the cold-question evidence isolates the question representation. We ran additional analyses under the paper's protocol; intervals are 95% and resample learners and questions.

  1. Fixed learner state. Scoring every withheld question a learner answered from one fixed state, the difference between text and permuted embeddings remains: +0.0963 [+0.0079, +0.1866] on MOOCRadar and +0.0795 [+0.0400, +0.1202] on XES3G5M. The absolute cold AUC drops (0.6041 → 0.5820; 0.6489 → 0.5706), so part of the original metric did reflect state changes.
  2. History-free control with the same question side. On first encounters it has the same point estimate as EoKT on MOOCRadar (0.6048 vs. 0.6048; difference 0.0000 [−0.0437, +0.0580]) and falls below it on XES3G5M (0.5693 vs. 0.6425).
  3. Third dataset. On NIPS34, text exceeds permuted embeddings by +0.0970 [+0.0469, +0.1538].
  4. Text-aware baselines on the same MOOCRadar holdout (new Table 5). EoKT 0.6041; EERNN-A 0.5950; AKT with fixed text embeddings 0.5481; a SINKT-style reimplementation, which also uses KC annotations and an LLM-generated concept graph, 0.6193. None of these three differences from EoKT has an interval that excludes zero.
  5. ExerCAKT, reimplemented from the authors' supplementary code: EoKT is higher in standard AUC on all three datasets and on cold questions. The reimplementation falls short of the published number, which we report.

We do not claim that EoKT outperforms text-based or KC-based methods in general. The revision states a simpler claim: on these datasets, with general-purpose question-text embeddings, neither KC annotations nor a question-ID table is needed for accuracy on seen questions, and unlike an ID table, the text also works for questions absent from training. Three results support it: on seen questions an ID table is at most 0.0023 higher; it cannot rank unseen questions from a fixed learner state; and on unseen questions the text content matters, with EoKT not below methods using KCs, concept graphs, or ID tables on the exploratory MOOCRadar holdout. We do not claim that KC information is unnecessary in general.

Revision. (a) A requirement comparison of prior text-aware methods is now Table 1 in the main text. (b) Section 4.2 adds Table 5, comparing cold AUC with text-aware and KC-based methods on the same holdout. (c) The introduction now states this claim, and the discussion and conclusion are reorganized around it. (d) Appendices C.1–C.4 report the new analyses, and Appendix I is rewritten around the new control. (e) Experiments 3 (embedding models), 4 (numeric variants), and 6 (pooled training) of the submission move to Appendices J, D, and E; Sections 4.2 and 4.4 summarize them. Tables 1, 3, and 4 of the submission are Tables 2, 4, and 38 of the revision. In the revised PDF, added or revised text is blue and text moved from the main text to the appendix is orange.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.