acceptodds
Under review as a conference paper at ICLR 2027

Understanding Semantic Generalization in Knowledge Tracing through Difficulty Supervision

Abstract

Knowledge tracing (KT) predicts students' responses from interaction histories, but new questions have no training responses. Knowledge components (KCs) can be shared across questions, while pretrained question semantics add limited or mixed gains beyond them. A privileged-information experiment shows that empirical item difficulty can be highly predictive. However, measuring empirical difficulty requires student responses, which are unavailable for newly introduced questions. We therefore study whether difficulty can link response patterns from training questions to the content of unseen ones. To study this transfer, we adapt a pretrained semantic encoder with discrete difficulty classification and continuous correctness regression, then use its frozen representations for KT. We also test generated solutions as an augmentation to question content. Across three datasets and six KT backbones, adaptation improves unseen-item prediction, with controls linking the gains to content–difficulty correspondence. Probes find more accessible correctness information, while exposure experiments show that the readout advantage diminishes as item-specific responses accumulate. Together, these results show how difficulty supervision transfers information from observed student responses through question content to unseen items.

Primary area:
applications to neuroscience & cognitive science
Keywords:
AI for Education, Knowledge Tracing, Difficulty Prediciton, Out of Distribution Generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vDBuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether difficulty supervision can improve semantic generalization in knowledge tracing (KT), especially for unseen questions. A pretrained semantic encoder is fine-tuned with difficulty labels derived from historical student responses, combining discrete difficulty classification with continuous correctness regression, and the adapted representations are then fed into downstream KT models. The authors also try LLM-generated solutions as additional question content. Results on three datasets and six KT backbones show consistent gains in unseen-item prediction, and the paper includes shuffled-label controls, representation probing, and controlled response-exposure analyses to examine what the difficulty signal actually contributes.

Critical strengths
  1. Consistent improvements across datasets and backbones. The paper evaluates difficulty-supervised representations on three datasets and six KT backbones, demonstrating consistent improvements in unseen-item AUC over the original pretrained text representations. This provides encouraging evidence that the proposed adaptation strategy is not limited to a particular KT architecture or dataset, supporting the practical relevance of the approach.

  2. Systematic investigation of difficulty supervision and semantic generalization.​ Beyond improving prediction performance, the paper provides a systematic empirical investigation of how difficulty supervision influences question representations and their generalization to unseen items. These analyses offer useful insights into the role of difficulty information in knowledge tracing and strengthen the overall contribution of the work.

Critical weaknesses
  1. Novelty over prior work is not well established. Using difficulty information derived from student responses to improve question representations has been studied before, e.g., PEBG (Improving Knowledge Tracing via Pre-training Question Embeddings (2020)), DCL4KT (Difficulty-Focused Contrastive Learning for Knowledge Tracing with a Large Language Model-Based Difficulty Prediction (2024)), and PMQE (Pre-Training Multi-Concept Question Embeddings for Knowledge Tracing (2025)), all of which are conceptually close to this work. I note that the paper does include some discussion of these prior works; however, the discussion remains at a conceptual level, and the stated differences do not convincingly show what the proposed method actually improves over them, or why its design should be preferred. In addition, the comparisons against QDCKT and RAGKT use simplified re-implementations rather than the full original methods, so the current results say little about how the approach stacks up against the actual state of the art.

  2. The joint classification + regression objective is not justified. Table 4 is hard to reconcile with the proposed design: the joint CE+MAE objective never achieves the best result on any of the three datasets — regression-only is better on XES3G5M, and classification-only is better on DBE-KT22 and MOOCRadar. If single-objective supervision is consistently stronger, it is unclear why the joint objective is adopted as the main method, and the paper offers little analysis of this discrepancy.

Questions for authors
  1. I notice the paper already contains some analysis and discussion of prior works such as PEBG, DCL4KT, and PMQE, but the differences identified there do not clearly demonstrate the proposed method's improvement over them. Could the authors clarify the key differences between the proposed method and these prior works? Given their substantial conceptual overlap, what are the specific advantages of the proposed approach, and what evidence supports these advantages over existing methods?

  2. Why is joint CE+MAE supervision the main objective when Table 4 shows single-objective supervision performing better on all three datasets? Please explain the motivation, and analyze why the better-performing objective flips across datasets.

3 comments

Thank you for recognizing the value of our experimental analysis and for your suggestions.

W1 / Q1: Relationship to prior work

We agree that PEBG, DCL4KT, and PMQE have already investigated the use of difficulty information in KT.

These studies establish the value of difficulty information for learning question representations and improving prediction. Our focus is semantic generalization to unseen questions. We observe limited gains from directly adding pretrained semantic embeddings to KT, and study whether difficulty supervision derived from training-question responses improves this generalization.

We have expanded the Related Work to explain this research question and its relationship to existing difficulty-related methods.

Because author-released code was unavailable to us, we implemented the difficulty modules described in QDCKT and RAG-KT and evaluated them within AKCT. The implementation details are provided in Appendix A.5.

W2 / Q2: Joint supervision objective

We agree that CE+MAE does not achieve the best result on every dataset.

The joint objective uses two forms of difficulty supervision: classification predicts coarse difficulty levels, and regression predicts continuous correctness rates. Our main finding concerns the value of difficulty supervision for generalization to unseen questions. CE and MAE are two ways to provide this supervision.

We have added experiments across all six KT backbones with multiple random seeds. The table below reports changes in unseen-question AUC relative to the original text representations, averaged across the six KT backbones, in percentage points:

DatasetCE-onlyMAE-onlyCE+MAE
XES3G5M+0.35+0.72+0.75
DBE-KT22+1.63+0.39+0.23
MOOCRadar+0.08-0.21+0.60

Classification and joint supervision give positive average gains on all three datasets. Regression improves average AUC on XES3G5M and DBE-KT22, but reduces it on MOOCRadar. Among these three objectives, joint supervision gives the largest average gain on XES3G5M and MOOCRadar, while classification alone gives the largest gain on DBE-KT22. Section 4.3 and Table 4 now report the added results and discussion, with the full per-backbone results in Appendix C.1.

Thanks for the additional experiments. A few points still need clarification:

  1. What do we learn beyond PEBG, DCL4KT, and PMQE? The focus on unseen questions is clearer now, but the specific new insight is still unclear.
  2. How do the simplifications in your QDCKT and RAG-KT implementations affect the comparisons? Which conclusions about the original methods can these results support?
  3. Why keep CE+MAE as the main objective? The new results help, but CE-only is still much better on DBE-KT22. What explains the different behavior across datasets?

We appreciate Reviewer vDBu for the follow-up questions.

1. What specific insights does this study provide beyond prior work?

We further examine the role of difficulty supervision when questions have no training responses of their own. In the representation analysis on XES3G5M, difficulty supervision makes it easier to predict correctness rates for questions with no training responses. As these questions accumulate training responses, this initial advantage diminishes, while the error on the fixed background questions changes relatively little (Figure 4). These results illustrate the role of difficulty supervision in unseen-question generalization and how its effect changes as responses accumulate. We hope this finding offers insight that helps the community better use semantic information for generalization.

2. What conclusions do the difficulty-module comparisons support?

For completeness, we added comparisons using the difficulty modules of QDCKT and RAG-KT to examine whether combining semantics and difficulty can provide further benefits for existing difficulty-based methods. Since we could not obtain the authors' code, we implemented their difficulty-related modules based on the descriptions available to us and incorporated them into AKCT. Implementation details are provided in Appendix A.5. Without fully reproducing the original methods, directly comparing our implementations with the numbers reported in the original papers would not provide a reliable comparison. We therefore conduct the comparisons under a common experimental setup. Table 2 shows that, within these implementations, difficulty-supervised adaptation stably provides additional gains.

3. Why use CE+MAE?

We use the joint objective to learn both coarse difficulty levels and continuous correctness rates. It can be written as LCE+λLMAE\mathcal{L}_{\mathrm{CE}}+\lambda\mathcal{L}_{\mathrm{MAE}}, where λ\lambda controls the relative contribution of the two supervision signals. Setting λ=0\lambda=0 gives CE-only, and the main experiments use equal weighting, λ=1\lambda=1. This is a simple supervision design; the joint objective or multi-task formulation itself is not considered as a contribution of our paper.

The additional DBE-KT22 comparison shows that the joint objectives have higher mean validation AUC than CE-only across all six KT backbones, whereas CE-only has higher mean test AUC. The AKCT results are shown below.

Supervision objectiveUnseen Validation AUCUnseen Test AUC
CE-only (λ=0)65.62 ± 0.8268.66 ± 0.78
CE+MAE (λ=1)67.81 ± 0.3467.21 ± 0.86
CE+0.5MAE (λ=0.5)69.29 ± 0.3765.63 ± 1.14

The validation results support joint supervision as a reasonable choice, although it does not achieve the best test performance in this comparison. We acknowledge the test advantage of CE-only on this fixed split and do not regard equal weighting as necessary or universally optimal. Users can explore the combination of the two losses that works best for their setting.

Reviewer_ZtgsOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates semantic generalization in knowledge tracing (KT), focusing on the unseen-item setting. The authors propose difficulty-supervised semantic encoder adaptation, using empirical item easiness derived from observed student responses to fine-tune a pretrained text encoder with difficulty classification and correctness regression objectives. The adapted item representations are then used by existing KT backbones. Experiments on three datasets and six KT backbones demonstrate consistent improvements in unseen-item AUC over pretrained text representations.

Critical strengths

S1. The paper addresses a practically relevant problem in knowledge tracing.

Predicting student performance on previously unseen questions is an important challenge in knowledge tracing, particularly when new exercises are introduced without historical student responses. The paper explores how difficulty information derived from existing questions can be transferred to unseen questions through semantic representations. This provides a reasonable and practically motivated direction for improving cold-start prediction.

S2. The experimental evaluation is relatively comprehensive.

The paper evaluates the proposed approach across three datasets and six KT backbones, demonstrating consistent improvements in unseen-item prediction. It also includes ablation studies, shuffled-label controls, representation probing, and controlled response-exposure experiments. These analyses provide useful empirical evidence that difficulty supervision can improve the predictive utility of semantic representations.

Critical weaknesses

W1. Limited novelty and insufficient comparisons with existing unseen-item KT methods.

Neither unseen-item generalization nor difficulty-aware knowledge tracing is a new research direction. Prior work such as SINKT has explored inductive KT, while RAG-KT has already combined unseen-item prediction with difficulty estimation. The proposed method mainly differs in how difficulty information is incorporated, making its methodological novelty relatively limited. Moreover, although SINKT is discussed in the paper, it is not included in the experimental comparisons. The evaluation also lacks broader comparisons with existing unseen-item KT methods, making it difficult to establish the proposed approach's advantages over prior work.

W2. Limited evaluation of knowledge mastery estimation.

Knowledge tracing aims to model students' evolving knowledge states, yet the evaluation focuses primarily on unseen-item response prediction. Although the proposed method improves prediction accuracy, it remains unclear whether these gains reflect better estimation of students' knowledge mastery at the KC level or simply better modeling of item difficulty.

Questions for authors

Q1. Since SINKT is explicitly discussed as related work on inductive KT, why is it not included in the experimental comparisons? Could the authors provide a broader comparison with existing unseen-item KT methods to better establish the effectiveness and novelty of the proposed approach?

Q2. Given that RAG-KT has already explored difficulty estimation for unseen questions, what is the key methodological advantage of difficulty-supervised semantic adaptation over existing approaches?

Q3. Do the improvements in response prediction also translate into more accurate estimation of students' knowledge mastery at the KC level?

1 comment

Thank you for your careful review and suggestions.

W1 / Q1: Novelty and comparison with SINKT

We study how difficulty information helps semantic representations support generalization to unseen questions. Starting from the limited gains of the original semantic representations, we investigate the role of difficulty supervision and its relationship to KT training.

Thank you for pointing out the missing SINKT comparison. We have added experiments using its official implementation. Replacing its question input embeddings with our difficulty-adapted embeddings gives the following results:

DatasetSINKTSINKT + OursΔAUC (pp)
XES3G5M71.3772.01+0.64
DBE-KT2269.7570.74+0.99
MOOCRadar80.5782.68+2.11

SINKT is designed for inductive KT. Replacing its question input embeddings with difficulty-adapted embeddings improves mean unseen-question AUC on all three datasets. This provides evidence that difficulty supervision can also help an existing inductive KT method.

These are the input-replacement results, reported in full in Appendix Table 20, including standard deviations. Table 2 separately reports experiments that add semantic vectors at SINKT's prediction head.

W2 / Q3: Evaluation of knowledge mastery

We agree that accurately measuring students' knowledge mastery is important. KT commonly evaluates students' knowledge states indirectly through prediction of their subsequent responses.

We have added an analysis of subsequent responses to questions sharing a KC. Using saved KT checkpoints without retraining, we evaluate each target response after adding an earlier recorded response to a different unseen question that shares at least one KC with the target. Target questions include both training questions and unseen questions. For each target question with both correct and incorrect responses, we calculate AUC across students and then average equally across target questions.

The following table reports AKCT's average within-question AUC after the earlier response is included in the history:

DatasetOriginal (T)Ours (T+D)ΔAUC (pp)
XES3G5M71.1872.08+0.90
DBE-KT2263.9465.50+1.56
MOOCRadar71.0671.48+0.42

For AKCT, difficulty adaptation improves mean AUC when distinguishing correct and incorrect responses from different students answering the same target question. These results provide evidence about response prediction. Direct assessment of knowledge mastery requires additional measurements. Full results, including standard deviations and the other KT backbones, are reported in Appendix Tables 45–47.

Q2: Relationship to RAG-KT

We recognize that RAG-KT also investigates difficulty information for unseen-question prediction. The two studies have different research emphases.

RAG-KT uses semantic retrieval and an LLM to estimate difficulty for new questions and supplies the resulting scalar to KT. Our starting point is the limited improvement obtained from directly adding pretrained semantic representations to KT. We investigate how difficulty supervision from training-question responses can help semantic representations improve generalization to unseen questions.

A practical advantage is that, after encoder adaptation, a new question can be encoded directly without retrieval or LLM-based difficulty estimation. Difficulty supervision is learned during training, simplifying the prediction procedure for new questions.

We also test whether difficulty supervision provides additional benefits in our AKCT implementation of RAG-KT's difficulty module, keeping its retrieval-based difficulty estimates fixed. Compared with unadapted question text, difficulty adaptation improves unseen-question AUC by 1.21, 4.22, and 1.88 percentage points on XES3G5M, DBE-KT22, and MOOCRadar, respectively. These results show that difficulty supervision can help even when estimated difficulty is already available to KT. Please see Table 2 for the comparison.

Reviewer_jbf5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors ask whether training question representations to predict empirical question difficulty can result in better downstream representations for knowledge tracing training. On experiments with XES3G5M, DBE-KT22 and MOOCRadar over multiple transformer backbones, the authors find that this finetuning does in fact lead to improved performance. In ablations varying training labels, the authors establish that the improvement derives from training to predict difficulty rather than taking mere gradient steps on the given dataset on noisy output.

Critical strengths
  • The authors do demonstrate that finetuning question representations to predict difficulty leads to better downstream performance. That this is the case should be intuitive from work on multi-task training and transfer learning on a range of deep learning tasks.
  • The ablation varying training labels was valuable for demonstrating the source of improvement.
Critical weaknesses

The main weakness of the paper is its writing. It is not sufficient for a paper to present convincing results, it must also be a strong instance of scientific communication. The paper falls short of this on multiple fronts:

  • The main motivation of the paper is presented in a rather convoluted manner and required reading through the entire paper to fully understand. The contribution itself is simple: Finetuning question representations on a difficulty-prediction task improves downstream performance. However, the paper presents this in a highly indirect manner through language such as: "We therefore study whether difficulty can link response patterns from training questions to the content of unseen ones" in the abstract or "We study how this connection transfers response experience" in the intro.
  • The introduction, following the research question statement, describes two entire experimental results that are orthogonal to the main experiment and whose setup are only explained in the appendix. Nothing in the text prepares the reader for a sentence like "Figure 2 (left) shows a privileged-information experiment on XES3G5M: unseen-item AUC rises from 0.6984 to 0.7951 with continuous difficulty and to 0.7775 with three difficulty levels." What is the privileged-information in question? What are the three difficulty levels or the continuous difficulty? These experiments break the flow of writing and distract from the main argument.
  • In Section 3.1, how \pi_g is defined, why p_{qg} is computed with the particular constants it is computed with or what m_q represents are not clearly defined. The reader must infer these from later context.
  • The experimental design in Section 4 particularly contains confusing sentences such as "Scalar-head and AKCT readout diagnostics cover all three datasets", unexplained terms such as "item fold 0 for Unseen Test" or unnecessary details such as an entire paragraph dedicated to experimental seeds. The experimental seeds, specifically, are one of many other markers of LLM generated text throughout the text.

Factors such as these make the paper difficult to understand even when the underlying motivation is simple. The other critique I would make is a lack of technical novelty. Past work, such as "Improving Knowledge Tracing via Pre-training Question Embeddings" by Liu et al (2020) have explored the utility of training question embeddings to predict difficulty, even if this may not have used semantic embeddings encoding text. That finetuning an embedding on an auxiliary task related to the downstream task improves performance, on the other hand, is not necessarily surprising given the long history of multi-task training and transfer learning within deep learning.

These together motivate my score.

Questions for authors

I have found the setup of Section 4.4 to be confusing. Where do the distinct training-student responses come from?

2 comments

Thank you for reviewing our paper. Your comments on the writing helped us clarify how to present the study. The original manuscript did not explain the motivation and experimental setup clearly enough. We have revised the Introduction and Experiments accordingly.

1. Introduction

We have substantially reorganized the Introduction and shortened the Abstract. The Introduction now states our question directly: Can question semantics help KT generalize to questions with no training responses, and can difficulty supervision improve this generalization?

We first show that directly adding pretrained semantic representations provides limited gains. We then introduce difficulty information, explain its value for unseen-question prediction, and identify the challenge: difficulty statistics are unavailable before responses to a new question have been collected. The correctness-rate comparison between semantically similar questions then motivates adapting semantic representations using difficulty supervision from training questions.

We have retained the two diagnostic experiments and clarified their settings and purpose. The difficulty-prior experiment shows the predictive value of difficulty information. The semantic-neighbor experiment shows that question content provides clues about difficulty. The former uses additional responses to unseen questions to calculate their difficulty statistics. The Introduction now explicitly explains where these statistics come from and why they are unavailable when predicting responses to a newly introduced question.

2. Variable definitions in Section 3.1

We now define the variables alongside the equations. For question qq and student group gg, cqgc_{qg} and nqgn_{qg} are the numbers of correct and total responses, respectively. πg\pi_g is the group's overall correctness rate on training questions. pqgp_{qg} is the smoothed correctness rate for that question within the group, and mqm_q is the equally weighted average of the smoothed rates over the five student groups.

The smoothing parameter κ\kappa is equivalent to adding κ\kappa observations with mean correctness πg\pi_g. When few responses are available, this pulls the estimate toward the group's overall correctness rate and reduces sampling noise. We have added this explanation to Section 3.1.

3. Experimental presentation in Section 4

We have reorganized the Experimental Setup to explain the datasets, student and question splits, comparison methods, and evaluation metrics. Details such as training steps, batch sizes, and random seeds have been moved to the appendix. We have also revised unclear terms and sentences throughout the experimental section. We apologize for the difficulty these caused in reading the original manuscript.

4. Research question and relationship to prior work

We agree that difficulty supervision for learning question representations has precedents. PEBG (Liu et al., 2020) pretrains question embeddings using question–skill relations, similarity relations in a graph, and a difficulty-prediction objective, and then uses these embeddings for KT.

Our emphasis is on the research question and the empirical investigation. We study how response statistics from training questions can help semantic representations support KT generalization to questions with no training responses. PEBG reports prediction performance under its original evaluation protocol; it does not separately evaluate questions held out from KT training. Starting from the limited gains of the original semantic representations, we investigate whether difficulty supervision improves unseen-question prediction. We also examine the role of correct difficulty labels, how accurately question correctness rates can be predicted from KT representations, and how this changes as questions accumulate training responses.

Multi-task learning and difficulty supervision are established techniques. Our contribution centers on the role of difficulty in this generalization problem and its relationship to KT training. The revised Related Work discusses PEBG and other difficulty-related methods, including QDCKT and RAG-KT, and states the question investigated in our study.

5. Analysis of training response counts

The response-count analysis in the original Section 4.4 is now Section 4.5, “Effect of Training Responses.”

All responses used in this experiment are real student responses from the dataset. We fix 50 target questions and gradually increase the number of responses to each question used for KT training. For a given question, the added responses come from different training students. Evaluation always uses the correctness rate calculated from a separate group of test students' responses to that question. These test responses are not used for KT training.

Reviewer_jbf5OpenReview ↗

Response to Authors

I thank the authors for their response. I will defer to the AC about to what extent a paper's writing can be allowed to be overhauled during the rebuttal period. Despite the improvements, I still have concerns regarding the paper's writing. Specifically, my issue with the introduction was the placement of three entire sets of experiments without proper introduction of datasets, the underlying models, the training protocols or key details like what model the pretrained embeddings come from. A reader should not be required to refer to the appendix in order to make sense of the introduction. I am maintaining my criticism about the paper's writing as a result.

Comments on the paper

2 comments

Dear Reviewer,

Thank you again for your valuable feedback. We have addressed your concerns in our rebuttal and would greatly appreciate it if you could take another look when you have a chance.

We would also be happy to provide any further clarifications if needed.

Best regards, The Authors

Dear Reviewer,

We sincerely apologize for following up again. As the discussion period will close in 24 hours, we would greatly appreciate it if you could take a moment to review our responses and share any remaining concerns.

Thank you very much for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.