- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether difficulty supervision can improve semantic generalization in knowledge tracing (KT), especially for unseen questions. A pretrained semantic encoder is fine-tuned with difficulty labels derived from historical student responses, combining discrete difficulty classification with continuous correctness regression, and the adapted representations are then fed into downstream KT models. The authors also try LLM-generated solutions as additional question content. Results on three datasets and six KT backbones show consistent gains in unseen-item prediction, and the paper includes shuffled-label controls, representation probing, and controlled response-exposure analyses to examine what the difficulty signal actually contributes.
Critical strengths
-
Consistent improvements across datasets and backbones. The paper evaluates difficulty-supervised representations on three datasets and six KT backbones, demonstrating consistent improvements in unseen-item AUC over the original pretrained text representations. This provides encouraging evidence that the proposed adaptation strategy is not limited to a particular KT architecture or dataset, supporting the practical relevance of the approach.
-
Systematic investigation of difficulty supervision and semantic generalization. Beyond improving prediction performance, the paper provides a systematic empirical investigation of how difficulty supervision influences question representations and their generalization to unseen items. These analyses offer useful insights into the role of difficulty information in knowledge tracing and strengthen the overall contribution of the work.
Critical weaknesses
-
Novelty over prior work is not well established. Using difficulty information derived from student responses to improve question representations has been studied before, e.g., PEBG (Improving Knowledge Tracing via Pre-training Question Embeddings (2020)), DCL4KT (Difficulty-Focused Contrastive Learning for Knowledge Tracing with a Large Language Model-Based Difficulty Prediction (2024)), and PMQE (Pre-Training Multi-Concept Question Embeddings for Knowledge Tracing (2025)), all of which are conceptually close to this work. I note that the paper does include some discussion of these prior works; however, the discussion remains at a conceptual level, and the stated differences do not convincingly show what the proposed method actually improves over them, or why its design should be preferred. In addition, the comparisons against QDCKT and RAGKT use simplified re-implementations rather than the full original methods, so the current results say little about how the approach stacks up against the actual state of the art.
-
The joint classification + regression objective is not justified. Table 4 is hard to reconcile with the proposed design: the joint CE+MAE objective never achieves the best result on any of the three datasets — regression-only is better on XES3G5M, and classification-only is better on DBE-KT22 and MOOCRadar. If single-objective supervision is consistently stronger, it is unclear why the joint objective is adopted as the main method, and the paper offers little analysis of this discrepancy.
Questions for authors
-
I notice the paper already contains some analysis and discussion of prior works such as PEBG, DCL4KT, and PMQE, but the differences identified there do not clearly demonstrate the proposed method's improvement over them. Could the authors clarify the key differences between the proposed method and these prior works? Given their substantial conceptual overlap, what are the specific advantages of the proposed approach, and what evidence supports these advantages over existing methods?
-
Why is joint CE+MAE supervision the main objective when Table 4 shows single-objective supervision performing better on all three datasets? Please explain the motivation, and analyze why the better-performing objective flips across datasets.
3 comments
Thanks for the additional experiments. A few points still need clarification:
- What do we learn beyond PEBG, DCL4KT, and PMQE? The focus on unseen questions is clearer now, but the specific new insight is still unclear.
- How do the simplifications in your QDCKT and RAG-KT implementations affect the comparisons? Which conclusions about the original methods can these results support?
- Why keep CE+MAE as the main objective? The new results help, but CE-only is still much better on DBE-KT22. What explains the different behavior across datasets?
We appreciate Reviewer vDBu for the follow-up questions.
1. What specific insights does this study provide beyond prior work?
We further examine the role of difficulty supervision when questions have no training responses of their own. In the representation analysis on XES3G5M, difficulty supervision makes it easier to predict correctness rates for questions with no training responses. As these questions accumulate training responses, this initial advantage diminishes, while the error on the fixed background questions changes relatively little (Figure 4). These results illustrate the role of difficulty supervision in unseen-question generalization and how its effect changes as responses accumulate. We hope this finding offers insight that helps the community better use semantic information for generalization.
2. What conclusions do the difficulty-module comparisons support?
For completeness, we added comparisons using the difficulty modules of QDCKT and RAG-KT to examine whether combining semantics and difficulty can provide further benefits for existing difficulty-based methods. Since we could not obtain the authors' code, we implemented their difficulty-related modules based on the descriptions available to us and incorporated them into AKCT. Implementation details are provided in Appendix A.5. Without fully reproducing the original methods, directly comparing our implementations with the numbers reported in the original papers would not provide a reliable comparison. We therefore conduct the comparisons under a common experimental setup. Table 2 shows that, within these implementations, difficulty-supervised adaptation stably provides additional gains.
3. Why use CE+MAE?
We use the joint objective to learn both coarse difficulty levels and continuous correctness rates. It can be written as , where controls the relative contribution of the two supervision signals. Setting gives CE-only, and the main experiments use equal weighting, . This is a simple supervision design; the joint objective or multi-task formulation itself is not considered as a contribution of our paper.
The additional DBE-KT22 comparison shows that the joint objectives have higher mean validation AUC than CE-only across all six KT backbones, whereas CE-only has higher mean test AUC. The AKCT results are shown below.
| Supervision objective | Unseen Validation AUC | Unseen Test AUC |
|---|---|---|
| CE-only (λ=0) | 65.62 ± 0.82 | 68.66 ± 0.78 |
| CE+MAE (λ=1) | 67.81 ± 0.34 | 67.21 ± 0.86 |
| CE+0.5MAE (λ=0.5) | 69.29 ± 0.37 | 65.63 ± 1.14 |
The validation results support joint supervision as a reasonable choice, although it does not achieve the best test performance in this comparison. We acknowledge the test advantage of CE-only on this fixed split and do not regard equal weighting as necessary or universally optimal. Users can explore the combination of the two losses that works best for their setting.
Thank you for recognizing the value of our experimental analysis and for your suggestions.
W1 / Q1: Relationship to prior work
We agree that PEBG, DCL4KT, and PMQE have already investigated the use of difficulty information in KT.
These studies establish the value of difficulty information for learning question representations and improving prediction. Our focus is semantic generalization to unseen questions. We observe limited gains from directly adding pretrained semantic embeddings to KT, and study whether difficulty supervision derived from training-question responses improves this generalization.
We have expanded the Related Work to explain this research question and its relationship to existing difficulty-related methods.
Because author-released code was unavailable to us, we implemented the difficulty modules described in QDCKT and RAG-KT and evaluated them within AKCT. The implementation details are provided in Appendix A.5.
W2 / Q2: Joint supervision objective
We agree that CE+MAE does not achieve the best result on every dataset.
The joint objective uses two forms of difficulty supervision: classification predicts coarse difficulty levels, and regression predicts continuous correctness rates. Our main finding concerns the value of difficulty supervision for generalization to unseen questions. CE and MAE are two ways to provide this supervision.
We have added experiments across all six KT backbones with multiple random seeds. The table below reports changes in unseen-question AUC relative to the original text representations, averaged across the six KT backbones, in percentage points:
Classification and joint supervision give positive average gains on all three datasets. Regression improves average AUC on XES3G5M and DBE-KT22, but reduces it on MOOCRadar. Among these three objectives, joint supervision gives the largest average gain on XES3G5M and MOOCRadar, while classification alone gives the largest gain on DBE-KT22. Section 4.3 and Table 4 now report the added results and discussion, with the full per-backbone results in Appendix C.1.