- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether knowledge tracing models with good prediction performance can be used to simulate learners. It proposes process contracts to evaluate model behavior under different learning orders and introduces process-contrastive regularization to reduce contract violations. Experiments are conducted on ASSISTments datasets and synthetic learning environments.
Critical strengths
-
The paper discusses an interesting issue that good prediction performance does not always mean reliable learner simulation.
-
The experiments consider different knowledge tracing models, random seeds, and data logging policies.
Critical weaknesses
-
I think the motivation of this paper is not clear enough. Why should a knowledge tracing model satisfy the proposed process contracts? The authors claim that prediction accuracy cannot guarantee simulation validity, but it is unclear how this issue affects real educational applications. More importantly, the paper does not provide experiments to show why solving this problem is necessary.
-
The novelty of this paper seems limited. The proposed framework mainly consists of predefined evaluation rules and an additional regularization loss. It is also unclear whether these predefined rules can reflect the real learning behaviors of learners.
-
The experimental results are not sufficiently informative. Most experiments only show whether different models satisfy the predefined contracts in simple synthetic environments. Although PCR reduces contract violations, it is unclear whether this improvement can lead to better learner simulation or downstream educational performance. In addition, the presentation of experimental results is not clear enough to show the practical advantages of this paper
Questions for authors
See above
Official comment by reviewer fqsW
The log-loss/Brier results and checkpoint-level audits answer my two questions. They show a small probability-quality cost in the blocked recency setting. PCR improves 57 checkpoints and worsens 14, so the average improvement still needs to be checked on the model actually deployed.
The planning results tell us more than the original contract counts. Repairing the original contracts does not reliably improve planning; matching the contract to the planner’s horizon reduces B→C regret from 0.0029 to 0.0002 at essentially unchanged AUC. That distinction is worth reporting, including the cases where other decisions get worse.
I would raise Informativeness from 2 to 3 for this additional finding. I am keeping the overall rating at 3: the experiments support the audit-and-repair approach, but the planning gains remain specific to the decisions covered by the contract.