acceptodds
Under review as a conference paper at ICLR 2027

Prediction Is Not Simulation: Process Contracts for Auditing and Repairing Knowledge-Tracing Models

Abstract

Knowledge-tracing models are selected as next-response predictors but reused as simulators for instructional decisions. Predictive fidelity on logged data does not identify how a model changes state under a controlled curriculum. This paper introduces process contracts as a second validation axis. They specify directional or practically null responses to matched trajectory transformations. The matched audit swaps early concept blocks while fixing the learner prefix, item-response multiset, exposure dose, delayed suffix, and probes. A contract passes only when its whole 95% interval lies on the prescribed side of zero or inside a predeclared ±0.01 band. On ASSISTments 2009 and a four-regime simulator, process contracts expose transition semantics that vary with architecture, seed, and logging policy and that test AUC does not determine. In a fresh 600-checkpoint matrix, the oracle and DKT pass all 24 endpoint contracts, whereas SAKT passes 16. Four of its eight failures occur at a mean AUC of at least 0.85. The paper also introduces Process-Contrastive Regularization (PCR), a hinge penalty added to the factual cross-entropy. The penalty pushes matched learner-level effects past a margin or into the band and uses item identities that the audit never scores. The promotion gate requires a matched test-AUC loss of at most 0.01 and strictly fewer failed contracts than the matched baseline. PCR passes this gate in an exploratory single-seed pilot (from 4/16 to 1/16) and for SAKT across fresh seeds under three logging policies. The gate refuses PCR trained against a misspecified contract.

Primary area:
learning on time series and dynamical systems
Keywords:
knowledge tracing, learner simulators, process contracts, simulator validation, interventional evaluation, equivalence testing, model auditing, model repair, AI for education

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ANyDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether knowledge tracing models with good prediction performance can be used to simulate learners. It proposes process contracts to evaluate model behavior under different learning orders and introduces process-contrastive regularization to reduce contract violations. Experiments are conducted on ASSISTments datasets and synthetic learning environments.

Critical strengths
  1. The paper discusses an interesting issue that good prediction performance does not always mean reliable learner simulation.

  2. The experiments consider different knowledge tracing models, random seeds, and data logging policies.

Critical weaknesses
  1. I think the motivation of this paper is not clear enough. Why should a knowledge tracing model satisfy the proposed process contracts? The authors claim that prediction accuracy cannot guarantee simulation validity, but it is unclear how this issue affects real educational applications. More importantly, the paper does not provide experiments to show why solving this problem is necessary.

  2. The novelty of this paper seems limited. The proposed framework mainly consists of predefined evaluation rules and an additional regularization loss. It is also unclear whether these predefined rules can reflect the real learning behaviors of learners.

  3. The experimental results are not sufficiently informative. Most experiments only show whether different models satisfy the predefined contracts in simple synthetic environments. Although PCR reduces contract violations, it is unclear whether this improvement can lead to better learner simulation or downstream educational performance. In addition, the presentation of experimental results is not clear enough to show the practical advantages of this paper

Questions for authors

See above

Reviewer_9XEdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces process contracts to evaluate whether knowledge tracing models behave appropriately when curriculum order changes, rather than relying solely on predictive accuracy. It also proposes Process-Contrastive Regularization (PCR) to improve contract compliance while preserving predictive performance. Experiments on real educational datasets and a synthetic simulator show that high AUC does not guarantee correct process behavior, and PCR can repair some failures.

Critical strengths
  1. This paper identifies an important gap between predicting student responses and simulating learning processes. This distinction is particularly relevant when knowledge tracing models support instructional planning. The motivation is clear and practically meaningful. It is not another "I have a better performance model"
  2. The matched audit controls item composition, exposure dose, learner history, and delayed suffixes. Directional and equivalence contracts provide explicit criteria for evaluating model behavior. This makes the proposed evaluation more rigorous and reproducible.
  3. Experiments show that models with similar AUC can exhibit different or even opposite responses to curriculum order changes (Which I am not suprised at all). The study covers multiple architectures, logging policies, and random seeds. These results provide convincing evidence that predictive metrics alone are insufficient.
Critical weaknesses
  1. The paper does not fully distinguish its approach from broader behavioral testing methods for learned models. The novelty needs stronger positioning.
  2. The synthetic environment focuses on two concepts, binary responses, and four predefined regimes. Real learning processes involve more complex interactions, forgetting, and instructional feedback. The generalizability of the framework remains uncertain.
  3. The real data experiments reveal model behavior but cannot establish whether the specified contracts reflect actual human learning. The authors acknowledge this limitation. Validation using prospective educational studies or an independently validated simulator would strengthen the claims.
  4. PCR repairs some failures but does not eliminate them, and multi seed repair experiments focus on SAKT. The practical null threshold also affects the number of contracts that pass. Broader architecture comparisons and stronger sensitivity analyses would improve confidence in the method.
Questions for authors
  1. How can the authors justify that the specified curriculum order contracts reflect meaningful instructional dynamics rather than arbitrary assumptions about how a knowledge tracing model should behave?
  2. The real log experiments use observational data and cannot establish causal effects of curriculum order on actual students. What evidence supports the practical relevance of the proposed contracts for real instructional planning?
Reviewer_fqsWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines how knowledge-tracing models respond to changes in curriculum order. It holds item-response content and other audit conditions fixed, checks the effects against directional or equivalence contracts, and adds these requirements to training through PCR. Models with similar AUC give different order effects. In the synthetic multi-seed study, PCR reduces aggregated SAKT failures from 11/24 to 6/24 without exceeding the allowed AUC drop.

Critical strengths
  • Matching item-response content and suffixes isolates the order change, and independent probes prevent the measurement itself from adding practice. For a null contract, the full interval must lie within the tolerance band. A wide interval containing zero therefore cannot be mistaken for evidence of no effect.
  • PCR repairs five aggregated failures across fresh seeds and three logging policies, while keeping all thirteen previously passing endpoints. The improvement therefore extends beyond the pilot. The tolerance-band and misspecified-contract analyses also show where the pass decisions depend on the audit settings.
Critical weaknesses
  • AUC checks ranking, not whether PCR keeps response probabilities accurate. A strictly increasing transformation can leave AUC unchanged while altering the probability differences used by the contracts. Keeping BCE in training does not settle this on held-out data. Without a test log loss or Brier score comparison, passing more contracts does not establish that PCR preserves the probability predictions needed from a simulator.
  • The contract decisions are made after averaging model seeds within each dataset. That estimates average behavior, but a practitioner ultimately selects one checkpoint. Opposite errors can cancel before a null contract is tested, and a correct average direction need not hold for individual runs. This is particularly relevant because the paper itself finds seed-dependent sign reversals. The bootstrap and multiplicity checks address uncertainty in the averages; they do not show how often an individual PCR run improves.
Questions for authors
  • How do matched test log loss or Brier scores change after PCR?
  • Within each dataset, how often does PCR reduce an individual checkpoint’s failure count, and how often does it increase it?
1 comment
Reviewer_fqsWOpenReview ↗

Official comment by reviewer fqsW

The log-loss/Brier results and checkpoint-level audits answer my two questions. They show a small probability-quality cost in the blocked recency setting. PCR improves 57 checkpoints and worsens 14, so the average improvement still needs to be checked on the model actually deployed.

The planning results tell us more than the original contract counts. Repairing the original contracts does not reliably improve planning; matching the contract to the planner’s horizon reduces B→C regret from 0.0029 to 0.0002 at essentially unchanged AUC. That distinction is worth reporting, including the cases where other decisions get worse.

I would raise Informativeness from 2 to 3 for this additional finding. I am keeping the overall rating at 3: the experiments support the audit-and-repair approach, but the planning gains remain specific to the decisions covered by the contract.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.