- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes that a tutoring agent’s memory for personalized education should be a mental world model of the learner. The world model is a fixed-size belief state that supports three operations, (a) write to update the belief after new exercise and response; (b) read to retrieve per-concept mastery; (c) imagine to roll a candidate exercise sequence forward and predict learner’s responses and mastery gain. Four experiments, on three common knowledge-tracing datasets, evaluate the memory fidelity, tutoring value, and memory cost.
Critical strengths
- The presentation and figures are very clear and easy to follow. I especially think fig.1-2 help me a lot in understanding the pipeline.
- The experiment results show clear evidence that world-model memory keeps both item-specific and learner-speciic information, while other readable memories lose one of them. I think this is very important for tutoring, since a tutor needs to know both which learner is behind and which exercises they will fail.
Critical weaknesses
- Based on the results, it seems like that world-model memory is not clearly different from, or better than, a knowledge-tracing model. The memory is a recurrent model so it is close to all deep-learning-based knowledge tracng models after DKT. The paper argues that the novelty lies in that the memory can be read and rolled forward. But appendix shows that these can also be added to DKT directly. I assume all KT models with a hidden state (which summarizes the history information) and an approxiamately learned forward dynamics (here is the recurrent model) should be able to do this?
- Related to my point 1, I want to understand the operation “imagine” better. I know this is a quite popular way in (deep) RL where you do the roll out to learn or select a policy. Here, “imagine” predicts the mastery gain by rolling the memory forward on its own sampled responses, but a change in mastery can mean either that the learner learned, or only that the memory updated its belief about what the learner already knew. Is there anything in the model that separates the two? The memory is also trained on logged sequences whose exercise order was not randomized, yet the planner uses it to rank candidate sequences that may not appear in the data. How do the authors know that the predicted mastery gain ranks candidates by learning gain? For example, does it predict the realized probe gain across the planner’s 32 candidates?
- All learners are simulated, and the learning gain is the change in correctness predicted by an AKT model. I am worried that the evaluation setup is too weak to support the tutoring results.
Questions for authors
See weakness.