- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper builds a benchmark, UniER, that compares item-level and path-level exercise recommenders on equal terms. Both kinds of method are run as an N-step sequence against a DKT student simulator and scored by Weighted Cognitive Gain (WCG), the weighted change in simulated concept mastery, under a targeted (TGA) and a global (GPP) weighting. With this setup the authors re-implement 18 methods, run them on nine datasets at budgets of 1 to 20 steps, and add sparsity, cold-start, noise and efficiency studies, a SimpleKT evaluator check, and conventional ranking metrics. They find that item-level methods lead at one step and path-level methods at ten.
Critical strengths
-
Nobody has compared these two families of methods under one protocol before, and the paper does it at scale: 18 methods in one codebase, nine datasets, a documented hyperparameter search, a public repository.
-
The authors stress-test their own protocol. Table 6 compares three ways of executing item-level lists; Table 7 swaps the evaluator to SimpleKT. The unfavourable results are reported, not hidden.
Critical weaknesses
- The simulator is on both sides of the measurement. Path-level policies are trained on DKT reward and scored by DKT-based WCG; item-level lists are executed by a DKT-guided rule that by itself lifts KG4Ex from -0.054 to 0.087 (Table 6). Switching to SimpleKT (Table 7) keeps the aggregate path-level advantage but scrambles the rankings: GEHRL goes from 16th to 1st, PKSD from 1st to 4th, Bridge2006 from near-zero everywhere to values up to 0.797. Negative gains (DRER -0.487 on Ednet; most path-level methods at @1) are never explained, and are more likely a DKT artifact than a property of the recommenders.
- The reporting is not tight enough for a benchmark. WCG is called "IRT-grounded" but Eq. 7 has no IRT in it; it is the mastery-gain reward CSEAL/GEHRL/PKSD already use, minus their normalisation. Tables 1, 2, 6, 7 show seven datasets but the captions say nine. The tuning objective and split are not stated. No seeds or variance, though many gaps are in the third decimal. Four item-level methods were trained on half of three datasets and DRER was cut at 72 h, with no comment. Sec. 4.1 describes Tables 1 and 2 the wrong way round.
Questions for authors
See Weakness.