acceptodds
Withdrawn from ICLR 2027

UniER: A Unified Benchmark for Item-level and Path-level Exercise Recommendation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Personalized exercise recommendation dynamically aligns pedagogical resources with individual knowledge mastery, which is crucial for satisfying students' dynamic learning needs in modern education. Existing approaches primarily follow two paradigms: Item-Level Exercise Recommendation (ILER) ranks exercises for immediate recommendation, whereas Path-Level Exercise Recommendation (PLER) plans ordered exercise sequences. Despite sharing the same ultimate objective, disparate evaluation setups have kept these two lines of research isolated, hindering unified benchmarking and fair comparison. To fill the gap, in this paper, we present a Unified Benchmark for Exercise Recommendation (UniER), a comprehensive evaluation framework that unifies ILER and PLER. Specifically, we introduce Weighted Cognitive Gain (WCG) to compare estimated knowledge gains under different instructional objectives and shared interaction budgets. We benchmark 18 representative methods spanning four recommendation categories on nine public datasets. Through analyses of effectiveness, generalizability, robustness, and efficiency, we find that the leading methods change with the interaction budget: ILER methods achieve the highest average scores at one step, whereas PLER methods lead at ten steps. Performance under data sparsity, cold-start, and label noise further varies across methods and datasets, providing evidence for selecting recommendation methods according to the intended learning setting.

Primary area:
datasets and benchmarks
Keywords:
Exercise Recommendation, Learning Path Recommendation, Personalized Education

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mVzxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper builds a benchmark, UniER, that compares item-level and path-level exercise recommenders on equal terms. Both kinds of method are run as an N-step sequence against a DKT student simulator and scored by Weighted Cognitive Gain (WCG), the weighted change in simulated concept mastery, under a targeted (TGA) and a global (GPP) weighting. With this setup the authors re-implement 18 methods, run them on nine datasets at budgets of 1 to 20 steps, and add sparsity, cold-start, noise and efficiency studies, a SimpleKT evaluator check, and conventional ranking metrics. They find that item-level methods lead at one step and path-level methods at ten.

Critical strengths
  1. Nobody has compared these two families of methods under one protocol before, and the paper does it at scale: 18 methods in one codebase, nine datasets, a documented hyperparameter search, a public repository.

  2. The authors stress-test their own protocol. Table 6 compares three ways of executing item-level lists; Table 7 swaps the evaluator to SimpleKT. The unfavourable results are reported, not hidden.

Critical weaknesses
  1. The simulator is on both sides of the measurement. Path-level policies are trained on DKT reward and scored by DKT-based WCG; item-level lists are executed by a DKT-guided rule that by itself lifts KG4Ex from -0.054 to 0.087 (Table 6). Switching to SimpleKT (Table 7) keeps the aggregate path-level advantage but scrambles the rankings: GEHRL goes from 16th to 1st, PKSD from 1st to 4th, Bridge2006 from near-zero everywhere to values up to 0.797. Negative gains (DRER -0.487 on Ednet; most path-level methods at @1) are never explained, and are more likely a DKT artifact than a property of the recommenders.
  2. The reporting is not tight enough for a benchmark. WCG is called "IRT-grounded" but Eq. 7 has no IRT in it; it is the mastery-gain reward CSEAL/GEHRL/PKSD already use, minus their normalisation. Tables 1, 2, 6, 7 show seven datasets but the captions say nine. The tuning objective and split are not stated. No seeds or variance, though many gaps are in the third decimal. Four item-level methods were trained on half of three datasets and DRER was cut at 72 h, with no comment. Sec. 4.1 describes Tables 1 and 2 the wrong way round.
Questions for authors

See Weakness.

Reviewer_TpsAOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UniER, a benchmark that unifies ILER and PLER. Three execution protocols convert unordered lists into ordered sequences under a shared interaction budget, and a simulator-based WCG metric is introduced, instantiated as TGA and GPP. 18 methods across 4 sub-categories are evaluated on 9 public datasets, covering effectiveness, generalizability, robustness, and efficiency. However, all evaluations rest on static KT logs, relying on a single deep model to simulate counterfactual feedback and knowledge gains, which has fundamental validity flaws.

Critical strengths

The paper identifies the pain point that ILER and PLER have long been evaluated in isolation, and bridges the two paradigms via a shared budget and unified execution protocols — a constructive perspective.

Massive reproduction effort: 18 baselines across 4 sub-categories (including recent NR4DER, ER-TGA, KnowLP), systematically benchmarked on 9 datasets. Appendix D provides ablations on sequencing strategies (Table 6) and alternative simulators (Table 7).

Well-structured formalization of both paradigms in Section 2; open-source code, hardware details, and exhaustive hyperparameter search spaces support reproducibility.

Critical weaknesses

All 9 datasets are passive KT logs, not recommendation datasets.

Tables 1 and 2 contain extreme negative WCG values (e.g., -0.487 for DRER on Ednet; -0.667 for AC on Ednet). Pedagogically, the worst case should be Gain ≈ 0, never a 50%–67% regression of mastered knowledge. This stems from DKT's well-known non-monotonic hidden-state fluctuation triggered by out-of-distribution recommendations; WCG defined via unconstrained DKT output differences severely distorts true pedagogical value.

In TGA, PLER methods receive the goal set GG as input, while ILER is goal-agnostic; comparing them directly under sparse target weights is unfair to ILER.

Questions for authors

How can DKT or SimpleKT trained on static logs predict counterfactual gains under severe off-policy distribution shift? Given that swapping DKT for SimpleKT reverses the rankings, why should the community trust this simulation framework as a benchmark?

How do the authors explain the negative gains of -0.487 or -0.667? Would KT models with monotonicity constraints eliminate state oscillation? What is used as ground truth for Recall, NDCG, F1, and MRR in Tables 8 and 9, given that unmastered KCs are unobserved?

Do native ILER methods receive Ctarget\mathcal{C}_{target} at inference in TGA? If not, is the comparison with goal-conditioned PLER fair? What is the precise mathematical definition and threshold of the "learnability criterion" in protocol (i)?

Why do AKTRec, SimpleKTRec, MulOER, and NR4DER OOM on 256GB RAM and 4× RTX 4090? Can full 100%-data results be provided via smaller batch sizes, gradient accumulation, or streaming data loading?

Reviewer_hR8bOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that although Item-Level Exercise Recommendation (ILER) and Path-Level Exercise Recommendation (PLER) both aim to improve students’ knowledge mastery, they emphasize immediate exercise selection and sequential learning-path planning, respectively. Differences in evaluation metrics, datasets, and experimental settings make it difficult to compare their strengths and applicable scenarios. To address this issue, the authors propose UniER, a unified benchmark that converts different methods’ outputs into exercise sequences executed under a shared interaction budget. They introduce Weighted Cognitive Gain (WCG), which aggregates changes in knowledge mastery estimated by a knowledge tracing model, with concept weights determined by instructional objectives. This supports the evaluation of both targeted remediation and overall proficiency improvement. Experiments involving 18 methods on nine public datasets show that ILER methods achieve the highest average scores under a one-interaction budget, whereas PLER methods lead under a ten-interaction budget. Method performance also varies with instructional objectives, data sparsity, cold-start conditions, and label noise. The authors aim to provide a unified, reproducible framework for selecting recommendation methods according to the intended learning setting.

Critical strengths

S1. The experimental evaluation is broad. The paper evaluates 18 methods on nine public datasets and examines interaction budgets, data sparsity, cold-start conditions, label noise, and computational efficiency, providing a useful benchmarking resource.

S2. The paper includes useful sensitivity analyses. The authors compare different recommendation-list execution strategies and replace the evaluation simulator while keeping the recommendation model parameters fixed, helping clarify how the findings depend on evaluation settings.

S3. The paper addresses fragmented evaluation across recommendation paradigms. Shared interaction budgets and evaluation objectives provide a common framework for comparing item-level and path-level exercise recommendation methods.

Critical weaknesses

W1. The necessity and added value of UniER are insufficiently established. Different evaluation settings may reflect legitimate differences in instructional objectives. The introduction lacks concrete evidence that existing evaluations lead to misleading method choices, leaving the practical consequences of the claimed evaluation gap unclear.

W2. Connections to recent content-based recommendation research are limited. Relevant directions, such as RAG/graph-based cold-start recommendation (e.g., ColdRAG) and multimodal recommendation, are insufficiently considered. Including relevant baselines would help assess whether UniER captures the benefits of auxiliary information under sparse interactions.

W3. Simulator dependence limits confidence in method-specific conclusions. DKT provides both simulated feedback and knowledge-gain estimates. Despite the SimpleKT sensitivity check, some results change substantially (e.g, in Tables 1 and 7, DQN’s GPP@10 on Ednet shifts from 0.486 to −0.340). Further validation is needed to clarify how well these simulated gains reflect actual learning outcomes.

Questions for authors

Please see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.