acceptodds
Under review as a conference paper at ICLR 2027

Who Scores the Tutor? Sampling Noise and Self-Preference in Simulated-Learner Evaluation

Abstract

Tutoring agents are compared by the learning gain they produce in a simulated learner. This learner is a knowledge-tracing model fitted to real interaction data, and the same model also scores the teaching. On one corpus, both DKT scorers rank a policy that plans against a DKT model first out of eight, but an AKT scorer ranks it seventh on the same learners and sequences. This paper isolates the effect of the scorer. Five planning policies (one per knowledge-tracing architecture) and three scripted controls choose exercises for the same learners. The chosen sequences are then frozen, and 10 scorers (five architectures with two seeds each) rank the policies on each of three corpora. Disagreement between scorers is measured against a resampling floor, the agreement of one scorer with itself on disjoint halves of the learners. This floor is already low (Kendall ), so with sampling noise alone a benchmark of this size barely determines its own ranking. Changing the architecture of the scorer (scorer identity) lowers by a further (bootstrap 95% interval , 2,062 simulated learners from 999 students). On the two ASSISTments corpora, changing only its seed adds no disagreement beyond resampling. Scorers also show a directional bias. A scorer ranks the policy planned against its own architecture positions higher than scorers of other architectures do (exact assignment tests, ). The bias is positive on all three corpora and depends on the planning. A greedy control that uses the same model without planning against it receives no detectable preference. On ASSIST2017, raising the search budget from 12 to 192 candidates nearly triples the gain that scorers of the same architecture as the planner see in excess of the other scorers. Averaging a larger panel of scorers does not produce a usable ranking on average at any panel size tested. Excluding scorers of the same architecture as the planner is free and reorders every corpus leaderboard. It changes the reported winner on NIPS34 in both runs and on ASSIST2017 in the second run. The protocol, exclusion estimator and instrument will be released.

Primary area:
datasets and benchmarks
Keywords:
simulated learners, knowledge tracing, intelligent tutoring, evaluation methodology, leaderboard reliability, self-preference bias, proxy overoptimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pJuLOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper deals with the issue that tutoring models are evaluated against student simulations, where the student simulation heavily influences the final ranking of the models. They show that this evaluation protocol is especially sensitive to changing the simulated student and to models being biased towards their own model family.

Critical strengths
  • I do find the overall question important of how to pick student simulators to evaluate tutoring models and progress on this can have significant impact in the community. Oftentimes, these choices are made out of convenience but make results less reliable.
Critical weaknesses
  • The paper is very hard to understand due to the writing. Despite trying my best and significant prior experience in the field I was unable to understand the exact set-up under which evaluation is done. For example, I was unable to even understand which exact tutoring agent is used for the evaluation? I checked Appendix D which outlines implementation details but could also not find it there. This is a significant weakness, because it is essential for readers to precisely understand the findings of the paper but, to me, these are not obvious because the set-up is not exactly clear and writing is very hard to parse.

  • As written above, many details are left out and only reported in the Appendix D which makes it hard for readers to follow the main text, as they have to skip forward, and even there important details are left out. In addition, I would expect clear recommendations for future practice but Sec. 5 is a) very short and b) also hard to parse which makes it unclear for readers how they can improve their benchmarks in the future.

Questions for authors
  • "The fitted model that simulates the learner also scores the teaching." Is this claim really true the way it is written? Take Dinucu-Jianu et al. 2025 as a counter-example where student simulator and LLM judge are different. The judge is also changed between training and testing. It is only true that student model of training and evaluation match but these do not handle the scoring. Could you please clarify this claim? Do you mean that the simulated student is used together with another LLM that establishes the scoring (which is not the simulated student)?

  • "The resulting advantage remains unmeasured" (L49) - what exactly is this advantage? Could you please clarify?

  • Can you please clearly define what you understand under knowledge tracing?

Reviewer_x3aTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the reliability of simulated-learner evaluations for tutoring agents, focusing on sampling noise and evaluator self-preference. Using five knowledge-tracing architectures, three datasets, and ten scorers, it demonstrates that leaderboard rankings are unstable and that scorers systematically favor policies optimized against their own architectures. The paper proposes a resampling-based reliability measure and an exclusion estimator that removes own-architecture scorers from policy evaluation.

Critical strengths
  1. Identifies systematic biases in simulated-learner benchmarks that can substantially affect tutoring-policy rankings.

  2. Frozen teaching sequences, resampling comparisons, difficulty-matched controls, and search-budget experiments help isolate scorer-dependent effects.

  3. Provides practical recommendations for reporting ranking reliability and mitigating evaluator self-preference without additional scoring costs.

Critical weaknesses
  1. Though excluding own-architecture scorers changes leaderboard rankings, the paper does not establish that the corrected rankings better reflect actual teaching effectiveness. Without external ground truth or real-student outcomes, the exclusion estimator's validity remains uncertain.

  2. Experiments cover only five knowledge-tracing architectures, three mathematics datasets, and short interventions of ten exercises. It remains unclear whether the observed biases extend to longer tutoring interactions, different educational domains, or LLM-based learner simulators.

  3. The proposed resampling floor provides a useful reference, but the threshold of Kendall is heuristic, and the sample-size extrapolation extends beyond the measured range. Moreover, the exclusion estimator does not resolve cross-architecture disagreement, leaving the reliability of the resulting rankings uncertain.

Questions for authors

Q1: Without real-student outcomes or an independent ground-truth evaluator, how can the authors establish that excluding own-architecture scorers produces more accurate rankings rather than simply different ones?

Q2: How sensitive are the reported disagreement estimates and conclusions to the resampling procedure, learner-level dependencies, and the choice of Kendall (0.8) as the reliability threshold?

Q3: Can the authors identify which architectural properties cause self-preference? For example, is it driven by differences in predicted learning gains, knowledge-state updates, or response calibration?

Q4: Does self-preference continue increasing with larger planning budgets, or does it eventually saturate? Would similar effects emerge under gradient-based policy optimization rather than candidate-sequence search?

Q5: Would the same conclusions hold for LLM-based simulated learners or longer, adaptive tutoring interactions where teaching sequences cannot be frozen independently of learner responses?

Reviewer_UhbeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the reliability of evaluating tutoring policies using simulated learners, focusing on sampling noise, disagreement between scoring models, and self-preference bias. The authors introduce a controlled evaluation protocol in which five model-based planning policies and three baseline policies are evaluated across three educational datasets (ASSIST2009, ASSIST2017, and NIPS34). The exercise sequences are fixed before evaluation and scored using ten knowledge-tracing models from five architectures with two random seeds each.

The experimental results show that leaderboard rankings are substantially affected by sampling variability and scorer architecture. Cross-architecture agreement falls below the within-scorer resampling baseline, and scorers systematically favor policies optimized against their own architectures. The authors further demonstrate that this self-preference increases with planning search budget, while simply averaging more scorers does not reliably resolve ranking disagreement. As a practical mitigation, the paper proposes excluding scorers that share the planning policy's architecture, which changes leaderboard rankings and some reported winners.

Critical strengths

The paper addresses an important methodological issue in simulation-based evaluation. The potential dependence between the model used for policy optimization and the model used for evaluation is worth examining, particularly as simulated learners become more common in tutoring research.

Freezing exercise sequences before scoring helps isolate the effects of scorer choice from differences in policy behavior. The comparison with a resampling baseline also provides a useful reference for distinguishing scorer disagreement from sampling variability.

The experiments cover three datasets, five knowledge-tracing architectures, and multiple random seeds. The inclusion of greedy and difficulty-matched controls, together with the search-budget analysis, strengthens the investigation of whether self-preference is associated with optimization against a particular model.

Critical weaknesses

All experiments rely on simulated learners and model-derived learning gains. Although the paper demonstrates disagreement and self-preference among evaluators, it does not establish how these effects relate to actual student learning outcomes. Consequently, it remains unclear whether the proposed correction produces rankings that better reflect real-world tutoring effectiveness.

The evaluation is restricted to five knowledge-tracing architectures, three educational datasets, and relatively short teaching sequences of ten exercises. The search-budget experiment is also conducted on only one dataset. It is therefore uncertain whether the observed patterns extend to longer tutoring interactions, more sophisticated tutoring agents, or other simulator-based evaluation settings.

Excluding same-architecture scorers is a reasonable heuristic for reducing the measured self-preference. However, the paper does not provide an independent reference ranking against which to assess the correction. Furthermore, different policies are averaged over different subsets of scorers, potentially introducing comparability issues when scorer baselines differ. The differences between raw-gain and difficulty-normalized rankings suggest that this aspect deserves further investigation.

Some findings differ across datasets and experimental runs. For example, same-architecture agreement is less consistent on NIPS34, and the correction does not change the ASSIST2017 winner in both runs. Although the paper discusses these differences, further analysis would help clarify the conditions under which the proposed conclusions and recommendations are most robust.

Questions for authors

Without ground-truth student learning outcomes, how can the proposed exclusion estimator be evaluated beyond demonstrating that it changes rankings? Could the authors provide additional evidence that the corrected rankings are more reliable or meaningful than the original rankings?

Since the exclusion estimator uses different scoring-model subsets for different policies, how sensitive are the results to differences in scorer calibration or baseline predicted gains? Would scorer-level normalization or comparisons relative to a common baseline lead to similar conclusions?

How do the authors expect self-preference and scorer disagreement to behave under longer tutoring interactions or policies that adapt to student responses over multiple teaching steps? Are there reasons to expect the observed effects to become stronger or weaker?

Can the authors further explain the differences observed on NIPS34 and between the two experimental runs? In particular, which findings should be considered broadly robust, and which depend on the specific dataset or sampling configuration?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.