- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces a controlled benchmark for evaluating LLM-based programming tutors. It combines student diagnostic states, acceptance behavior, cognitive-load modeling, and structured feedback interactions to measure multiple aspects of tutoring. The evaluation includes comparisons across tutor models and learner profiles, component analyses, sensitivity analyses, and human studies. The benchmark is best interpreted as a standardized diagnostic testbed rather than a validated simulator of long-term learning in real students. Within that scope, it offers a useful framework for examining differences in tutor behavior and outcomes.
Critical strengths
- The benchmark makes several otherwise conflated aspects of tutoring explicit and experimentally controllable. This diagnostic structure is useful for understanding why tutors differ, beyond a single aggregate success metric.
- The evaluation covers multiple tutors, learner configurations, component analyses, human evidence, and paired uncertainty estimates. The sensitivity analysis in Appendix G.3 provides evidence that the reported results are not simply an artifact of one load-credit calculation, although it does not validate every simulator assumption.
Critical weaknesses
-
The behavioral validity of the overload-related information-retention mechanism remains insufficiently established. Recomputing scores on fixed trajectories under alternative credit rules does not establish robustness to changing which information the simulated student actually retains. This limits how strongly benchmark differences can be interpreted as differences in teaching effectiveness.
-
The detector validation in Appendix H is unclear about independence from calibration. The text describes selecting weights and thresholds to maximize F1 on 150 expert-labeled turns, and Table 26 reports performance on a set of the same size without an explicit held-out split. Moreover, half the sample is enriched for suspected positives. Thus, the reported precision and accuracy cannot directly be interpreted as deployment-distribution performance, and possible calibration optimism needs to be addressed.
Questions for authors
-
Were the examples used for Table 26 independent of threshold and weight selection? Please clarify the split, preferably at the problem level, and quantify how detector errors affect the main tutor comparisons. Please also distinguish performance on the enriched sample from performance under the natural operating distribution.
-
How sensitive are the main tutor differences to changing the overload-related retention policy and rerunning the interactions, rather than rescoring fixed trajectories? This would clarify the extent to which the findings depend on the simulator’s behavioral assumptions.
-
Teach2Eval (ICLR 2026) already evaluates models through teaching interactions and the resulting improvements in weaker students, but it does not appear to be discussed in the submission. Could the authors clarify what CogTeachBench contributes beyond this existing evaluation paradigm? In particular, under matched tasks, student backbones, and interaction budgets, what evidence shows that the acceptance and cognitive-load controls improve evaluation validity relative to an unconstrained weak-student interaction protocol? Such evidence would affect my assessment of the benchmark’s contribution.
3 comments
(Part 2/2)
Q3 (relation to Teach2Eval). We were aware of Teach2Eval (Zhou et al., 2026) but left it out of the submission for space; the revision discusses it in Related Work.
The two benchmarks have different goals. Teach2Eval is a general-capability benchmark: it maps how much a model improves weak student models on multiple-choice versions of existing tasks onto the model's overall ability, and it is validated against general leaderboards. CogTeachBench targets tutoring as a vertical setting, where general capability is a weak proxy for teaching ability: fluent, capable responses can be misaligned with the intended learning outcome (Hardy & Kim, 2026). In our results, GPT-5.4 attributes root causes most accurately, yet its dense turns overload the learner and lower its teaching gain (Section 5.1 and Appendix E.7).
Our contribution is built around the learner, at four levels.
- Screened and verified scenarios. Problems are released after the learner models' pretraining cutoff, so no answer can be recalled. A zero-intervention screen removes scenarios the learner repairs on its own (181 of 546), and an oracle screen keeps only scenarios that become repairable under correct guidance, so any gain must come from teaching.
- Controlled learners. LLM students accept plausible corrections readily, retain every message, and behave uniformly. The acceptance control α and the overload control β, calibrated on student load ratings, address these behaviors, and two learner sizes crossed with these controls give twelve profiles, so results are not fitted to one kind of student.
- Constraints over the whole process. The anti-spooning check counts answer transfer as failure, and hidden executable tests decide repair, so neither copying from context nor agreement without repair yields a teaching gain.
- Measurement grounded in educational theory. Overload follows cognitive-load theory, and the process dimensions follow cognitive diagnosis, conceptual change, and scaffolding research. Teaching is measured on gain, ability sensitivity, process quality, and load efficiency, which is what lets the benchmark explain why tutors differ.
Q3 (evidence relative to an unconstrained weak-student protocol). Our α-off, β-off configuration is this protocol under matched conditions: the same 316 scenarios, the same Qwen2.5 learner backbones, the same interaction budget and post-test, with only the acceptance and load controls removed. The table reports CBCS gaps for the three teachers run on the complete control grid, with 95% CIs from a task-clustered bootstrap (Section 5.2, Table 3; Appendix F.1).
| Configuration | Qwen3.5 Plus − Flash | GPT-5.4 − Qwen3.5 Flash |
|---|---|---|
| α off, β off (unconstrained) | +0.052 [−0.030, +0.135] | +0.079 [+0.000, +0.162] |
| α only | +0.132 [+0.060, +0.209] | +0.152 [+0.086, +0.216] |
| β only | +0.180 [+0.091, +0.269] | +0.199 [+0.121, +0.279] |
| α + β (main setting) | +0.153 [+0.089, +0.219] | +0.196 [+0.136, +0.260] |
| Human post-test success | +16.7 pp | +25.0 pp |
Under the unconstrained protocol, the learner proxy shows confusion in only 0.65% of turns and outputs code immediately in 30.09% of turns (Appendix F), and the benchmark no longer separates Qwen3.5 Plus from Qwen3.5 Flash, although human learners taught by Plus succeed 16.7 pp more often. With either control, the separation returns. Under matched tasks, backbones, and budgets, the controls therefore recover teaching differences that human learners show and the unconstrained protocol misses.
We hope these additions resolve your questions, and we would be glad to clarify anything further.
Thank you for the response, which addressed most of my concerns. I believe this is a good paper, so I will maintain my positive assessment.
(Part 1/2)
Thank you for the careful review and for describing the benchmark as a standardized diagnostic testbed that is useful for understanding why tutors differ. That reading matches our intent. We address the three questions below.
W1 and Q2 (rerunning interactions under a different retention policy). We agree that rescoring fixed trajectories, as in Appendix G.3, does not change what the learner retains. We had run the interaction-level test before submission but omitted it from the manuscript. The revision reports it in Appendix G.4.
The learner proxy is a standardized learner that applies one retention rule to every teacher. The head–tail rule approximates limited working-memory capacity under overload (Sweller, 1988; Cowan, 2001) and was specified with educational experts (Appendix G.2). We varied the retention ratio and reran the interactions with head/tail 15%/15% and 35%/35%, under α-low+β-on, with the 3B and 7B learners, five teachers, and the 316 scenarios. From the first overload turn onward, the learner sees a different retained message, and all later learner replies, teacher turns, and post-tests are regenerated. The dialogue prefix before the first overload turn is identical to the original run and is served from the request-hash cache.
Mean CBCS over the 3B and 7B learners.
The teacher ranking is identical under all three ratios (Spearman ρ = 1.0, no pairwise inversions), and the main gaps keep their size (Gemini 3.1 Pro − GPT-5.4: 0.058/0.059/0.051; Qwen3.5 Plus − Flash: 0.136/0.131/0.123). The spread across ratios grows with the teacher's overload rate, from 0.012 for Gemini 3.1 Pro to 0.059 for Qwen3.5 Flash. The tutor differences thus depend on whether a teacher overloads the learner, not on the exact retention ratio.
W2 and Q1 (independence of Table 26 from calibration). Table 26 (Table 30 in the revision) reports in-sample performance on the 150 turns used for calibration. To quantify calibration optimism, we repeated the threshold selection inside a nested 5-fold cross-validation grouped by problem, so that no problem appears in both the selection and evaluation folds. Over 30 repetitions, the held-out F1 is 0.825 (range 0.812–0.840), with precision 0.795 and recall 0.858, close to the reported 0.837. The revision adds this analysis and the error-propagation results below to Appendix H.
Q1 (enriched sample, natural distribution, and effect on the main comparisons). The enriched half of the 150 turns oversamples suspected positives, so its precision and accuracy are not deployment-distribution estimates. In deployment, positives are rare: the detector flags 2.39% of formal benchmark records (Appendix H). What matters for the paper's conclusions is how detector errors propagate to the tutor comparison, and we bound this in both directions. Removing the detector entirely, which treats every flag as a false positive, leaves the ranking unchanged (0.588, 0.509, 0.490, 0.485, 0.347). We also propagated the measured error rates by Monte Carlo, reversing flags according to precision and miss rate over 3,000 draws, and the full ranking was preserved in every draw. The tutor comparison is therefore insensitive to detector errors at any plausible operating point.