acceptodds
Under review as a conference paper at ICLR 2027

CogTeachBench: A Standardized Learner-State Benchmark for Evaluating LLM Tutors

Abstract

Large language models (LLMs) are increasingly embedded in learners’ study processes as explainers, feedback providers, and personalized tutors. However, evaluating whether an LLM can actually teach remains challenging, especially when teaching is treated as measurable change in a learner’s misconception state. Existing educational benchmarks typically assess static response quality, task-solving accuracy, or isolated pedagogical rubrics, while learner variation is often reduced to coarse role prompts or ability labels. To address this gap, we introduce CogTutorBench, a benchmark for evaluating LLM tutors through controlled misconception repair. We instantiate it in algorithmic programming using LiveCodeBench problems released after the learner-proxy models’ pretraining cutoff, resulting in 316 oracle-verified misconception-repair scenarios. We further define standardized learner profiles by varying ability tier, acceptance behavior, and cognitive-load constraints, and evaluate tutors with four complementary metrics, including Cognitive Bottleneck Clearance Score (CBCS), Adaptive ZPD Span (AZS), Process-level Scaffolding Quality (PSQ), and Pedagogical Cognitive Efficiency (PCE). Experiments show that tutoring quality has multiple rankings. Gemini 3.1 Pro achieves the strongest post-teaching clearance, while GPT-5.4 achieves the highest process quality and load-normalized efficiency. Learner profiles substantially change measured outcomes. Cognitive-load constraints reduce teaching gain, whereas high-acceptance or higher-ability learners may produce fluent agreement signals that some tutors mistake for executable understanding. These findings suggest that robust LLM tutor evaluation must jointly measure learner improvement, pedagogical process, cognitive efficiency, and variation across learner profiles.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, AI for Education, LLM Tutors, Benchmarking, Standardized Learner Proxies, Interactive LLM Tutoring

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FxGVOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a controlled benchmark for evaluating LLM-based programming tutors. It combines student diagnostic states, acceptance behavior, cognitive-load modeling, and structured feedback interactions to measure multiple aspects of tutoring. The evaluation includes comparisons across tutor models and learner profiles, component analyses, sensitivity analyses, and human studies. The benchmark is best interpreted as a standardized diagnostic testbed rather than a validated simulator of long-term learning in real students. Within that scope, it offers a useful framework for examining differences in tutor behavior and outcomes.

Critical strengths
  1. The benchmark makes several otherwise conflated aspects of tutoring explicit and experimentally controllable. This diagnostic structure is useful for understanding why tutors differ, beyond a single aggregate success metric.
  2. The evaluation covers multiple tutors, learner configurations, component analyses, human evidence, and paired uncertainty estimates. The sensitivity analysis in Appendix G.3 provides evidence that the reported results are not simply an artifact of one load-credit calculation, although it does not validate every simulator assumption.
Critical weaknesses
  1. The behavioral validity of the overload-related information-retention mechanism remains insufficiently established. Recomputing scores on fixed trajectories under alternative credit rules does not establish robustness to changing which information the simulated student actually retains. This limits how strongly benchmark differences can be interpreted as differences in teaching effectiveness.

  2. The detector validation in Appendix H is unclear about independence from calibration. The text describes selecting weights and thresholds to maximize F1 on 150 expert-labeled turns, and Table 26 reports performance on a set of the same size without an explicit held-out split. Moreover, half the sample is enriched for suspected positives. Thus, the reported precision and accuracy cannot directly be interpreted as deployment-distribution performance, and possible calibration optimism needs to be addressed.

Questions for authors
  1. Were the examples used for Table 26 independent of threshold and weight selection? Please clarify the split, preferably at the problem level, and quantify how detector errors affect the main tutor comparisons. Please also distinguish performance on the enriched sample from performance under the natural operating distribution.

  2. How sensitive are the main tutor differences to changing the overload-related retention policy and rerunning the interactions, rather than rescoring fixed trajectories? This would clarify the extent to which the findings depend on the simulator’s behavioral assumptions.

  3. Teach2Eval (ICLR 2026) already evaluates models through teaching interactions and the resulting improvements in weaker students, but it does not appear to be discussed in the submission. Could the authors clarify what CogTeachBench contributes beyond this existing evaluation paradigm? In particular, under matched tasks, student backbones, and interaction budgets, what evidence shows that the acceptance and cognitive-load controls improve evaluation validity relative to an unconstrained weak-student interaction protocol? Such evidence would affect my assessment of the benchmark’s contribution.

3 comments

(Part 1/2)

Thank you for the careful review and for describing the benchmark as a standardized diagnostic testbed that is useful for understanding why tutors differ. That reading matches our intent. We address the three questions below.

W1 and Q2 (rerunning interactions under a different retention policy). We agree that rescoring fixed trajectories, as in Appendix G.3, does not change what the learner retains. We had run the interaction-level test before submission but omitted it from the manuscript. The revision reports it in Appendix G.4.

The learner proxy is a standardized learner that applies one retention rule to every teacher. The head–tail rule approximates limited working-memory capacity under overload (Sweller, 1988; Cowan, 2001) and was specified with educational experts (Appendix G.2). We varied the retention ratio and reran the interactions with head/tail 15%/15% and 35%/35%, under α-low+β-on, with the 3B and 7B learners, five teachers, and the 316 scenarios. From the first overload turn onward, the learner sees a different retained message, and all later learner replies, teacher turns, and post-tests are regenerated. The dialogue prefix before the first overload turn is identical to the original run and is served from the request-hash cache.

TeacherOverload turns15%/15%25%/25% (main)35%/35%
Gemini 3.1 Pro7.3%0.5860.5930.598
Claude Opus 4.610.2%0.5390.5490.558
GPT-5.419.4%0.5290.5340.547
Qwen3.5 Plus31.2%0.4460.4700.492
Qwen3.5 Flash36.4%0.3100.3390.369

Mean CBCS over the 3B and 7B learners.

The teacher ranking is identical under all three ratios (Spearman ρ = 1.0, no pairwise inversions), and the main gaps keep their size (Gemini 3.1 Pro − GPT-5.4: 0.058/0.059/0.051; Qwen3.5 Plus − Flash: 0.136/0.131/0.123). The spread across ratios grows with the teacher's overload rate, from 0.012 for Gemini 3.1 Pro to 0.059 for Qwen3.5 Flash. The tutor differences thus depend on whether a teacher overloads the learner, not on the exact retention ratio.

W2 and Q1 (independence of Table 26 from calibration). Table 26 (Table 30 in the revision) reports in-sample performance on the 150 turns used for calibration. To quantify calibration optimism, we repeated the threshold selection inside a nested 5-fold cross-validation grouped by problem, so that no problem appears in both the selection and evaluation folds. Over 30 repetitions, the held-out F1 is 0.825 (range 0.812–0.840), with precision 0.795 and recall 0.858, close to the reported 0.837. The revision adds this analysis and the error-propagation results below to Appendix H.

Q1 (enriched sample, natural distribution, and effect on the main comparisons). The enriched half of the 150 turns oversamples suspected positives, so its precision and accuracy are not deployment-distribution estimates. In deployment, positives are rare: the detector flags 2.39% of formal benchmark records (Appendix H). What matters for the paper's conclusions is how detector errors propagate to the tutor comparison, and we bound this in both directions. Removing the detector entirely, which treats every flag as a false positive, leaves the ranking unchanged (0.588, 0.509, 0.490, 0.485, 0.347). We also propagated the measured error rates by Monte Carlo, reversing flags according to precision and miss rate over 3,000 draws, and the full ranking was preserved in every draw. The tutor comparison is therefore insensitive to detector errors at any plausible operating point.

(Part 2/2)

Q3 (relation to Teach2Eval). We were aware of Teach2Eval (Zhou et al., 2026) but left it out of the submission for space; the revision discusses it in Related Work.

The two benchmarks have different goals. Teach2Eval is a general-capability benchmark: it maps how much a model improves weak student models on multiple-choice versions of existing tasks onto the model's overall ability, and it is validated against general leaderboards. CogTeachBench targets tutoring as a vertical setting, where general capability is a weak proxy for teaching ability: fluent, capable responses can be misaligned with the intended learning outcome (Hardy & Kim, 2026). In our results, GPT-5.4 attributes root causes most accurately, yet its dense turns overload the learner and lower its teaching gain (Section 5.1 and Appendix E.7).

Our contribution is built around the learner, at four levels.

  1. Screened and verified scenarios. Problems are released after the learner models' pretraining cutoff, so no answer can be recalled. A zero-intervention screen removes scenarios the learner repairs on its own (181 of 546), and an oracle screen keeps only scenarios that become repairable under correct guidance, so any gain must come from teaching.
  2. Controlled learners. LLM students accept plausible corrections readily, retain every message, and behave uniformly. The acceptance control α and the overload control β, calibrated on student load ratings, address these behaviors, and two learner sizes crossed with these controls give twelve profiles, so results are not fitted to one kind of student.
  3. Constraints over the whole process. The anti-spooning check counts answer transfer as failure, and hidden executable tests decide repair, so neither copying from context nor agreement without repair yields a teaching gain.
  4. Measurement grounded in educational theory. Overload follows cognitive-load theory, and the process dimensions follow cognitive diagnosis, conceptual change, and scaffolding research. Teaching is measured on gain, ability sensitivity, process quality, and load efficiency, which is what lets the benchmark explain why tutors differ.

Q3 (evidence relative to an unconstrained weak-student protocol). Our α-off, β-off configuration is this protocol under matched conditions: the same 316 scenarios, the same Qwen2.5 learner backbones, the same interaction budget and post-test, with only the acceptance and load controls removed. The table reports CBCS gaps for the three teachers run on the complete control grid, with 95% CIs from a task-clustered bootstrap (Section 5.2, Table 3; Appendix F.1).

ConfigurationQwen3.5 Plus − FlashGPT-5.4 − Qwen3.5 Flash
α off, β off (unconstrained)+0.052 [−0.030, +0.135]+0.079 [+0.000, +0.162]
α only+0.132 [+0.060, +0.209]+0.152 [+0.086, +0.216]
β only+0.180 [+0.091, +0.269]+0.199 [+0.121, +0.279]
α + β (main setting)+0.153 [+0.089, +0.219]+0.196 [+0.136, +0.260]
Human post-test success+16.7 pp+25.0 pp

Under the unconstrained protocol, the learner proxy shows confusion in only 0.65% of turns and outputs code immediately in 30.09% of turns (Appendix F), and the benchmark no longer separates Qwen3.5 Plus from Qwen3.5 Flash, although human learners taught by Plus succeed 16.7 pp more often. With either control, the separation returns. Under matched tasks, backbones, and budgets, the controls therefore recover teaching differences that human learners show and the unconstrained protocol misses.

We hope these additions resolve your questions, and we would be glad to clarify anything further.

Thank you for the response, which addressed most of my concerns. I believe this is a good paper, so I will maintain my positive assessment.

Reviewer_sQYTOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

The paper introduces a benchmark for evaluating LLM tutors: 316 oracle-verified misconception-repair scenarios, a configurable learner proxy, and four metrics. The authors report that the tutor ranking agrees with human transfer success (ρ = 1.00).

Critical strengths
  1. The artifact is real: post-cutoff problems, oracle verification, executable post-tests, and frozen splits.
  2. Awkward numbers are not hidden: confidence intervals overlap in the middle of the ranking, and the load penalty differs across tutors.
Critical weaknesses

There is a serious problem with the writing: the paper reads more like an evaluation report than a scientific study. What it lacks is analysis of the results. For example, why do the five tutors end up in this order? The main text does not answer this.

  1. The central construct of the benchmark is diagnosis, but diagnosis is never measured. The tutor is given the verified reference solution, and only the diagnostic key is withheld. With the answer in context, diagnosis reduces to comparing against it. No diagnostic accuracy is reported. The only proxy is the RCA sub-score of PSQ, which is scored by an LLM judge.

  2. The human study cannot carry the ρ = 1.00 claim in the abstract. It is a Spearman correlation over five tutors, with no confidence interval and no test, and the paper's own first-task analysis gives ρ = 0.975 with one tie. The 316 scenarios come from only 166 tasks, so scenarios from the same task are not independent and the reported intervals are too narrow.

Questions for authors
  1. What is the diagnostic accuracy, and can part of the study be rerun with the reference solution withheld?
  2. What is the confidence interval and test for ρ = 1.00, and can the bootstrap be clustered by task?
  3. Are the four metrics independent? Most AZS intervals include zero, and PCE is normalized by load.
4 comments

(Part 1/2)

Thank you for the detailed review and for noting that the artifact is real and that the paper reports its intervals openly. We address the writing concern first, then the three weaknesses and questions.

Readability. The revision adds an overview figure (Figure 1) that shows a scenario, the teaching dialogue under the learner controls, and the post-test, with each metric placed at the stage where it is measured. The submission omitted it because of the page limit; the revised main text includes it.

Analysis of the results: why do the five tutors end up in this order? The submission analyzes the sources of the ranking in three places, though spread across subsections. Section 5.3 decomposes process quality: GPT-5.4 has the strongest root-cause attribution (RCA 0.929), while Gemini 3.1 Pro leads on guided participation and small repair slices (SGR 0.778, MSS 0.889). Section 5.4 shows that enabling overload costs Qwen3.5 Flash the most and GPT-5.4 the least (0.116 vs. 0.057 under high acceptance), so the β control separates tutors that pace and localize repair from those that do not. Section 5.5 and Appendix J show, with paired logs, that some tutors end the lesson once a high-acceptance learner signals agreement, before the repair is executable. FxGV describes this decomposition as "useful for understanding why tutors differ".

The revision draws these findings together in a paragraph at the end of Section 5.1 and adds dialogue-level evidence in Appendix E.7. Overload is the common factor: for every tutor, dialogues with an overloaded turn have lower post-test success (Gemini 3.1 Pro 0.671 vs. 0.391; Qwen3.5 Flash 0.435 vs. 0.279).

TeacherCBCSWords / turnTurns with codeOverloaded dialogues
Gemini 3.1 Pro0.58110910.4%21.8%
GPT-5.40.50917914.4%40.0%
Claude Opus 4.60.48612014.2%31.5%
Qwen3.5 Plus0.46616418.4%64.8%
Qwen3.5 Flash0.31317634.2%69.0%

Gemini 3.1 Pro leads because it teaches in short, finely sliced steps and rarely overloads the learner. GPT-5.4 diagnoses most accurately but packs the explanation into long turns and ends after 3.25 turns on average, so 40% of its dialogues reach overload; the 7B learner tolerates denser turns (overload threshold 0.787 vs. 0.685), which explains why GPT-5.4 is the only tutor with a clearly positive AZS. Claude Opus 4.6 rarely overloads but scores lower on root-cause attribution and Socratic guidance. The Qwen tutors overload the learner in about two thirds of dialogues, and Qwen3.5 Flash relies most on code and most often ends the lesson while the learner still fails the post-test (66% of the lessons it ends, against 39% for Gemini 3.1 Pro).

W1 and Q1 (diagnosis and the reference solution). The central construct of CogTeachBench is clearing the learner's misconception, the cognitive bottleneck that CBCS measures: whether a learner who failed because of the misconception passes hidden tests after teaching. Diagnosis is a means to this end and enters as one process dimension (RCA) within PSQ. We revised the sentence in Section 3 whose wording could suggest that diagnosis is the evaluated outcome.

The reference solution is given to every teacher so that score differences reflect teaching rather than problem-solving ability; Reviewer 9BZL notes that this provides "a clearer basis for comparing teaching effectiveness". Withholding it would reintroduce differences in solving ability into the comparison, which the design removes on purpose. It also mirrors tutoring systems supplied with trusted answer keys.

Having the solution does not reduce diagnosis to comparison. The solution shows what correct code looks like, not why the learner wrote the bug. The learner's belief, recorded in the diagnostic key, is withheld from the teacher, and the learner's opening query reports symptoms and may attribute the failure to a plausible but wrong cause, so the teacher must infer the misconception. Pointing to the difference between the buggy and correct code is also not enough to score: code-level answers are flagged and count as failures, and a low-acceptance learner rejects corrections offered without evidence. The data reflect this. Although all teachers see the same reference solution, RCA ranges from 0.683 to 0.929 across teachers, which would not happen if diagnosis reduced to reading off the answer. RCA is scored against the withheld diagnostic key by a judge selected for agreement with three experts (87.6% within-one agreement, against 91.3% between experts; Appendix I.3).

(Part 2/2)

W2 and Q2 (test and interval for ρ = 1.00). With five tutors there are 120 possible orderings and only one gives ρ = 1, so the exact permutation p-value is 0.008 (one-sided) and 0.017 (two-sided). Resampling each tutor's successes from its observed rate gives a 95% bootstrap interval of [0.56, 1.00] for ρ. The strongest and weakest tutors differ by 30.6 pp (Fisher's exact test, p = 0.0001). For the first-task analysis, ρ = 0.975 has an exact one-sided p-value of 0.017; the tie is between Claude Opus 4.6 and Qwen3.5 Plus (5/7 each). The revised abstract, Section 5.2, and Appendix B.1 report ρ with the number of tutors and these tests.

W2 and Q2 (clustering by task). We reran the CBCS bootstrap clustered by the 166 source tasks. The intervals widen slightly, for example from [+0.014, +0.131] to [+0.008, +0.135] for Gemini 3.1 Pro vs. GPT-5.4, and the significance conclusion of every one of the ten pairwise comparisons is unchanged. Qwen3.5 Plus vs. Flash remains separated (+0.153 [+0.089, +0.219]), and the AZS of GPT-5.4 remains above zero (0.109 [0.015, 0.203]). The revision reports the task-clustered intervals in Section 5.1 and Appendix E.5.

Q3 (are the four metrics independent?). They are designed as related views of the same teaching episode, and they are not redundant. Across the 20 teacher–profile cells, CBCS correlates with PSQ at ρ = 0.82, which is expected, since better scaffolding should produce more repair. Yet the metrics select different best tutors: Gemini 3.1 Pro on CBCS and GPT-5.4 on PCE. Within each teacher, CBCS and load are nearly uncorrelated (r = −0.16), so load captures a separate property of the teaching (Appendix E.6). PCE is normalized by load by definition, and we present it as an efficiency summary that combines the two. AZS is an ability-sensitivity measure, and a value near zero is itself a finding: the teacher's gain does not depend on learner ability. The one clearly positive AZS, for GPT-5.4, is explained by its dense turns, as described above.

Ethics flag. The paper involves two low-risk studies with human participants: 13 student volunteers rated the cognitive load of teaching segments, and 36 CS undergraduates read prerecorded tutoring dialogues and then solved the corresponding programming tasks. Participants did not interact with any model, teacher identities were hidden, participation was voluntary and organized as mutual help among students, all participants gave informed consent and could withdraw at any time, no sensitive personal information was collected, and responses were anonymized before analysis.

We hope these additions address your concerns, and we would be grateful if you would reconsider your assessment in light of the revision.

Thank you for your rebuttal. Your response resolved my doubts about the paper, especially regarding the evaluation method and the diagnostic process. Therefore, I have increased my score.

Thank you for reading our response and the revised paper, and for raising your score. We appreciate the time and care you have given to the review and to this discussion. If any question remains, or if any part of the paper could be made clearer, please let us know. We will answer it fully and revise the paper accordingly.

Reviewer_9BZLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CogTeachBench to evaluate the ability of large language model teachers to correct students' misunderstandings. The authors constructed 316 programming teaching scenarios, controlling scenario difficulty through unguided correction screening and teachability verification under explicit guidance, and setting an acceptance control α and a cognitive overload control β for simulated students. Evaluation combines four categories of indicators: program correction, ability sensitivity, teaching process quality, and cognitive efficiency. Experiments show that different teachers have their own advantages in error correction effectiveness and teaching efficiency, and students' acceptance and information retention limitations affect the teaching process and outcomes. Teacher rankings remained largely stable after changing student model families and were consistent with the transfer performance rankings in a 36-person dialogue playback study.

Critical strengths
  1. Unguided screening, teachability verification, reference solutions provided to teachers, hidden tests, and answer leakage detection reduce interference from students' existing knowledge, teachers' problem-solving limitations, and direct answer copying, providing a clearer basis for comparing teaching effectiveness.
  2. Cognitive load calibration using student ratings, expert annotation checks, experiments across learner model families, and a human transfer study provide initial support for the benchmark. Paired dialogues also illustrate how alpha affects teacher explanations and verification through student follow-up questions, linking interaction patterns to repair outcomes.
Critical weaknesses
  1. Whether the addition of student controls improves the correlation between assessments and real-person learning outcomes has not been directly verified. Behavioral statistics and paired cases of alpha indicate that it can alter the interaction process, but it is uncertain whether this change improves the reliability of teacher evaluations. The real-person experiment only reports the ranking of the five teachers under the complete configuration, without comparing whether assessments with alpha/β disabled could still predict real-person results. Furthermore, the load calibration of β cannot directly verify the information retention rule of "keeping the two ends and deleting the middle," and the existing sensitivity analysis did not change this rule and rerun the teaching interactions. Therefore, the assessment value brought by the core controls remains unclear.

  2. Successful program repair still provides limited support for "correction of misunderstandings." Simulated students can still read the instructional dialogue during post-tests, and the repaired program is for the original question; answer leakage detection mainly targets the transmission of answers in code form. Therefore, within the permissible range of natural language guidance, students may still complete the repair by executing specific modification steps without mastering the corresponding concepts. The human transfer experiment provided positive evidence, but its scope was limited to 20 medium-weight scenarios, and the paper lacked conceptual judgments or transfer verification of simulated student success cases. This limits the teaching conclusions that the key indicators can support.

Questions for authors
  1. In the scenarios corresponding to the human study, which evaluation configuration can better predict human performance: the full configuration, α disabled, beta disabled, or both disabled? Has the addition of controls brought quantifiable improvements?
  2. Replace the information retention method of β with other reasonable rules or different retention ratios, and rerun the teaching. Will the teacher ranking and main conclusions be maintained?
  3. For simulated students who have successfully repaired the original program, are there variant questions testing the same concept, transfer questions,, or conceptual judgment results that can distinguish between concept mastery and executing specific modification instructions?
6 comments

(Part 1/3)

Thank you for the constructive review and for recognizing that the screening, reference solutions, hidden tests, and leakage detection give a clearer basis for comparing teaching effectiveness. We answer the three questions in order.

W1 and Q1 (do the learner controls improve prediction of human outcomes?). Yes, and the improvement shows up as resolution between teachers that humans separate. Three teachers, GPT-5.4, Qwen3.5 Plus, and Qwen3.5 Flash, were run on the complete control grid, and they span the top, middle, and bottom of the human ranking. The table reports their CBCS gaps under each configuration, with 95% CIs from a bootstrap clustered by task.

ConfigurationQwen3.5 Plus − FlashGPT-5.4 − Qwen3.5 Flash
α off, β off+0.052 [−0.030, +0.135]+0.079 [+0.000, +0.162]
α only+0.132 [+0.060, +0.209]+0.152 [+0.086, +0.216]
β only+0.180 [+0.091, +0.269]+0.199 [+0.121, +0.279]
α + β (main setting)+0.153 [+0.089, +0.219]+0.196 [+0.136, +0.260]
Human post-test success+16.7 pp+25.0 pp

Without learner controls, the benchmark cannot distinguish Qwen3.5 Plus from Qwen3.5 Flash, although human learners taught by Plus succeed 16.7 pp more often. Each control alone restores this separation, and the full configuration keeps it. On the w2 scenarios, the bucket used in the human study, the configurations without β place Qwen3.5 Plus above GPT-5.4, whereas the configurations with β recover the human order GPT-5.4 > Plus > Flash. The controls therefore make the benchmark resolve teaching differences that human learners exhibit and that an unconstrained learner proxy compresses. The revision adds this analysis to Section 5.2 (Table 3), with details in Appendix F.1.

A direct test of each configuration against human outcomes would require a separate human study per configuration. A participant cannot be returned to the pre-teaching state once taught, so each configuration needs its own cohort reading its own dialogues (Appendix B.1). More broadly, validating simulated learners against human learning is an open problem: recent work documents systematic gaps between prompted LLM students and real learners and calls for constrained, cognitively grounded learner proxies (Wu et al., 2025; Yuan et al., 2026; Salhan et al., 2026). CogTeachBench answers this call, and its controls are supported at three levels. Each control targets a documented artifact of LLM learners: α counters post-training compliance, and β follows cognitive-load theory with overload thresholds calibrated on student load ratings. Both shift learner behavior in the intended direction, with more defensive dialogues under α-low (58.3% vs. 50.3%, p = 0.002) and more confusion and fewer immediate code outputs under overload (Appendix F). As shown above, they also recover teacher differences that human learners exhibit. The released pipeline accepts any learner simulator, so new simulators can be evaluated under the same protocol as validation methods for simulated learners mature.

(Part 2/3)

W1 and Q2 (retention rule). We ran this experiment before submission but omitted it from the manuscript. The revision reports it in Appendix G.4.

The learner proxy is designed as a standardized learner. One retention rule is applied identically to every teacher, so score differences reflect the teaching and not the learner. The head–tail rule is an engineering approximation of limited working-memory capacity under overload (Sweller, 1988; Cowan, 2001), specified with educational experts (Appendix G.2). The relevant question is whether the teacher comparison depends on the specific retention ratio.

We reran the teaching interactions with two alternative ratios, head/tail 15%/15% and 35%/35%, under α-low+β-on with the 3B and 7B learners, for all five teachers on the 316 scenarios. From the first overload turn onward, the learner receives a different retained message, and all later learner replies, teacher turns, and post-tests are regenerated. Requests identical to the original run, which form the dialogue prefix before the first overload turn, are served from the request-hash cache, so each variant differs from the original only where the retention rule takes effect.

TeacherOverload turns15%/15%25%/25% (main)35%/35%
Gemini 3.1 Pro7.3%0.5860.5930.598
Claude Opus 4.610.2%0.5390.5490.558
GPT-5.419.4%0.5290.5340.547
Qwen3.5 Plus31.2%0.4460.4700.492
Qwen3.5 Flash36.4%0.3100.3390.369

Mean CBCS over the 3B and 7B learners.

The teacher ranking is identical under all three ratios (Spearman ρ = 1.0, no pairwise inversions). The Gemini 3.1 Pro − GPT-5.4 gap is 0.058/0.059/0.051 and the Qwen3.5 Plus − Flash gap is 0.136/0.131/0.123. The largest change from the reported rule is 0.030 (Qwen3.5 Flash). The spread between the 15% and 35% settings grows with the teacher's overload rate, from 0.012 for Gemini 3.1 Pro to 0.059 for Qwen3.5 Flash. The retention ratio therefore matters mainly for teachers that frequently overload the learner, which is the behavior β is designed to expose, and it does not change the teacher comparison.

(Part 3/3)

W2 and Q3 (is the misconception actually repaired?). Yes. CogTeachBench measures misconception repair the way cognitive diagnostic assessment measures any latent cognitive state: through responses to tasks built so that the response is diagnostic of that state (Leighton & Gierl, 2007). The design closes each route by which a learner could pass without revising the misconception.

The task is diagnostic of the misconception. Each buggy program is derived from the misconception through an explicit failure chain from belief to inference to behavior, and experts checked this causal link and excluded unrelated implementation errors (Section 3; Appendix C.2). Two screens then tie the outcome to the belief. The zero-intervention screen shows that a learner holding the misconception cannot repair the program, and the oracle screen shows that correcting this misconception is sufficient to repair it. Passing the post-test therefore indicates that the behavior produced by the misconception is gone.

Fixing one example does not pass. The hidden test suite, never shown to the learner, probes the misconception's failure mode on inputs that do not appear in the dialogue. A patch for the failing example discussed with the teacher fails these tests. Passing requires a program that follows the corrected rule on every input, which is evidence at the level of the rule rather than the instance.

Following instructions without revising the belief does not pass. Teacher code that solves the task or matches the reference solution counts as a failure, so the learner cannot copy a fix (Appendix H). Long step-by-step instructions trigger overload, after which the learner keeps only part of the message, and the post-test sees only what the learner retained. A low-acceptance learner revises a belief only when confronted with evidence, the condition conceptual change theory places on belief revision (Posner et al., 1982). The paired dialogues you highlight in Strength 2 show the consequence: in ctb_189, with the same teacher and learner, the high-acceptance learner accepts a surface repair and fails, while the low-acceptance learner first asks for trace evidence, revises its belief, and passes. Agreement without belief revision does not produce CBCS.

Human learners confirm the ordering without the dialogue. Participants read the dialogue and then solved the task without returning to it, so they could rely only on what they had understood. Their success orders the five tutors exactly as CBCS does (ρ = 1.00, exact p = 0.017).

We have also corrected the description of the human study in Section 5.2 and Appendix B.1. The previous wording "structurally different transfer task" was meant to say that no participant repeated a task with a different teacher; participants solved the same task as in the dialogue, without returning to the dialogue.

We hope these results address your concerns, and we would be grateful if you would consider them in your final assessment.

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted their responses to the reviews. Please go through their rebuttal at your earliest convenience, and engage in a discussion or leave any follow-up comments.

The discussion period ends on November 18.

Thanks,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.