- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces StudentBench, an evaluation suite and public dataset for AI tutoring. A study with 2,383 adult participants compares one-hour GRE tutoring by autonomous LLMs, expert human tutors, and a no-tutoring control. The main result is statistical equivalence between pooled AI and human tutoring on immediate, combined GRE learning gains within the stated equivalence bounds. A second study collects 2,028 expert pairwise reviews of lesson plans and practice problems. Additional analyses examine conversational teaching behaviors, cost, latency, and engagement.
Critical strengths
-
Direct learning evidence with a meaningful human reference. Unaided post-tests with different questions, counterbalanced assessment forms, and live expert human tutors make the primary evaluation relevant to whether students acquire knowledge through AI tutoring. Explicit equivalence tests, adjustments for shared tutors and repeated students, and sensitivity analyses strengthen this evidence.
-
A substantial and useful research resource. The study brings together multiple autonomous LLMs, live expert human tutoring, unaided assessments, and linked public data under a common protocol. Assessment responses, tutoring conversations, practice records, expert judgments, and analysis code provide useful infrastructure for further research. The resulting resource is valuable beyond the paper's headline comparisons.
Critical weaknesses
-
Strength and scope of the equivalence claims. Although the pooled equivalence result is supported by several sensitivity analyses, its generalizability remains limited by the relatively small human-tutor sample (140 sessions from 13 tutors). The main CR2/Satterthwaite analysis yields only 1.59 effective degrees of freedom. Excluding repeat participants increases this to 7.44 while preserving the equivalence result, and the leave-one-tutor-out checks are reassuring (Appendices B.2–B.4). Still, some uncertainty remains about the small-sample inference and generalization beyond the sampled tutors.
-
Validation and interpretation of secondary evidence. Expert ratings can provide an independent assessment of teaching materials, but their relationship to actual learning has not been clearly established. Comparison plans were generated for expert review and did not replace the plans used during tutoring; correlations with student answer disputes provide evidence about student reports, not independently verified errors or learning gains. The conversational evaluation raises a more direct measurement concern: lexical and turn-order rules have not been shown to reliably identify appropriate pedagogical behavior. Their scores can depend on verbosity and interaction length, and spoken human tutoring differs from typed AI chat. Comparing the rule outputs with human annotations would help establish whether they capture the intended behaviors. The paper should distinguish material quality, detected conversational patterns, and demonstrated learning effectiveness.
-
Latency and learning. The abstract appropriately describes correlations in the Quantitative section, but the proposed chain from faster responses through more interaction and correct practice to higher learning gains remains exploratory. Separate associations do not identify a mediated causal effect. Task difficulty, the clarity of student errors, student state, and response length could jointly affect these variables. Moreover, correct practice is credited using AI-generated answer keys, and students may consult the tutor before submitting. Temporal analyses provide less consistent support (Appendix D.2). The weak model-level relationship between latency and learning gains further limits the practical implication that choosing a faster model would improve learning outcomes.Together, these limitations reduce confidence in the secondary explanations of teaching effectiveness and learning mechanisms, rather than directly overturning the primary pooled comparison.
Questions for authors
-
Validity of the teaching metrics. What validation evidence is available for the expert scores and conversational indicators, especially agreement between the deterministic rules and human annotations? Are these intended as independent material-quality and behavior measures, or as proxies for learning effectiveness? If the latter, what evidence supports that interpretation?
-
Robustness to participation thresholds. Appendix B.9 shows that restoring 184 sessions excluded solely for insufficient AI participation changes the combined AI–human point estimate only slightly. Could the authors also report the corresponding 90% confidence interval and equivalence test using the same CR2/Satterthwaite method and fixed equivalence bounds, to assess whether the primary conclusion remains robust to these exclusions?
We are glad to see the reviewer finds that, in addition to our study of AI tutoring, “the resulting resource is valuable beyond the paper's headline comparisons.” We believe the dataset and data-collection application to be major contributions of this work.
We agree, and the reviewer summarizes the checks correctly. Appendix B.2 states that the small-sample approximation remains uncertain; we will add to the Limitations that the result may not generalize beyond the sampled human tutors.
The submission draws these distinctions, and we agree they should be more prominent. Section 8 states that the Figure 3 evaluations are not based on how much learning occurs; Section 4.1 states that answer disputes are student disagreements and not verified errors; and Section 4.2 and Appendix F state the verbosity and spoken-versus-typed caveats. We have not compared the rule outputs with human annotations and will say so in Section 4.2.
We agree that this chain is exploratory and does not identify a causal effect, and the submission presents it that way (Section 5.2; Appendix D.2, which also notes the AI-authored answer keys and the less consistent temporal comparisons). Table D.1 shows the weak tutor-level association between reply time and learning gain (ρ=−0.007). We will qualify two phrases that go further: “linking AI latency to learning gain” (Section 5.2) and “may indirectly influence” (Appendix D.2).
They are intended as independent measures of material quality and conversational behavior, not as proxies for learning (Section 8; Appendix F). For the expert scores, the evidence is blinded review with randomized order (Appendix E.1) and agreement with an independent student signal: tutors that experts preferred for practice design received fewer answer disputes (Spearman ρ=0.71 in Quantitative and 0.77 combined; 0.32 and not significant in Verbal; Appendix F.2). For the conversation indicators, we have no comparison with human annotations; please see our response above.