acceptodds
Under review as a conference paper at ICLR 2027

StudentBench: AI and human tutoring yield equivalent GRE learning gains

Abstract

AI offers an unprecedented opportunity to augment human capabilities by teaching us new knowledge and skills. However, the extent to which AI can do this had not been measured at scale. We introduce StudentBench, a suite of LLM teaching evaluations and a public platform that enables large-scale data collection with over 175,000 student–AI messages to study whether LLMs produce learning gains equivalent to human tutoring. Using StudentBench, we measured learning gains on Quantitative and Verbal GRE questions across 2,383 human participants receiving AI tutoring, human tutoring, or no tutoring. We establish that pooled AI tutoring is statistically equivalent to expert human tutoring for combined GRE learning gains (). In a second study, expert human tutors compared LLM-generated lesson plans and practice problems through 2,028 pairwise rubric evaluations. The two studies clearly separate LLMs for tutoring across (1) lesson planning, (2) practice-problem creation, (3) conversational pedagogy, (4) cost, and (5) engagement. One model achieved learning gains equivalent to human tutoring () at lower cost per percentage point of learning gain. For Quantitative GRE sessions, faster AI replies correlated with more student messages, more messages with more correct practice, and more correct practice with larger learning gains.

Primary area:
datasets and benchmarks
Keywords:
Tutoring, AI tutoring, RHSI, Evaluation, Dataset, Teaching, GRE

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JV9sOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces StudentBench, an evaluation suite and public dataset for AI tutoring. A study with 2,383 adult participants compares one-hour GRE tutoring by autonomous LLMs, expert human tutors, and a no-tutoring control. The main result is statistical equivalence between pooled AI and human tutoring on immediate, combined GRE learning gains within the stated equivalence bounds. A second study collects 2,028 expert pairwise reviews of lesson plans and practice problems. Additional analyses examine conversational teaching behaviors, cost, latency, and engagement.

Critical strengths
  1. Direct learning evidence with a meaningful human reference. Unaided post-tests with different questions, counterbalanced assessment forms, and live expert human tutors make the primary evaluation relevant to whether students acquire knowledge through AI tutoring. Explicit equivalence tests, adjustments for shared tutors and repeated students, and sensitivity analyses strengthen this evidence.

  2. A substantial and useful research resource. The study brings together multiple autonomous LLMs, live expert human tutoring, unaided assessments, and linked public data under a common protocol. Assessment responses, tutoring conversations, practice records, expert judgments, and analysis code provide useful infrastructure for further research. The resulting resource is valuable beyond the paper's headline comparisons.

Critical weaknesses
  1. Strength and scope of the equivalence claims. Although the pooled equivalence result is supported by several sensitivity analyses, its generalizability remains limited by the relatively small human-tutor sample (140 sessions from 13 tutors). The main CR2/Satterthwaite analysis yields only 1.59 effective degrees of freedom. Excluding repeat participants increases this to 7.44 while preserving the equivalence result, and the leave-one-tutor-out checks are reassuring (Appendices B.2–B.4). Still, some uncertainty remains about the small-sample inference and generalization beyond the sampled tutors.

  2. Validation and interpretation of secondary evidence. Expert ratings can provide an independent assessment of teaching materials, but their relationship to actual learning has not been clearly established. Comparison plans were generated for expert review and did not replace the plans used during tutoring; correlations with student answer disputes provide evidence about student reports, not independently verified errors or learning gains. The conversational evaluation raises a more direct measurement concern: lexical and turn-order rules have not been shown to reliably identify appropriate pedagogical behavior. Their scores can depend on verbosity and interaction length, and spoken human tutoring differs from typed AI chat. Comparing the rule outputs with human annotations would help establish whether they capture the intended behaviors. The paper should distinguish material quality, detected conversational patterns, and demonstrated learning effectiveness.

  3. Latency and learning. The abstract appropriately describes correlations in the Quantitative section, but the proposed chain from faster responses through more interaction and correct practice to higher learning gains remains exploratory. Separate associations do not identify a mediated causal effect. Task difficulty, the clarity of student errors, student state, and response length could jointly affect these variables. Moreover, correct practice is credited using AI-generated answer keys, and students may consult the tutor before submitting. Temporal analyses provide less consistent support (Appendix D.2). The weak model-level relationship between latency and learning gains further limits the practical implication that choosing a faster model would improve learning outcomes.Together, these limitations reduce confidence in the secondary explanations of teaching effectiveness and learning mechanisms, rather than directly overturning the primary pooled comparison.

Questions for authors
  1. Validity of the teaching metrics. What validation evidence is available for the expert scores and conversational indicators, especially agreement between the deterministic rules and human annotations? Are these intended as independent material-quality and behavior measures, or as proxies for learning effectiveness? If the latter, what evidence supports that interpretation?

  2. Robustness to participation thresholds. Appendix B.9 shows that restoring 184 sessions excluded solely for insufficient AI participation changes the combined AI–human point estimate only slightly. Could the authors also report the corresponding 90% confidence interval and equivalence test using the same CR2/Satterthwaite method and fixed equivalence bounds, to assess whether the primary conclusion remains robust to these exclusions?

1 comment

We are glad to see the reviewer finds that, in addition to our study of AI tutoring, “the resulting resource is valuable beyond the paper's headline comparisons.” We believe the dataset and data-collection application to be major contributions of this work.

Strength and scope of the equivalence claims. Although the pooled equivalence result is supported by several sensitivity analyses, its generalizability remains limited by the relatively small human-tutor sample (140 sessions from 13 tutors). The main CR2/Satterthwaite analysis yields only 1.59 effective degrees of freedom. Excluding repeat participants increases this to 7.44 while preserving the equivalence result, and the leave-one-tutor-out checks are reassuring (Appendices B.2–B.4). Still, some uncertainty remains about the small-sample inference and generalization beyond the sampled tutors.

We agree, and the reviewer summarizes the checks correctly. Appendix B.2 states that the small-sample approximation remains uncertain; we will add to the Limitations that the result may not generalize beyond the sampled human tutors.

Validation and interpretation of secondary evidence. Expert ratings can provide an independent assessment of teaching materials, but their relationship to actual learning has not been clearly established. Comparison plans were generated for expert review and did not replace the plans used during tutoring; correlations with student answer disputes provide evidence about student reports, not independently verified errors or learning gains. The conversational evaluation raises a more direct measurement concern: lexical and turn-order rules have not been shown to reliably identify appropriate pedagogical behavior. Their scores can depend on verbosity and interaction length, and spoken human tutoring differs from typed AI chat. Comparing the rule outputs with human annotations would help establish whether they capture the intended behaviors. The paper should distinguish material quality, detected conversational patterns, and demonstrated learning effectiveness.

The submission draws these distinctions, and we agree they should be more prominent. Section 8 states that the Figure 3 evaluations are not based on how much learning occurs; Section 4.1 states that answer disputes are student disagreements and not verified errors; and Section 4.2 and Appendix F state the verbosity and spoken-versus-typed caveats. We have not compared the rule outputs with human annotations and will say so in Section 4.2.

Latency and learning. The abstract appropriately describes correlations in the Quantitative section, but the proposed chain from faster responses through more interaction and correct practice to higher learning gains remains exploratory. Separate associations do not identify a mediated causal effect. Task difficulty, the clarity of student errors, student state, and response length could jointly affect these variables. Moreover, correct practice is credited using AI-generated answer keys, and students may consult the tutor before submitting. Temporal analyses provide less consistent support (Appendix D.2). The weak model-level relationship between latency and learning gains further limits the practical implication that choosing a faster model would improve learning outcomes. Together, these limitations reduce confidence in the secondary explanations of teaching effectiveness and learning mechanisms, rather than directly overturning the primary pooled comparison.

We agree that this chain is exploratory and does not identify a causal effect, and the submission presents it that way (Section 5.2; Appendix D.2, which also notes the AI-authored answer keys and the less consistent temporal comparisons). Table D.1 shows the weak tutor-level association between reply time and learning gain (ρ=−0.007). We will qualify two phrases that go further: “linking AI latency to learning gain” (Section 5.2) and “may indirectly influence” (Appendix D.2).

Validity of the teaching metrics. What validation evidence is available for the expert scores and conversational indicators, especially agreement between the deterministic rules and human annotations? Are these intended as independent material-quality and behavior measures, or as proxies for learning effectiveness? If the latter, what evidence supports that interpretation?

They are intended as independent measures of material quality and conversational behavior, not as proxies for learning (Section 8; Appendix F). For the expert scores, the evidence is blinded review with randomized order (Appendix E.1) and agreement with an independent student signal: tutors that experts preferred for practice design received fewer answer disputes (Spearman ρ=0.71 in Quantitative and 0.77 combined; 0.32 and not significant in Verbal; Appendix F.2). For the conversation indicators, we have no comparison with human annotations; please see our response above.

Reviewer_kyt6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tests whether today’s LLMs can tutor as well as human tutors. About 2,400 adults took a short GRE practice test, then spent one hour with one of 13 AI tutors, with an experienced human tutor over video call, or watching unrelated videos, and then took a similar second test. Learning gains with AI tutors were about the same as with human tutors, and both were clearly larger than in the video group; the authors conclude that AI and human tutoring are statistically equivalent. In a second study, expert tutors compare lesson plans written by different AI tutors. The paper also compares the AI tutors by cost, response speed and conversation style, and reports that one small open model matches human tutors at a fraction of the cost. Data and code are released.

Critical strengths
  1. A real study with real learners. Most work on AI tutors judges chat transcripts or simulated students. This paper measures what people actually learned, in a large randomized study with a control group and carefully constructed tests. That is rare and valuable, and the released data will be useful to others.
  2. An informative null result. The AI tutors, from a small open model to the most expensive commercial ones, lead to very similar learning gains. This is a useful, if sobering, finding for anyone who builds benchmarks of tutoring ability.
Critical weaknesses
  1. “Equivalent” is a stronger claim than the study supports. First, the bar for “equivalent” is generous. The allowed difference (about 4 percentage points) is more than half of the whole benefit of human tutoring over the control group (about 7 points). An AI tutor that delivered less than half the benefit of a human tutor would still count as equivalent. Second, the human side is small: nine tutors per section, some of whom taught many of the students. The analysis corrects for this, but with so few tutors the result depends on how the correction is done, and the authors’ own sensitivity check does not find equivalence. Third, the human group started with lower pre-test scores in both sections, and in the verbal section most of them received the same test version first. This is surprising for random assignment, and the paper does not explain how human sessions were scheduled or how many people dropped out of each group. Finally, “pooled AI tutoring” averages 13 different systems, the best AI tutor is a different model in every topic area, and there is no group that simply practiced on their own. I find “AI tutors achieve most of the benefit of human tutors in one hour” supported, but not yet “equivalent”.

  2. The framing and some conclusions go beyond the data. The idea of “recursive human self-improvement” and a “singularity for human intelligence” is not grounded in the learning sciences, and one hour of tutoring before a 27-question test is a weak test of it. The participants were mostly students from all majors, recruited through a career platform and paid $50; it is unclear how many were actually preparing for the GRE. The rankings that do separate the models (expert ratings of lesson plans, conversation style, cost, engagement) are never linked to learning, which is the one outcome where the models do not differ. For example, the model that experts rate best for lesson planning has one of the lowest learning gains. Several smaller claims also go too far: the advantage of Gemini models for strong students is not statistically supported, and the “918× cheaper” result relies on a single test out of twelve and on a human tutoring price taken from a 2018 website.

Questions for authors
  1. How were participants assigned to the human, AI and control groups, and how were the human sessions scheduled? How many people dropped out of each group?
  2. Does equivalence still hold with a stricter margin, and with an analysis that accounts for differences between individual human tutors?
  3. How many participants were preparing for the GRE, and do the results hold for them?
  4. Appendix A.1 mentions excluded “earlier control conditions”. What were they, and do they tell us how much people learn from practice alone?

Minor:

  • Each post-test question is matched by topic to a pre-test question, and the tutors see the pre-test. The study therefore measures near transfer, which should be stated.
  • The conversation-style scores compare typed AI chats with spoken human sessions and favor longer sessions, so they should not be presented as a ranking of teaching quality.
4 comments

“Equivalent” is a stronger claim than the study supports. First, the bar for “equivalent” is generous. The allowed difference (about 4 percentage points) is more than half of the whole benefit of human tutoring over the control group (about 7 points). An AI tutor that delivered less than half the benefit of a human tutor would still count as equivalent. Second, the human side is small: nine tutors per section, some of whom taught many of the students. The analysis corrects for this, but with so few tutors the result depends on how the correction is done, and the authors’ own sensitivity check does not find equivalence. Third, the human group started with lower pre-test scores in both sections, and in the verbal section most of them received the same test version first. This is surprising for random assignment, and the paper does not explain how human sessions were scheduled or how many people dropped out of each group. Finally, “pooled AI tutoring” averages 13 different systems, the best AI tutor is a different model in every topic area, and there is no group that simply practiced on their own. I find “AI tutors achieve most of the benefit of human tutors in one hour” supported, but not yet “equivalent”.

Several of these points are covered in the submission. The pooled result also meets the stricter ±0.20-SD bounds (p=.023; Section 3), and the 90% interval for the AI − human difference is [−2.18, 1.03] points, against a human − control difference of 7.04 (Table B.2). The sensitivity check that did not find equivalence (Appendix B.3) omits the pre-test and assessment-form covariates; under the primary model, equivalence holds when each human tutor is left out in turn (Appendix B.2). The baseline and form imbalances are reported in Table A.1 and the primary model adjusts for them. The missing practice-only control is stated in Section 8.

How the human sessions were scheduled, which explains the form imbalance in Verbal, and the exclusions in each condition are in our answer to your first question below. We will add both to the appendix.

The framing and some conclusions go beyond the data. The idea of “recursive human self-improvement” and a “singularity for human intelligence” is not grounded in the learning sciences, and one hour of tutoring before a 27-question test is a weak test of it. The participants were mostly students from all majors, recruited through a career platform and paid $50; it is unclear how many were actually preparing for the GRE. The rankings that do separate the models (expert ratings of lesson plans, conversation style, cost, engagement) are never linked to learning, which is the one outcome where the models do not differ. For example, the model that experts rate best for lesson planning has one of the lowest learning gains. Several smaller claims also go too far: the advantage of Gemini models for strong students is not statistically supported, and the “918× cheaper” result relies on a single test out of twelve and on a human tutoring price taken from a 2018 website.

On rankings and learning: Section 8 states that the Figure 3 evaluations are not based on how much learning occurs, and learning gains do not differ significantly among the AI tutors (p=.755; Appendix B.1), so the learning-gain order of individual tutors, including the one experts rate best, is not resolved. On Gemini and strong students: Appendix C.3 reports the comparison as inconclusive (p=.120); we will state this in Section 3. On cost: six of the twelve AI tutors pass the individual equivalence test, uncorrected (Appendix B.7), and the cost multiple scales with whatever human price is assumed (Figure 5). On GRE preparation: we did not record this; please see our answer to that question below.

How were participants assigned to the human, AI and control groups, and how were the human sessions scheduled? How many people dropped out of each group?

All participants came from the same pool: approximately 70,000 randomly selected students across all majors, invited through a career platform (Section 2.1). Each invitation led to one condition, and the students invited to each condition were drawn at random from this pool. Students invited to AI tutoring were placed at random among the AI tutors with the fewest students so far, which keeps the tutors balanced. Students invited to human tutoring booked a session time that suited them, without knowing who their tutor would be, and joined a live video call with the tutor for that slot. Students invited to the control condition enrolled through a separate link.

Each human tutor worked with only one test form, so that no tutor ever saw a student's post-test, and students took their tutor's form as the pre-test. In Verbal, the form P tutors taught more sessions, which is why most human-tutored Verbal students took form P first. Our main analysis accounts for pre-test score and form order.

The exclusions in each condition are below. Every rule was fixed without reference to post-test score or learning gain, and all are defined in Appendix A.2. The counts sum to the totals in Table A.2, which reports them by section only. We will add this table and the scheduling details to the appendix.

AIHumanControl
Started3,439160311
Pre-test score outside the eligible range (5–24 of 27 in Quantitative, 4–24 in Verbal)5381064
Insufficient assessment effort231436
Did not complete both assessments, or no recorded score317320
Did not take part in the tutoring (AI only)186––
Repeat attempt, or tutoring outside the assigned condition2831
Retained in the analysis2,139140190

Does equivalence still hold with a stricter margin, and with an analysis that accounts for differences between individual human tutors?

On the margin, yes: the pooled result also meets ±0.20-SD bounds (p=.023; Section 3). On tutors: the primary analysis clusters sessions that share a human tutor, and equivalence holds when each tutor is left out in turn (Appendix B.2). Please also see our response to the first weakness on Appendix B.3.

How many participants were preparing for the GRE, and do the results hold for them?

We did not collect this, so we cannot analyze that subgroup; we will note it in the Limitations. The GRE is an admissions test for graduate programs, so the people who take it are mainly upper-level undergraduates and recent graduates, and that is the population we recruited: participants had a median age of 21 and 83.3% were aged 18 to 24 (Appendix A.1), 73.5% of sessions were from undergraduates, and about half were from students in their third undergraduate year or later. We will add these demographics to the appendix.

Appendix A.1 mentions excluded “earlier control conditions”. What were they, and do they tell us how much people learn from practice alone?

They do not measure practice alone: as stated in Section 8, the study has no condition in which students worked through GRE practice problems on their own. In the final control condition, students reviewed their graded pre-test for at least five minutes and then watched videos unrelated to the GRE (Section 2.1; Appendix A.2).

Each post-test question is matched by topic to a pre-test question, and the tutors see the pre-test. The study therefore measures near transfer, which should be stated.

We agree. The design is described (questions matched on subtopic, Appendix A.1; tutors see the pre-test mistakes but never the post-test, Section 2.1), but the term is not used. We will state in Section 2 and in the Limitations that the study measures near transfer.

The conversation-style scores compare typed AI chats with spoken human sessions and favor longer sessions, so they should not be presented as a ranking of teaching quality.

We agree with these caveats, and the submission states them: Section 4.2 notes the bias toward longer sessions, and the Figure 3 caption says that the human reference is spoken tutoring and that scores should be compared only within panels. We will more clearly describe these scores as the prevalence of detected conversation patterns, and avoid framing them as a ranking of teaching quality.

Thank you for your detailed response and the additional information. The scheduling details explain the test-form imbalance. The leave-one-tutor-out results are also reassuring. However, your exclusion table also shows that the human group lost far fewer participants than the AI and control groups (about 12% vs. 38%), already at the pre-test, which suggests a more selected group, so I would still describe the result as “comparable” rather than “equivalent” and report how many invited students took part in each condition. Since my concerns about the “equivalent” claim and the broader framing (“recursive human self-improvement” and “singularity for human intelligence”) remain, I will keep my score.

Thank you Reviewer kyt6 for your prompt response! After some discussion among the authors, we have decided to remove the RHSI and singularity framing from our paper and will replace it with broader motivations grounded in more academic terms—specifically, that AI has potential to augment and amplify human intelligence, but the extent to which it can do so is not yet well-understood, thus motivating new evaluations.

However, your exclusion table also shows that the human group lost far fewer participants than the AI and control groups (about 12% vs. 38%), already at the pre-test, which suggests a more selected group

This is a good observation, and is to be expected: participants in the human group had to book a virtual video-on Zoom call with a human tutor, so they were less likely to do things that jeopardized the quality of our data , like randomly picking answers, tabbing away, dropping the study midway, etc. We accounted for this in our analysis and when making our claims of equivalence.

Also, to clarify, whenever we mention "equivalence" we mean "statistical equivalence," e.g., the pre-post score differential is not statistically significant between human and AI tutors. Of course, we recognize that AI and human tutoring are not "equivalent" in an experiential sense. We will clarify the scope of the term "equivalence" in our paper.

Reviewer_8iTaOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents StudentBench, a large-scale benchmark with over 175k student-AI interactions with 2,383 participants for quantitative and verbal GRE learning sessions. The authors additionally conduct a large-scale study with human experts to validate and assess the quality of lesson plans and practice problems generated by AI tutors in the benchmark's learning interactions. The paper conducts several comparative analyses of different LLMs in tutoring performance, cost, and engagement to draw additional findings from the benchmark.

Critical strengths
  • The paper conducts a large-scale data collection study, which is additionally augmented by expert-human annotations for a rich benchmark for GRE education. The resource will be very helpful within the AI and education community, given the dearth of high quality data in education.
  • The statistical analysis for the quantitative experiments helps ground the variation in model-human performance for the tutoring task, which helps establish the comparative performance of frontier LLMs to human tutors.
Critical weaknesses
  • While the paper is heavily inclined towards quantitative analysis, including some errors or examples of expert-human annotations would help improve the reading experience, as well as bring out the details of the final benchmark.
  • The study overtly emphasizes the pre-test/post-test differences to establish meaningful tutoring, not including other important aspects of learning, ranging from meta-cognitive learning, tutoring experience, affective nature of learning, and learner feedback. This is further visible in the analysis experiment measuring student engagement and AI reply time, where engagement is viewed only as the number of student messages, disregarding other aspects of meaningful engagement beyond the number of utterances, such as the quality of doubts being asked, the depth of topics detailed by the AI tutor, etc.
  • The expert-human analysis is limited to AI lesson plans, resulting in ungrounded relative findings, which do not express their quality compared to human tutors.

Minor feedback -

  • While the limitations acknowledge that the study is done with adults, who are often not the target of GRE education, and it is a genuine limitation of the benchmark, it should ideally not be called a student benchmark if it is not grounded in student interactions.
  • The abstract and intro establish the equivalence of human and AI tutor performance by quoting (p=0.015), without describing the conditions or statistical test until later.
Questions for authors
  • Why did the authors only consider pairwise evaluation for AI models (Fig. 3 results)? Why were only AI-generated lessons and embedded problems the artifacts for this evaluation study?
  • Was first-person feedback (especially open-ended responses) taken from the participants in the study after the post-test? It would be good to include the qualitative, experience-driven attributes in the findings, if possible.
1 comment

The study overtly emphasizes the pre-test/post-test differences to establish meaningful tutoring, not including other important aspects of learning, ranging from meta-cognitive learning, tutoring experience, affective nature of learning, and learner feedback. This is further visible in the analysis experiment measuring student engagement and AI reply time, where engagement is viewed only as the number of student messages, disregarding other aspects of meaningful engagement beyond the number of utterances, such as the quality of doubts being asked, the depth of topics detailed by the AI tutor, etc.

Learning gain is the primary outcome, but the submission reports other measures too. Figure 3C (Section 4.2) scores what happens in the dialogue with six conversation indicators grounded in literature in the learning and cognitive sciences, such as requests for explanation and reasoning checks. Students in the AI condition also rated how helpful the conversation was for their learning (Appendix D.1, Figure D.4), and Appendix D.2 repeats the engagement analysis with message length and practice measures in place of message counts. We did not measure metacognitive or affective outcomes, which is a limitation. Also in our limitations section, we briefly mentioned our narrow definition of learning as pre/post-test score difference but will expand on it to make this limitation clearer.

The expert-human analysis is limited to AI lesson plans, resulting in ungrounded relative findings, which do not express their quality compared to human tutors.

Human tutors had only a short time to prepare for each session and usually wrote a few notes, not a full lesson plan. A pairwise comparison of those notes with AI-written plans would favor the AI tutors on length alone and would say little about teaching quality. We therefore compare AI and human tutors where both produce the same thing: learning gains (Section 3) and conversation behavior (Figure 3C). We will add this explanation to Section 2.3.

While the limitations acknowledge that the study is done with adults, who are often not the target of GRE education, and it is a genuine limitation of the benchmark, it should ideally not be called a student benchmark if it is not grounded in student interactions.

Participants were students: we invited about 70,000 randomly selected students across all majors (Section 2.1), and the analyzed participants had a median age of 21, with 83.3% aged 18 to 24 (Appendix A.1). The "all adults" sentence in Section 8 is about generalizing to other ages and settings. We will reword it in the paper. We also have additional statistics about participants that we can add to the appendix, which also addresses some of Reviewer kyt6’s questions about participant info: 1,815 of the 2,469 main-cohort sessions (73.5%) are from undergraduates, 223 (9.0%) from graduate students and 381 (15.4%) from people with post-secondary education who are not currently in school; 39 are missing and 11 list other levels.

The abstract and intro establish the equivalence of human and AI tutor performance by quoting (p=0.015), without describing the conditions or statistical test until later.

The test is specified in Section 2.2 and shown in Figure 1B (two one-sided tests with ±0.25-SD equivalence bounds). We agree that it should be named where the p-value first appears, and will revise the abstract and introduction.

Why did the authors only consider pairwise evaluation for AI models (Fig. 3 results)? Why were only AI-generated lessons and embedded problems the artifacts for this evaluation study?

Human tutors' notes are not comparable with AI lesson plans; please see our response above on the expert analysis. Among the AI tutors, each pair of plans is generated from the same student's pre-test, so reviewers judge both against the same mistakes, blind to tutor identity (Section 2.3; Appendix E.1). Lesson plans are also not the only artifact we evaluate: Figure 3C scores the tutoring conversations of AI and human tutors, and Section 3 compares their learning gains.

Was first-person feedback (especially open-ended responses) taken from the participants in the study after the post-test? It would be good to include the qualitative, experience-driven attributes in the findings, if possible.

Yes, in part. After tutoring, students in the AI condition rated how helpful the conversation was for their learning on a five-point scale; per-tutor means range from 3.73 to 4.33 (Appendix D.1, Figure D.4). The submission does not analyze open-ended responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.