- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The manuscript examines whether LLMs can select examples and organize concepts into lessons that support human learning. It addresses an important question using both formally defined tasks and human learning experiments.
Critical strengths
Strengths:
-
Clear motivation. The study separates the contribution of instructional content from dialogue and feedback in LLM tutoring and issues an important question that even a responsive AI tutor must decide which examples and concepts to present.
-
Theory-grounded evaluation. In the tasks, the study compares LLM choices with a Bayesian teaching model and then measure how human learners respond, which provides an opportunity to assess both the theoretical informativeness and practical effectiveness of the examples.
-
Insightful comparison of teaching strategies. The study shows both contrastive and positive example can be optimal when constructed appropriately. In the human evaluation, LLM-generated examples, which more often use contrastive pairs, produce higher accuracy and shorter response times on correct trials than those from the Bayesian teaching model. This suggests that equally informative examples can differ in how easily learners identify the rule.
-
Extension to compositional learning. The study moves beyond simple concepts to curricula that introduce intermediate concepts before compositional targets. Besides, in the human experiment, participants receiving these curricula identified target rules more accurately than those receiving target-only examples.
-
Evaluation in different domains. The study extends to logic circuits, z-scores, and Mendelian genetics, and compares several curriculum conditions using the same assessment within each subject, which strengthens the evidence for generalizability.
Critical weaknesses
Weakness:
-
Although the full curriculum performs better than target-only instruction, it also provides more content and practice, so it is hard to isolate the effects of decomposition and ordering. It would be helpful to add a control group to control these factors.
-
Although each naturalistic domain is evaluated by an exam generated by an LLM, it is essential to make sure the exam covers the curriculum knowledge and document item coverage, answer validation, and potential overlap with instruction.
-
The naturalistic experiments do not compare LLM-generated curricula with a teacher-designed lesson or a simple prerequisite-based sequence matched for content and study time. Please add the control group to explore the practical benefit specifically attributable to LLM-generated curriculum design.
-
The prerequisite order is generated by an LLM and annotated by another LLM. But the prompt may imply the order. Please validate these relations with independent subject experts and inspect the curricula actually shown to human participants.
-
Each participant answers 12 questions, so the answers may not be considered as independent. Please repeat the primary analyses at the participant level.
Questions for authors
Please see the weakness part.
3 comments
Response to Reviewer jTeP [2/2]
The naturalistic experiments do not compare LLM-generated curricula with a teacher-designed lesson or a simple prerequisite-based sequence matched for content and study time. Please add the control group to explore the practical benefit specifically attributable to LLM-generated curriculum design.
We agree that comparison with human-designed curricula is valuable, but leave this comparison for future work. We thank the reviewer for noting this point, and agree that such a comparison would quantify the benefit of LLM-generated curricula with respect to human teaching in naturalistic domains; we will leave this comparison for future work. The current evidence presented demonstrates that LLMs, to a certain degree, can scale principled curricula generation from simple to complex, real-world domains. Still, it is valuable to explore how LLM curriculum quality compares and adds to those designed by human teachers, which we believe warrants a separate study to address this question in full.
The prerequisite order is generated by an LLM and annotated by another LLM. But the prompt may imply the order. Please validate these relations with independent subject experts and inspect the curricula actually shown to human participants.
We manually validated all 43 prerequisite relations and confirmed their correct ordering in every single-pass curriculum shown to participants. All 43 prerequisite relations generated in the decomposition plans were judged to be faithful by the authors, and in each curriculum we also confirmed that prerequisites were taught before their dependent concepts. Because verification was done by the authors rather than independent subject experts, we will make the curricula available after the review period.
Shuffling the listed objectives confirms that prerequisite ordering is largely robust to prompt order. We also show that the ordering of the learning objectives within the prompt does not determine the model’s own prerequisite ordering, which we further explain in our response to Reviewer dwPe. When the listed objectives were shuffled, curricula still taught prerequisites before their dependent concepts in 95.8% of prerequisite pairs (vs. 50% chance; Wilcoxon p < .001; n = 90; 100% with the original ordering), including 90.5% of cases where the prompt listed the dependent concept before its prerequisite (p < .001). Prompt order therefore has a small influence on the ordering of the curricula but these data suggest that models still make choices about what content should be taught first even under different orderings of objectives listed in the prompt. Moreover, we note that while the learning objectives specify which skills to teach, around half the time models restructure these objectives in their decomposition plans, splitting or merging them, or introduce new subskills (e.g., gametes and segregation for Mendelian genetics).
Each participant answers 12 questions, so the answers may not be considered as independent. Please repeat the primary analyses at the participant level.
Participant-level analyses preserve the main comparisons and their significance. We repeated the analyses both with participant random intercepts (Tables R1 and R3) and using each participant's mean assessment accuracy as the outcome (linear regression with domain fixed effects and robust standard errors; Exp. 1: +29.1 pp [19.2, 39.0], p < .001; C1: +20.4 pp [10.4, 30.3], p < .001; C2: +9.1 pp [1.6, 16.6], p = .009; C3: −0.2 pp [−7.6, 7.2], p = .96; C1 and C2 one-sided).
I thank the authors for the response. I maintain my rating.
Response to Reviewer jTeP [1/2]
We thank the reviewer for their positive assessment of our motivation, comparisons to theoretical models, analysis of teaching strategies, extensions to compositional learning, and evaluation across domains. We respond to the reviewer’s feedback below.
We conducted an additional experiment matching the instructional volume of target-only and two-stage curricula, finding a trending advantage for the two-stage condition (Tables R6–R8). Following the reviewer’s suggestion, we conducted a further experiment to isolate the effect of knowledge organization from additional teaching volume. To do this, we re-prompted Gemini 3.1 Pro Preview to extend the target-only curriculum from Exp. 2 to match the two-stage curriculum in the number of pages, practice questions, and words; the resulting curricula closely match the two-stage condition in the amount of instructional content (Table R6). Overall, the two-stage condition had higher accuracy, but neither accuracy nor total experiment duration differed significantly in two-sided Welch t-tests (Table R7, Table R8); z-score differences reached p < .05 before correction (accuracy p = .034; duration p = .049) but not after Holm correction across the three domains (p = .101 and .147). Timeout rates were similar across both conditions (two-stage 18.3% vs. length-matched 19.6%).
Importantly, all practice questions in the length-matched target-only curricula address the full target skill, while the two-stage curricula also budget their practice questions across prerequisite skills. As a consequence the two-stage condition has less practice on the full target skill. Despite this, the two-stage condition shows a trending advantage in assessment accuracy over the length-matched target-only condition (80.7% vs. 74.3%; t = 1.67, p = .096).
We manually validated the coverage and correctness of the assessment questions and provide a mapping to learning objectives (Table R5). We address the reviewer’s concerns about 1) the coverage of the assessment with respect to the curriculum and 2) the correctness of the assessment.
The assessment was generated by ensuring that each question mapped to at least one of the predefined learning objectives. We include this mapping in Table R5, which the authors (who have coursework in these subjects) manually verified while taking the assessment. During our inspection, we also validated that each of the answers for the assessment questions is correct.
We additionally checked for overlap between curriculum content and assessments, and confirmed that removing duplicate questions preserves our main results. We address the reviewer’s note on overlap between the assessment and curriculum content. The curricula and assessments are independently generated through separate prompts. Using regex, we extracted key values and expressions that were relevant for identifying worked examples of circuit structures, genetic crosses, or z-score computations from the LLM-generated curricula, and matched them to what was tested in each assessment question.
This algorithm flagged 112 candidates in which the underlying structure of a worked example or practice question possibly appeared in a later assessment question. We manually reviewed every instance, which resulted in us identifying 10 of the 36 total assessment questions as containing duplicate material. We observe that the duplicates for Mendelian genetics are more pronounced (7 of the 10 questions), but this is because a simple Punnett square has only six cross combinations that can be tested.
Regardless, after removing the duplicate assessment questions, all significant comparisons remain significant and in the same direction, and C3 remains nonsignificant (Exp. 1: 1.86 [1.21, 2.52], p < .001; C1: 1.23 [0.54, 1.92], p < .001; C2: 0.69 [0.09, 1.30], p = .013; C3: 0.06 [−0.57, 0.68], p = .85; C1 and C2 one-sided).