acceptodds
Under review as a conference paper at ICLR 2027

Evaluating the Ability of Large Language Models to Generate Structured Curricula for Human Learners

Abstract

Large language models (LLMs) are increasingly used to teach human learners, but it remains unclear what aspects of their instruction support learning. Most work has focused on interactive tutoring, where models can respond to questions, provide feedback, and adapt to a learner. Here, we isolate a complementary component: the curriculum a model chooses to present in advance. We evaluate LLM-generated curricula with 1,428 human learners across four concept-learning settings, progressing from controlled tasks in which teaching strategies can be characterized formally to more complex scientific and mathematical subjects. Where tractable, we compare LLM curricula with Bayesian pedagogy, a normative account in which teaching depends on selecting information with a learner's inference in mind. LLM-generated curricula align with the Bayes-optimal model, while exhibiting systematic preferences among distinct strategies that the theory treats as equally optimal, and these preferences benefit human learners. As the concepts become more compositional, models spontaneously introduce intermediate concepts and organize them into structured curricula that improve learning. The same organization appears in naturalistic subjects, where static LLM-generated curricula substantially improve human performance. Together, these results suggest that an important component of LLM teaching lies in how models select and structure instructional content, and that this organization extends from choosing examples to decomposing complex concepts into learnable parts for human learners.

Primary area:
applications to neuroscience & cognitive science
Keywords:
large language models, pedagogy, Bayesian pedagogy, concept learning, cognitive science, curriculum generation, compositionality

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jTePOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript examines whether LLMs can select examples and organize concepts into lessons that support human learning. It addresses an important question using both formally defined tasks and human learning experiments.

Critical strengths

Strengths:

  1. Clear motivation. The study separates the contribution of instructional content from dialogue and feedback in LLM tutoring and issues an important question that even a responsive AI tutor must decide which examples and concepts to present.

  2. Theory-grounded evaluation. In the tasks, the study compares LLM choices with a Bayesian teaching model and then measure how human learners respond, which provides an opportunity to assess both the theoretical informativeness and practical effectiveness of the examples.

  3. Insightful comparison of teaching strategies. The study shows both contrastive and positive example can be optimal when constructed appropriately. In the human evaluation, LLM-generated examples, which more often use contrastive pairs, produce higher accuracy and shorter response times on correct trials than those from the Bayesian teaching model. This suggests that equally informative examples can differ in how easily learners identify the rule.

  4. Extension to compositional learning. The study moves beyond simple concepts to curricula that introduce intermediate concepts before compositional targets. Besides, in the human experiment, participants receiving these curricula identified target rules more accurately than those receiving target-only examples.

  5. Evaluation in different domains. The study extends to logic circuits, z-scores, and Mendelian genetics, and compares several curriculum conditions using the same assessment within each subject, which strengthens the evidence for generalizability.

Critical weaknesses

Weakness:

  1. Although the full curriculum performs better than target-only instruction, it also provides more content and practice, so it is hard to isolate the effects of decomposition and ordering. It would be helpful to add a control group to control these factors.

  2. Although each naturalistic domain is evaluated by an exam generated by an LLM, it is essential to make sure the exam covers the curriculum knowledge and document item coverage, answer validation, and potential overlap with instruction.

  3. The naturalistic experiments do not compare LLM-generated curricula with a teacher-designed lesson or a simple prerequisite-based sequence matched for content and study time. Please add the control group to explore the practical benefit specifically attributable to LLM-generated curriculum design.

  4. The prerequisite order is generated by an LLM and annotated by another LLM. But the prompt may imply the order. Please validate these relations with independent subject experts and inspect the curricula actually shown to human participants.

  5. Each participant answers 12 questions, so the answers may not be considered as independent. Please repeat the primary analyses at the participant level.

Questions for authors

Please see the weakness part.

3 comments
AuthorsOpenReview ↗

Response to Reviewer jTeP [1/2]

We thank the reviewer for their positive assessment of our motivation, comparisons to theoretical models, analysis of teaching strategies, extensions to compositional learning, and evaluation across domains. We respond to the reviewer’s feedback below.

Although the full curriculum performs better than target-only instruction, it also provides more content and practice, so it is hard to isolate the effects of decomposition and ordering. It would be helpful to add a control group to control these factors.

We conducted an additional experiment matching the instructional volume of target-only and two-stage curricula, finding a trending advantage for the two-stage condition (Tables R6–R8). Following the reviewer’s suggestion, we conducted a further experiment to isolate the effect of knowledge organization from additional teaching volume. To do this, we re-prompted Gemini 3.1 Pro Preview to extend the target-only curriculum from Exp. 2 to match the two-stage curriculum in the number of pages, practice questions, and words; the resulting curricula closely match the two-stage condition in the amount of instructional content (Table R6). Overall, the two-stage condition had higher accuracy, but neither accuracy nor total experiment duration differed significantly in two-sided Welch t-tests (Table R7, Table R8); z-score differences reached p < .05 before correction (accuracy p = .034; duration p = .049) but not after Holm correction across the three domains (p = .101 and .147). Timeout rates were similar across both conditions (two-stage 18.3% vs. length-matched 19.6%).

Importantly, all practice questions in the length-matched target-only curricula address the full target skill, while the two-stage curricula also budget their practice questions across prerequisite skills. As a consequence the two-stage condition has less practice on the full target skill. Despite this, the two-stage condition shows a trending advantage in assessment accuracy over the length-matched target-only condition (80.7% vs. 74.3%; t = 1.67, p = .096).

Although each naturalistic domain is evaluated by an exam generated by an LLM, it is essential to make sure the exam covers the curriculum knowledge and document item coverage, answer validation, and potential overlap with instruction.

We manually validated the coverage and correctness of the assessment questions and provide a mapping to learning objectives (Table R5). We address the reviewer’s concerns about 1) the coverage of the assessment with respect to the curriculum and 2) the correctness of the assessment.

The assessment was generated by ensuring that each question mapped to at least one of the predefined learning objectives. We include this mapping in Table R5, which the authors (who have coursework in these subjects) manually verified while taking the assessment. During our inspection, we also validated that each of the answers for the assessment questions is correct.

We additionally checked for overlap between curriculum content and assessments, and confirmed that removing duplicate questions preserves our main results. We address the reviewer’s note on overlap between the assessment and curriculum content. The curricula and assessments are independently generated through separate prompts. Using regex, we extracted key values and expressions that were relevant for identifying worked examples of circuit structures, genetic crosses, or z-score computations from the LLM-generated curricula, and matched them to what was tested in each assessment question.

This algorithm flagged 112 candidates in which the underlying structure of a worked example or practice question possibly appeared in a later assessment question. We manually reviewed every instance, which resulted in us identifying 10 of the 36 total assessment questions as containing duplicate material. We observe that the duplicates for Mendelian genetics are more pronounced (7 of the 10 questions), but this is because a simple Punnett square has only six cross combinations that can be tested.

Regardless, after removing the duplicate assessment questions, all significant comparisons remain significant and in the same direction, and C3 remains nonsignificant (Exp. 1: 1.86 [1.21, 2.52], p < .001; C1: 1.23 [0.54, 1.92], p < .001; C2: 0.69 [0.09, 1.30], p = .013; C3: 0.06 [−0.57, 0.68], p = .85; C1 and C2 one-sided).

AuthorsOpenReview ↗

Response to Reviewer jTeP [2/2]

The naturalistic experiments do not compare LLM-generated curricula with a teacher-designed lesson or a simple prerequisite-based sequence matched for content and study time. Please add the control group to explore the practical benefit specifically attributable to LLM-generated curriculum design.

We agree that comparison with human-designed curricula is valuable, but leave this comparison for future work. We thank the reviewer for noting this point, and agree that such a comparison would quantify the benefit of LLM-generated curricula with respect to human teaching in naturalistic domains; we will leave this comparison for future work. The current evidence presented demonstrates that LLMs, to a certain degree, can scale principled curricula generation from simple to complex, real-world domains. Still, it is valuable to explore how LLM curriculum quality compares and adds to those designed by human teachers, which we believe warrants a separate study to address this question in full.

The prerequisite order is generated by an LLM and annotated by another LLM. But the prompt may imply the order. Please validate these relations with independent subject experts and inspect the curricula actually shown to human participants.

We manually validated all 43 prerequisite relations and confirmed their correct ordering in every single-pass curriculum shown to participants. All 43 prerequisite relations generated in the decomposition plans were judged to be faithful by the authors, and in each curriculum we also confirmed that prerequisites were taught before their dependent concepts. Because verification was done by the authors rather than independent subject experts, we will make the curricula available after the review period.

Shuffling the listed objectives confirms that prerequisite ordering is largely robust to prompt order. We also show that the ordering of the learning objectives within the prompt does not determine the model’s own prerequisite ordering, which we further explain in our response to Reviewer dwPe. When the listed objectives were shuffled, curricula still taught prerequisites before their dependent concepts in 95.8% of prerequisite pairs (vs. 50% chance; Wilcoxon p < .001; n = 90; 100% with the original ordering), including 90.5% of cases where the prompt listed the dependent concept before its prerequisite (p < .001). Prompt order therefore has a small influence on the ordering of the curricula but these data suggest that models still make choices about what content should be taught first even under different orderings of objectives listed in the prompt. Moreover, we note that while the learning objectives specify which skills to teach, around half the time models restructure these objectives in their decomposition plans, splitting or merging them, or introduce new subskills (e.g., gametes and segregation for Mendelian genetics).

Each participant answers 12 questions, so the answers may not be considered as independent. Please repeat the primary analyses at the participant level.

Participant-level analyses preserve the main comparisons and their significance. We repeated the analyses both with participant random intercepts (Tables R1 and R3) and using each participant's mean assessment accuracy as the outcome (linear regression with domain fixed effects and robust standard errors; Exp. 1: +29.1 pp [19.2, 39.0], p < .001; C1: +20.4 pp [10.4, 30.3], p < .001; C2: +9.1 pp [1.6, 16.6], p = .009; C3: −0.2 pp [−7.6, 7.2], p = .96; C1 and C2 one-sided).

I thank the authors for the response. I maintain my rating.

Reviewer_dwPeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether LLMs can generate effective curricula in advance of teaching. It evaluates static instruction with 1,428 human learners across causal, primitive, compositional and naturalistic concept-learning settings. In tractable domains, model choices are compared with Bayesian pedagogy. Equally informative strategies can differ in how well people learn from them. In more complex domains, models introduce intermediate concepts and arrange them into lessons that improve immediate assessment performance. Naturalistic experiments compare full curricula, target-only instruction and single-pass generation, finding no detectable additional benefit from explicitly prompting decomposition.

Critical strengths

[+] Experiments with 1,428 human learners across several tasks and models directly test whether the curricula help people learn.

[+] The Bayesian comparison shows that equally informative examples can differ in human accuracy and response time.

[+] The single-pass and two-stage comparisons test whether explicit decomposition adds to ordinary curriculum generation.

Critical weaknesses

[-] Decomposed curricula include more content and practice, so the gains cannot be attributed to organization alone. Appendix D.8 does not control for this difference.

[-] The naturalistic regressions omit participant and curriculum effects despite repeated responses and shared curricula (Appendix E.3). This may understate uncertainty.

[-] The prompts supply ordered objectives, and curricula follow this order in 92.8% of pairs (Table 10). Prompt order could explain some of the apparent prerequisite ordering.

Questions for authors
  1. Do the contrasts hold with participant, item, and curriculum clustering, and with chance-level participants retained?
  2. Which evidence separates organization from extra content and practice? Please report instructional time and practice counts by condition.
  3. Does prerequisite ordering persist when the listed objectives are shuffled? Table 10 shows 92.8% agreement with prompt order.
3 comments
AuthorsOpenReview ↗

Response to Reviewer dwPe [1/2]

We thank the reviewer for their positive assessment of our experiments with human learners, comparisons with Bayesian pedagogy, and analysis of decomposition in LLM-generated curricula. We answer the reviewer’s questions below.

Do the contrasts hold with participant, item, and curriculum clustering, and with chance-level participants retained?

The direction and significance of all main comparisons are preserved after accounting for participant, item, and curriculum-level clustering (Table R4). The causal, primitive, and compositional learning tasks include participant and concept/item random intercepts. However, as the reviewer notes, the model for the naturalistic experiment does not account for participant, item, and curriculum-level clustering, so we implement our pre-registered model with these three added intercepts, and report their results with and without participant exclusions (Table R4).

We did not exclude participants for low accuracy, and have corrected the exclusion criteria in Appendix A.2 (Table R2). We would also like to clarify that we did not exclude any participant for low accuracy, so chance-level participants are retained in all reported analyses; the exclusion criteria we applied are detailed in Table R2. Appendix A.2 states that chance-level participants are excluded from analyses, but this statement did not reflect the exclusions we implemented in practice; we have revised the paper to list the actual criteria. Preregistration deviations and an analysis of participants who timed out before the assessment are detailed in our response to Reviewer 1cy3.

Which evidence separates organization from extra content and practice? Please report instructional time and practice counts by condition.

We conducted an additional experiment matching the amount of instructional content and practice across conditions (Table R6). To do this, we re-prompted Gemini 3.1 Pro Preview to extend the target-only curriculum from Exp. 2 to match the two-stage curriculum in the number of pages, practice questions, and words; the resulting curricula closely match the two-stage condition in the amount of instructional content (Table R6). Curricula in either condition contained five pages and five practice questions and participants on average took similar amounts of time to complete the experiment (two-stage 16.9 minutes, length-matched 17.7 minutes).

The two-stage curricula yielded higher accuracy, although the differences were not statistically significant (Tables R7 and R8). Overall, the two-stage condition had higher accuracy, but neither accuracy nor total experiment duration differed significantly in two-sided Welch t-tests (Table R7, Table R8); z-score differences reached p < .05 before correction (accuracy p = .034; duration p = .049) but not after Holm correction across the three domains (p = .101 and .147). Timeout rates were similar across both conditions (two-stage 18.3% vs. length-matched 19.6%).

We highlight that all practice questions in the length-matched target-only curricula address the full target skill, while the two-stage curricula also budget their practice questions across prerequisite skills. As a consequence the two-stage condition has less practice on the full target skill. Despite this, the two-stage condition shows a trending advantage in assessment accuracy over the length-matched target-only condition (80.7% vs. 74.3%; t = 1.67, p = .096).

AuthorsOpenReview ↗

Response to Reviewer dwPe [2/2]

Does prerequisite ordering persist when the listed objectives are shuffled? Table 10 shows 92.8% agreement with prompt order.

We find that prerequisite ordering is largely preserved when the order of listed objectives is shuffled (95.8% vs. 50% chance; p < .001). Using the same freeform prompt specified in Appendix E.5, we shuffled the order of the listed objectives and generated the curricula across the three domains and LLMs used in the naturalistic experiment. We then analyzed the rate at which the curricula taught prerequisites before their dependent concepts, using the prerequisite relations validated by the authors, relative to an equal-chance baseline with no preference for teaching the prerequisite or dependent concept earlier (pairs introduced on the same page were left out).

Running our original prompts with the original ordering of listed objectives (regenerated in the same run), all resulting curricula taught prerequisites before their dependent concepts (100% vs 50% chance, Wilcoxon p < 0.001, n=45 curricula). When we shuffled the listed objectives within the prompt, a large majority of the resulting curricula taught prerequisites before their dependent concepts (95.8% vs 50% chance, Wilcoxon p < 0.001, n=90). Notably, even when the dependent concept was listed before the prerequisite concept in the prompt, models still significantly chose to teach the prerequisite first (90.5% vs. 50% chance: Wilcoxon p < .001, n = 85). Moreover, the order in which objectives are introduced in the curricula under the shuffled prompting condition does not significantly follow the ordering of the objectives listed in the prompt (53.2% vs. 50% chance: Wilcoxon p = .25, n = 90). These data suggest that while the ordering of objectives in the prompt may affect the final ordering of the curricula, models still make choices about the overall structure of their curricula and order it in a manner largely robust to the presented ordering of objectives. The high agreement with prompt order in Table 10 (Table 12 in the revision) therefore reflects that our listed objectives already followed prerequisite order, rather than models copying the listed order.

To clarify, this measure is slightly different than what is presented in Table 10 in that it uses the author-validated prerequisite relations and determines order from the first page on which each objective appears (as labeled by the generating model), rather than using an LLM to segment the curriculum into instructional units; this is because we thought it was a more faithful measure that does not rely on a second LLM, as noted by Reviewer jTeP.

Thanks for the reanalysis. The length-matched comparison still leaves open whether decomposition itself improves learning. I’m keeping my rating at 2.

Reviewer_1cy3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This article investigates the ability of large language models to select teaching examples and organize static courses for real people. A total of 1428 participants were recruited for tasks involving causality, simple visual concepts, compositional visual concepts, and three real-world subject areas. In causal tasks, the models' selections are similar to Bayesian teaching predictions for some causal structures; in simple visual tasks, almost all of the example pairs generated by the models belong to the theoretically optimal set, and the models prefer contrastive pairs that make real people answer more accurately and quickly. In compositional tasks, the models introduce intermediate concepts, and related courses improve the recognition rate of target rules. Real-world subject experiments show that a complete course is superior to no course or only target modules; Gemini's course ablation experiment did not detect any additional benefits of explicit planning compared to generating a complete course at once. Additional analysis of freely generated courses also found that models usually introduce prerequisite knowledge first, and then introduce the content that depends on it.

Critical strengths
  1. Combining theoretical references with human learning performance distinguishes information sufficiency from comprehensibility. LLMs favored contrastive pairs, while the Bayesian model favored positive pairs. Although both strategies can be theoretically optimal, LLM-generated examples yielded higher accuracy (85.4% vs. 80.6%) and faster responses (median 13 vs. 23 seconds), showing the importance of instructional presentation.
  2. The experiments connect controlled tasks with real-world subjects, covering five models in the first three task categories and three in the main real-world experiment. A separate Gemini ablation compares target-only, single-pass, and two-stage curricula. Analyses of decomposition strategies and prerequisite ordering in freely generated curricula provide further evidence about course organization.
Critical weaknesses
  1. The statistical processing of repeated responses in real-world subject experiments is not clear enough. Although the summary statistics at the participant level were reported in the article, the reported item-level regression model, with each person answering 12 questions, did not reflect within-participant correlations, nor were other methods for handling them explained. It is necessary to confirm whether the confidence intervals and significance of the main course comparisons are robust.
  2. Excluding participants based on their grades may affect the estimation of teaching effectiveness. The authors excluded participants with chance-level scores according to the preregistration rules, but did not fully explain the specific criteria, exclusion details for each group, and the results with these participants retained. Low grades may also reflect learning difficulties or ineffective teaching, so it is necessary to verify whether the conclusion is sensitive to this screening.
  3. The contributions of course organization and increased teaching volume have not been fully distinguished. The authors have analyzed the interface practice effect and the correlation between subconcept performance and target performance, but the complete course still provides additional explanations, examples, and exercises, so the existing analyses are not sufficient to separate the specific contributions of knowledge organization. The authors acknowledge this limitation, which primarily limits the explanation of the sources of learning gains.
Questions for authors
  1. Does the formula in Appendix E.3 fully reflect the statistical model actually used? After dealing with the correlation of multiple responses from the same participant, are the effect estimates, confidence intervals, and significance of the main course comparisons preserved?
  2. What are the specific criteria for the “chance level,” how is this rule applied to each experiment, and how many participants were excluded for each condition? Do analyses that retain these participants support the same conclusions?
  3. In addition to the existing analysis in Appendix D.8, are there any controls with comparable amounts of instruction or other evidence that could further distinguish the contribution of knowledge organization from that of additional explanations, examples, and exercises?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 1cy3 [1/2]

We thank the reviewer for their positive assessment of our evaluations with respect to both Bayesian models and human learning, and the progression of tasks to real-world domains. We answer the reviewer’s questions below.

Does the formula in Appendix E.3 fully reflect the statistical model actually used? After dealing with the correlation of multiple responses from the same participant, are the effect estimates, confidence intervals, and significance of the main course comparisons preserved?

Accounting for participant-level dependence preserves the direction and significance of all main comparisons (Table R1). Appendix E.3 correctly reports the preregistered models used in the original submission. As the reviewer notes, the models do not account for participant random effects, so we ran analyses for new models which include an additional random intercept for participant.

What are the specific criteria for the “chance level,” how is this rule applied to each experiment, and how many participants were excluded for each condition? Do analyses that retain these participants support the same conclusions?

We did not exclude any participant for low accuracy, and have corrected the exclusion criteria reported in Appendix A.2 (Table R2). We removed participants and responses that suggested insufficient attention, deliberation, or completion, and participants who received LLM-generated curricula that violated the generation protocol (n=3, compositional experiment). Appendix A.2 incorrectly states that chance-level participants were excluded and misstates the preregistered RT cutoffs; we have replaced this statement with the correct per-experiment criteria in Table R2.

Next, we note that there are two deviations for exclusions from the original preregistered analyses. i) For the primitive experiment, all participants were included and preregistered exclusions for at or just above chance-level performance (0 or 1 of 10 trials correct; chance is 1/36) were omitted. Following the preregistered exclusion removes 13 of 283 participants and strengthens the result for LLM advantage over the Bayesian model (β=+0.60[0.26,0.95],p<.001\beta = +0.60 [0.26, 0.95], p < .001). ii) For the compositional experiment, our preregistration included an accuracy-based exclusion without specifying a threshold; because chance-level performance is effectively 0, we chose not to apply an accuracy-based exclusion.

Retaining every participant and response with assessment data leaves all conclusions unchanged (Table R3). Table R3 cannot retain responses from participants who timed out before the assessment, and timeouts were more frequent with a curriculum (14–21% across curriculum conditions vs. 4% without). We performed an additional analysis where we replaced the accuracy of those participants who timed out with the average accuracy of participants who received no curriculum in the same domain (that is, we assume i) participants who timed out under the curriculum condition have similar priors to those under the no-curriculum condition and ii) receiving a curriculum does not help learning in this case). All significant contrasts remain significant, and C3 remains nonsignificant (Exp. 1: 1.35 [0.77, 1.92], p < .001; C1: 0.84 [0.26, 1.42], p = .002; C2: 0.49 [0.03, 0.96], p = .019; C3: 0.18 [−0.30, 0.67], p = .46; C1 and C2 one-sided). Together, our results show that the exclusions do not drive our main conclusions of teaching efficacy.

AuthorsOpenReview ↗

Response to Reviewer 1cy3 [2/2]

In addition to the existing analysis in Appendix D.8, are there any controls with comparable amounts of instruction or other evidence that could further distinguish the contribution of knowledge organization from that of additional explanations, examples, and exercises?

We conducted an additional experiment matching instructional volume across conditions (Table R6), finding a trending advantage for the two-stage curriculum that does not reach statistical significance. To do this, we re-prompted Gemini 3.1 Pro Preview to extend the target-only curriculum from Exp. 2 to match the two-stage curriculum in the number of pages, practice questions, and words; the resulting curricula closely match the two-stage condition in the amount of instructional content (Table R6). Overall, the two-stage condition had higher accuracy, but neither accuracy nor total experiment duration differed significantly in two-sided Welch t-tests (Table R7, Table R8); z-score differences reached p < .05 before correction (accuracy p = .034; duration p = .049) but not after Holm correction across the three domains (p = .101 and .147). Timeout rates were similar across both conditions (two-stage 18.3% vs. length-matched 19.6%).

Importantly, all practice questions in the length-matched target-only curricula address the full target skill, while the two-stage curricula also budget their practice questions across prerequisite skills. As a consequence the two-stage condition has less practice on the full target skill. Despite this, the two-stage condition shows a trending advantage in assessment accuracy over the length-matched target-only condition (80.7% vs. 74.3%; t = 1.67, p = .096).

Thank you for the additional analyses, which address some of my concerns. However, the benefit of curriculum organization beyond providing more content and practice remains unclear. Given the importance of this claim to the paper, I maintain my rating and leave the final decision to the AC.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response: summary of revisions and Tables R1-R9 [1/2]

We thank the reviewers for their time and careful reviews. We have uploaded a revised PDF that adds all new analyses to the appendix, with the main text unchanged. Table R9 lists where each item appears. Appendix tables after Table 1 are renumbered by +2 (e.g., original Table 10 is now Table 12), and the naturalistic prompts moved from Appendix E.4 to E.5.

Table R1. Mixed-effects logistic regression, without (preregistered) and with newly added participant intercept (log-odds scale; 95% CI). Following preregistered analyses, p-values for C1 and C2 are one-sided, and Exp. 1 and C3 are two-sided.

ContrastModel without participant intercept (preregistered)Model with participant intercept
Exp. 1: Two-stage vs. no curriculum1.31 [1.10, 1.52], p < .0011.97 [1.30, 2.63], p < .001
C1: Target-only vs. no curriculum0.79 [0.58, 1.01], p < .0011.18 [0.51, 1.85], p < .001
C2: Single-pass vs. target-only0.53 [0.32, 0.74], p < .0010.83 [0.24, 1.42], p = .003
C3: Two-stage vs. single-pass−0.06 [−0.27, 0.16], p = .61−0.08 [−0.69, 0.52], p = .79

Table R2. Exclusion criteria and excluded participants for each experiment.

ExperimentParticipant exclusionResponse exclusionExcluded / recruited
CausalCompleted < 80% of trials; near-uniform ratings (range ≤5\leq 5) on every trialRT < 3 s12/217 (11 incomplete, 1 uniform); 8 responses removed*
PrimitiveNoneNone0/283*
CompositionalDid not complete exactly four target trialsNoneDecomposed 4/209; Target-only 2/174
Naturalistic Exp. 1Answered < 80% of assessment questionsRT < 3 s or > participant mean + 3 SDNo curriculum 2/50; Two-stage 36/172; 108 responses removed
Naturalistic Exp. 2Same as Exp. 1Same as Exp. 1Target-only 20/109; Single-pass 22/111; Two-stage 14/103; 121 responses removed (163 with the shared no-curriculum group)

*Note that teaching source varies within participants for these experiments.

Table R3. Contrasts for each experiment, with and without reported exclusions (estimates with 95% CIs). All models include participant random intercepts. Causal: linear mixed model on the normalized rating of the true graph; we omit one participant who rated every structure 0 on every trial, as their measure is undefined. Primitive: logistic model for accuracy and linear model for log RT; the reported analysis already includes all participants. Compositional and naturalistic: logistic models (log-odds); naturalistic models include the added participant intercept. C1 and C2 p-values are one-sided, as preregistered. Participants who timed out before the assessment could not be included, as they have no responses.

ExperimentContrastReported exclusionsAll retained
CausalLLM vs. Bayesian0.034 [0.001, 0.067], p = .047 (n = 205)0.034 [0.001, 0.067], p = .043 (n = 216)
PrimitiveLLM vs. Bayesian, accuracy0.54 [0.19, 0.88], p = .002 (n = 283)Same; no exclusions applied
PrimitiveLLM vs. Bayesian, log RT−0.52 [−0.58, −0.47], p < .001Same; no exclusions applied
CompositionalDecomposed vs. target-only1.41 [0.60, 2.22], p < .001 (n = 377)1.45 [0.64, 2.26], p < .001 (n = 383)
Naturalistic Exp. 1Two-stage vs. no curriculum1.97 [1.30, 2.63], p < .001 (n = 184)1.90 [1.24, 2.56], p < .001 (n = 184)
Naturalistic Exp. 2C1: Target-only vs. no curriculum1.18 [0.51, 1.85], p < .001 (n = 314)1.12 [0.45, 1.80], p < .001 (n = 315)
C2: Single-pass vs. target-only0.83 [0.24, 1.42], p = .0030.72 [0.13, 1.31], p = .009
C3: Two-stage vs. single-pass−0.08 [−0.69, 0.52], p = .790.05 [−0.55, 0.66], p = .86

Table R4. Mixed-effects logistic regression, with participant, item, and curriculum-level random intercepts added to the preregistered model (log-odds scale; 95% CI). C1 and C2 p-values are one-sided, as preregistered.

ContrastWith exclusionsAll retained
Exp. 1: Two-stage vs. no curriculum2.03 [1.34, 2.72], p < .0011.97 [1.29, 2.66], p < .001
C1: Target-only vs. no curriculum1.23 [0.53, 1.94], p < .0011.17 [0.47, 1.87], p < .001
C2: Single-pass vs. target-only0.86 [0.24, 1.48], p = .0030.74 [0.13, 1.35], p = .009
C3: Two-stage vs. single-pass−0.08 [−0.71, 0.55], p = .800.06 [−0.57, 0.68], p = .86
AuthorsOpenReview ↗

General response: summary of revisions and Tables R1-R9 [2/2]

Table R5. Learning objectives specified in the generation prompt and their corresponding assessment questions. Every curriculum shown to participants contains at least one page covering each objective.

DomainObjectiveDescriptionQuestions
Logic circuitsgatesInterpret binary inputs and AND, OR, and NOT gate behaviorq01–q03
tracingTrace outputs through up to three connected gatesq04–q06
missing_inputInfer a missing input when the information determines it uniquelyq07–q09
comparisonCompare the outputs of small circuits for stated input settingsq10–q12
Mendelian geneticsnotationInterpret allele, genotype, and phenotype notation and complete dominanceq01–q03
outcomesPredict possible offspring genotypes and phenotypes from stated parental genotypesq03–q06
probabilityCalculate probabilities for a single offspring of a monohybrid crossq07–q09
comparisonCompare offspring outcomes and probabilities across monohybrid crossesq10–q12
z-scorescalculationCalculate a z-score from a score, mean, and positive standard deviationq01–q03
interpretationInterpret the sign and magnitude of a z-scoreq04–q06
comparisonCompare relative performance across classes using z-scoresq07–q09
referencePredict how changing the reference mean or standard deviation changes a z-scoreq10–q12

Table R6. Instructional content per curriculum (mean; range in brackets). Worked problems are counted automatically.

Two-stageLength-matched target-only
Pages55
Practice questions55
Teaching words1,639 [1,402–1,797]1,656 [1,540–1,761]
Words including practice questions2,1542,261
Worked problems4.86.7

Table R7. Entire experiment duration in minutes (n). Two-sided Welch t-tests are presented.

ScopeTwo-stageLength-matched target-onlyt (df)Two-sided pHolm p
Aggregate16.92 (83)17.74 (86)−0.67 (165.2).501—
Genetics17.73 (27)16.47 (30)0.65 (44.4).5191.000
Z-scores15.82 (28)20.52 (27)−2.01 (51.7).049.147
Logic circuits17.24 (28)16.47 (29)0.39 (54.1).7021.000

Table R8. Assessment accuracy (n). All participants who reached the assessment are included. Two-sided Welch t-tests are presented.

ScopeTwo-stageLength-matched target-onlyt (df)Two-sided pHolm p
Aggregate80.7% (89)74.3% (90)1.67 (176.7).096—
Genetics83.3% (30)72.5% (31)1.83 (55.3).073.146
Z-scores83.9% (29)69.8% (29)2.18 (52.8).034.101
Logic circuits75.0% (30)80.6% (30)−0.76 (54.6).452.452

Table R9. Location of rebuttal content in the revised PDF.

Rebuttal contentRevised PDF
Exclusion criteria (R2), pre-registration deviations, all participants retained (R3), timeoutsAppendix A.2, Tables 2–3
Participant, item, and curriculum clustering (R1, R4)Appendix E.4.1, Tables 16–17
Assessment coverage, correctness, and overlap (R5)Appendix E.4.2, Table 18
Prerequisite validationAppendix E.4.3
Shuffled-objective experimentAppendix E.4.4
Length-matched experiment (R6–R8)Appendix E.4.5, Tables 19–21

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.