acceptodds
Under review as a conference paper at ICLR 2027

Anticipating the Consequences of Curriculum Decisions with Large Language Models

Abstract

Automatic curriculum learning can improve the effectiveness of reinforcement learning by selecting the training experiences presented to the agent over time. Predicting the consequences of such decisions can, however, be difficult. We analyze automatic curriculum learning as a sequential decision-making problem, highlighting a gap between the quantities that determine the value of curriculum decisions and the information captured by local learning signals commonly used to guide them. We then investigate whether Large Language Models (LLMs) can exploit richer information about the learning problem to better anticipate the consequences of curriculum decisions. We introduce a method that combines online learning-progress estimates with LLM-informed estimates of (i) the potential downstream benefits of learning on each task and (ii) whether direct training on a task is currently likely to produce progress. We evaluate the approach on a custom benchmark of 256 textual goals in Craftax under different curriculum objectives. We observe the strongest gains when optimizing for individual target tasks. When optimizing across the full task set, the benefits vary across learners with different mechanisms for cross-task transfer, ranging from modest improvements in learning speed to larger gains that persist through the end of training.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, automatic curriculum learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hqQWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an automatic curriculum learning method for multi-task reinforcement learning, built on the idea that large language models (LLMs) can draw on knowledge about the tasks and environment to anticipate how a curriculum decision will affect subsequent training. The authors formulate task selection as a sequential decision-making problem and augment absolute learning progress (ALP)-based sampling with two static graphs constructed by an LLM before training. The first represents the potential downstream benefits of training on one task for other tasks (Graph GLCG_{LC}), while the second identifies competencies that provide evidence that a task is ready for productive direct training (Graph GFEG_{FE}). During training, these graphs are combined with online performance estimates to adapt task selection. The method is evaluated on 256 custom goals in Craftax under single-target and uniformly weighted multi-task objectives. Compared with the ALP baseline, the reported advantages are most pronounced on the selected single-target tasks. In the multi-task setting, the largest improvements occur for a learner that can invoke learned behaviors as subroutines, while benefits for a learner restricted to primitive actions mainly appear early in training.

Critical strengths

S1. Clear motivation. The paper makes a useful distinction between a task showing recent learning progress and that task being the best choice for further training. Its analysis separates progress on the selected task, immediate effects on other tasks, and effects on future learning. This gives a clear rationale for considering downstream benefits and readiness for training alongside learning progress.

S2. Meaningful empirical gains. The improvements on several selected targets are substantial. For example, final success rates on Dungeon and Furnace reach approximately 44% and 79%, compared with 2% and 0% for ALP. In the multi-task setting, the learner with subroutine calls also shows a sustained improvement, with average success increasing from 25.5% to 31.3%. These results provide encouraging evidence that task-relation priors can help in settings with sparse rewards and long chains of prerequisite skills.

S3. Useful diagnostic experiments. The evaluation includes five paired training seeds, component ablations, graph permutation, hyperparameter sensitivity analysis, and an alternative weighting of target and auxiliary tasks. The curriculum trajectories also help explain how training allocation changes over time. I appreciate that the authors report where the benefits are limited, including the similar final performance of the primitive-action learners and the inconsistent contribution of feasibility guidance.

Critical weaknesses

W1. The contribution of the task relationships is not fully isolated. In the single-target setting, zero objective weights prevent ALP from prioritizing auxiliary tasks by their learning progress. Appendix E.2 partly addresses this through reweighting. A further comparison with ALP using absolute learning progress without objective weighting, while retaining the same fixed target share, would help distinguish the benefit of adaptive auxiliary-task selection from that of the specific LLM-generated relationships.

W2. The role of the LLM itself is underexplored. All graphs come from a single model, Gemini-3.7-Flash, so we don't know how much the results depend on the choice or strength of the LLM. A related issue is that the prompts already describe the game mechanics in detail, including crafting recipes and tool requirements (App. B.2). This makes it hard to tell whether the LLM contributes knowledge or reasoning of its own, or mostly turns the given rules into a graph. A simple graph built directly from these rules and plugged into the same selection procedure would be a useful reference point. Results from one or two other LLMs (e.g., a commercial model such as GPT-6 or an open model such as Qwen3), along with a measure of graph consistency, would also be useful.

W3. No comparison with other LLM-guided curricula. The baselines are either LLM-free (uniform sampling, intermediate difficulty, ALP, direct target training) or ablations of the proposed method. OMNI, CurricuLLM and LEACL are cited in the related work but not evaluated, and ELLM [1], which uses LLM-suggested goals to guide exploration in Crafter, is not cited at all. Even one representative LLM-guided baseline, adapted to the fixed goal set and given a comparable interaction budget, would make the case for this particular way of using LLMs much stronger. I would also suggest discussing MAGELLAN [2], which learns to predict competence and learning progress across goals. Its setting differs from offline graph construction, but explaining how the two differ would make the contribution clearer.

W4. A visual overview would help. The method has several moving parts, and I found it hard to see how they fit together until I had worked through all the equations. Algorithm 1 in Appendix A.1 helps, but a figure in the main text would make the paper much easier to follow. It should show how the two LLM-generated graphs combine with the online performance and learning-progress estimates to produce the sampling distribution.

[1] Yuqing Du et al., Guiding Pretraining in Reinforcement Learning with Large Language Models. ICML 2023

[2] Loris Gaven et al., MAGELLAN: Metacognitive Predictions of Learning Progress Guide Autotelic LLM Agents in Large Goal Spaces. ICML 2025

Questions for authors

Please see the Critical Weaknesses section.

I look forward to the authors’ response and will carefully consider their clarifications when updating my review.

3 comments

Thank you for your time and constructive feedback. We respond to your concerns and questions below.

The contribution of the task relationships is not fully isolated. In the single-target setting, zero objective weights prevent ALP from prioritizing auxiliary tasks by their learning progress. Appendix E.2 partly addresses this through reweighting. A further comparison with ALP using absolute learning progress without objective weighting, while retaining the same fixed target share, would help distinguish the benefit of adaptive auxiliary-task selection from that of the specific LLM-generated relationships.

We have added the requested comparison to Appendix E.2. The results show that the higher performance of ALP+D+F on Dungeon, Wood, and Furnace persists, as does its earlier learning on Skeleton.

The role of the LLM itself is underexplored. All graphs come from a single model, Gemini-3.7-Flash, so we don't know how much the results depend on the choice or strength of the LLM. A related issue is that the prompts already describe the game mechanics in detail, including crafting recipes and tool requirements (App. B.2). This makes it hard to tell whether the LLM contributes knowledge or reasoning of its own, or mostly turns the given rules into a graph. A simple graph built directly from these rules and plugged into the same selection procedure would be a useful reference point. Results from one or two other LLMs (e.g., a commercial model such as GPT-6 or an open model such as Qwen3), along with a measure of graph consistency, would also be useful.

We have added new experiments that remove the detailed environment description fed to the language model (Appendix E.3, Figures 15–16). These show that, for the tested settings, the detailed environment context does not consistently improve outcomes and is not required to obtain substantial performance benefits from the approach.

We would also like to clarify that the environment information supplied in the original experiments describes Craftax’s rules, actions, and mechanics, much like documentation for the environment. While potentially relevant, this is not the information the LLM is asked to estimate (learning connectivity and feasibility evidence relationships). Moreover, prerequisite relations are not the only kinds of relations that can indicate potential for transfer or evidence of feasibility. For feasibility, competence on one task can indicate capabilities relevant to productive training on another, without the first task being a prerequisite for the second. To illustrate this broader distinction for learning connectivity:

  1. Training to defeat one type of creature can develop combat skills that improve performance or facilitate subsequent learning against another. For example, learning to defeat zombies can benefit learning to defeat skeletons, without either being a prerequisite for the other.
  2. Learning to be adjacent to a chest can develop navigation and exploration skills and help the agent learn what the instruction “be adjacent to” requires in the environment. These capabilities can facilitate learning to be adjacent to a fountain, even though neither goal requires completing the other.
  3. Learning to reach a floor where diamonds are more common, such as floor 2, can facilitate subsequent learning to find them. Reaching that floor is not necessary for seeing diamond, since diamonds can also occur on the starting floor (floor 0), but it can make rewarded experiences easier to encounter.

All three relationships appear in the learning-connectivity graphs generated for both learners. The graphs are included in the supplementary material.

Regarding model choice, we did not search over LLMs or select Gemini 3.7 Flash based on downstream performance. We prioritized other experimental comparisons over repeating the evaluation with additional LLMs, so sensitivity to model choice remains untested. Our expectation is that stronger LLMs will generally be better at making the estimates our method requires and, more broadly, at informing curriculum decisions.

No comparison with other LLM-guided curricula. The baselines are either LLM-free (uniform sampling, intermediate difficulty, ALP, direct target training) or ablations of the proposed method. OMNI, CurricuLLM and LEACL are cited in the related work but not evaluated, and ELLM [1], which uses LLM-suggested goals to guide exploration in Crafter, is not cited at all. Even one representative LLM-guided baseline, adapted to the fixed goal set and given a comparable interaction budget, would make the case for this particular way of using LLMs much stronger. I would also suggest discussing MAGELLAN [2], which learns to predict competence and learning progress across goals. Its setting differs from offline graph construction, but explaining how the two differ would make the contribution clearer.

We have now added ProCuRL-Target to the single-target experiments. Although it was not originally proposed as an LLM-guided curriculum, our implementation uses distances between language-model embeddings of goal descriptions as the method's proxy for transfer between tasks. It achieves lower final mean success than our method (ALP+D+F) on all eight targets. The updated results are presented in Section 5.3.1.

Regarding the suggested LLM-guided approaches, we agree that comparisons could be valuable, but we did not find them to be particularly direct comparisons for our experimental setting. CurricuLLM generates intermediate tasks with their reward functions and goal distributions, while LEACL generates task variations by modifying initial-state and goal specifications. In our experiments, the available goals, rewards, and initial-state distributions are fixed, and the curriculum only determines how to allocate training among them. Adapting these methods would therefore require substantial modifications to their mechanisms or to the experimental setting. OMNI could be more directly applied, but it primarily addresses a different problem. Its use of LLMs to identify tasks that humans would consider interesting is motivated by open-ended learning and could complement our considerations of feasibility and inter-task transfer. Moreover, it is not clear how to faithfully adapt its interestingness-based guidance to our explicitly defined curriculum objectives, particularly in the single-target setting.

We did not discuss ELLM because it addresses a different setting and research question, which we considered adjacent but less central to our contribution. It focuses on reward-free pretraining, dynamically generating exploration goals and intrinsic rewards. Our experiments consider a predefined set of tasks and study how training should be allocated among them. ELLM therefore does not directly translate into a baseline for our setting.

Regarding MAGELLAN, we did not discuss it separately because it focuses on a different limitation of LP-based curriculum learning. It addresses how to efficiently estimate competence and learning progress in large language-based goal spaces. In the notation of our paper, its contribution primarily concerns estimating LP^i(m)\widehat{\mathrm{LP}}_i(m), whereas our work examines why prioritizing tasks based on local LP can be insufficient even when recent progress is accurately estimated. The two approaches are therefore complementary: MAGELLAN's LP^i(m)\widehat{\mathrm{LP}}_i(m) estimator could be used within our method. In our experiments, where we consider a fixed and manageable goal set, we instead use the simpler estimator described in Appendix A.2. We hope this clarifies the distinction. We have also added MAGELLAN to the LP-based methods cited in our related work.

A visual overview would help. The method has several moving parts, and I found it hard to see how they fit together until I had worked through all the equations. Algorithm 1 in Appendix A.1 helps, but a figure in the main text would make the paper much easier to follow. It should show how the two LLM-generated graphs combine with the online performance and learning-progress estimates to produce the sampling distribution.

Thank you for pointing this out. We have added a visual overview of the method early in Section 4 (Figure 1).

I have read the authors’ response and decided to raise my score to 3 (Weak Accept). However, I am really not familiar with this domain (so I keep my confidence score unchanged). Good luck!

Reviewer_FJmgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper analyzes curriculum selection as a sequential decision problem and identifies information missing from local learning-progress estimates. Its method constructs two offline LLM-informed graphs over tasks: one for potential downstream learning benefits and one for evidence that direct training is currently feasible. These priors combine with online performance and absolute learning progress to sample goals. Experiments use 256 textual Craftax goals, single-target objectives, and two learners with different mechanisms for reusing learned behavior.

Critical strengths
  1. The paper addresses a meaningful limitation of local progress criteria: progress on an auxiliary task can matter because of what it enables elsewhere. The offline graph construction gives a practical way to incorporate richer task knowledge without querying an LLM throughout training. The task-selection procedure and its relationship to online observations are clearly specified.

  2. The experiments reveal an informative interaction with the learner's transfer mechanism. Gains on several designated targets are large, while all-task improvements are modest and mainly transient for the primitive-action learner but larger and persistent with subroutine reuse. Five paired training seeds, D/F ablations, sensitivity analysis, and the auxiliary-weight control make this more informative than a single favorable curriculum example.

Critical weaknesses
  1. The single-target evidence is conditioned on an outcome-based target-selection procedure. Appendix C.5 reports screening candidate targets with one shared seed, retaining targets where at least one method achieves nonzero final success, and including that screening seed in the reported five-seed comparisons. This does not establish selection for a relative advantage, but it makes the strongest results conditional on observed learnability and partially reuses selection data for reporting. Given the substantial seed variation, sensitivity to removing the screening seed is important for assessing the strength of these results.

  2. The experiments do not sufficiently isolate the value of LLM-inferred transfer relationships from simpler structural task priors. PERM preserves the feasibility graph and can retain useful prerequisite connections; the 50% auxiliary-weight control in Appendix E.2 is useful but does not prioritize structurally relevant auxiliary goals. Since several goals have explicit threshold or prerequisite relationships, and the path scores are acknowledged heuristics rather than calibrated transfer predictions, the evidence primarily establishes the utility of the supplied prior, rather than its specificity to anticipating general learning consequences. This distinction affects the breadth of the contribution, especially because the primitive learner shows little final all-task advantage.

Overall assessment: 3 — Weak acceptance, close to the boundary. The paper provides a useful curriculum design and an informative empirical result about when task-relationship priors help. The strong target gains and learner-dependent all-task findings justify cautious support, while selection-conditioned results and limited isolation of the LLM graph's contribution constrain confidence in broader claims.

Questions for authors
  1. Can you report the four-seed target comparisons excluding the seed used for screening, together with the screening results for the two excluded targets? Do the principal target conclusions remain unchanged?

  2. How much of the graph's useful prioritization is explained by simple goal structure, such as resource thresholds or mechanical prerequisites? A comparison using the same sampler with a simple structural or description-based graph, or a focused analysis of non-obvious predicted relationships, would clarify the distinct value of the LLM judgments.

  3. Can you provide the actual generated graphs and generation records, including query/token cost? Are results sensitive to independently reconstructing the graph pair, as opposed to repeating learner seeds with a single fixed graph pair?

2 comments

Thank you for your time and constructive feedback. We respond to your concerns and questions below.

The single-target evidence is conditioned on an outcome-based target-selection procedure. Appendix C.5 reports screening candidate targets with one shared seed, retaining targets where at least one method achieves nonzero final success, and including that screening seed in the reported five-seed comparisons. This does not establish selection for a relative advantage, but it makes the strongest results conditional on observed learnability and partially reuses selection data for reporting. Given the substantial seed variation, sensitivity to removing the screening seed is important for assessing the strength of these results.

Can you report the four-seed target comparisons excluding the seed used for screening, together with the screening results for the two excluded targets? Do the principal target conclusions remain unchanged?

As you note, the screening criterion did not depend on any method having an advantage. We therefore chose to document the procedure and retain the additional seed. As the procedure in Appendix C.5 describes, all 5 screened methods had zero success by the end of training on the excluded targets. No learning occurred on these targets with any of the screening seeds.

The principal conclusions still hold when considering only the 4 seeds not used for screening. The table below compares the final average performance for the main method and baseline using 4 and 5 seeds. We have also added an additional discussion and a figure showing all the learning curves with 4 seeds to section C.5. In Appendix E.1 one can additionally find the learning curves for each ALP+D+F seed on the targets for the learner with primitive actions.

LearnerTargetALP+D+F (5 seeds)ALP+D+F (4 seeds)ALP (5 seeds)ALP (4 seeds)
Primitive actionsDungeon43.8%49.8%2.0%0%
Primitive actionsSkeleton67.2%67.4%66.1%66.0%
Primitive actionsWood16.6%20.7%0%0%
Primitive actionsFurnace79.2%74.2%0%0%
SubroutinesCoal12.5%15.0%0%0%
SubroutinesIron Pickaxe78.1%97.7%0%0%
SubroutinesOrc Soldier3.0%0%0%0%
SubroutinesFountain7.8%9.8%0%0%

The experiments do not sufficiently isolate the value of LLM-inferred transfer relationships from simpler structural task priors. PERM preserves the feasibility graph and can retain useful prerequisite connections; the 50% auxiliary-weight control in Appendix E.2 is useful but does not prioritize structurally relevant auxiliary goals. Since several goals have explicit threshold or prerequisite relationships, and the path scores are acknowledged heuristics rather than calibrated transfer predictions, the evidence primarily establishes the utility of the supplied prior, rather than its specificity to anticipating general learning consequences. This distinction affects the breadth of the contribution, especially because the primitive learner shows little final all-task advantage.

How much of the graph's useful prioritization is explained by simple goal structure, such as resource thresholds or mechanical prerequisites? A comparison using the same sampler with a simple structural or description-based graph, or a focused analysis of non-obvious predicted relationships, would clarify the distinct value of the LLM judgments.

We agree that simple structural relationships, such as resource thresholds and mechanical prerequisites, can provide useful information for curriculum decisions. However, these are only some of the relationships that can indicate potential learning transfer or feasibility. As motivated by our analysis in Section 3, the value of curriculum decisions depends on how training affects performance on other tasks and the effectiveness of subsequent learning, which cannot generally be characterized by mechanical prerequisite relationships alone. The quantities our method seeks to estimate are more general.

For example, with regard to learning connectivity:

  1. Training to defeat one type of creature can develop combat skills that improve performance or facilitate subsequent learning against another. For instance, learning to defeat zombies can benefit learning to defeat skeletons, without either being a prerequisite for the other.
  2. Learning to be adjacent to a chest can develop navigation and exploration skills and help the agent learn what the instruction “be adjacent to” requires in the environment. These capabilities can facilitate learning to be adjacent to a fountain, even though neither goal requires completing the other.
  3. Learning to reach a floor where diamonds are more common, such as floor 2, can facilitate subsequent learning to find them. Reaching that floor is not necessary for seeing diamond, since diamonds can also occur on the starting floor (floor 0), but it can make rewarded experiences easier to encounter.

These example relationships appear in the learning-connectivity graphs generated for both learners. The graphs are included in the supplementary material.

We do not claim that simple structural priors cannot provide useful curriculum guidance, nor that our experiments isolate the contribution of each kind of relationship. Our aim is to investigate whether LLMs can provide useful estimates of the more general learning consequences motivated by our analysis, without restricting their predictions to manually defined classes of relationships.

Can you provide the actual generated graphs and generation records, including query/token cost? Are results sensitive to independently reconstructing the graph pair, as opposed to repeating learner seeds with a single fixed graph pair?

The supplementary material now includes the graph pairs used in our main experiments, together with their generation records and per-query token usage. We have also added a summary of generation costs to Appendix B. Under our configuration, constructing each pair cost approximately US$30–35 and used approximately 5 million input tokens and 7 million output tokens, including reasoning tokens.

We have not systematically evaluated sensitivity to reconstructing the graph pair under identical generation settings. However, in the additional experiment in Appendix E.3, we generated a new pair without the detailed environment description and still obtained substantial performance benefits.

Reviewer_ijtKOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies automatic curriculum learning (ACL) for RL, where training tasks are selected sequentially to accelerate learning. The paper highlights a mismatch between local learning signals commonly used for curriculum selection and the quantities that determine the long-term value of curriculum decisions, particularly their effects on downstream learning and future curriculum choices. To address this, the authors investigate whether LLMs can use richer descriptions of the tasks, learner, and training setup to anticipate the consequences of curriculum decisions. The proposed method combines online learning-progress estimates with LLM-derived estimates of (i) the downstream learning potential of each task and (ii) whether training on a task is currently likely to yield progress. These estimates are constructed prior to training as directed learning-connectivity and feasibility-evidence graphs. Experiments in Craftax evaluate whether incorporating these signals improves learning under different curriculum objectives.

Critical strengths
  • Addresses an important problem in RL: improving sample efficiency through automatic curriculum design.
  • The sequential decision-making perspective provides an interesting analysis of the limitations of predominantly local learning-progress signals.
  • Using LLMs to incorporate semantic/task-level information into curriculum decisions is intuitive and potentially useful.
  • The paper is generally clear and well written, although some important context is deferred to the appendix.
Critical weaknesses
  • Tzannetos et al. (2024) considers expected learning progress on the target task distribution rather than only the selected training task, seemingly capturing part of the non-local effect formalized in Eqs. (2)-(3). Why is Tzannetos et al. (2024) not included as a baseline?
  • The approach relies on an LLM's ability to infer learning connectivity and feasibility from descriptions of the tasks, learner, and training setup. It is unclear how much domain-specific information or manual task description is required, and consequently how readily the approach transfers beyond Craftax.
  • The proposed method introduces LLM inference into curriculum construction. The computational/API cost of obtaining these judgments should be compared with the resulting improvement in RL sample efficiency.
  • The final task-selection distribution in Section 4.3 contains several hyperparameters. Without sensitivity analysis, it is difficult to determine whether the gains are robust or depend on careful tuning, which itself may introduce additional computational cost.
Questions for authors
  • Why is Tzannetos et al. (2024) not included as a baseline? Their method estimates learning progress on the target task distribution and therefore already captures some of the downstream effect motivating the first terms of Eq. (3), although not the effect on future curriculum decisions.
  • How much domain-specific information must be provided to the LLM? For example, how sensitive are the inferred connectivity and feasibility graphs to the granularity and quality of task/environment descriptions?
  • How sensitive are the results to the hyperparameters introduced in Section 4.3? How were these values selected?
6 comments

Thank you for your time and constructive feedback. We respond to your concerns and questions below.

The approach relies on an LLM's ability to infer learning connectivity and feasibility from descriptions of the tasks, learner, and training setup. It is unclear how much domain-specific information or manual task description is required, and consequently how readily the approach transfers beyond Craftax.

How much domain-specific information must be provided to the LLM? For example, how sensitive are the inferred connectivity and feasibility graphs to the granularity and quality of task/environment descriptions?

We have added new experiments that remove the detailed environment description (Appendix E.3, Figures 15–16). These show that (for the tested settings) the detailed environment context does not consistently improve outcomes and is not required to obtain substantial performance benefits from the approach.

Beyond these new results, there are several things we would like to clarify:

An important motivation we argue for using LLMs in curriculum decisions is precisely the flexibility in applying them across domains and in incorporating different degrees and kinds of information. For any particular domain, relevant information could come from knowledge acquired during pretraining, information explicitly supplied in the prompt, or external information automatically accessed through tools. Our method does not impose fixed requirements on the type or amount of supplementary domain information provided to the LLM.

In the original experiments, we decided to supply information on Craftax mechanics because Craftax is a relatively specialized environment, and the LLM was not given access to external documentation or environment code. Importantly, the information supplied is a specification of how the Craftax environment works; its rules, actions, and mechanics, much like documentation for the environment. While potentially relevant, this is not the information the LLM needs to estimate (learning connectivity or feasibility evidence).

Regarding task descriptions specifically, the LLM receives only the concise goal specifications listed in Appendix C.1, such as "Place a furnace" or "Have at least 64 wood." These are the original descriptions defining the benchmark tasks, also used to condition the trained policies across all methods, including the LLM-free curriculum baselines. We do not introduce additional goal-specific explanations or information.

The prompts were not tuned based on the method's downstream performance.

Finally, neither the mathematical formulation nor the proposed curriculum algorithm contains Craftax-specific components.

Tzannetos et al. (2024) considers expected learning progress on the target task distribution rather than only the selected training task, seemingly capturing part of the non-local effect formalized in Eqs. (2)-(3). Why is Tzannetos et al. (2024) not included as a baseline?

Why is Tzannetos et al. (2024) not included as a baseline? Their method estimates learning progress on the target task distribution and therefore already captures some of the downstream effect motivating the first terms of Eq. (3), although not the effect on future curriculum decisions.

Thank you for suggesting this comparison. We have now added ProCuRL-Target to the single-target experiments. It achieves lower final mean success than our method (ALP+D+F) on all eight targets. The updated results and a discussion of the observed curriculum behavior are presented in Section 5.3.1.

The proposed method introduces LLM inference into curriculum construction. The computational/API cost of obtaining these judgments should be compared with the resulting improvement in RL sample efficiency.

We have added a paragraph on computational cost to the graph-construction appendix (Appendix B). Under our configuration, constructing both graphs cost approximately US$30-35 per learner. This is a one-time offline cost, with the graphs reused across curriculum objectives. The additional curriculum computation during training is negligible, taking approximately 0.01 ms per iteration compared with approximately 52 s for a training iteration.

We would also like to point out that our method improves not only sample efficiency but also final performance in most settings within the evaluated training budgets.

The final task-selection distribution in Section 4.3 contains several hyperparameters. Without sensitivity analysis, it is difficult to determine whether the gains are robust or depend on careful tuning, which itself may introduce additional computational cost.

How sensitive are the results to the hyperparameters introduced in Section 4.3? How were these values selected?

As explained in Appendix C.6, we did not perform any hyperparameter tuning on our method (ALP+D+F). All tuning was conducted using the ALP baseline. In particular, the mixture weights in Section 4.3 were not tuned for our method.

We retained the 80% allocation to the LP-based component selected for ALP to keep the comparison consistent across methods. When adding feasibility guidance, we simply divided the remaining 20% equally between uniform and feasibility-guided sampling.

Regarding sensitivity, we did not study sensitivity specifically to the mixture coefficients. Appendix C.7 examines sensitivity to the maximum path length LL and downstream coefficient α\alpha. This analysis was conducted after the main experiments and was not used to select their hyperparameter values.

Thanks for clarifications, additional experiments, and addressing my concerns. Re. Figure 15: your method w/o detailed description is performing considerably better than with detailed description; does it flag any prompt sensitivity of the method (depending on domain)?

Comments on the paper

1 comment

We thank the reviewers for their time and constructive feedback. Alongside our individual responses, we summarize the main changes to the revised paper:

  • Doubled the number of targets in the single-target experiments, adding four new targets using the learner with access to subroutine calls (Section 5.3.1).
  • Added ProCuRL-Target (Tzannetos et al., 2024) as a baseline for the single-target experiments (Section 5.3.1 and Appendix D.1).
  • Added ALP+D and ALP+F ablations under uniform task weights for both learners (Section 5.3.2).
  • Added an ALP baseline with uniform task weights to the single-target comparisons (Appendix E.2).
  • Added experiments removing the detailed environment description supplied to the LLM (Appendix E.3).
  • Added single-target results using only the four non-screening seeds for each target (Appendix C.5).
  • Added a diagram illustrating the method (Section 4).
  • Added computational-cost estimates for offline graph construction and online curriculum updates (Appendix B.1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.