- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an automatic curriculum learning method for multi-task reinforcement learning, built on the idea that large language models (LLMs) can draw on knowledge about the tasks and environment to anticipate how a curriculum decision will affect subsequent training. The authors formulate task selection as a sequential decision-making problem and augment absolute learning progress (ALP)-based sampling with two static graphs constructed by an LLM before training. The first represents the potential downstream benefits of training on one task for other tasks (Graph ), while the second identifies competencies that provide evidence that a task is ready for productive direct training (Graph ). During training, these graphs are combined with online performance estimates to adapt task selection. The method is evaluated on 256 custom goals in Craftax under single-target and uniformly weighted multi-task objectives. Compared with the ALP baseline, the reported advantages are most pronounced on the selected single-target tasks. In the multi-task setting, the largest improvements occur for a learner that can invoke learned behaviors as subroutines, while benefits for a learner restricted to primitive actions mainly appear early in training.
Critical strengths
S1. Clear motivation. The paper makes a useful distinction between a task showing recent learning progress and that task being the best choice for further training. Its analysis separates progress on the selected task, immediate effects on other tasks, and effects on future learning. This gives a clear rationale for considering downstream benefits and readiness for training alongside learning progress.
S2. Meaningful empirical gains. The improvements on several selected targets are substantial. For example, final success rates on Dungeon and Furnace reach approximately 44% and 79%, compared with 2% and 0% for ALP. In the multi-task setting, the learner with subroutine calls also shows a sustained improvement, with average success increasing from 25.5% to 31.3%. These results provide encouraging evidence that task-relation priors can help in settings with sparse rewards and long chains of prerequisite skills.
S3. Useful diagnostic experiments. The evaluation includes five paired training seeds, component ablations, graph permutation, hyperparameter sensitivity analysis, and an alternative weighting of target and auxiliary tasks. The curriculum trajectories also help explain how training allocation changes over time. I appreciate that the authors report where the benefits are limited, including the similar final performance of the primitive-action learners and the inconsistent contribution of feasibility guidance.
Critical weaknesses
W1. The contribution of the task relationships is not fully isolated. In the single-target setting, zero objective weights prevent ALP from prioritizing auxiliary tasks by their learning progress. Appendix E.2 partly addresses this through reweighting. A further comparison with ALP using absolute learning progress without objective weighting, while retaining the same fixed target share, would help distinguish the benefit of adaptive auxiliary-task selection from that of the specific LLM-generated relationships.
W2. The role of the LLM itself is underexplored. All graphs come from a single model, Gemini-3.7-Flash, so we don't know how much the results depend on the choice or strength of the LLM. A related issue is that the prompts already describe the game mechanics in detail, including crafting recipes and tool requirements (App. B.2). This makes it hard to tell whether the LLM contributes knowledge or reasoning of its own, or mostly turns the given rules into a graph. A simple graph built directly from these rules and plugged into the same selection procedure would be a useful reference point. Results from one or two other LLMs (e.g., a commercial model such as GPT-6 or an open model such as Qwen3), along with a measure of graph consistency, would also be useful.
W3. No comparison with other LLM-guided curricula. The baselines are either LLM-free (uniform sampling, intermediate difficulty, ALP, direct target training) or ablations of the proposed method. OMNI, CurricuLLM and LEACL are cited in the related work but not evaluated, and ELLM [1], which uses LLM-suggested goals to guide exploration in Crafter, is not cited at all. Even one representative LLM-guided baseline, adapted to the fixed goal set and given a comparable interaction budget, would make the case for this particular way of using LLMs much stronger. I would also suggest discussing MAGELLAN [2], which learns to predict competence and learning progress across goals. Its setting differs from offline graph construction, but explaining how the two differ would make the contribution clearer.
W4. A visual overview would help. The method has several moving parts, and I found it hard to see how they fit together until I had worked through all the equations. Algorithm 1 in Appendix A.1 helps, but a figure in the main text would make the paper much easier to follow. It should show how the two LLM-generated graphs combine with the online performance and learning-progress estimates to produce the sampling distribution.
[1] Yuqing Du et al., Guiding Pretraining in Reinforcement Learning with Large Language Models. ICML 2023
[2] Loris Gaven et al., MAGELLAN: Metacognitive Predictions of Learning Progress Guide Autotelic LLM Agents in Large Goal Spaces. ICML 2025
Questions for authors
Please see the Critical Weaknesses section.
I look forward to the authors’ response and will carefully consider their clarifications when updating my review.
3 comments
No comparison with other LLM-guided curricula. The baselines are either LLM-free (uniform sampling, intermediate difficulty, ALP, direct target training) or ablations of the proposed method. OMNI, CurricuLLM and LEACL are cited in the related work but not evaluated, and ELLM [1], which uses LLM-suggested goals to guide exploration in Crafter, is not cited at all. Even one representative LLM-guided baseline, adapted to the fixed goal set and given a comparable interaction budget, would make the case for this particular way of using LLMs much stronger. I would also suggest discussing MAGELLAN [2], which learns to predict competence and learning progress across goals. Its setting differs from offline graph construction, but explaining how the two differ would make the contribution clearer.
We have now added ProCuRL-Target to the single-target experiments. Although it was not originally proposed as an LLM-guided curriculum, our implementation uses distances between language-model embeddings of goal descriptions as the method's proxy for transfer between tasks. It achieves lower final mean success than our method (ALP+D+F) on all eight targets. The updated results are presented in Section 5.3.1.
Regarding the suggested LLM-guided approaches, we agree that comparisons could be valuable, but we did not find them to be particularly direct comparisons for our experimental setting. CurricuLLM generates intermediate tasks with their reward functions and goal distributions, while LEACL generates task variations by modifying initial-state and goal specifications. In our experiments, the available goals, rewards, and initial-state distributions are fixed, and the curriculum only determines how to allocate training among them. Adapting these methods would therefore require substantial modifications to their mechanisms or to the experimental setting. OMNI could be more directly applied, but it primarily addresses a different problem. Its use of LLMs to identify tasks that humans would consider interesting is motivated by open-ended learning and could complement our considerations of feasibility and inter-task transfer. Moreover, it is not clear how to faithfully adapt its interestingness-based guidance to our explicitly defined curriculum objectives, particularly in the single-target setting.
We did not discuss ELLM because it addresses a different setting and research question, which we considered adjacent but less central to our contribution. It focuses on reward-free pretraining, dynamically generating exploration goals and intrinsic rewards. Our experiments consider a predefined set of tasks and study how training should be allocated among them. ELLM therefore does not directly translate into a baseline for our setting.
Regarding MAGELLAN, we did not discuss it separately because it focuses on a different limitation of LP-based curriculum learning. It addresses how to efficiently estimate competence and learning progress in large language-based goal spaces. In the notation of our paper, its contribution primarily concerns estimating , whereas our work examines why prioritizing tasks based on local LP can be insufficient even when recent progress is accurately estimated. The two approaches are therefore complementary: MAGELLAN's estimator could be used within our method. In our experiments, where we consider a fixed and manageable goal set, we instead use the simpler estimator described in Appendix A.2. We hope this clarifies the distinction. We have also added MAGELLAN to the LP-based methods cited in our related work.
A visual overview would help. The method has several moving parts, and I found it hard to see how they fit together until I had worked through all the equations. Algorithm 1 in Appendix A.1 helps, but a figure in the main text would make the paper much easier to follow. It should show how the two LLM-generated graphs combine with the online performance and learning-progress estimates to produce the sampling distribution.
Thank you for pointing this out. We have added a visual overview of the method early in Section 4 (Figure 1).
I have read the authors’ response and decided to raise my score to 3 (Weak Accept). However, I am really not familiar with this domain (so I keep my confidence score unchanged). Good luck!
Thank you for your time and constructive feedback. We respond to your concerns and questions below.
We have added the requested comparison to Appendix E.2. The results show that the higher performance of ALP+D+F on Dungeon, Wood, and Furnace persists, as does its earlier learning on Skeleton.
We have added new experiments that remove the detailed environment description fed to the language model (Appendix E.3, Figures 15–16). These show that, for the tested settings, the detailed environment context does not consistently improve outcomes and is not required to obtain substantial performance benefits from the approach.
We would also like to clarify that the environment information supplied in the original experiments describes Craftax’s rules, actions, and mechanics, much like documentation for the environment. While potentially relevant, this is not the information the LLM is asked to estimate (learning connectivity and feasibility evidence relationships). Moreover, prerequisite relations are not the only kinds of relations that can indicate potential for transfer or evidence of feasibility. For feasibility, competence on one task can indicate capabilities relevant to productive training on another, without the first task being a prerequisite for the second. To illustrate this broader distinction for learning connectivity:
All three relationships appear in the learning-connectivity graphs generated for both learners. The graphs are included in the supplementary material.
Regarding model choice, we did not search over LLMs or select Gemini 3.7 Flash based on downstream performance. We prioritized other experimental comparisons over repeating the evaluation with additional LLMs, so sensitivity to model choice remains untested. Our expectation is that stronger LLMs will generally be better at making the estimates our method requires and, more broadly, at informing curriculum decisions.