- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CRAFT, an algorithm for automatic curriculum generation for a distribution of K tasks with binary success outcomes for GRPO. CRAFT up-weights tasks with 1) a higher deficit from the target success rate 2) faster learning progress 3) higher variation in outcomes. The learning progress is measured as change in success, which practically is the difference between a slow and fast moving exponential moving average (EMA). Additionally, CRAFT enforces a minimum probability given to each task, to ensure coverage, and up-weights tasks with low visitation counts and tasks that have not been visited recently. An EMA is used to compute most of these statistics (1-3), in addition to being applied to the linear combination of them (Q). CRAFT is evaluated on LIBERO, and variations of the LIBERO-PRO Long task, generally with improvements over the pi_RL baseline when evaluated after 30 updates.
Critical strengths
The strengths of this paper are the simplicity of the method, at a high-level, and exploring a new set of diversity criteria for task prioritization, beyond variation in return.
Critical weaknesses
The primary weaknesses are the lack of relevant baselines and the weakness of the empirical results .
First, the authors claim “CRAFT is the first framework to use group feedback to allocate rollouts”; however, VCRL (Jiang et al., 2025, https://arxiv.org/pdf/2509.19803), CGPO (Li et al., 2026 https://arxiv.org/pdf/2605.17807v1) and PAPRIKA (Tajwar et al., 2025 https://arxiv.org/pdf/2502.17543) all use variance of feedback or the coefficient of variation with GRPO or DPO to prioritize rollouts. There are papers that do not divide by the group standard deviation, which can be seen as implicitly multiplying by the (root of) variance. And, there are papers with “zero-variance filtering”. Additionally, a standard algorithm, Prioritized Level Replay (Jiang et al., 2021), while potentially not tested yet with GRPO, is highly relevant here. This method is cited but not tested, and I do not understand why they could not test it. Ablations of all components would also help make this significantly more convincing, since this method can be seen as an extension of some of those methods.
Second, the results are not very strong. On LIBERO, CRAFT improves the backbone in two domains and decreases performance in two domains, with a large improvement only on “Long”. It is then perhaps not surprising, that the majority of the subsequent results focus on LIBERO-Long (with additional perturbations), which is the environment that was most amenable to CRAFT. Still, CRAFT only helps after 30 updates. At the same time, without knowing whether there was a held-out validation set for tuning, even this result could be an artifact of hyper-parameter’s overfit to this setting. Finally, There are experiments reported in the appendix showing variation over three seeds to be ~0.5 percent on LIBERO-Long without perturbations. However, the variance of the backbone and variance in the other settings are not clear.
Questions for authors
There are experiments reported in the appendix showing variation over three seeds to be ~0.5 percent on LIBERO-Long. However, the variance on other tasks is not clear. Could you quantify the variation on other experiments in the main body. For example, variance of the baseline pi_RL and LIBERO-Long tasks under four perturbation types?
Were the hyper-parameters tuned on a heldout validation set? Could they be overfit?
Could you perform ablations of the individual terms used to compute the priority scores?
More Minor: v = 4¯ r(1−¯r). Why is this quantity scaled by 4?
Is there a reason CRAFT is evaluated on top of pi_RL but none of the other baselines? For example, VLA-Adapter seems stronger on many of these settings.
If L and C use EMA, why does Q need to as well? Is this because D does not use EMA? And why does D not use EMA?
Why are b_visit and b_age not added at the same time as D, L, and C?
Is there a reason the definitions for softmax, standardization, and EMA (twice) appear the main body, but the definition of b_age and b_visit are only defined in the appendix?
Is b_visit derived from a UCB-style exploration bonus? This connection should probably be mentioned.
Why does b_visit count completed groups while b_age count reserved groups?
Why does the algorithm need locks? It was unclear to me that there was asynchronicity in the sampling until I saw that it was implied by the locks. Can you explain the need for this?
Does time spent on a task during training correlate with improvements in performance over baseline? The subsection on allocation dynamics gestures to this, but the takeaway is unclear, and I mostly see text that recapitulates the results.
6 comments
AW2: Magnitude and Timing of the Gains
Thanks for asking where the gains occur. Table 1 compares and + CRAFT after 30 updates, using three matched training seeds for Spatial, Object, and Goal, and five seeds per method for Long. Each seed is evaluated over 500 episodes.
| LIBERO suite | (%) | + CRAFT (%) | Difference (%) |
|---|---|---|---|
| Spatial | 96.7 | 96.9 | +0.2 |
| Object | 97.7 | 99.2 | +1.5 |
| Goal | 98.5 | 97.6 | −0.9 |
| Long | 89.9 | 92.5 | +2.6 |
Baseline success rates for Spatial, Object, and Goal are 96.7%–98.5%, near the ceiling. Spatial and Object improve by 0.2 and 1.5%, while Goal drops 0.9%. At these levels, a few tasks can shift the overall score, and online RL stochasticity may also cause minor fluctuations.
We repeated evaluations on LIBERO-Long, where longer tasks and an 89.9% baseline allow more improvement. Long shows the biggest Table 1 gain, reaching 92.5% (+2.6%) averaged over five training seeds per method.
In a matched LIBERO-Long run, both methods complete 232 rollout groups and 890,880 environment steps. Mixed success/failure groups comprise 60.8% with + CRAFT vs 45.7% with , a 15.1% increase, yielding more within-group reward contrast at the same interaction budget (Section 5.2). Figure 2 reports the final slow success EMA summaries: the bottom 40%, bottom 20%, and weakest-task values are 6.8%, 16.1%, and 20.0% higher with CRAFT.
On LIBERO-PRO, models are trained with Object perturbation feedback and evaluated on four perturbations of the same ten Long tasks. Table 4 reports pooled success rates over three seeds.
| PRO perturbation | + CRAFT | Difference (%) | |
|---|---|---|---|
| Object | 161/300 (53.67%) | 178/300 (59.33%) | +5.66 |
| Semantic | 98/150 (65.33%) | 107/150 (71.33%) | +6.00 |
| Position | 30/150 (20.00%) | 25/150 (16.67%) | −3.33 |
| Task | 58/150 (38.67%) | 65/150 (43.33%) | +4.66 |
| Overall | 347/750 (46.27%) | 375/750 (50.00%) | +3.73 |
Table 4 reports the episode counts and perturbation rates; Table 5 places these rates alongside published methods.
Earlier checkpoints are mixed: the original PRO run shows differences of −2.0 and −0.8% after 10 and 20 updates, while the original Spatial run gains 1.4% after 10 updates. Since both start from the same checkpoint, rollout-allocation differences may have limited time to affect the policy early, and some gaps may be run-to-run variability. All runs use a fixed configuration (AQ2).
See Tables 1, 4, 5, and 6 and Appendix B (Table 9) of the revised PDF for the suite, perturbation, published-context, and checkpoint results; Section 5.2 reports the mixed-group diagnostic, and Figure 2 shows the final slow success EMA summaries.
AQ1: Seed Variation Across Benchmarks
Thank you for the suggestion. We report 30-update success rates as mean ± sample SD across training seeds for both methods.
LIBERO-Long (five training seeds per method, 500 episodes per run).
| Method | Episode counts (successful/total) | Mean success rate ± sample SD (%) |
|---|---|---|
| 2,247/2,500 | 89.88 ± 0.76 | |
| + CRAFT | 2,313/2,500 | 92.52 ± 0.99 |
LIBERO-PRO (three training seeds per method, 250 evaluation episodes per run).
| Method | Episode counts (successful/total) | Mean success rate ± sample SD (%) |
|---|---|---|
| 347/750 | 46.27 ± 1.67 | |
| + CRAFT | 375/750 | 50.00 ± 1.83 |
The PRO paired difference is 3.73 ± 3.35% (mean ± sample SD). AW2 reports pooled results for the four perturbations.
See Section 5.2 and Tables 1 and 4 of the revised PDF for the benchmark results and evaluation protocols.
AQ2: Held-Out Validation and Hyperparameter Tuning
Sorry for the confusion. We have used a fixed configuration and have not tuned it on LIBERO-Long or LIBERO-PRO. We have used settings from the original references/codebases without a held-out hyperparameter search for these benchmarks. Section 5.1 states the configuration is fixed across runs, and Appendix A, Table 7 lists the settings. The equal-weight and coverage ablations probe specific design choices; broader hyperparameter sensitivity remains open.
See Section 5.1 and Appendix A (Table 7) of the revised PDF for the fixed configuration and its parameter values.
AQ3: Priority Signal Ablations
Thanks for suggesting the component study. + CRAFT achieves the highest success rate (94.20%) among the variants. We evaluated each configuration on 500 LIBERO-Long episodes after 30 updates. , , and denote the target-success deficit, group outcome variation, and the discrepancy between fast and slow success estimates, respectively.
Online RL experiments are compute- and time-intensive. Given our compute budget and discussion window, we evaluated each ablation in one training run. Repeating the full study across multiple seeds would greatly increase training cost.
| Allocation variant | Episode counts (successful/total) | Success rate (%) |
|---|---|---|
| 451/500 | 90.20 | |
| + CRAFT ( only) | 452/500 | 90.40 |
| + CRAFT ( only) | 452/500 | 90.40 |
| + CRAFT ( only) | 451/500 | 90.20 |
| + CRAFT (equal weights) | 457/500 | 91.40 |
| + CRAFT (without coverage) | 460/500 | 92.00 |
| + CRAFT | 471/500 | 94.20 |
The full controller scores 2.20% higher than the variant without coverage.
See Table 3 of the revised PDF for the full component study.
AQ4: Scaling of Binary Outcome Variation
Sorry for the confusion. The factor of four rescales the statistic to 0–1. For binary outcomes, peaks at when successes and failures are evenly split. Therefore, equals 1 at an even split and 0 when all outcomes match, making endpoints intuitive and preserving the group-variation ranking (Section 4.2).
See Section 4.2 and Appendix A.1 of the revised PDF for the variation statistic and its implementation.
Thank you for the careful review and suggestions, which helped us strengthen CRAFT’s evaluation and presentation. We compared CRAFT against four adaptive samplers under a shared training budget, added component ablations to study the allocation design, and repeated the 30-update experiments for both methods. CRAFT achieves mean gains of 2.6% on LIBERO-Long and 3.73% on LIBERO-PRO. Below, we relate our design to prior work and clarify the controller implementation.
We have included the above discussions in the paper and updated Figure 1.
AW1: Related Work, Novelty, and Adaptive Baselines
Indeed, VCRL, CGPO, and PAPRIKA study how feedback steers sampling. That said, our method differs substantially from these approaches, particularly in the feedback signal and how it updates the allocation.
VCRL adjusts LLM sample difficulty using group reward variance. CRAFT adds a target-success deficit from a slower success estimate: even when groups are all-success or all-failure (zero variance), persistently failing tasks can be prioritized via a larger deficit, giving the allocator a signal without mixed outcomes.
CGPO prioritizes text-to-image prompts by group reward variance and calibrates category weights via proportional fairness and category-level rewards. CRAFT reprioritizes robot tasks using completed visits, reservation-based staleness, and a probability floor to ensure coverage, keeping a minimum sampling probability even when outcome variation is low.
PAPRIKA uses the coefficient of variation of trajectory lengths as a learnability signal for its Twenty Questions curriculum. CRAFT updates task priorities online using binary success outcomes from completed groups. The absolute gap between fast and slow success estimates reflects recent improvements and declines, based directly on success outcomes.
These choices allow CRAFT to combine unmet success targets, within-group reward contrast, and task-performance changes when allocating future rollout groups. We keep the learner objective, advantage computation, and group-retention rule fixed. The default πRL samples tasks uniformly, while πRL + CRAFT uses the same learner with CRAFT task allocation.
We adapted DRATS, PLR, SFL, and CGPO to the RLinf task-selection interface. Along with πRL and πRL + CRAFT, we compared six configurations across three matched training seeds (18 runs, 9,000 evaluation episodes). Each run uses 30 updates, 1,920 rollout trajectories, and 500 evaluation episodes.
πRL + CRAFT has the highest mean success rate (91.80%), 0.40% above the best adaptive comparator, πRL + CGPO. SD is the sample standard deviation across three training seeds. Component results are in AQ3.
We add the relevant citations and clarify these distinctions in Section 2 of the revised PDF.
See Sections 2 and 4 and Table 2 of the revised PDF for the related work, allocation design, and matched sampler comparison.