acceptodds
Under review as a conference paper at ICLR 2027

CRAFT: Constrained Rollout Allocation from Feedback for Vision Language Action Reinforcement Learning

Abstract

Online reinforcement fine-tuning of Vision Language Action (VLA) policies is often constrained by limited training budgets, with only a finite number of policy update rounds and rollout opportunities available for multi-task reinforcement learning. In this setting, every rollout must be assigned to a specific task, making the allocation of limited rollout opportunities across tasks a direct determinant of training efficiency and final performance. Existing methods typically employ uniform task sampling, which may repeatedly sample already saturated tasks without producing further performance gains, leave difficult tasks generating homogeneous failures that provide little effective within-group learning signal, and fail to prevent previously acquired capabilities from deteriorating during subsequent training. To address these limitations, we propose CRAFT, a rollout allocation framework for budget-constrained Vision Language Action reinforcement learning. CRAFT uses feedback from completed rollout groups to adaptively distribute future rollout opportunities across tasks. Specifically, CRAFT jointly models target success deficits, within-group outcome variation, and differences between fast and slow success estimates to identify tasks with greater current training value. It further employs priority smoothing, coverage bonuses, and a minimum sampling probability to convert group feedback into a stable next-task distribution while preventing individual tasks from being persistently overlooked. Under identical training settings and matched evaluation protocols, CRAFT achieves consistent aggregate gains over uniform task allocation within fixed update budgets: after 30 updates, it improves LIBERO-Long success by 4.0%; after 10 updates, it improves LIBERO-Spatial success by 1.4%; after 30 updates, it improves the four-suite LIBERO average by 0.9%; and after 30 updates, it improves weighted LIBERO-PRO success by 7.6%. These results demonstrate that CRAFT allocates rollout opportunities more effectively for update-constrained multi-task reinforcement learning.

Primary area:
reinforcement learning
Keywords:
vision language action models, embodied AI, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_H7w7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CRAFT, an algorithm for automatic curriculum generation for a distribution of K tasks with binary success outcomes for GRPO. CRAFT up-weights tasks with 1) a higher deficit from the target success rate 2) faster learning progress 3) higher variation in outcomes. The learning progress is measured as change in success, which practically is the difference between a slow and fast moving exponential moving average (EMA). Additionally, CRAFT enforces a minimum probability given to each task, to ensure coverage, and up-weights tasks with low visitation counts and tasks that have not been visited recently. An EMA is used to compute most of these statistics (1-3), in addition to being applied to the linear combination of them (Q). CRAFT is evaluated on LIBERO, and variations of the LIBERO-PRO Long task, generally with improvements over the pi_RL baseline when evaluated after 30 updates.

Critical strengths

The strengths of this paper are the simplicity of the method, at a high-level, and exploring a new set of diversity criteria for task prioritization, beyond variation in return.

Critical weaknesses

The primary weaknesses are the lack of relevant baselines and the weakness of the empirical results .

First, the authors claim “CRAFT is the first framework to use group feedback to allocate rollouts”; however, VCRL (Jiang et al., 2025, https://arxiv.org/pdf/2509.19803), CGPO (Li et al., 2026 https://arxiv.org/pdf/2605.17807v1) and PAPRIKA (Tajwar et al., 2025 https://arxiv.org/pdf/2502.17543) all use variance of feedback or the coefficient of variation with GRPO or DPO to prioritize rollouts. There are papers that do not divide by the group standard deviation, which can be seen as implicitly multiplying by the (root of) variance. And, there are papers with “zero-variance filtering”. Additionally, a standard algorithm, Prioritized Level Replay (Jiang et al., 2021), while potentially not tested yet with GRPO, is highly relevant here. This method is cited but not tested, and I do not understand why they could not test it. Ablations of all components would also help make this significantly more convincing, since this method can be seen as an extension of some of those methods.

Second, the results are not very strong. On LIBERO, CRAFT improves the backbone in two domains and decreases performance in two domains, with a large improvement only on “Long”. It is then perhaps not surprising, that the majority of the subsequent results focus on LIBERO-Long (with additional perturbations), which is the environment that was most amenable to CRAFT. Still, CRAFT only helps after 30 updates. At the same time, without knowing whether there was a held-out validation set for tuning, even this result could be an artifact of hyper-parameter’s overfit to this setting. Finally, There are experiments reported in the appendix showing variation over three seeds to be ~0.5 percent on LIBERO-Long without perturbations. However, the variance of the backbone and variance in the other settings are not clear.

Questions for authors

There are experiments reported in the appendix showing variation over three seeds to be ~0.5 percent on LIBERO-Long. However, the variance on other tasks is not clear. Could you quantify the variation on other experiments in the main body. For example, variance of the baseline pi_RL and LIBERO-Long tasks under four perturbation types?

Were the hyper-parameters tuned on a heldout validation set? Could they be overfit?

Could you perform ablations of the individual terms used to compute the priority scores?

More Minor: v = 4¯ r(1−¯r). Why is this quantity scaled by 4?

Is there a reason CRAFT is evaluated on top of pi_RL but none of the other baselines? For example, VLA-Adapter seems stronger on many of these settings.

If L and C use EMA, why does Q need to as well? Is this because D does not use EMA? And why does D not use EMA?

Why are b_visit and b_age not added at the same time as D, L, and C?

Is there a reason the definitions for softmax, standardization, and EMA (twice) appear the main body, but the definition of b_age and b_visit are only defined in the appendix?

Is b_visit derived from a UCB-style exploration bonus? This connection should probably be mentioned.

Why does b_visit count completed groups while b_age count reserved groups?

Why does the algorithm need locks? It was unclear to me that there was asynchronicity in the sampling until I saw that it was implied by the locks. Can you explain the need for this?

Does time spent on a task during training correlate with improvements in performance over baseline? The subsection on allocation dynamics gestures to this, but the takeaway is unclear, and I mostly see text that recapitulates the results.

6 comments

Thank you for the careful review and suggestions, which helped us strengthen CRAFT’s evaluation and presentation. We compared CRAFT against four adaptive samplers under a shared training budget, added component ablations to study the allocation design, and repeated the 30-update experiments for both methods. CRAFT achieves mean gains of 2.6% on LIBERO-Long and 3.73% on LIBERO-PRO. Below, we relate our design to prior work and clarify the controller implementation.

We have included the above discussions in the paper and updated Figure 1.

AW1: Related Work, Novelty, and Adaptive Baselines

Indeed, VCRL, CGPO, and PAPRIKA study how feedback steers sampling. That said, our method differs substantially from these approaches, particularly in the feedback signal and how it updates the allocation.

  • VCRL adjusts LLM sample difficulty using group reward variance. CRAFT adds a target-success deficit from a slower success estimate: even when groups are all-success or all-failure (zero variance), persistently failing tasks can be prioritized via a larger deficit, giving the allocator a signal without mixed outcomes.

  • CGPO prioritizes text-to-image prompts by group reward variance and calibrates category weights via proportional fairness and category-level rewards. CRAFT reprioritizes robot tasks using completed visits, reservation-based staleness, and a probability floor to ensure coverage, keeping a minimum sampling probability even when outcome variation is low.

  • PAPRIKA uses the coefficient of variation of trajectory lengths as a learnability signal for its Twenty Questions curriculum. CRAFT updates task priorities online using binary success outcomes from completed groups. The absolute gap between fast and slow success estimates reflects recent improvements and declines, based directly on success outcomes.

These choices allow CRAFT to combine unmet success targets, within-group reward contrast, and task-performance changes when allocating future rollout groups. We keep the learner objective, advantage computation, and group-retention rule fixed. The default πRL\pi_{\mathrm{RL}} samples tasks uniformly, while πRL\pi_{\mathrm{RL}} + CRAFT uses the same learner with CRAFT task allocation.

We adapted DRATS, PLR, SFL, and CGPO to the RLinf task-selection interface. Along with πRL\pi_{\mathrm{RL}} and πRL\pi_{\mathrm{RL}} + CRAFT, we compared six configurations across three matched training seeds (18 runs, 9,000 evaluation episodes). Each run uses 30 updates, 1,920 rollout trajectories, and 500 evaluation episodes.

Task samplerEpisode counts (successful/total)Mean success rate ± SD (%)
πRL\pi_{\mathrm{RL}}1,349/1,50089.93 ± 1.47
πRL\pi_{\mathrm{RL}} + DRATS1,364/1,50090.93 ± 0.50
πRL\pi_{\mathrm{RL}} + PLR1,366/1,50091.07 ± 2.10
πRL\pi_{\mathrm{RL}} + SFL1,367/1,50091.13 ± 1.50
πRL\pi_{\mathrm{RL}} + CGPO1,371/1,50091.40 ± 0.20
πRL\pi_{\mathrm{RL}} + CRAFT1,377/1,50091.80 ± 0.35

πRL\pi_{\mathrm{RL}} + CRAFT has the highest mean success rate (91.80%), 0.40% above the best adaptive comparator, πRL\pi_{\mathrm{RL}} + CGPO. SD is the sample standard deviation across three training seeds. Component results are in AQ3.

We add the relevant citations and clarify these distinctions in Section 2 of the revised PDF.

See Sections 2 and 4 and Table 2 of the revised PDF for the related work, allocation design, and matched sampler comparison.

AW2: Magnitude and Timing of the Gains

Thanks for asking where the gains occur. Table 1 compares πRL\pi_{\mathrm{RL}} and πRL\pi_{\mathrm{RL}} + CRAFT after 30 updates, using three matched training seeds for Spatial, Object, and Goal, and five seeds per method for Long. Each seed is evaluated over 500 episodes.

LIBERO suiteπRL\pi_{\mathrm{RL}} (%)πRL\pi_{\mathrm{RL}} + CRAFT (%)Difference (%)
Spatial96.796.9+0.2
Object97.799.2+1.5
Goal98.597.6−0.9
Long89.992.5+2.6

Baseline success rates for Spatial, Object, and Goal are 96.7%–98.5%, near the ceiling. Spatial and Object improve by 0.2 and 1.5%, while Goal drops 0.9%. At these levels, a few tasks can shift the overall score, and online RL stochasticity may also cause minor fluctuations.

We repeated evaluations on LIBERO-Long, where longer tasks and an 89.9% baseline allow more improvement. Long shows the biggest Table 1 gain, reaching 92.5% (+2.6%) averaged over five training seeds per method.

In a matched LIBERO-Long run, both methods complete 232 rollout groups and 890,880 environment steps. Mixed success/failure groups comprise 60.8% with πRL\pi_{\mathrm{RL}} + CRAFT vs 45.7% with πRL\pi_{\mathrm{RL}}, a 15.1% increase, yielding more within-group reward contrast at the same interaction budget (Section 5.2). Figure 2 reports the final slow success EMA summaries: the bottom 40%, bottom 20%, and weakest-task values are 6.8%, 16.1%, and 20.0% higher with CRAFT.

On LIBERO-PRO, models are trained with Object perturbation feedback and evaluated on four perturbations of the same ten Long tasks. Table 4 reports pooled success rates over three seeds.

PRO perturbationπRL\pi_{\mathrm{RL}}πRL\pi_{\mathrm{RL}} + CRAFTDifference (%)
Object161/300 (53.67%)178/300 (59.33%)+5.66
Semantic98/150 (65.33%)107/150 (71.33%)+6.00
Position30/150 (20.00%)25/150 (16.67%)−3.33
Task58/150 (38.67%)65/150 (43.33%)+4.66
Overall347/750 (46.27%)375/750 (50.00%)+3.73

Table 4 reports the episode counts and perturbation rates; Table 5 places these rates alongside published methods.

Earlier checkpoints are mixed: the original PRO run shows differences of −2.0 and −0.8% after 10 and 20 updates, while the original Spatial run gains 1.4% after 10 updates. Since both start from the same checkpoint, rollout-allocation differences may have limited time to affect the policy early, and some gaps may be run-to-run variability. All runs use a fixed configuration (AQ2).

See Tables 1, 4, 5, and 6 and Appendix B (Table 9) of the revised PDF for the suite, perturbation, published-context, and checkpoint results; Section 5.2 reports the mixed-group diagnostic, and Figure 2 shows the final slow success EMA summaries.

AQ1: Seed Variation Across Benchmarks

Thank you for the suggestion. We report 30-update success rates as mean ± sample SD across training seeds for both methods.

LIBERO-Long (five training seeds per method, 500 episodes per run).

MethodEpisode counts (successful/total)Mean success rate ± sample SD (%)
πRL\pi_{\mathrm{RL}}2,247/2,50089.88 ± 0.76
πRL\pi_{\mathrm{RL}} + CRAFT2,313/2,50092.52 ± 0.99

LIBERO-PRO (three training seeds per method, 250 evaluation episodes per run).

MethodEpisode counts (successful/total)Mean success rate ± sample SD (%)
πRL\pi_{\mathrm{RL}}347/75046.27 ± 1.67
πRL\pi_{\mathrm{RL}} + CRAFT375/75050.00 ± 1.83

The PRO paired difference is 3.73 ± 3.35% (mean ± sample SD). AW2 reports pooled results for the four perturbations.

See Section 5.2 and Tables 1 and 4 of the revised PDF for the benchmark results and evaluation protocols.

AQ2: Held-Out Validation and Hyperparameter Tuning

Sorry for the confusion. We have used a fixed configuration and have not tuned it on LIBERO-Long or LIBERO-PRO. We have used settings from the original references/codebases without a held-out hyperparameter search for these benchmarks. Section 5.1 states the configuration is fixed across runs, and Appendix A, Table 7 lists the settings. The equal-weight and coverage ablations probe specific design choices; broader hyperparameter sensitivity remains open.

See Section 5.1 and Appendix A (Table 7) of the revised PDF for the fixed configuration and its parameter values.

AQ3: Priority Signal Ablations

Thanks for suggesting the component study. πRL\pi_{\mathrm{RL}} + CRAFT achieves the highest success rate (94.20%) among the variants. We evaluated each configuration on 500 LIBERO-Long episodes after 30 updates. DD, LL, and CC denote the target-success deficit, group outcome variation, and the discrepancy between fast and slow success estimates, respectively.

Online RL experiments are compute- and time-intensive. Given our compute budget and discussion window, we evaluated each ablation in one training run. Repeating the full study across multiple seeds would greatly increase training cost.

Allocation variantEpisode counts (successful/total)Success rate (%)
πRL\pi_{\mathrm{RL}}451/50090.20
πRL\pi_{\mathrm{RL}} + CRAFT (DD only)452/50090.40
πRL\pi_{\mathrm{RL}} + CRAFT (LL only)452/50090.40
πRL\pi_{\mathrm{RL}} + CRAFT (CC only)451/50090.20
πRL\pi_{\mathrm{RL}} + CRAFT (equal weights)457/50091.40
πRL\pi_{\mathrm{RL}} + CRAFT (without coverage)460/50092.00
πRL\pi_{\mathrm{RL}} + CRAFT471/50094.20

The full controller scores 2.20% higher than the variant without coverage.

See Table 3 of the revised PDF for the full component study.

AQ4: Scaling of Binary Outcome Variation

Sorry for the confusion. The factor of four rescales the statistic to 0–1. For binary outcomes, rˉ(1−rˉ)\bar r(1-\bar r) peaks at 1/41/4 when successes and failures are evenly split. Therefore, v=4rˉ(1−rˉ)v=4\bar r(1-\bar r) equals 1 at an even split and 0 when all outcomes match, making endpoints intuitive and preserving the group-variation ranking (Section 4.2).

See Section 4.2 and Appendix A.1 of the revised PDF for the variation statistic and its implementation.

Reviewer_VVt1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies multi-task online RL fine-tuning of VLA policies under a fixed number of policy updates and rollout groups. It proposes CRAFT, which allocates future rollout groups using a target-success deficit, within-group binary-outcome variation, and a fast-versus-slow success discrepancy. Priority normalization/smoothing, coverage bonuses, and a probability floor are intended to prevent unstable allocation and starvation. CRAFT is implemented in a Flow-SDE VLA RL pipeline and evaluated against uniform allocation on LIBERO and LIBERO-PRO, with task-level allocation and outcome analyses.

Critical strengths
  1. The method addresses a distinct and relevant decision problem. CRAFT allocates future data collection while holding the learner and policy-update budget fixed, rather than conflating allocation with reward shaping or a stronger policy optimizer. This is a clean contribution and the coverage safeguards are sensible.

  2. This paper is transparent about evaluation protocol and heterogeneous effects. It reports both average and task-level results, notes that larger allocation share need not imply a larger gain, and explicitly labels the ten-update paired study as a pilot. This makes the empirical story easier to audit than a single aggregate success number.

Critical weaknesses
  1. This paper lacks an in-depth analysis of the results presented in Table 1. Specifically, the authors fail to explain why the proposed method only achieves a significant improvement on libero-long, while experiencing performance degradation on the other three variants

  2. Although a RoboDojo-based platform is introduced, the paper does not report success rate comparisons for multiple VLAs (as done in the simulator), and fails to analyze any specific phenomena from the real-world experiments.

Questions for authors

See above

3 comments

Thank you for the careful review and for noting our matched-budget design and the importance of reporting aggregate and task-level results. Your feedback helped us strengthen the evaluation and clarify the scope of our conclusions. To explain suite-dependent gains, we added three-seed comparisons on Spatial, Object, and Goal, alongside the five-seed LIBERO-Long comparison, and analyzed rollout groups collected under a matched budget. We also compared CRAFT to four adaptive samplers with the same learner. We clarify that the RoboDojo trials are preliminary. The default πRL\pi_{\mathrm{RL}} uses uniform task sampling; πRL\pi_{\mathrm{RL}} + CRAFT is the same learner with CRAFT task allocation.

We have updated Figure 1 in the revised PDF to illustrate our method better.

AW1: Differences Across LIBERO Suites

Thanks for asking about differences across LIBERO suites. The table reports success after 30 updates using three matched training seeds for Spatial, Object, and Goal, and five seeds per method for Long. Each seed is evaluated over 500 episodes.

LIBERO suiteπRL\pi_{\mathrm{RL}} (%)πRL\pi_{\mathrm{RL}} + CRAFT (%)Difference (%)
Spatial96.796.9+0.2
Object97.799.2+1.5
Goal98.597.6−0.9
Long89.992.5+2.6

LIBERO-Long shows the largest gain across the four suites (+2.6%). Its longer-horizon tasks start from a lower baseline (89.9%), leaving more room to improve. Spatial, Object, and Goal have higher baselines (96.7%–98.5%). Spatial and Object increase by 0.2 and 1.5%, while Goal drops by 0.9%. At these near-ceiling rates, a few tasks can shift suite scores, and online RL stochasticity may cause small fluctuations.

On Long, mixed-outcome groups are 60.8% with πRL\pi_{\mathrm{RL}} + CRAFT vs. 45.7% with πRL\pi_{\mathrm{RL}} (+15.1%), in a matched run of 232 completed groups and 890,880 environment steps per method (Section 5.2). These groups include both successes and failures, providing within-group reward contrast. Figure 2 separately compares final slow success EMA summaries: the bottom 40%, bottom 20%, and weakest-task values are 6.8%, 16.1%, and 20.0% higher with CRAFT. With a fixed rollout budget, prioritizing one task reduces experience for others.

See Table 1 and Section 5.2 of the revised PDF for the suite results and mixed-group diagnostic, and Figure 2 for final slow success EMA summaries.

AW2: Scope of the RoboDojo Evaluation

Thanks for suggesting success-rate comparisons across VLA backbones and deeper rollout analysis on RoboDojo. We planned to evaluate more tasks and backbones, but our RoboDojo setup requires RTX GPUs for large-scale simulation and online RL; with our available compute, we have conducted only preliminary integration trials. Appendix D details block-level integration, and Figure 6 shows rollout frames from the initial π0.5\pi_{0.5} ARX X5 joint-action baseline.

Given the discussion-period training resources, we broadened our comparison of task-allocation methods on LIBERO-Long. We added DRATS, PLR, SFL, and CGPO, yielding six configurations, three matched training seeds (18 runs) and 9,000 evaluation episodes. The table reports results after 30 updates using the same learner and rollout budget for all configurations (Table 2 in the revised PDF).

Task samplerEpisode counts (successful/total)Mean success rate ± SD (%)
πRL\pi_{\mathrm{RL}}1,349/1,50089.93 ± 1.47
πRL\pi_{\mathrm{RL}} + DRATS1,364/1,50090.93 ± 0.50
πRL\pi_{\mathrm{RL}} + PLR1,366/1,50091.07 ± 2.10
πRL\pi_{\mathrm{RL}} + SFL1,367/1,50091.13 ± 1.50
πRL\pi_{\mathrm{RL}} + CGPO1,371/1,50091.40 ± 0.20
πRL\pi_{\mathrm{RL}} + CRAFT1,377/1,50091.80 ± 0.35

πRL\pi_{\mathrm{RL}} + CRAFT achieves the highest mean success rate, outperforming the strongest adaptive baseline, πRL\pi_{\mathrm{RL}} + CGPO, by 0.40%. We also added repeated training runs on LIBERO-Long and LIBERO-PRO.

We regret that we could not include the multi-backbone RoboDojo comparisons you suggested in this revision. For each backbone, we plan to compare default sampling with CRAFT on the same tasks under matched rollout budgets and evaluation protocols, and analyze rollout behavior alongside success rates.

See Table 2 (revised PDF) for new sampler comparisons, and Appendix D and Figure 6 for preliminary RoboDojo integration and baseline rollouts.

If the additional experiments and clarifications address your main concerns, we would appreciate your reconsidering the score in light of the revised evidence. Thanks!

Dear Reviewer VVt1,

As the discussion phase is approaching its end, we kindly request you to let us know if the above clarifications and the previously added experiments have addressed the remaining questions. If you are satisfied, we kindly request that you consider updating the score to reflect the newly added results and discussion. We would be happy to address any additional points the reviewer may have during the remaining time of the discussion phase. We thank the reviewer for engaging in the discussion with us.

Thank you for your time and consideration.

Best regards, The Authors

As the absence of the multi-backbone RoboDojo comparisons, I will maintain my rating.

Reviewer_vvj1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CRAFT, a method for allocating rollout groups during multi-task reinforcement fine-tuning of vision-language-action policies. It combines success-rate deficits, variation in group outcomes, and changes in estimated success to prioritize tasks, with normalization, smoothing, and coverage safeguards to stabilize allocation. CRAFT changes data collection while retaining the underlying policy optimization procedure. Experiments use a π0.5-based training pipeline on LIBERO and LIBERO-PRO under matched initialization, update, and trajectory budgets. The reported results show improvements on LIBERO-Long and several perturbation conditions, although the benefits vary across task suites and training checkpoints.

Critical strengths
  1. The main comparison carefully controls the policy setup and training budget. Section 5.1 initializes CRAFT and uniform sampling from the same checkpoint and uses the same learner configuration, 30 policy updates, and 1,920 rollout trajectories. This makes the comparison directly relevant to the proposed allocation problem and avoids attributing gains to additional updates or trajectories. The design also integrates task allocation into an existing training pipeline without changing rewards or introducing another policy objective.

  2. The reported results show promising improvements on difficult tasks and some transfer conditions. Table 1 reports an increase in LIBERO-Long success from 90.2% to 94.2%, while Table 2 reports an increase in episode-weighted LIBERO-PRO success from 44.4% to 52.0% at 30 updates. Although adaptation uses Object perturbations, Semantic success increases from 64.0% to 76.0% and Task success from 30.0% to 40.0%. These results suggest that changing rollout allocation can benefit evaluation conditions beyond those used for adaptation.

Critical weaknesses
  1. The experiments do not establish the benefit of the proposed allocation design over simpler adaptive alternatives. Section 2 discusses Prioritized Level Replay and Sampling for Learnability, but the matched experiments compare CRAFT only with uniform sampling. Sections 4.2-4.3 combine several feedback signals and coverage mechanisms without component ablations. Consequently, it remains unclear whether the gains require the proposed combination or could be achieved by a simpler success-deficit or outcome-variation sampler. This limits the evidence for CRAFT’s added value over established adaptive-sampling ideas.

  2. The reliability of the main performance gains across training runs remains unclear. Tables 1 and 2 report the 30-update improvements without uncertainty estimates from repeated training runs for both methods. Section 5.3 provides additional experiments, but its three-seed result concerns CRAFT at 10 updates, and the paired comparison also uses 10-update checkpoints. These results therefore may not quantify uncertainty in the headline 30-update gains over uniform sampling.

Questions for authors
  1. Under the same learner and rollout budget, how does CRAFT compare with an established adaptive sampler or a simpler success-deficit rule, and which proposed components account for its advantage?

  2. Do the 30-update gains persist across matched training seeds for CRAFT and uniform sampling, with uncertainty estimates reported for their performance differences on LIBERO and LIBERO-PRO?

5 comments

Thank you for noting our matched-budget setup; your feedback improved the paper’s rigor and clarity. The revised manuscript includes two sets of additional experiments. For allocation evaluation, we added four adaptive samplers and a component analysis; πRL\pi_{\mathrm{RL}} + CRAFT achieves the best average success rate. To assess reliability after 30 updates, we conducted multiple training repeats on LIBERO-Long and LIBERO-PRO, improving mean performance by 2.6 and 3.73%, respectively. πRL\pi_{\mathrm{RL}} samples tasks uniformly at random, whereas πRL\pi_{\mathrm{RL}} + CRAFT uses the same learner with CRAFT-based task allocation.

AW1: Adaptive Baselines and Component Ablations

Thanks for the suggestion. We expanded the comparison to four adaptive sampling methods (DRATS, PLR, SFL, CGPO). With πRL\pi_{\mathrm{RL}} and πRL\pi_{\mathrm{RL}} + CRAFT, this yields six configurations across three matched seeds (18 training runs; 9,000 evaluation episodes). We adapted the released sampler implementations to RLinf’s task-selection interface. All runs use the same πRL\pi_{\mathrm{RL}} learner and initialization (30 updates; 1,920 rollout trajectories). Each run is evaluated on 500 LIBERO-Long episodes.

Task samplerEpisode counts (successful/total)Mean success rate ± SD (%)
πRL\pi_{\mathrm{RL}}1,349/1,50089.93 ± 1.47
πRL\pi_{\mathrm{RL}} + DRATS1,364/1,50090.93 ± 0.50
πRL\pi_{\mathrm{RL}} + PLR1,366/1,50091.07 ± 2.10
πRL\pi_{\mathrm{RL}} + SFL1,367/1,50091.13 ± 1.50
πRL\pi_{\mathrm{RL}} + CGPO1,371/1,50091.40 ± 0.20
πRL\pi_{\mathrm{RL}} + CRAFT1,377/1,50091.80 ± 0.35

πRL\pi_{\mathrm{RL}} + CRAFT has the highest mean success rate, outperforming the best adaptive baseline, πRL\pi_{\mathrm{RL}} + CGPO, by 0.40%. SD is the standard deviation over three training seeds. Table 3 also includes allocation variants, including a success-deficit-only variant (AQ1).

To clarify our method, we updated Figure 1 in the revised PDF.

See Section 5.2 and Tables 2–3 in the revised PDF for the sampler comparison and component study.

AW2 & AQ2: Consistency Across Training Seeds After 30 Updates

We agree that repeated runs at the main evaluation endpoint are needed to assess reliability. We have evaluated both methods in single runs after 30 updates. For this rebuttal, we repeated the evaluations on LIBERO-Long and LIBERO-PRO across training seeds. We report mean success rates and seed-to-seed variation. The 10-update analysis is reported separately.

LIBERO-Long. Across five training seeds, πRL\pi_{\mathrm{RL}} + CRAFT increases mean success by 2.6%. Each run is evaluated over 500 episodes.

MethodEpisode counts (successful/total)Mean success rate ± sample SD (%)
πRL\pi_{\mathrm{RL}}2,247/2,50089.88 ± 0.76
πRL\pi_{\mathrm{RL}} + CRAFT2,313/2,50092.52 ± 0.99

LIBERO-PRO. Across three matched training seeds, πRL\pi_{\mathrm{RL}} + CRAFT increases overall success by 3.73%. Each run is evaluated over 250 episodes.

MethodEpisode counts (successful/total)Mean success rate ± sample SD (%)
πRL\pi_{\mathrm{RL}}347/75046.27 ± 1.67
πRL\pi_{\mathrm{RL}} + CRAFT375/75050.00 ± 1.83

On LIBERO-PRO, the paired improvement is 3.73 ± 3.35% (mean ± sample SD). The overall rate aggregates evaluation episodes across the four perturbation types.

We incorporate the above discussions and findings into Section 5.2 as well as Tables 1 and 4.

AQ1: Success-Deficit Baseline and Component Effects

Thanks for the question. In our ablation study, πRL\pi_{\mathrm{RL}} + CRAFT achieves the highest success rate (94.20%) among the tested variants. Each variant is evaluated on 500 LIBERO-Long episodes after 30 updates. DD, LL, and CC refer to the target-success shortfall, the variability in group outcomes, and the gap between the fast and slow success estimates, respectively.

Online RL experiments are compute- and time-intensive. Given limited time and resources, we evaluated each ablation in one training run; repeating across multiple seeds would greatly increase training cost.

Allocation variantEpisode counts (successful/total)Success rate (%)
πRL\pi_{\mathrm{RL}}451/50090.20
πRL\pi_{\mathrm{RL}} + CRAFT (DD only)452/50090.40
πRL\pi_{\mathrm{RL}} + CRAFT (LL only)452/50090.40
πRL\pi_{\mathrm{RL}} + CRAFT (CC only)451/50090.20
πRL\pi_{\mathrm{RL}} + CRAFT (equal weights)457/50091.40
πRL\pi_{\mathrm{RL}} + CRAFT (without coverage)460/50092.00
πRL\pi_{\mathrm{RL}} + CRAFT471/50094.20

AW1 compares PLR and SFL under the same learner, initialization, and rollout budget. On LIBERO-Long (three matched seeds), πRL\pi_{\mathrm{RL}} + CRAFT achieves 91.80% mean success vs. 91.07% with πRL\pi_{\mathrm{RL}} + PLR and 91.13% with πRL\pi_{\mathrm{RL}} + SFL.

See Table 3 of the revised PDF for the full component study.

If the additional experiments and clarifications address your main concerns, we would greatly appreciate your reconsideration of the score in light of the revised evidence. Thanks!

The rebuttal substantially addresses the concerns raised by this reviewer. However, as noted by other reviewers, concerns regarding the limited improvements over existing adaptive sampling methods, the reliability of component ablations, and generalizability remain. The reviewer therefore maintains the original rating and encourages the authors to further improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.