acceptodds
Under review as a conference paper at ICLR 2027

A GPU-Parallel Framework for Heterogeneous Multi-Task Reinforcement Learning

Abstract

GPU-parallel simulation provides abundant robot interaction, but existing benchmarks rarely combine this scale with heterogeneous manipulation tasks and standardized multi-task RL evaluation. We introduce Hebero (Heterogeneous Benchmark for Robot Learning), a GPU-parallel Isaac Lab benchmark that enables efficient joint training and evaluation of a single policy across all 40 heterogeneous tasks. Scaling experiments show that increasing parallel replicas per task improves success under a fixed wall-clock budget. To support learning with sparse rewards and limited demonstrations, we propose Demonstration-Guided Policy Optimization (DGPO), which reuses demonstrations for dense tracking rewards and asymmetric value learning. Its shared stack supports controlled comparisons of learner-specific demonstration interfaces within PPO. Within DGPO framework, we introduce IW-ABC, which uses a lightweight per-task learning progress signal to coordinate adaptive behavior cloning (ABC), relaxing demonstration guidance with task progress, and importance weighting (IW), emphasizing lagging tasks in PPO updates. With 50 demonstrations per task, IW-ABC achieves 90.1% state-input mean success, outperforming the strongest baseline FAMO-ABC by 7.8 percentage points. Its visual counterpart reaches 93.5% mean success. Real-world experiments further demonstrate that a single multi-task policy trained in simulation can successfully perform four tasks on a physical Piper robot. A preview of the benchmark, dataset and training code is provided in the supplementary materials.

Primary area:
reinforcement learning
Keywords:
Multi-task reinforcement learning, Robot manipulation, Learning from demonstrations, GPU-parallel simulation, Reinforcement learning benchmarks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5VcUOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. This paper introduces Hebero, a GPU-parallel benchmark for jointly training and evaluating a single policy on 40 heterogeneous robotic manipulation tasks.

  2. The authors further propose DGPO and IW-ABC, which use demonstrations and task-level learning progress to adapt imitation guidance and prioritize underperforming tasks during PPO training.

  3. Experiments show that IW-ABC achieves 90.1% state-based and 93.5% visual mean success rates, while the simulation-trained policy obtains 82.5% success on four real-robot tasks.

Critical strengths

The paper presents a scalable GPU-parallel multi-task RL benchmark and an effective demonstration-guided training method, supported by strong simulation results and real-robot transfer.

Critical weaknesses
  1. The paper combines many components, but the individual algorithmic contributions appear largely incremental, making the core technical novelty difficult to identify.

  2. The evaluation is restricted to adapted LIBERO tasks without held-out-task testing, so the generalization of Hebero and IW-ABC to broader task distributions remains unclear.

  3. The real-world evaluation covers only four tasks with 20 trials per task and lacks competing baselines, providing limited evidence for robust sim-to-real transfer.

  4. Given the unusually broad use of generative AI in experimental design, implementation, analysis, and writing, the paper should disclose the AI-assisted components and their independent verification procedures more precisely.

Questions for authors
  1. Could the authors clarify the key technical novelty of IW-ABC and explain how it differs fundamentally from existing demonstration-guided and task-weighting methods?

  2. How well do Hebero and IW-ABC generalize to unseen tasks, different task distributions, or robotic environments beyond the adapted LIBERO suite?

  3. Could the authors provide stronger real-world evaluations, including more tasks and trials as well as comparisons with relevant baselines under the same deployment setting?

  4. Could the authors specify which parts of the experimental design, implementation, analysis, and manuscript preparation involved generative AI, and describe how each AI-assisted artifact was independently verified?

5 comments
AuthorsOpenReview ↗

Response 1/3: AI use statement (Weakness 4; Q4)

Thank you for your careful review. We address the scope of AI assistance and how it was verified first.

Generative AI was used mainly for code debugging and polishing the wording of the paper. All final decisions on the research design, experimental design, and analysis of results were made by the authors. All of our code is included in the supplementary material, and we plan to open-source all datasets and the code framework. We did not use any automated-research pipeline for the paper's main experiments or for constructing the method. AI was used in two places:

  • Paper writing. For the appendix, the authors first wrote the key points in Chinese; an AI then produced an English first draft, which the authors revised and verified item by item. In all other sections, AI use was limited to polishing wording and expression. The authors checked every AI-generated or AI-rewritten technical description against the actual implementation and results, and verified the citations.
  • Code debugging. AI helped locate errors and suggested code changes. The authors verified each change through manual code inspection, unit tests, and run results.

We will state this scope explicitly in the revised AI use statement.

AuthorsOpenReview ↗

Response 2/3: Technical novelty (Weakness 1; Q1)

Thank you for this important question.

First, we would like to restate that our main contributions are a scalable GPU-parallel heterogeneous multi-task RL benchmark and a training framework for fair comparison of demonstration-guided methods; IW-ABC is the reference algorithm within it. We do not claim that performance-based task prioritization is new, and we will clarify how IW and ABC differ from prior work.

What IW-ABC adds is one signal used twice: DGPO maintains a single per-task online signal and uses it both to weight PPO gradients across tasks (IW) and to set demonstration-guidance strength (ABC). Below we compare each use with prior work and point to the supporting evidence.

2.1 IW vs. multi-task optimization. By feedback signal, prior methods fall into three groups; IW belongs to M2.

CategorySignalPrior work
M1Loss/gradient-basedLoss improvement, gradient relationsFAMO [1]; PCGrad [2]
M2Performance/difficulty-basedTask metrics, interaction-based difficultyDTP [3]; Hard Tasks First [4]; IW (ours)
M3Uncertainty-basedLearned task noiseKendall et al. [5]
  • DTP [3] weights supervised losses by task KPIs. IW extends this to continuous-control online RL, using the EMA of episode success rates from training rollouts to assign bounded PPO weights.
  • Hard Tasks First [4] also favors hard tasks, but reallocates the interaction budget using return/entropy-based difficulty. IW keeps each task's rollout allocation fixed (a fixed number of environment replicas per task, so all methods use the same sampling budget) and reweights the PPO objective instead.
  • FAMO [1] uses critic loss improvement; our success-rate signal measures task completion directly, which does not necessarily move in step with the progress of critic fitting. With weighting on the critic in both cases, IW-ABC outperforms FAMO-ABC by 5.8 / 13.4 points (Mean / Long SR).

We will add these references and positioning to Related Work.

2.2 ABC vs. demonstration-guided RL. Methods grouped by how demonstrations enter training (bold = compared in Sec. 4.2):

Use of demosPrior workIn DGPO
D1Policy initializationBC init. in DAPG [6]BC→PPO
D2Policy regularization (scheduled, filtered, or adaptive)DAPG [6]; Q-filtered BC [7]; Adaptive BC [8]DAPG: offline demo (obs, action) pairs. ABC (ours): online observations paired with cursor-aligned demo actions; strength set by success-rate EMA
D3Demo-based rewardDeepMimic [9]; AMP [10]Tracking reward, shared by all methods
D4Reset/exploration curriculumRFCL [11]RFCL-inspired reset curriculum on the shared PPO backbone
D5Demo replay, off-policyRLPD [12]Out of scope (on-policy PPO comparison)
  • DGPO places the demonstration-based tracking reward (D3) in the shared framework, and provides representative, method-specific demonstration usages from D1, D2, and D4 for a controlled comparison. The compared methods share the PPO backbone, reward, observations, training budget, and evaluation protocol.
  • With the same IW weighting, ABC (D2) achieves the highest mean and Long success in our state-input evaluation.
  • The success-rate term in ABC contributes measurably: replacing it with time-only annealing (all else unchanged) lowers Mean SR from 90.1 to 86.1 and Long SR from 70.0 to 56.7.

In summary, IW-ABC differs clearly from prior work on both sides: IW differs from existing multi-task optimization methods in its signal (task success rate, not loss or gradients) and in where it acts (PPO weights under fixed rollout allocation, not interaction budget), and ABC differs from existing demonstration-guided methods in how demonstrations are used (online observations paired with cursor-aligned demonstration actions, with success-adaptive strength).

References

[1] Liu et al. FAMO: Fast Adaptive Multitask Optimization. NeurIPS 2023.

[2] Yu et al. Gradient Surgery for Multi-Task Learning. NeurIPS 2020.

[3] Guo et al. Dynamic Task Prioritization for Multitask Learning. ECCV 2018.

[4] Cho et al. Hard Tasks First: Multi-Task RL Through Task Scheduling. ICML 2024.

[5] Kendall et al. Multi-Task Learning Using Uncertainty to Weigh Losses. CVPR 2018.

[6] Rajeswaran et al. Learning Complex Dexterous Manipulation with Deep RL and Demonstrations. RSS 2018.

[7] Nair et al. Overcoming Exploration in RL with Demonstrations. ICRA 2018.

[8] Zhao et al. Adaptive Behavior Cloning Regularization for Stable Offline-to-Online RL. ESANN 2022.

[9] Peng et al. DeepMimic. ACM TOG 2018.

[10] Peng et al. AMP: Adversarial Motion Priors. ACM TOG 2021.

[11] Tao et al. Reverse Forward Curriculum Learning for Extreme Sample and Demo Efficiency. ICLR 2024.

[12] Ball et al. Efficient Online Reinforcement Learning with Offline Data. ICML 2023.

AuthorsOpenReview ↗

Response 3/3: Generalization and real-world evaluation (Weaknesses 2, 3; Q2, Q3)

Thank you for pointing out these limitations. We respond to the generalization question first and then the real-world evaluation.

3.1 Varied robot environments (Weakness 2; Q2). Besides the LIBERO suites, our paper also adapts RoboTwin tasks for real-robot deployment; these differ from LIBERO in task types and robot embodiments.

3.2 Unseen tasks. We acknowledge this limitation: our experiments do not yet verify generalization to unseen tasks. Simulation experiments evaluate the 40 jointly trained tasks, and the real-robot experiment uses a separately trained four-task policy to test sim-to-real transfer on those tasks. This work focuses on efficient joint learning and task balancing over a predefined task set, and the predefined task encoding limits direct extension to new tasks.

3.3 Different task distributions (new). We evaluated generalization on known tasks under shifted initial-state distributions. We expand the object initial XY sampling box, around its center, to 2× and 3× the base evaluation range, and separately test robot joint initial noise (supplementary Figure 10; same protocol as the paper: 50 episodes, 3 training seeds).

ConditionObject XY rangeJoint noise (rad)IW-ABC Mean SR (%)Vis IW-ABC Mean SR (%)
Base1×090.1±3.893.5±2.6
Larger object range2×088.5±3.380.6±2.0
Larger object range3×074.3±2.960.4±2.4
Joint noise only1×0.0190.0±3.093.4±2.2
Joint noise only1×0.0290.4±3.193.1±2.5
  • The visual policy degrades more than the state policy as the object layout range grows.
  • The RL-trained policy tolerates small changes in the robot's initial joint configuration.

In future work, we will explore language- or goal-conditioned representations, stronger policy architectures, and pretrained perception models for systematic evaluation.

The revision will clearly distinguish these kinds of generalization and state the limits of the current experimental scope and task encoding.

3.4 More real-world evaluations (Weakness 3; Q3). Constrained by the robot embodiments available to us, we designed an additional six-task sim-to-real experiment on a UR12e arm with a Robotiq 2F-85 gripper. It shows cross-embodiment transfer and raises task difficulty: the tasks differ clearly in semantics and precision requirements and are not only pick-and-place. The task definitions and MDP interfaces for joint learning were straightforward to implement in simulation, and we collected 50 motion-planned expert trajectories per task. Given time and compute limits, we compare IW-ABC only with the strongest baseline, FAMO-ABC, under identical training and deployment (20 trials per task):

TaskFAMO-ABCIW-ABC
Click Bell20/2020/20
Shake Bottle18/2019/20
Move Pill Bottle onto Pad18/2018/20
Place Container on Plate20/2019/20
Put Apple onto Plate11/2015/20
Stack Three Cubes8/2012/20
Total95/120 (79.2%)103/120 (85.8%)

Videos: anonymous link. We will add these results to Sec. 5.4. Thank you again for these suggestions.

Reviewer_Qc9jOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

This paper introduces Hebero, a GPU parallelized Isaac Lab benchmark of the CPU bound LIBERO which makes it easier to enable joint training and a singular evaluation across the 40 tasks. Learning with limited demos and a sparse reward loop, they propose DGPO (Demonstration Guided Policy Optimization) where it gets a tracking reward for following a demonstration and uses an asymmetric critic that gets extra information of the environment as the judge. The final part of their recipe is IW-ABC melded into the DGPO - where the IW (Importance Weighting helps it prioritize lagging tasks and the ABC(Adaptive Behavior cloning) pulls it towards the demonstrated actions and reduces the pull when it has learnt a behaviour well. Experimentally, Hebero simulates about 3.4x faster than the original MuJoCo LIBERO, IW-ABC reaches 90.1% mean success vs 82.3% for the strongest baseline (FAMO-ABC) over 3 seeds and a four-task policy succeeds in 66 of 80 real-robot trials.

Critical strengths
  1. It is Very Useful infrastructure. Hebero runs the 40 tasks of LIBERO’s four standard suites in a singular GPU-parallel Isaac Lab process, so that makes it a very good candidate for demo guided multitask RL esp on long horizon ones
  2. It's a very controlled environment with a clear goal. All the configurations use the same command stream, reward, Critic inputs, PPO backbone and network capacity. And on top of that adding IW to ABC clearly helps: The jump from 75.5 to 90% for mean SR and the jump from 40 to 70% for Long SR
Critical weaknesses
  1. The IW-ABC do not seem to be isolated and its margin over the baseline seem to be slim. IW-ABC weights both actor and critic, while FAMO-ABC weights only the critic. Actor-only IW-ABC (78.2%) is below FAMO-ABC (82.3%), and there is no critic-only IW run. ABC’s success term, the tracking reward and the asymmetric critic are not ablated. With 3 seeds, the gain over FAMO-ABC is borderline
  2. There is no comparison to the original LIBERO showing that the Isaac Lab port reproduces it. And the number of evaluation episodes per task is not stated. Six of eight state learners report zero-variance Long SRs that are exact multiples of 10 which might point to a few episodes or non deterministic evals. The task code is hand designed with no language and makes it not directly comparable to standard LIBERO numbers
Questions for authors
  1. How many evaluation episodes does each task get ? Do all Long tasks in the state runs end at exactly 0% or 100%?
  2. Can you report critic-only IW-ABC runs ? and FAMO applied to both actor and critic?
  3. What %age of the LIBERO demonstrations succeed when their actions are replayed in HEBERO ?
  4. Could we add a few more ablations for ABC with time-only annealing, IW-ABC without the tracking reward and IW-ABC with a symmetric critic?
  5. How do IW and ABC differ from Dynamic Task Prioritization (Guo et al., 2018) and adaptive BC regularization (Zhao et al., 2022)?
5 comments
AuthorsOpenReview ↗

Response 1/3: Ablations and attribution of IW-ABC (Weakness 1; Q2, Q4)

Thank you for recognizing the importance of our infrastructure and controlled RL framework, and for your careful review. We answer in three responses and mark which of your comments each addresses.

1.1 Same-placement comparison with FAMO (Weakness 1, Q2). We added a critic-only IW-ABC run (added to Table 2) so that the weighting placement matches FAMO-ABC:

MethodWeighting placementMean SRLong SR
FAMO-ABCcritic82.3±2.953.3±5.8
IW-ABCcritic88.1±5.466.7±11.5
IW-ABCactor+critic90.1±3.870.0±10.0
  • Same-placement attribution. With weighting on the critic in both cases, IW-ABC exceeds FAMO-ABC by 5.8 / 13.4 points (Mean / Long SR). Applying IW to the actor as well adds a further 2.0 / 3.3 points, mostly on long-horizon tasks.

  • Why FAMO has no actor+critic version (Q2). Our FAMO-ABC baseline applies FAMO's task weighting to the critic losses only, adapting FAMO's actor–critic (SAC-based) RL setting [1] to our PPO framework (Appendix B.4). FAMO estimates task progress from changes in log losses, so it needs positive losses. The value-regression loss satisfies this after adding ε, but the unmodified PPO policy surrogate can take either sign. Applying FAMO to it would require constructing an extra positive loss, and the offset choice would affect FAMO's progress estimates and effective weights.

  • To separate the weighting rule from its placement, we added the critic-only IW-ABC run, and we report this comparison separately from the full actor+critic IW-ABC so the two effects can be assessed independently. We will include both comparisons and this explanation in the revised paper.

1.2 The three ablations you requested (Q4). Each starts from the full IW-ABC and changes one factor, with the same training budget and evaluation protocol (30k iterations, 3 training seeds):

ConfigurationChangeMean SR (%)Long SR (%)Task coverage
IW-ABC (default)none90.1±3.870.0±10.035/40
ABC time-only annealingABC coefficient anneals with global training progress only; success-rate term removed86.1±2.856.7±5.834/40
w/o tracking rewardrtdemo=0r^{\text{demo}}_t=0; critic keeps reference-relative errors0.0±0.00.0±0.00/40
symmetric criticall reference-relative errors removed from critic input; rtdemor^{\text{demo}}_t kept0.0±0.00.0±0.00/40
  • Time-only annealing. Removing the ABC coefficient's dependence on task success lowers the harder Long SR from 70.0 to 56.7, supporting the benefit of success-adaptive ABC guidance on long-horizon tasks.
  • w/o tracking reward / symmetric critic. Both drop to 0, showing that the tracking reward and the asymmetric critic are prerequisites of the whole DGPO training stack.

Thank you again for these constructive suggestions. Together, the same-placement comparison and these single-factor ablations let us attribute the gains more precisely: to the weighting rule (IW vs. FAMO at the same placement), to its placement (critic-only vs. actor+critic), to success-adaptive ABC, and to the shared tracking reward and asymmetric critic. We will add all results to the revised paper.

Reference

[1] Liu et al. FAMO: Fast Adaptive Multitask Optimization. NeurIPS 2023.

AuthorsOpenReview ↗

Response 2/3: Evaluation protocol and demonstration replay (Weakness 2; Q1, Q3)

Thank you for these careful questions. We address the evaluation protocol and demonstration replay here, and the comparability with original LIBERO in the next response.

2.1 Evaluation protocol and the discreteness of Long SR (Weakness 2; Q1).

  • Protocol. Sec. 3.4 describes how initial states are sampled and the evaluation metrics: in each episode, the object's initial XY position is drawn independently and uniformly from a box determined by the 50 demonstration initial positions of that task, with collision rejection sampling. To be explicit, we evaluate 50 episodes per task with a deterministic policy, and we will add these two details to Sec. 3.4.
  • Every Libero-10 task is 0% or 100% (Q1). For IW-ABC, over all three training seeds, each of the 10 Libero-10 tasks has SR exactly 0 or 100%, so Long SR is a multiple of 10% and the zero variance follows from this. Such polarized per-task success is also common in multi-task RL; e.g., in the per-task results on Meta-World MT10 of Cho et al. [1] (Table 1), most tasks are 100±0.0 or 0±0.0 for most methods. Per-task results for IW-ABC:
Training seedSR on L0–L9Long SR (%)
01, 0, 0, 1, 1, 0, 1, 1, 1, 170±0.0
11, 0, 1, 1, 1, 1, 1, 1, 0, 180±0.0
21, 0, 0, 1, 1, 0, 1, 1, 0, 160±0.0
  • This is not caused by too few evaluation episodes. Changing the evaluation seed (re-drawing object XY) leaves the Long-task results unchanged, so success is determined by policy capability. The ±10 across training seeds comes from whether individual tasks (e.g., L2, L8) are learned within the current training budget, not from evaluation noise.
  • Evaluation randomness appears mainly in a few contact-rich Goal tasks (G0 "open the middle drawer of the cabinet", G3 "open the top drawer and put the bowl inside", G5 "push the plate to the front of the stove", etc.), whose Isaac Lab simulation is somewhat stochastic, giving SR between 0.3 and 0.8.

2.2 Demonstration replay success and simulator dynamics gap (Q3). We use all 50 demonstrations per task with no replay-based filtering. We tried two ways of transferring LIBERO (MuJoCo) demonstrations to Isaac Lab and measured replay success (50 demos per task, averaged by suite):

SuiteOSC-pose action replay (%)Joint-space replay (%)
Spatial47.892.6
Object33.684.2
Goal36.270.8
Long11.674.4
All 40 tasks32.380.5
  • OSC-pose action replay replays the recorded end-effector pose-delta actions through an identically configured OSC controller. Controller error is amplified after contact (object error 3–11 mm before first contact vs. 38–186 mm after), so success is low, with 7 tasks at 0/50.
  • Joint-space replay bypasses the controller and drives the joints directly to the recorded joint positions. Success is much higher but not 100%; the residual failures come from contact and object dynamics (PD-tracking joint error about 0.02–0.03 rad).

Conclusion. Both transfers show a MuJoCo-to-PhysX sim2sim dynamics gap, mainly from contact dynamics and accumulated controller-timing error, not from infeasible demonstrations. DGPO does not treat demonstrations as directly replayable trajectories: they serve only as the reference for the tracking reward, the reference-relative errors for the critic, and the matched-action regression target. Hence, even with only 32.3% OSC action-replay success, DGPO reaches 90.1% Mean SR, suggesting the framework is fairly robust to sim2sim dynamics gaps and usable for cross-simulator demonstration transfer.

Reference

[1] Cho et al. Hard Tasks First: Multi-Task Reinforcement Learning Through Task Scheduling. ICML 2024.

AuthorsOpenReview ↗

Response 3/3: Comparability with original LIBERO and difference from existing methods (Weakness 2; Q5)

Thank you for raising these points. We address comparability with original LIBERO and the relation of IW-ABC to the two works you mention.

3.1 Comparability with original LIBERO (Weakness 2). You note that Hebero's task code is hand-designed and contains no language instructions. We agree that results should not be compared directly, but language conditioning is not the focus of this work:

  • We study multi-task RL with known task identities, not language understanding. Our policy is conditioned on a task encoding, with no language conditioning and no unseen-task language generalization, so comparison with language-based VLA results is not needed.
  • Standard LIBERO is itself a weak test of language. LIBERO-PRO [1] and LIBERO-Plus [2] both find that VLAs trained on standard LIBERO keep essentially the same outputs and success rates even when instructions are corrupted, or replaced with gibberish. Removing language conditioning therefore does not make LIBERO tasks unrepresentative, and standard LIBERO success does not imply language understanding.
  • Hebero's positioning. It is a unified, GPU-efficient parallel benchmark for heterogeneous multi-task RL in which all methods are compared under one protocol. We inherit LIBERO's structured task-scene definitions and success criteria, use the same expert trajectories, and implement heterogeneous multi-task parallel simulation and multi-task RL interfaces. The original language-instruction definitions are also kept in the task configuration files, so users can easily replace the task encoding with language conditioning for downstream research.

3.2 Difference between IW-ABC and existing methods (Q5).

DTP [3]. DTP also dynamically raises the weight of lagging tasks based on a smoothed estimate of each task's performance (KPI), and our IW follows this idea. The differences are:

  • We move it from supervised multi-task learning to online RL for continuous robot control, where the weights act on the PPO gradient budget.
  • Our signal is the EMA of task success rate, obtained directly from training rollouts with no extra evaluation, at nearly zero cost in 40-task parallel training.
  • The same signal also sets the demonstration strength (ABC), whereas DTP only weights tasks.

Adaptive BC regularization [4]. We share the motivation of adapting imitation strength to policy performance, but target a different training regime.

  • Zhao et al. stabilize off-policy online fine-tuning after offline RL pretraining, whereas our on-policy PPO learner starts from scratch and uses demonstration guidance to bootstrap learning.
  • Our BC term pairs online rollout observations with demonstration actions aligned by the trajectory cursor (Appendix B.2).
  • Moreover, their controller adapts to return-target gaps and sudden return drops, whereas we regulate BC with a lightweight per-task success-rate EMA and time annealing.

One signal, two uses. DGPO maintains a single per-task online signal, the success-rate EMA, and uses it to weight PPO gradient updates (IW) and, together with a global anneal, to set demonstration-guidance strength (ABC). Our contribution is unifying these two uses under one lightweight signal, with clear empirical gains in the LIBERO 40-task setting.

We thank you again for the helpful review. We will add the references you recommended and update the paper accordingly.

References

[1] Zhou et al. LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization. arXiv:2510.03827, 2025.

[2] Fei et al. LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models. arXiv:2510.13626, 2025.

[3] Guo et al. Dynamic Task Prioritization for Multitask Learning. ECCV 2018.

[4] Zhao et al. Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning. ESANN 2022.

Reviewer_tfprOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Hebero, a GPU-parallel benchmark supporting joint reinforcement learning across 40 heterogeneous LIBERO tasks. The authors also propose DGPO, a demonstration-guided PPO framework, and IW-ABC, which combines adaptive behavior cloning with success-based task weighting. Experiments show that IW-ABC achieves 90.1% mean success, outperforming FAMO-ABC by 7.8 percentage points. The paper further demonstrates multi-task policy deployment on a physical robot.

Critical strengths
  1. Hebero provides a unified GPU-parallel framework for heterogeneous multi-task RL, with standardized training and evaluation interfaces.

  2. IW-ABC achieves substantial improvements over the evaluated baselines, particularly on long-horizon tasks.

  3. The authors demonstrate successful deployment of a jointly trained policy on four physical robot tasks.

Critical weaknesses
  1. IW-ABC primarily combines success-based task weighting with adaptive imitation loss, both of which are relatively straightforward extensions of existing ideas. The paper does not clearly establish its algorithmic novelty over prior multi-task optimization and demonstration-guided RL methods.

  2. FAMO-ABC applies adaptive weighting only to the critic, whereas IW-ABC weights both actor and critic objectives. This difference makes it difficult to attribute the reported improvements specifically to the proposed weighting strategy.

  3. The main experiments evaluate the same 40 tasks used during training. The real-world experiments also use a separately trained four-task policy, leaving generalization to unseen tasks largely unexplored.

  4. Given existing GPU-parallel platforms such as ManiSkill3 and MTBench, the paper should better clarify the unique technical contributions of Hebero beyond integrating existing tasks into a unified training pipeline.

Questions for authors

see weaknesses

6 comments
AuthorsOpenReview ↗

Response 1/4: Novelty of IW-ABC (Weakness 1)

Thank you for your positive assessment of our work and for your careful review.

First, we would like to restate that our main contributions are a scalable GPU-parallel heterogeneous multi-task RL benchmark and a training framework for fair comparison of demonstration-guided methods; IW-ABC is the reference algorithm within it. We do not claim that performance-based task prioritization is new, and we will clarify how IW and ABC differ from prior work.

What IW-ABC adds is one signal used twice: DGPO maintains a single per-task online signal and uses it both to weight PPO gradients across tasks (IW) and to set demonstration-guidance strength (ABC). Below we compare each use with prior work and point to the supporting evidence.

1.1 IW vs. multi-task optimization. By feedback signal, prior methods fall into three groups; IW belongs to M2.

CategorySignalPrior work
M1Loss/gradient-basedLoss improvement, gradient relationsFAMO [1]; PCGrad [2]
M2Performance/difficulty-basedTask metrics, interaction-based difficultyDTP [3]; Hard Tasks First [4]; IW (ours)
M3Uncertainty-basedLearned task noiseKendall et al. [5]
  • DTP [3] weights supervised losses by task KPIs. IW extends this to continuous-control online RL, using the EMA of episode success rates from training rollouts to assign bounded PPO weights.
  • Hard Tasks First [4] also favors hard tasks, but reallocates the interaction budget using return/entropy-based difficulty. IW keeps each task's rollout allocation fixed (a fixed number of environment replicas per task, so all methods use the same sampling budget) and reweights the PPO objective instead.
  • FAMO [1] uses critic loss improvement; our success-rate signal measures task completion directly, which does not necessarily move in step with the progress of critic fitting. With weighting on the critic in both cases, IW-ABC outperforms FAMO-ABC by 5.8 / 13.4 points (Mean / Long SR).

We will add these references and positioning to Related Work.

1.2 ABC vs. demonstration-guided RL. Methods grouped by how demonstrations enter training (bold = compared in Sec. 4.2):

Use of demosPrior workIn DGPO
D1Policy initializationBC init. in DAPG [6]BC→PPO
D2Policy regularization (scheduled, filtered, or adaptive)DAPG [6]; Q-filtered BC [7]; Adaptive BC [8]DAPG: offline demo (obs, action) pairs. ABC (ours): online observations paired with cursor-aligned demo actions; strength set by success-rate EMA
D3Demo-based rewardDeepMimic [9]; AMP [10]Tracking reward, shared by all methods
D4Reset/exploration curriculumRFCL [11]RFCL-inspired reset curriculum on the shared PPO backbone
D5Demo replay, off-policyRLPD [12]Out of scope (on-policy PPO comparison)
  • DGPO places the demonstration-based tracking reward (D3) in the shared framework, and provides representative, method-specific demonstration usages from D1, D2, and D4 for a controlled comparison. The compared methods share the PPO backbone, reward, observations, training budget, and evaluation protocol.
  • With the same IW weighting, ABC (D2) achieves the highest mean and Long success in our state-input evaluation.
  • The success-rate term in ABC contributes measurably: replacing it with time-only annealing (all else unchanged) lowers Mean SR from 90.1 to 86.1 and Long SR from 70.0 to 56.7.

In summary, IW-ABC differs clearly from prior work on both sides: IW differs from prior multi-task optimization in its signal (task success, not loss or gradients) and where it acts (PPO weights, not interaction budget); ABC differs from prior demonstration-guided methods in how demonstrations are used (online observations paired with cursor-aligned demonstration actions, success-adaptive strength).

References

[1] Liu et al. FAMO: Fast Adaptive Multitask Optimization. NeurIPS 2023.

[2] Yu et al. Gradient Surgery for Multi-Task Learning. NeurIPS 2020.

[3] Guo et al. Dynamic Task Prioritization for Multitask Learning. ECCV 2018.

[4] Cho et al. Hard Tasks First: Multi-Task RL Through Task Scheduling. ICML 2024.

[5] Kendall et al. Multi-Task Learning Using Uncertainty to Weigh Losses. CVPR 2018.

[6] Rajeswaran et al. Learning Complex Dexterous Manipulation with Deep RL and Demonstrations. RSS 2018.

[7] Nair et al. Overcoming Exploration in RL with Demonstrations. ICRA 2018.

[8] Zhao et al. Adaptive Behavior Cloning Regularization for Stable Offline-to-Online RL. ESANN 2022.

[9] Peng et al. DeepMimic. ACM TOG 2018.

[10] Peng et al. AMP: Adversarial Motion Priors. ACM TOG 2021.

[11] Tao et al. Reverse Forward Curriculum Learning for Extreme Sample and Demo Efficiency. ICLR 2024.

[12] Ball et al. Efficient Online Reinforcement Learning with Offline Data. ICML 2023.

AuthorsOpenReview ↗

Response 2/4: Same-placement comparison with FAMO (Weakness 2)

Thank you for this valuable suggestion; a same-placement comparison is indeed needed to attribute the gain correctly. We added a critic-only IW-ABC run (added to Table 2) so that the weighting placement matches FAMO-ABC:

MethodWeighting placementMean SRLong SR
FAMO-ABCcritic82.3±2.953.3±5.8
IW-ABCcritic88.1±5.466.7±11.5
IW-ABCactor+critic90.1±3.870.0±10.0
  • Same-placement attribution. With weighting on the critic in both cases, IW-ABC exceeds FAMO-ABC by 5.8 / 13.4 points (Mean / Long SR). Applying IW to the actor as well adds a further 2.0 / 3.3 points, mostly on long-horizon tasks.

  • Why FAMO has no actor+critic version. Our FAMO-ABC baseline applies FAMO's task weighting to the critic losses only, adapting FAMO's actor–critic (SAC-based) RL setting to our PPO framework (Appendix B.4 of our paper). FAMO estimates task progress from changes in log losses, so it needs positive losses. The value-regression loss satisfies this after adding ε, but the unmodified PPO policy surrogate can take either sign. Applying FAMO to it would require constructing an extra positive loss, and the offset choice would affect FAMO's progress estimates and effective weights. We therefore report the critic-only comparison separately from the full actor+critic IW-ABC, so the effects of the weighting rule and of its placement can be assessed independently. We will include both comparisons and this explanation in the revised paper.

Reference [1] Liu et al. FAMO: Fast Adaptive Multitask Optimization. NeurIPS 2023.

AuthorsOpenReview ↗

Response 3/4: Generalization and real-robot evidence (Weakness 3)

Thank you for pointing out these limitations. We respond to the three aspects of this weakness below.

3.1 Unseen tasks. We agree this is a limitation. Simulation experiments evaluate the 40 jointly trained tasks, and the real-robot experiment uses a separately trained four-task policy to test sim-to-real transfer on those tasks. This work targets efficient joint learning and task balancing over a predefined task set, and the predefined task encoding limits direct extension to new tasks. We will state this explicitly in the appendix and explore language- or goal-conditioned representations, stronger architectures, and pretrained perception models in future work.

3.2 Initial-state generalization (new). On known tasks, we expand the object initial XY sampling box around its center to 2× and 3× the base evaluation range, and separately add robot joint initial noise (same protocol as the paper: 50 episodes, 3 training seeds). We will add a section in appendix to discuss the result.

ConditionObject XY rangeJoint noise (rad)IW-ABC Mean SR (%)Vis IW-ABC Mean SR (%)
Base1×090.1±3.893.5±2.6
Larger object range2×088.5±3.380.6±2.0
Larger object range3×074.3±2.960.4±2.4
Joint noise only1×0.0190.0±3.093.4±2.2
Joint noise only1×0.0290.4±3.193.1±2.5

The visual policy degrades more than the state policy as the object layout range grows, while the RL-trained policy tolerates small changes in initial joint configuration.

This adds within-task distribution evidence; unseen-task generalization remains unverified, and we will distinguish the two clearly in the revision.

3.3 More real-robot experiments (new). To show cross-embodiment transfer and higher task difficulty, we built a six-task sim-to-real experiment on a UR12e arm with a Robotiq 2F-85 gripper. The tasks differ in semantics and precision requirements and are not only pick-and-place. We implemented the task definitions and MDP interfaces in simulation and collected 50 motion-planned expert trajectories per task. Given time and compute limits, we compare IW-ABC only with the strongest baseline, FAMO-ABC, under identical training and deployment (20 trials per task):

TaskFAMO-ABCIW-ABC
Click Bell20/2020/20
Shake Bottle18/2019/20
Move Pill Bottle onto Pad18/2018/20
Place Container on Plate20/2019/20
Put Apple onto Plate11/2015/20
Stack Three Cubes8/2012/20
Total95/120 (79.2%)103/120 (85.8%)

Videos: anonymous link. We will add these results to Sec. 5.4. We thank you again for pushing us to broaden the evaluation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Dear Reviewers and Area Chair,

Thank you for the time and care you have devoted to reviewing our manuscript. Your comments helped us identify where additional evidence and clearer explanations were needed. We have revised the manuscript with additional controlled comparisons, ablations, evaluation diagnostics, and real-robot experiments. The main revisions and their locations are summarized below.

  • Clarified the contributions and positioning relative to prior work

    We sharpened the distinction between Hebero as a heterogeneous GPU-parallel benchmark, DGPO as a shared demonstration-guided training framework, and IW-ABC as its reference learning recipe. We expanded comparisons with ManiSkill3 and MTBench and discussions of Dynamic Task Prioritization, Hard Tasks First, and adaptive BC regularization, clarifying the shared success-rate signal, loss weighting versus interaction allocation, and differences in training regime and demonstration usage.

    (Section 1, pp. 1–2; Sections 2.1–2.2, p. 3; corresponding bibliography entries)

  • Added a matched-placement comparison to separate the weighting rule from its placement

    We added critic-only IW-ABC whose weighting placement matches FAMO-ABC. The main results table now explicitly identifies the weighting placement for each learner. In addition, we explain why our PPO adaptation of FAMO applies weighting to the critic-regression loss only.

    (Section 5.2 and Table 2, pp. 8–9; Appendix B.4, pp. 20–21; Appendix D.1 and Table 6, p. 24)

  • Added ablations of success-adaptive ABC and the shared DGPO training components

    We added the three requested ablations: replacing success-adaptive ABC with time-only annealing, removing the demonstration-tracking reward, and removing the critic’s reference-relative errors. Time-only annealing reduces Mean SR from 90.1% to 86.1% and Long SR from 70.0% to 56.7%. Removing either shared component yields zero success under the evaluated configuration and training budget.

    (Section 5.2, p. 8; Section 5.3, p. 9; Appendix D.2 and Table 7, pp. 24–25)

  • Made the evaluation protocol and interpretation of Long SR more explicit

    We now specify that evaluation uses 50 episodes per task with a deterministic policy. We added a dedicated appendix subsection reporting per-task Long SR for three independently trained state-input IW-ABC policies. It explains the observed 0%/100% task-level outcomes, and the resulting discrete suite averages.

    (Section 3.4, p. 5; Section 5.2, p. 8; Appendix D.5 and Table 9, p. 26)

  • Added cross-simulator demonstration-replay diagnostics

    We evaluated both OSC-pose action replay and joint-space replay of the original LIBERO demonstrations in Isaac Lab, obtaining overall replay success rates of 32.3% and 80.5%, respectively. The new analysis documents control and contact-related deviations and clarifies how DGPO uses demonstrations despite the MuJoCo-to-PhysX gap. We explicitly state that all 50 demonstrations per task are retained without replay-based filtering.

    (Section 3.2, p. 4; Appendix A.5 and Table 4, pp. 18–19)

  • Expanded initial-state generalization experiments and clarified their scope

    We added evaluations with object-XY sampling ranges expanded to 2× and 3× the base range, together with separate tests of initial joint noise at 0.01 and 0.02 rad. These results quantify robustness within known tasks and reveal greater sensitivity of the visual policy to object-layout shifts. We also explicitly state that generalization to unseen tasks remains untested.

    (Section 5.3, p. 9; Appendix D.4 and Table 8, pp. 25–26; Section 6, p. 10)

  • Extended real-world evaluation with a second robot platform and a competing baseline

    In addition to the existing four-task Piper deployment, we added a six-task study on a UR12e with a Robotiq 2F-85 gripper. Figure 6 now presents both platforms and the per-task comparison, and Appendix E includes the UR12e observation layout. We updated the Abstract, Introduction, and Conclusion to reflect the expanded evidence and clarify that the robot policies are trained separately.

    (Section 5.4 and Figure 6, p. 10; Appendix E, p. 26; Abstract, p. 1; Section 1, pp. 1–2; Section 6, p. 10)

We hope these additions and clarifications, considered together with our point-by-point rebuttal responses, provide a clearer basis for evaluating the contributions, empirical evidence, and scope of our work. We would be grateful for your consideration of the revised manuscript.

Thank you again for your time and constructive feedback.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.