- Rating
- 3(+2)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
-
This paper introduces Hebero, a GPU-parallel benchmark for jointly training and evaluating a single policy on 40 heterogeneous robotic manipulation tasks.
-
The authors further propose DGPO and IW-ABC, which use demonstrations and task-level learning progress to adapt imitation guidance and prioritize underperforming tasks during PPO training.
-
Experiments show that IW-ABC achieves 90.1% state-based and 93.5% visual mean success rates, while the simulation-trained policy obtains 82.5% success on four real-robot tasks.
Critical strengths
The paper presents a scalable GPU-parallel multi-task RL benchmark and an effective demonstration-guided training method, supported by strong simulation results and real-robot transfer.
Critical weaknesses
-
The paper combines many components, but the individual algorithmic contributions appear largely incremental, making the core technical novelty difficult to identify.
-
The evaluation is restricted to adapted LIBERO tasks without held-out-task testing, so the generalization of Hebero and IW-ABC to broader task distributions remains unclear.
-
The real-world evaluation covers only four tasks with 20 trials per task and lacks competing baselines, providing limited evidence for robust sim-to-real transfer.
-
Given the unusually broad use of generative AI in experimental design, implementation, analysis, and writing, the paper should disclose the AI-assisted components and their independent verification procedures more precisely.
Questions for authors
-
Could the authors clarify the key technical novelty of IW-ABC and explain how it differs fundamentally from existing demonstration-guided and task-weighting methods?
-
How well do Hebero and IW-ABC generalize to unseen tasks, different task distributions, or robotic environments beyond the adapted LIBERO suite?
-
Could the authors provide stronger real-world evaluations, including more tasks and trials as well as comparisons with relevant baselines under the same deployment setting?
-
Could the authors specify which parts of the experimental design, implementation, analysis, and manuscript preparation involved generative AI, and describe how each AI-assisted artifact was independently verified?
5 comments
Response 2/3: Technical novelty (Weakness 1; Q1)
Thank you for this important question.
First, we would like to restate that our main contributions are a scalable GPU-parallel heterogeneous multi-task RL benchmark and a training framework for fair comparison of demonstration-guided methods; IW-ABC is the reference algorithm within it. We do not claim that performance-based task prioritization is new, and we will clarify how IW and ABC differ from prior work.
What IW-ABC adds is one signal used twice: DGPO maintains a single per-task online signal and uses it both to weight PPO gradients across tasks (IW) and to set demonstration-guidance strength (ABC). Below we compare each use with prior work and point to the supporting evidence.
2.1 IW vs. multi-task optimization. By feedback signal, prior methods fall into three groups; IW belongs to M2.
| Category | Signal | Prior work | |
|---|---|---|---|
| M1 | Loss/gradient-based | Loss improvement, gradient relations | FAMO [1]; PCGrad [2] |
| M2 | Performance/difficulty-based | Task metrics, interaction-based difficulty | DTP [3]; Hard Tasks First [4]; IW (ours) |
| M3 | Uncertainty-based | Learned task noise | Kendall et al. [5] |
- DTP [3] weights supervised losses by task KPIs. IW extends this to continuous-control online RL, using the EMA of episode success rates from training rollouts to assign bounded PPO weights.
- Hard Tasks First [4] also favors hard tasks, but reallocates the interaction budget using return/entropy-based difficulty. IW keeps each task's rollout allocation fixed (a fixed number of environment replicas per task, so all methods use the same sampling budget) and reweights the PPO objective instead.
- FAMO [1] uses critic loss improvement; our success-rate signal measures task completion directly, which does not necessarily move in step with the progress of critic fitting. With weighting on the critic in both cases, IW-ABC outperforms FAMO-ABC by 5.8 / 13.4 points (Mean / Long SR).
We will add these references and positioning to Related Work.
2.2 ABC vs. demonstration-guided RL. Methods grouped by how demonstrations enter training (bold = compared in Sec. 4.2):
| Use of demos | Prior work | In DGPO | |
|---|---|---|---|
| D1 | Policy initialization | BC init. in DAPG [6] | BC→PPO |
| D2 | Policy regularization (scheduled, filtered, or adaptive) | DAPG [6]; Q-filtered BC [7]; Adaptive BC [8] | DAPG: offline demo (obs, action) pairs. ABC (ours): online observations paired with cursor-aligned demo actions; strength set by success-rate EMA |
| D3 | Demo-based reward | DeepMimic [9]; AMP [10] | Tracking reward, shared by all methods |
| D4 | Reset/exploration curriculum | RFCL [11] | RFCL-inspired reset curriculum on the shared PPO backbone |
| D5 | Demo replay, off-policy | RLPD [12] | Out of scope (on-policy PPO comparison) |
- DGPO places the demonstration-based tracking reward (D3) in the shared framework, and provides representative, method-specific demonstration usages from D1, D2, and D4 for a controlled comparison. The compared methods share the PPO backbone, reward, observations, training budget, and evaluation protocol.
- With the same IW weighting, ABC (D2) achieves the highest mean and Long success in our state-input evaluation.
- The success-rate term in ABC contributes measurably: replacing it with time-only annealing (all else unchanged) lowers Mean SR from 90.1 to 86.1 and Long SR from 70.0 to 56.7.
In summary, IW-ABC differs clearly from prior work on both sides: IW differs from existing multi-task optimization methods in its signal (task success rate, not loss or gradients) and in where it acts (PPO weights under fixed rollout allocation, not interaction budget), and ABC differs from existing demonstration-guided methods in how demonstrations are used (online observations paired with cursor-aligned demonstration actions, with success-adaptive strength).
References
[1] Liu et al. FAMO: Fast Adaptive Multitask Optimization. NeurIPS 2023.
[2] Yu et al. Gradient Surgery for Multi-Task Learning. NeurIPS 2020.
[3] Guo et al. Dynamic Task Prioritization for Multitask Learning. ECCV 2018.
[4] Cho et al. Hard Tasks First: Multi-Task RL Through Task Scheduling. ICML 2024.
[5] Kendall et al. Multi-Task Learning Using Uncertainty to Weigh Losses. CVPR 2018.
[6] Rajeswaran et al. Learning Complex Dexterous Manipulation with Deep RL and Demonstrations. RSS 2018.
[7] Nair et al. Overcoming Exploration in RL with Demonstrations. ICRA 2018.
[8] Zhao et al. Adaptive Behavior Cloning Regularization for Stable Offline-to-Online RL. ESANN 2022.
[9] Peng et al. DeepMimic. ACM TOG 2018.
[10] Peng et al. AMP: Adversarial Motion Priors. ACM TOG 2021.
[11] Tao et al. Reverse Forward Curriculum Learning for Extreme Sample and Demo Efficiency. ICLR 2024.
[12] Ball et al. Efficient Online Reinforcement Learning with Offline Data. ICML 2023.
Response 3/3: Generalization and real-world evaluation (Weaknesses 2, 3; Q2, Q3)
Thank you for pointing out these limitations. We respond to the generalization question first and then the real-world evaluation.
3.1 Varied robot environments (Weakness 2; Q2). Besides the LIBERO suites, our paper also adapts RoboTwin tasks for real-robot deployment; these differ from LIBERO in task types and robot embodiments.
3.2 Unseen tasks. We acknowledge this limitation: our experiments do not yet verify generalization to unseen tasks. Simulation experiments evaluate the 40 jointly trained tasks, and the real-robot experiment uses a separately trained four-task policy to test sim-to-real transfer on those tasks. This work focuses on efficient joint learning and task balancing over a predefined task set, and the predefined task encoding limits direct extension to new tasks.
3.3 Different task distributions (new). We evaluated generalization on known tasks under shifted initial-state distributions. We expand the object initial XY sampling box, around its center, to 2× and 3× the base evaluation range, and separately test robot joint initial noise (supplementary Figure 10; same protocol as the paper: 50 episodes, 3 training seeds).
| Condition | Object XY range | Joint noise (rad) | IW-ABC Mean SR (%) | Vis IW-ABC Mean SR (%) |
|---|---|---|---|---|
| Base | 1× | 0 | 90.1±3.8 | 93.5±2.6 |
| Larger object range | 2× | 0 | 88.5±3.3 | 80.6±2.0 |
| Larger object range | 3× | 0 | 74.3±2.9 | 60.4±2.4 |
| Joint noise only | 1× | 0.01 | 90.0±3.0 | 93.4±2.2 |
| Joint noise only | 1× | 0.02 | 90.4±3.1 | 93.1±2.5 |
- The visual policy degrades more than the state policy as the object layout range grows.
- The RL-trained policy tolerates small changes in the robot's initial joint configuration.
In future work, we will explore language- or goal-conditioned representations, stronger policy architectures, and pretrained perception models for systematic evaluation.
The revision will clearly distinguish these kinds of generalization and state the limits of the current experimental scope and task encoding.
3.4 More real-world evaluations (Weakness 3; Q3). Constrained by the robot embodiments available to us, we designed an additional six-task sim-to-real experiment on a UR12e arm with a Robotiq 2F-85 gripper. It shows cross-embodiment transfer and raises task difficulty: the tasks differ clearly in semantics and precision requirements and are not only pick-and-place. The task definitions and MDP interfaces for joint learning were straightforward to implement in simulation, and we collected 50 motion-planned expert trajectories per task. Given time and compute limits, we compare IW-ABC only with the strongest baseline, FAMO-ABC, under identical training and deployment (20 trials per task):
| Task | FAMO-ABC | IW-ABC |
|---|---|---|
| Click Bell | 20/20 | 20/20 |
| Shake Bottle | 18/20 | 19/20 |
| Move Pill Bottle onto Pad | 18/20 | 18/20 |
| Place Container on Plate | 20/20 | 19/20 |
| Put Apple onto Plate | 11/20 | 15/20 |
| Stack Three Cubes | 8/20 | 12/20 |
| Total | 95/120 (79.2%) | 103/120 (85.8%) |
Videos: anonymous link. We will add these results to Sec. 5.4. Thank you again for these suggestions.
Response 1/3: AI use statement (Weakness 4; Q4)
Thank you for your careful review. We address the scope of AI assistance and how it was verified first.
Generative AI was used mainly for code debugging and polishing the wording of the paper. All final decisions on the research design, experimental design, and analysis of results were made by the authors. All of our code is included in the supplementary material, and we plan to open-source all datasets and the code framework. We did not use any automated-research pipeline for the paper's main experiments or for constructing the method. AI was used in two places:
We will state this scope explicitly in the revised AI use statement.