- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces RoboMARL-Bench, a benchmark for offline multi-agent reinforcement learning (OMARL) in robotic manipulation. The main motivation is that a dataset’s quality label alone doesn’t show what agents can learn from it, because coordination also depends on which agent produced the good behavior and when. The paper makes three contributions. First, it offers 65 task configurations (PyBullet, MuJoCo and discrete) grouped into seven interaction families defined by environment rules, such as Cooperation, Competition, Multi-Goal and Transition, each with its own evaluation metric. Second, it provides eight data constructions across five layers (Source, Episode, Role, Temporal and Skill), so that within the same task and data budget one can vary data quality, episode mixing, which agent gets which data source, and the order in which controllers act. Third, it evaluates seven continuous-action methods plus discrete and goal-conditioned baselines across these conditions. The authors report that no single method leads every family, though OMIGA is strongest overall.
Critical strengths
-
The main contribution is the eight data constructions across the Source, Episode, Role, Temporal and Skill layers. They let the user change how offline experience is put together while keeping the task and the transition budget the same. Existing benchmarks such as OG-MARL and D4RL cannot isolate role and timing effects at a matched budget. This is a new axis of evaluation that changes how future OMARL results should be reported and read.
-
Evaluation can be broken down by the interaction type with interesting findings. The seven rule-defined families show what a single aggregate score would hide. No method leads every family. OMIGA is strongest overall but weak on Competition, and ITD3+BC leads on the Hybrid tasks with Medium data. The learning-gain analysis then finds a concrete open problem: on data from suboptimal policies, the best methods beat the recorded success rate on only about half the tasks, mostly failing on Multi-Goal and Transition.
Critical weaknesses
-
The paper discusses data quality and provides datasets in three different qualities: Good, Medium and Poor. Despite these being useful, a more detailed description and quantitative characterization of data quality would make it more useful. It would be interesting to see metrics for each dataset, for example, trajectory length, control energy for trajectories, diversity, among others.
-
A correlation between performance in sim and real is not stablished. I believe a benchmark is as useful as informing how a policy will perform in real given performance in sim. Without this analysis, it is difficult to assess the impact of this benchmark and datasets. At the very least, one would expect that the ranking of policies in sim and real would be similar.
Questions for authors
- Which data pool, observation setting and checkpoint were used for each deployed policy?
- What was the simulated success of those exact checkpoints? Without this, Table 2 cannot be read against the simulation results. In particular, the method ranking flips between simulation and the real robot for Lift, Handover and Connect.
- For families with only 2–5 tasks (Hybrid, Intervention, one-shot Trajectory-Tracking), can you give confidence intervals or rank stability across seeds and tasks, to show that the claimed winners in each family are reliable?
5 comments
2/4
Part 2/4
W2: Correlation between simulated and real-robot performance, including whether method rankings agree
For the deployed checkpoints, on the tasks where simulation separates the methods, the rankings are the same or differ only by margins that are not statistically significant (see Q2 for each task). With only 30 hardware trials per policy, the statistical precision of this conclusion is limited. A.1 gives confidence intervals and significance tests for the hardware results and describes the systematic differences between hardware and simulation (for example, the hardware executes joint targets through MoveIt planning) and how we handle them (domain randomization and a fixed end-effector orientation).
The conclusions of the paper about data composition and interaction structure come from matched comparisons in simulation (same pools, same budget, five seeds per condition). The real-robot part is a deployment study, which tests whether policies trained in simulation can run on two Franka Panda arms through the same interface without real-robot data or fine-tuning. The same interface can also store real-robot episodes in the benchmark's data format, which provides a basis for studying the correlation between simulation and hardware on more tasks.
Q1: Data pool, observation setting and checkpoint of each deployed policy
The table below gives, for the 12 deployed policies (4 tasks × 3 methods), the data pool, the observation setting, the simulated success of the deployed checkpoint and the real-robot result.
- Data pool. All policies were trained only on Good data collected in simulation, and no real-robot data or fine-tuning was used for deployment. For every task except Reach, the Good data were collected again in simulation in a configuration that supports domain randomization, with the same pipeline and the same observation types as the benchmark pools. Most deployed checkpoints were trained with domain randomization.
- Observation setting. The hardware tasks keep the observation layout of the benchmark tasks, Relative for Reach and Lift and Partial for Handover and Connect-Insert. Compared with simulation, only the objects, their placement range and the success tolerance were adapted to the physical setup.
- Checkpoint. Each policy deploys one checkpoint saved during training, and the simulated success in the table was measured over 100 simulated evaluation episodes when it was saved. The intervals after the hardware results are binomial 95% intervals, as in Table 3 of A.1.
Reply table. The 12 deployed policies with data pool, observation, simulated and real-robot results (hardware intervals as in Table 3 of A.1).
| Task | Method | Data pool | Observation | Simulation, deployed checkpoint (%) | Real robot (successes of 30) | Real robot 95% CI (%) |
|---|---|---|---|---|---|---|
| Reach | MABC | Good | Relative | 100 | 30 | [89, 100] |
| Reach | OMIGA | Good | Relative | 100 | 30 | [89, 100] |
| Reach | InSPO | Good | Relative | 100 | 30 | [89, 100] |
| Lift | MABC | Good | Relative | 100 | 30 | [89, 100] |
| Lift | OMIGA | Good | Relative | 100 | 23 | [59, 88] |
| Lift | InSPO | Good | Relative | 100 | 24 | [63, 90] |
| Handover | MABC | Good | Partial | 63 | 11 | [22, 54] |
| Handover | OMIGA | Good | Partial | 77 | 17 | [39, 73] |
| Handover | InSPO | Good | Partial | 93 | 18 | [42, 75] |
| Connect-Insert | MABC | Good | Partial | 92 | 11 | [22, 54] |
| Connect-Insert | OMIGA | Good | Partial | 83 | 16 | [36, 70] |
| Connect-Insert | InSPO | Good | Partial | 100 | 18 | [42, 75] |
Details of the deployment setup are given in A.1.
3/4
Part 3/4
Q2: Simulated success of the deployed checkpoints, and whether rankings flip between simulation and hardware
The deployed checkpoints show no significant ranking flip between simulation and hardware. The flip mentioned in the review appears when the real-robot results (Table 2) are compared with the main-experiment results (Table 4 in A.6), but these two tables come from different policies, and the real-robot Connect-Insert corresponds to Insert in the main experiments, not to Connect. In Table 2, the number before the parentheses is the number of successes in 30 hardware trials, and the number in parentheses is the simulated success of the deployed checkpoint (%, 100 evaluation episodes).
Table 2 (main text). Real-robot successes and simulated success of the deployed checkpoints.
| Task | MABC | OMIGA | InSPO |
|---|---|---|---|
| Reach | 30/30 (100) | 30/30 (100) | 30/30 (100) |
| Lift | 30/30 (100) | 23/30 (100) | 24/30 (100) |
| Handover (partial) | 11/30 (63) | 17/30 (77) | 18/30 (93) |
| Connect-Insert (partial) | 11/30 (92) | 16/30 (83) | 18/30 (100) |
The same checkpoints in simulation and on hardware. All p-values below come from pairwise comparisons of the success rates of two methods (Fisher's exact test).
- Reach: all three checkpoints reach 100% in simulation and 30/30 on hardware.
- Lift: all three checkpoints reach 100% in simulation, so there is no simulated ranking that could flip. On hardware, MABC is better than OMIGA (p = 0.011) and InSPO (p = 0.024). This difference arises in the transfer to hardware, where joint targets are executed through MoveIt planning and the joint limits and response speed also differ from simulation (A.1).
- Handover: the order is the same in simulation and on hardware, InSPO, OMIGA, MABC (93%, 77% and 63% vs 18, 17 and 11 of 30).
- Connect-Insert: InSPO is first in both (100% and 18/30). MABC and OMIGA exchange ranks, but their gap is not significant in simulation (92% vs 83%, p = 0.09) or on hardware (11/30 vs 16/30, p = 0.30), so this is not a ranking flip.
Differences from the main experiments. The main-experiment results (Table 4) and the real-robot results come from different policies, which differ in three respects. First, the main-experiment policies were trained on the benchmark pools without domain randomization, whereas the deployed policies were trained separately for the hardware (see Q1). Second, the main experiments mostly use Relative observations, whereas the deployed Handover and Connect-Insert policies use Partial observations. Third, the real-robot Connect-Insert corresponds to Insert in the main experiments, not to Connect (A.1). With Good data in the main experiments, InSPO leads on Lift (97.0%, OMIGA 84.8%, MABC 81.6%), MABC and OMIGA lead on Handover (99.2% and 98.8%, InSPO 82.8%), OMIGA and MABC lead on Connect (91.4% and 89.6%, InSPO 79.0%), and on Insert the three methods are within 2 pp (MABC 92.8%, InSPO 91.4%, OMIGA 90.8%). The ranking differences between Table 2 and Table 4 therefore come from comparing different policies and tasks, not from the same policies changing rank from simulation to hardware.
Table 4 in A.6, which was already in the submitted version, gives these results (Good data, Final success in %, mean ± standard deviation over five seeds).
Table 4 (A.6, excerpt). Main-experiment results on these tasks.
| Task (Good data) | MABC | ITD3+BC | ICQL | MABCQ | IICQ | OMIGA | InSPO |
|---|---|---|---|---|---|---|---|
| Lift | 81.6 ± 3.9 | 85.6 ± 19.4 | 86.0 ± 3.1 | 39.4 ± 6.0 | 81.8 ± 5.8 | 84.8 ± 4.0 | 97.0 ± 1.2 |
| Handover | 99.2 ± 0.8 | 99.8 ± 0.4 | 77.6 ± 3.6 | 42.2 ± 18.7 | 99.0 ± 0.7 | 98.8 ± 1.1 | 82.8 ± 17.5 |
| Connect | 89.6 ± 3.2 | 93.6 ± 2.9 | 85.4 ± 9.2 | 80.6 ± 3.7 | 91.8 ± 2.6 | 91.4 ± 3.1 | 79.0 ± 20.1 |
| Insert | 92.8 ± 4.3 | 45.4 ± 24.7 | 82.2 ± 2.8 | 62.8 ± 6.5 | 92.4 ± 2.9 | 90.8 ± 5.2 | 91.4 ± 5.9 |
1/4
Part 1/4
Thank you for the detailed review and for recognizing the data constructions as a new axis of evaluation. As you summarized, the starting point of the paper is that a quality label alone does not show what agents can learn from the data, because coordination also depends on which agent produced the competent behavior and when. RoboMARL-Bench therefore provides 65 task configurations grouped by environment rules into seven interaction families, together with eight data constructions in five layers that change the data composition within a task at a matched budget. The findings you mentioned, that no single method leads every family and that on data from suboptimal policies even the best methods exceed the recorded success on only about half of the tasks, come from this evaluation by interaction structure and data composition. Among the benchmarks compared in Table 1, including OG-MARL and D4RL, only RoboMARL-Bench deploys policies trained in simulation on real robots. A.1 describes the pipeline from simulation to hardware and how we handle the differences (domain randomization, planner differences and an end-effector orientation constraint), and the same interface can store real-robot episodes in the benchmark's data format for future sim-to-real studies. In response to your comments, we added statistics for every data pool, the full information and simulated success of every deployed policy, and confidence intervals and rank stability for the winner of each family, and we improved the readability of the main text and the appendices. If these revisions address your concerns, we would be grateful if you would take them into account in your final rating. Below, we first list the changes in the revision and then respond to each point.
Table 1 (excerpt). Whether each benchmark reports real-robot results.
Changes in the revision addressing your comments
W1: Quantitative characterization of data quality with per-dataset metrics
Following your suggestion, we added these metrics for every released data pool in the new A.29. The submitted version already contained part of this characterization. A.20 reports, per quality level, the action coverage and state–action coverage of the pools of each main task, and the learning-gain analysis in Sec. 5.3 uses the recorded success of every Medium pool. A.29 extends this to all pools and lists, for each pool, the numbers of episodes and transitions, the mean and median trajectory length, the termination rate, the recorded success, the mean return, the control energy (mean squared action), the action smoothness (mean squared difference between consecutive actions), the action entropy, the state diversity, and the coverage of the Good reference states (computed as in A.20). State diversity measures how spread out the states are. For each state we take the distance to its fifth nearest neighbor and average the logarithm of these distances, so that larger values mean more spread-out states. The states are first standardized with the statistics of the task's Good pool.
Table 42 (A.29). Median statistics of the main pools by quality level.