acceptodds
Under review as a conference paper at ICLR 2027

RoboMARL-Bench: How Task Structure and Data Composition Matter in Offline Multi-Agent Robotic Manipulation

Abstract

Offline multi-agent reinforcement learning (OMARL) requires agents to learn coordinated behavior from a fixed dataset. Yet task performance and overall dataset quality alone do not reveal how the mix of coordination experiences in the dataset affects what agents can learn. We introduce RoboMARL-Bench, a robotics-oriented benchmark with 65 task configurations, 7 rule-defined interaction structures, and 8 controlled data constructions spanning the source, episode, role, temporal, and skill layers. This design enables matched-budget comparisons within the same task while varying how experience is collected and organized. Across extensive evaluations, we find that algorithm rankings change across interaction structures, datasets with the same nominal quality can yield substantially different outcomes, and role assignment and controller ordering introduce additional task-dependent variation. Current OMARL methods also leave clear room for improvement: on data collected by suboptimal policies, they exceed the recorded data level on only about half of the tasks, and most often fall short on multi-goal and transition tasks. RoboMARL-Bench provides a controlled testbed for studying how interaction structure and offline data composition jointly shape multi-agent learning.

Primary area:
datasets and benchmarks
Keywords:
Offline Multi-Agent Reinforcement Learning, Multi-Robot Systems, Benchmarks, Data Analysis, Structure Study

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6RZQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces RoboMARL-Bench, a benchmark for offline multi-agent reinforcement learning (OMARL) in robotic manipulation. The main motivation is that a dataset’s quality label alone doesn’t show what agents can learn from it, because coordination also depends on which agent produced the good behavior and when. The paper makes three contributions. First, it offers 65 task configurations (PyBullet, MuJoCo and discrete) grouped into seven interaction families defined by environment rules, such as Cooperation, Competition, Multi-Goal and Transition, each with its own evaluation metric. Second, it provides eight data constructions across five layers (Source, Episode, Role, Temporal and Skill), so that within the same task and data budget one can vary data quality, episode mixing, which agent gets which data source, and the order in which controllers act. Third, it evaluates seven continuous-action methods plus discrete and goal-conditioned baselines across these conditions. The authors report that no single method leads every family, though OMIGA is strongest overall.

Critical strengths
  1. The main contribution is the eight data constructions across the Source, Episode, Role, Temporal and Skill layers. They let the user change how offline experience is put together while keeping the task and the transition budget the same. Existing benchmarks such as OG-MARL and D4RL cannot isolate role and timing effects at a matched budget. This is a new axis of evaluation that changes how future OMARL results should be reported and read.

  2. Evaluation can be broken down by the interaction type with interesting findings. The seven rule-defined families show what a single aggregate score would hide. No method leads every family. OMIGA is strongest overall but weak on Competition, and ITD3+BC leads on the Hybrid tasks with Medium data. The learning-gain analysis then finds a concrete open problem: on data from suboptimal policies, the best methods beat the recorded success rate on only about half the tasks, mostly failing on Multi-Goal and Transition.

Critical weaknesses
  1. The paper discusses data quality and provides datasets in three different qualities: Good, Medium and Poor. Despite these being useful, a more detailed description and quantitative characterization of data quality would make it more useful. It would be interesting to see metrics for each dataset, for example, trajectory length, control energy for trajectories, diversity, among others.

  2. A correlation between performance in sim and real is not stablished. I believe a benchmark is as useful as informing how a policy will perform in real given performance in sim. Without this analysis, it is difficult to assess the impact of this benchmark and datasets. At the very least, one would expect that the ranking of policies in sim and real would be similar.

Questions for authors
  1. Which data pool, observation setting and checkpoint were used for each deployed policy?
  2. What was the simulated success of those exact checkpoints? Without this, Table 2 cannot be read against the simulation results. In particular, the method ranking flips between simulation and the real robot for Lift, Handover and Connect.
  3. For families with only 2–5 tasks (Hybrid, Intervention, one-shot Trajectory-Tracking), can you give confidence intervals or rank stability across seeds and tasks, to show that the claimed winners in each family are reliable?
5 comments

Part 1/4

Thank you for the detailed review and for recognizing the data constructions as a new axis of evaluation. As you summarized, the starting point of the paper is that a quality label alone does not show what agents can learn from the data, because coordination also depends on which agent produced the competent behavior and when. RoboMARL-Bench therefore provides 65 task configurations grouped by environment rules into seven interaction families, together with eight data constructions in five layers that change the data composition within a task at a matched budget. The findings you mentioned, that no single method leads every family and that on data from suboptimal policies even the best methods exceed the recorded success on only about half of the tasks, come from this evaluation by interaction structure and data composition. Among the benchmarks compared in Table 1, including OG-MARL and D4RL, only RoboMARL-Bench deploys policies trained in simulation on real robots. A.1 describes the pipeline from simulation to hardware and how we handle the differences (domain randomization, planner differences and an end-effector orientation constraint), and the same interface can store real-robot episodes in the benchmark's data format for future sim-to-real studies. In response to your comments, we added statistics for every data pool, the full information and simulated success of every deployed policy, and confidence intervals and rank stability for the winner of each family, and we improved the readability of the main text and the appendices. If these revisions address your concerns, we would be grateful if you would take them into account in your final rating. Below, we first list the changes in the revision and then respond to each point.

Table 1 (excerpt). Whether each benchmark reports real-robot results.

Row of Table 1D4RLOG-MARLPyMARLzoo+OGBenchBiGymMangoBenchRoboFactoryRoboMARL-Bench
Real-robot results✗✗✗✗✗✗✗✓

Changes in the revision addressing your comments

  • New A.29: statistics of every released data pool, including trajectory length, termination, recorded success, return, control energy, action smoothness, action entropy, state diversity and coverage (W1).
  • Sec. 5.4 is retitled "Real-Robot Deployment", and Sec. 5.4 and Sec. 6 state the scope of the real-robot study (W2).
  • Table 2 in the main text now includes the simulated success of each deployed checkpoint. A.1 describes the training data and observation setting of each deployed policy and gives 95% intervals for the hardware results and Fisher's exact tests between methods (Q1, Q2).
  • New Table 18 in A.12: a two-level bootstrap over tasks and seeds for all 7 families, giving confidence intervals and rank stability for the winner of each family. Sec. 5.1 no longer names a single winner for Hybrid, Intervention and one-shot Trajectory-Tracking and refers to their means and intervals in A.12 (Q3).

W1: Quantitative characterization of data quality with per-dataset metrics

Following your suggestion, we added these metrics for every released data pool in the new A.29. The submitted version already contained part of this characterization. A.20 reports, per quality level, the action coverage and state–action coverage of the pools of each main task, and the learning-gain analysis in Sec. 5.3 uses the recorded success of every Medium pool. A.29 extends this to all pools and lists, for each pool, the numbers of episodes and transitions, the mean and median trajectory length, the termination rate, the recorded success, the mean return, the control energy (mean squared action), the action smoothness (mean squared difference between consecutive actions), the action entropy, the state diversity, and the coverage of the Good reference states (computed as in A.20). State diversity measures how spread out the states are. For each state we take the distance to its fifth nearest neighbor and average the logarithm of these distances, so that larger values mean more spread-out states. The states are first standardized with the statistics of the task's Good pool.

Table 42 (A.29). Median statistics of the main pools by quality level.

GoodMediumPoor
Mean episode length (steps)46.663.0120.0
Terminated episodes (%)96.753.40.0
Recorded success (%)96.851.20.0
Control energy1.091.011.00
Action smoothness0.761.072.00
Action entropy0.780.911.00
State diversity0.660.961.36
Coverage of Good reference states (%)91.830.23.2

Part 2/4

W2: Correlation between simulated and real-robot performance, including whether method rankings agree

For the deployed checkpoints, on the tasks where simulation separates the methods, the rankings are the same or differ only by margins that are not statistically significant (see Q2 for each task). With only 30 hardware trials per policy, the statistical precision of this conclusion is limited. A.1 gives confidence intervals and significance tests for the hardware results and describes the systematic differences between hardware and simulation (for example, the hardware executes joint targets through MoveIt planning) and how we handle them (domain randomization and a fixed end-effector orientation).

The conclusions of the paper about data composition and interaction structure come from matched comparisons in simulation (same pools, same budget, five seeds per condition). The real-robot part is a deployment study, which tests whether policies trained in simulation can run on two Franka Panda arms through the same interface without real-robot data or fine-tuning. The same interface can also store real-robot episodes in the benchmark's data format, which provides a basis for studying the correlation between simulation and hardware on more tasks.

Q1: Data pool, observation setting and checkpoint of each deployed policy

The table below gives, for the 12 deployed policies (4 tasks × 3 methods), the data pool, the observation setting, the simulated success of the deployed checkpoint and the real-robot result.

  • Data pool. All policies were trained only on Good data collected in simulation, and no real-robot data or fine-tuning was used for deployment. For every task except Reach, the Good data were collected again in simulation in a configuration that supports domain randomization, with the same pipeline and the same observation types as the benchmark pools. Most deployed checkpoints were trained with domain randomization.
  • Observation setting. The hardware tasks keep the observation layout of the benchmark tasks, Relative for Reach and Lift and Partial for Handover and Connect-Insert. Compared with simulation, only the objects, their placement range and the success tolerance were adapted to the physical setup.
  • Checkpoint. Each policy deploys one checkpoint saved during training, and the simulated success in the table was measured over 100 simulated evaluation episodes when it was saved. The intervals after the hardware results are binomial 95% intervals, as in Table 3 of A.1.

Reply table. The 12 deployed policies with data pool, observation, simulated and real-robot results (hardware intervals as in Table 3 of A.1).

TaskMethodData poolObservationSimulation, deployed checkpoint (%)Real robot (successes of 30)Real robot 95% CI (%)
ReachMABCGoodRelative10030[89, 100]
ReachOMIGAGoodRelative10030[89, 100]
ReachInSPOGoodRelative10030[89, 100]
LiftMABCGoodRelative10030[89, 100]
LiftOMIGAGoodRelative10023[59, 88]
LiftInSPOGoodRelative10024[63, 90]
HandoverMABCGoodPartial6311[22, 54]
HandoverOMIGAGoodPartial7717[39, 73]
HandoverInSPOGoodPartial9318[42, 75]
Connect-InsertMABCGoodPartial9211[22, 54]
Connect-InsertOMIGAGoodPartial8316[36, 70]
Connect-InsertInSPOGoodPartial10018[42, 75]

Details of the deployment setup are given in A.1.

Part 3/4

Q2: Simulated success of the deployed checkpoints, and whether rankings flip between simulation and hardware

The deployed checkpoints show no significant ranking flip between simulation and hardware. The flip mentioned in the review appears when the real-robot results (Table 2) are compared with the main-experiment results (Table 4 in A.6), but these two tables come from different policies, and the real-robot Connect-Insert corresponds to Insert in the main experiments, not to Connect. In Table 2, the number before the parentheses is the number of successes in 30 hardware trials, and the number in parentheses is the simulated success of the deployed checkpoint (%, 100 evaluation episodes).

Table 2 (main text). Real-robot successes and simulated success of the deployed checkpoints.

TaskMABCOMIGAInSPO
Reach30/30 (100)30/30 (100)30/30 (100)
Lift30/30 (100)23/30 (100)24/30 (100)
Handover (partial)11/30 (63)17/30 (77)18/30 (93)
Connect-Insert (partial)11/30 (92)16/30 (83)18/30 (100)

The same checkpoints in simulation and on hardware. All p-values below come from pairwise comparisons of the success rates of two methods (Fisher's exact test).

  • Reach: all three checkpoints reach 100% in simulation and 30/30 on hardware.
  • Lift: all three checkpoints reach 100% in simulation, so there is no simulated ranking that could flip. On hardware, MABC is better than OMIGA (p = 0.011) and InSPO (p = 0.024). This difference arises in the transfer to hardware, where joint targets are executed through MoveIt planning and the joint limits and response speed also differ from simulation (A.1).
  • Handover: the order is the same in simulation and on hardware, InSPO, OMIGA, MABC (93%, 77% and 63% vs 18, 17 and 11 of 30).
  • Connect-Insert: InSPO is first in both (100% and 18/30). MABC and OMIGA exchange ranks, but their gap is not significant in simulation (92% vs 83%, p = 0.09) or on hardware (11/30 vs 16/30, p = 0.30), so this is not a ranking flip.

Differences from the main experiments. The main-experiment results (Table 4) and the real-robot results come from different policies, which differ in three respects. First, the main-experiment policies were trained on the benchmark pools without domain randomization, whereas the deployed policies were trained separately for the hardware (see Q1). Second, the main experiments mostly use Relative observations, whereas the deployed Handover and Connect-Insert policies use Partial observations. Third, the real-robot Connect-Insert corresponds to Insert in the main experiments, not to Connect (A.1). With Good data in the main experiments, InSPO leads on Lift (97.0%, OMIGA 84.8%, MABC 81.6%), MABC and OMIGA lead on Handover (99.2% and 98.8%, InSPO 82.8%), OMIGA and MABC lead on Connect (91.4% and 89.6%, InSPO 79.0%), and on Insert the three methods are within 2 pp (MABC 92.8%, InSPO 91.4%, OMIGA 90.8%). The ranking differences between Table 2 and Table 4 therefore come from comparing different policies and tasks, not from the same policies changing rank from simulation to hardware.

Table 4 in A.6, which was already in the submitted version, gives these results (Good data, Final success in %, mean ± standard deviation over five seeds).

Table 4 (A.6, excerpt). Main-experiment results on these tasks.

Task (Good data)MABCITD3+BCICQLMABCQIICQOMIGAInSPO
Lift81.6 ± 3.985.6 ± 19.486.0 ± 3.139.4 ± 6.081.8 ± 5.884.8 ± 4.097.0 ± 1.2
Handover99.2 ± 0.899.8 ± 0.477.6 ± 3.642.2 ± 18.799.0 ± 0.798.8 ± 1.182.8 ± 17.5
Connect89.6 ± 3.293.6 ± 2.985.4 ± 9.280.6 ± 3.791.8 ± 2.691.4 ± 3.179.0 ± 20.1
Insert92.8 ± 4.345.4 ± 24.782.2 ± 2.862.8 ± 6.592.4 ± 2.990.8 ± 5.291.4 ± 5.9
Reviewer_72w8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces RoboMARL-Bench, a robotics-oriented benchmark for offline multi-agent RL. It has 65 task configurations grouped into seven interaction families, plus eight controlled data constructions that compare offline data compositions under a matched budget within the same task. The experiments suggest that OMARL performance depends jointly on interaction structure and data composition, beyond nominal data quality. A small dual-arm real-robot study is also included.

Critical strengths

The paper treats how offline data are composed as an explicit experimental variable, instead of relying only on coarse quality labels. Substantial engineering effort. The benchmark covers two physics platforms and a wide range of manipulation settings, and a real-robot interface. Results are averaged over five seeds, with sample standard deviations and t-based confidence intervals. Final and validation-selected Best checkpoints are reported separately.

Critical weaknesses

Stitch switches controllers at a fixed step 10, while task horizons range from 50 to over 500 steps. So G→R pools are mostly Random data and R→G pools are mostly Guided data, and Fig. 5A matches this (G→R is close to Random, R→G is close to Guided). The reported ordering effect may largely reflect the share of each source rather than the order itself. The realized source shares should be reported and controlled for. The “same quality label, different outcome” finding is partly built in. The Medium label is assigned by collection procedure, not calibrated to data quality. PretrainedSuboptimal and Mixed pools differ substantially in recorded success (32.1% vs. 69.5%), so different training outcomes are expected. A comparison between pools with matched recorded success but different composition is needed to support this claim. All methods, including OMIGA and InSPO, are trained as independent learners, with their cross-agent mechanisms removed (Appendix A.18). This weakens the central claim about how interaction structure shapes OMARL. The original CTDE versions should be included at least on cooperative families, along with representative coordination-based methods. The real-robot study adds limited evidence. As the authors acknowledge, the study shows deployability rather than validating the paper’s main findings.

Questions for authors

What is the realized share of transitions from each source in every Stitch pool? If R→G and G→R pools were given equal source shares (e.g., by switching at a fixed fraction of the horizon), how much of the ordering effect would remain? Have you compared PretrainedSuboptimal and Mixed pools whose recorded success rates are matched? Would the outcome gap between these two Medium sources persist under that control? How do the original CTDE versions of OMIGA and InSPO perform on the Cooperation and Transition families, compared with the Independent adaptations reported in the paper?

8 comments

Part 1/7

Thank you for the positive assessment and the concrete suggestions. You recognized that the paper treats the composition of offline data as an explicit experimental variable rather than relying only on coarse quality labels, the engineering effort behind the two physics platforms, the range of manipulation tasks and the real-robot interface, and the reporting with five seeds, confidence intervals and separate Final and Best checkpoints. You correctly pointed out that a fixed switch at step 10 confounds the order effect with the source shares, and this point changed how we present the order results. We have added experiments and revised the paper accordingly. We report the realized share of every Stitch pool and compare the two orders again at equal shares, compare the two Medium sources at matched recorded success, add CTDE versions of OMIGA and InSPO that keep their original coordination mechanisms, and state explicitly that the real-robot study is a deployment study. Under these controls the main conclusions still hold, and they are now stated more precisely. We have also revised the wording of the main text and the appendices to improve readability. We hope that these additions and revisions address your concerns. Below, we first list the changes in the revision and then respond to each point.

Changes in the revision addressing your comments

  • Sec. 4.3, Sec. 5.2, A.5 and the caption of Fig. 5 now present the step-10 result as one instance, obtained with the switch at step 10, and give the realized shares next to it (W1).
  • Sec. 4.1 states that the quality level of a pool is determined by its data source rather than by a threshold on recorded success (W2).
  • Table 27 in A.22 gives the realized shares of all 56 Stitch pools (Q1).
  • New A.26: the switch at H/2, a sweep over switch steps, order comparisons at matched shares, and 50% recomposed pools. Sec. 5.2 reports the matched comparison, and Sec. 1 and Sec. 6 state that the order effect persists at matched shares and that its size depends on the share (W1, Q1).
  • New A.27, with Sec. 5.2 revised accordingly: the comparison of the two Medium sources at matched recorded success (W2, Q2).
  • New A.28, with the corresponding results in Sec. 5.1: CTDE versions of OMIGA and InSPO that keep their original coordination mechanisms, paired with the Independent versions. The main table still uses the Independent setting (W3, Q3).
  • Sec. 5.4 is retitled "Real-Robot Deployment", and Sec. 5.4 and Sec. 6 state the scope of the real-robot study. Table 2 in the main text now includes the simulated success of the deployed checkpoints, and A.1 describes the training data and observation setting of each deployed policy (W4).

W1: With a fixed switch at step 10, the order effect may reflect the source shares

We agree with your observation. With the switch at step 10 on Lift, Connect, Insert and Handover, R→G pools consist mostly of Guided data (62–77%) and G→R pools mostly of Random data (12–20%), so the step-10 comparison largely reflects the source shares. The revision no longer attributes this gap to order alone. It presents the result as one instance, obtained with the switch at step 10, and gives the realized shares. The shares of every pool, and how much of the order effect remains at equal shares, are given in our answer to Q1.

W2: The Medium label, and the gap in recorded success between PretrainedSuboptimal and Mixed (32.1% vs 69.5%)

Medium is one quality level, which can be provided by either of two sources, PretrainedSuboptimal or Mixed. We agree that this level is assigned by data source rather than by recorded success. Following your suggestion, we ran a comparison at matched recorded success. After matching, the gap persists and its direction depends on the method, so this finding is not explained by the difference in recorded success alone (see Q2, Sec. 5.2 and the new A.27).

W3: Independent training of OMIGA and InSPO, their original CTDE versions, and coordination mechanisms

With the CTDE versions, the family-level conclusions about interaction structure still hold, while the per-task results of InSPO change substantially (Q3). As for representative coordination-based methods, the revision covers three coordination mechanisms with centralized training: value decomposition with a mixing network (CTDE OMIGA, results in Q3, and MAICQ on the discrete tasks, results in A.6), teammate-aware sequential policy optimization (InSPO), and centralized critics (MACQL and MATD3+BC on 22 tasks, A.19). Centralized training gives no consistent advantage in these comparisons.

Part 2/7

W4: The role of the real-robot study

All data-composition comparisons are carried out in simulation. The real-robot study tests deployment, namely whether policies trained offline on data collected in simulation can run directly on two Franka Panda arms through the same interface. No real-robot data and no fine-tuning are used. To narrow the gap between simulation and hardware, the training data for every task except Reach were collected in a configuration that supports domain randomization, the deployed checkpoints are mostly ones trained with domain randomization, and the end-effector orientation is fixed on the hardware to reduce the differences in motion between MoveIt and the simulated planner. The observation types are the same as in the benchmark (Relative for Reach and Lift, Partial for Handover and Connect-Insert). The same interface can also store real-robot episodes in the benchmark's data format for future sim-to-real studies.

For the deployed checkpoints, the method order on hardware largely agrees with simulation. On Handover, both give InSPO, OMIGA, MABC. On Connect-Insert, InSPO is first in both, and MABC and OMIGA exchange places, but the difference between these two is not significant in simulation or on hardware (p = 0.09 and p = 0.30). On Reach and Lift, all three checkpoints reach 100% in simulation. Table 2 in the main text gives the simulated success (%, 100 evaluation episodes) in parentheses after each real-robot result (successes out of 30 trials).

Table 2 (main text). Real-robot successes and simulated success of the deployed checkpoints.

TaskMABCOMIGAInSPO
Reach30/30 (100)30/30 (100)30/30 (100)
Lift30/30 (100)23/30 (100)24/30 (100)
Handover (partial)11/30 (63)17/30 (77)18/30 (93)
Connect-Insert (partial)11/30 (92)16/30 (83)18/30 (100)

Part 3/7

Q1: The realized share of each source in every Stitch pool, and how much of the order effect remains at equal shares

Realized shares. The realized shares agree with your reading of Fig. 5A. With the switch at step 10, Guided accounts for 62.1–77.2% of the R→G pools on Lift, Connect, Insert and Handover but for only 12.5–20.4% of the G→R pools, so in Fig. 5A R→G is close to Guided and G→R is close to Random.

The same holds for all 56 pools. The purposeful controller (Guided or Expert) accounts for 42.7–98.1% of the transitions when it acts second and for only 1.4–36.6% when it acts first. This is because, when it acts second, it runs until the task is completed or the time limit is reached, whereas when it acts first, it runs for at most the first 10 steps. Table 27 in A.22 lists, for all 56 pools, the realized share, the fraction of episodes that end before the switch, the recorded success and the seven-method mean Final success.

Table 27 (A.22). Realized source shares of all 56 Stitch pools.

TaskOrderSwitch stepEpisodesEnded before switch (%)G/E share (%)Recorded success (%)Mean of 7 methods (%)
CompetitionButtonR→G1012470.075.192.711.5
G→R1010030.120.01.20.3
R→E1018040.063.982.30.1
E→R10170742.532.853.70.0
CompetitionPushR→G1010881.078.340.05.0
G→R1010850.121.711.25.5
R→E1015880.868.370.710.4
E→R1011701.623.419.615.8
ConnectR→G1016370.067.391.155.5
G→R1010010.020.00.10.9
R→E1016040.067.972.448.6
E→R10150637.529.141.243.1
CurlingR→G106250.087.570.692.7
G→R106250.012.510.147.8
R→E106250.087.582.260.6
E→R106250.012.515.028.7
HandoverR→G1011400.177.299.745.6
G→R106270.012.50.00.0
R→E1014900.170.295.772.7
E→R106270.012.50.30.1
InsertR→G1013770.172.592.247.3
G→R106250.012.50.00.0
R→E109210.081.653.248.9
E→R106320.612.61.31.7
LiftR→G1018960.062.198.268.5
G→R1010220.520.43.518.7
R→E1015710.068.671.371.4
E→R10189651.636.660.265.2
PandaHandoverR→G101330.097.373.73.6
G→R10720.01.40.00.0
R→E10970.098.144.314.1
E→R10720.01.40.00.0
PointMaze3v1ArenaR→G1028680.742.7100.063.0
G→R10110080.615.596.060.0
R→E1023670.852.799.660.7
E→R1098165.414.796.270.5
PointMultiGoalArenaR→G106180.087.699.855.6
G→R101000.02.00.00.0
R→E105040.089.995.651.8
E→R101000.02.00.00.0
PointReachRaceR→G1019600.160.899.966.1
G→R1030028.05.653.028.7
R→E1011880.076.293.957.8
E→R1030229.55.654.030.0
PointTrajectoryHIR→G101250.097.5100.071.8
G→R101250.02.50.00.0
R→E101250.097.582.43.7
E→R101250.02.50.00.0
PushInterventionR→G1014570.070.997.454.9
G→R1010750.221.514.024.9
R→E1019920.160.294.567.8
E→R10162221.931.152.846.0
ReachSequentialR→G1018820.162.494.566.5
G→R10114613.522.715.735.5
R→E1014790.070.474.047.8
E→R1010655.921.27.520.3

The definition and computation of the realized share are given in A.22.

How much remains at equal shares. With the switch at step 10, the two orders differ by 46–55 pp on the four dual-arm tasks. At equal shares of 25–50%, the purposeful controller acting first is still better by 10.7–23.1 pp (seven-method mean, all intervals excluding zero, for both the native matched pools and the 50% recomposed pools), and at 65–75% the two orders do not differ significantly. The step-10 gap therefore comes mainly from the share, but order still matters at equal shares, by an amount that depends on the share.

Reviewer_42ZMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RoboMARL-Bench studies how task interaction structure and offline data composition affect multi-agent robotic learning. It provides 65 task configurations, including 44 robotic-arm configurations, organized into seven rule-defined families. Eight data constructions span source, episode, role, temporal and skill levels. The paper combines behavioral diagnostics with within-task, matched-budget comparisons of data construction, and evaluates existing learning methods under several observation and goal-conditioning settings. Four physical dual-arm tasks provide deployment evidence. The results identify task-dependent algorithm rankings, sensitivity to role assignment and controller ordering, and limited gains over some suboptimal data sources. The main contribution is a benchmark and empirical study.

Critical strengths
  1. The motivation is clear and the experimental design is relevant. Aggregate dataset quality can obscure which agent contributes competent behavior and when that behavior occurs. The role-assignment and within-episode controller-switching constructions directly address this question under matched data budgets. Combining these controls with a substantial robotic task suite creates a useful resource for studying coordination data (Secs. 1, 3 and 4.1).
  2. The empirical study provides useful structure and substantial supporting evidence. The reward-event, action-dependence and phase-concentration diagnostics, together with task-matched responses, go beyond aggregate rankings. The greater role-swap sensitivity observed for Transition tasks is an informative result (Secs. 4.2-4.3). Per-task results, robustness checks and physical deployment further support the benchmark's potential utility. The authors appropriately disclose baseline adaptations and qualify the descriptive and real-robot conclusions.
Critical weaknesses
  1. The incremental empirical contribution beyond existing data-centric work is not yet sufficiently established. Formanek et al. (2024) already examine dataset characteristics in offline MARL, and Tian et al. (2023) study agent-wise quality imbalance. RoboFactory provides multi-robot data generation with temporal constraints, and MangoBench provides goal-conditioned offline MARL evaluation. These works are cited, and the paper does identify a distinct combination of role/time controls. However, a more direct comparison, including RoboFactory in Table 1, would clarify the additional research capability. More importantly, the composition experiments change realized controller exposure, visited states and termination patterns alongside controller order (Appendix A.22). They establish differences between collection procedures, but provide less evidence for what the proposed organization explains beyond accompanying dataset properties. This limits the depth of the empirical contribution.
  2. The main continuous evaluation has limited coverage of the original coordination-aware methods. Appendix A.18 explains that Independent OMIGA removes cross-agent value mixing, while Independent InSPO omits teammate-density modelling and importance correction. The common Independent setting is disclosed and reasonably motivated across cooperative and competitive tasks. Nevertheless, the centralized-training comparison in A.19 covers MACQL and MATD3+BC, so it does not establish whether the OMIGA/InSPO rankings and learning-gain patterns persist with their original coordination mechanisms on compatible tasks. This limits the benchmark's empirical reference for those methods. It does not establish an implementation error or imply that the original methods must perform better.

Updated comments: I improved my score, since the paired stock experiment and the paired CTDE comparison added by the author have addressed my main concerns.

Questions for authors
  1. Could you use one central role/order result to demonstrate the additional insight enabled by the protocol relative to the closest prior work? Please report the relevant recorded performance and realized controller fractions, and explain whether existing analyses or a small matched comparison support an interpretation beyond those differences. A precise comparison with RoboFactory's temporal constraints would also clarify the contribution. An exhaustive cross-platform study or a universal causal model is unnecessary.
  2. On a small, compatible shared-reward task subset, is there existing evidence or a paired comparison showing how the original and Independent OMIGA/InSPO variants differ under the same data, actor observations and comparable budgets? Whether the main rankings and gains persist could affect my assessment. I am not asking that cooperative objectives be applied to competitive tasks.
9 comments

Part 1/9

Note. Following your further comments, this reply has been condensed. The central result for each concern and its location in the paper are summarized in the table of our follow-up comment, and the full tables are in the revised paper.

Thank you for the careful and detailed review. You found the motivation clear and the experimental design relevant, and you noted that aggregate dataset quality can obscure which agent contributes competent behavior and when that behavior occurs. The role-assignment and within-episode controller-switching constructions address this question directly under matched data budgets, and it is also the question that motivated RoboMARL-Bench. Both of your concerns are well founded, and we have added experiments and revised the paper accordingly. Regarding the incremental contribution, we added RoboFactory to Table 1, state how our work differs from each of the four works you named, and show with share-matched experiments that the order effect persists at approximately matched shares and that dataset statistics do not account for it. Regarding the original coordination mechanisms, we added CTDE versions of OMIGA and InSPO that keep these mechanisms and paired them with the Independent versions under the same data, observations and budget. Within-family rankings change little and the learning gains of OMIGA are preserved, while the per-task results of our InSPO adaptation change substantially. We have also revised the wording of the main text and the appendices to improve readability. If these revisions address your concerns, we would be grateful if you would take them into account in your final rating. Below, we first list the changes in the revision and then respond to each point.

Changes in the revision addressing your comments

  • Table 1 now includes a RoboFactory column, and its notes define every row (W1).
  • Sec. 2 states how the Role layer relates to Tian et al. (2023) and how our tasks relate to RoboFactory (W1).
  • Sec. 4.3, Sec. 5.2 and A.5 now present the step-10 order result as one instance, obtained with the switch at step 10, and give the realized shares of these pools. Sec. 5.2 reports the comparison at matched shares, Sec. 1 states that the order effect persists at matched shares, and Sec. 6 states that its size depends on the share. We no longer claim as a general rule that handing control to the purposeful controller last is better (W1, Q1).
  • Table 27 in A.22 gives the realized shares and recorded success of all 56 Stitch pools (Q1).
  • New A.26: the order comparison at matched shares, including the switch at H/2 and the sequentially dependent tasks ReachSequential and PushButton (W1, Q1).
  • New A.27: the comparison within the Medium level at matched recorded success (Q1).
  • New A.29: statistics of every pool, and whether these statistics identify the pool that trains better (Table 45) (W1).
  • New A.28, with the corresponding results in Sec. 5.1: CTDE versions of OMIGA and InSPO that keep their original coordination mechanisms, paired with the Independent versions (W2, Q2).

Part 2/9

W1: Incremental contribution

Central result. At approximately matched shares, the order still changes the outcome, and no tested statistic identifies the better pool more reliably than the order itself (A.26, A.29).

Contribution relative to the cited works, and comparison with RoboFactory

Relative to the four works named in the review, RoboMARL-Bench adds the following.

  • Formanek et al. (2024) relate offline MARL results to dataset statistics such as quality, coverage and diversity. Our work instead fixes the behavior sources and the budget within a task and changes only how the data are composed, and we find that the summary statistics we tested do not identify which pool trains better. Pools with the same quality label, the same recorded success and even approximately the same realized share can still lead to very different training results.
  • Tian et al. (2023) propose a method for learning from data whose quality differs across agents. We turn this imbalance into a controlled condition of the benchmark (Role layer) and measure how the outcome depends on which role receives the better data. We also vary when each behavior acts within an episode (Temporal layer), which their study does not address.
  • RoboFactory generates multi-robot demonstrations that satisfy temporal constraints, for imitation learning. We provide offline MARL pools that keep failed episodes and organize 44 multi-arm manipulation tasks (RoboFactory has 8) into seven interaction families with task-aligned metrics. Beyond the same kinds of task-level dependency, we vary the timing of the data within an episode, evaluate several offline MARL algorithms, and report real-robot results (detailed comparison below).
  • MangoBench evaluates only offline goal-conditioned MARL and contains 2 multi-arm manipulation tasks. We run both standard and goal-conditioned multi-agent learning on the same physical pools, across 44 multi-arm manipulation tasks, three observation modes and the data constructions above.

We have also added RoboFactory to Table 1 of the paper. The new column is shown below next to ours, where (✓) means partial support. The full Table 1 is in the revised paper.

Table 1 (excerpt). The new RoboFactory column.

Row of Table 1RoboFactory (new)RoboMARL-Bench
Multi-arm tasks8 of 1144 of 65
Agent settingBothBoth
Learning settingOffline (imitation learning from validated demonstrations)Both
Data generation✓✓
Interaction taxonomy✗ (tasks are grouped by the number of agents)✓
Role / timing control(✓) (timing between agents is constrained during generation, but roles and phases are not varied)✓
Observability contrast✓✓
Standard & GC pool✗✓
Real-robot results✗✓

What the data organization explains beyond the accompanying properties

We agree with your observation that when Stitch changes the order, the controller share, the visited states and the termination pattern change with it. These three properties differ in nature. The controller share can be controlled separately, whereas the visited states and the termination pattern are direct consequences of the order, because the controller that acts first determines which states the second one sees and whether the episode ends before the switch. We therefore added two analyses.

First, controlling the share. We added experiments at matched shares (A.26). Once the realized shares are approximately matched, the order effect remains. At about 45% share, Guided→Random is better by 20–30 pp on every task, although its recorded success is lower. The effect becomes smaller at higher shares (Q1 and A.22).

Second, testing whether the remaining properties explain the outcome. Visited states and termination cannot be separated from the order, so we report them next to every order comparison and test whether they identify the pool that trains better (new Table 45 in A.29). In the 20 native matched pairs, the two pools of each pair share sources, budget and realized share. Guided→Random trains better in 16 pairs and Random→Guided in 4. The table counts how often the pool with the higher value of a statistic is the one that trains better (seven-method mean Final success).

Table 45 (A.29). Dataset statistics against the training outcome.

StatisticG→R higherPicks the better pool
Recorded success8 / 2010 / 20
State diversity11 / 207 / 20
Coverage of Guided reference states20 / 2016 / 20
Early-state coverage by purposeful transitions20 / 2016 / 20
Shorter mean episode length20 / 2016 / 20

Recorded success and state diversity perform close to chance. Coverage and episode length reach 16/20 because they favor Guided→Random in all 20 pairs, so in these pairs they cannot be separated from the order and may contribute to its effect.

Part 3/9

W2: OMIGA and InSPO with their original coordination mechanisms

Central result. With their original coordination mechanisms, the within-family rankings change little and the learning gain of OMIGA is preserved, while the per-task results of our InSPO adaptation change substantially (A.28).

A.19 covered only MACQL and MATD3+BC. The new A.28 adds CTDE versions of OMIGA and InSPO that keep their original coordination mechanisms, on all 19 Cooperation tasks and all 10 Transition tasks. Each CTDE run is paired with an Independent run on the same pool, actor observations, budget and seeds. Within each family and quality level, the seven-method ranking changes little (Kendall rank correlation τ = 0.81–1.00, where τ = 1 means identical rankings). Over all 29 tasks, OMIGA keeps its rank in 40 of 58 task–quality cells, and its mean learning gain on Medium data is preserved (+10.6 pp Independent, +11.3 pp CTDE). Sec. 5.1 of the paper now reports this comparison. Detailed results, including the shared-reward subset you asked for, are given in our answer to Q2.

Q1: A central order result with recorded performance, realized controller fractions and a matched comparison, and a precise comparison with RoboFactory's temporal constraints

Central result. At approximately matched shares (0.2–4.2 pp apart), the purposeful controller acting first is better in 11 of the 12 native pool pairs at 25–45% share and on 11 of 12 tasks with recomposed 50% pools, and the difference is not significant at 65–75% (Tables 33–35 in A.26).

Additional insight. We use the controller order in Stitch as the central result. The closest prior work (for example Formanek et al., 2024) explains offline MARL results through dataset-level descriptions such as quality labels, recorded success and coverage. Our protocol can fix the behavior sources and the budget, approximately match the realized share and manipulate only the order. This yields three results that dataset-level descriptions do not reveal.

  1. At approximately the same realized share (about 45%), Guided→Random is better by 20–30 pp on every task, although its recorded success is lower. Selecting by recorded success picks the wrong pool, and no other dataset statistic adds information beyond the order itself (Table 45 under W1).
  2. The size of the order effect depends on the share. It is significant at 25–50% share and no longer significant at 65–75%.
  3. In tasks with sequential dependencies, the outcome also depends on which stage the purposeful data cover. When the purposeful data are in the first stage, the learned policies complete the first stage 19–50 pp more often.

Points (i)–(v) below answer your requests in turn: (i) realized controller fractions and recorded performance, (ii) the comparison at matched shares, (iii) other shares and other constructions, (iv) whether visited states and termination explain the outcome, and (v) results of the same kind beyond order. The comparison with RoboFactory's temporal constraints follows at the end.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Concerns reply 1/2

Part 1/2

Thank you for the further review and the concrete suggestions. We agree with all four points and have revised the paper and our original reply accordingly. We address the four concerns in turn below.

1. Central result for each concern and where to find it

In an effort to present all results directly in the reply, our original response became long and was distributed over many comments, which made it difficult to review. We apologize for this. The table below summarizes the central result for each of your weaknesses and questions, together with its location in the revised paper and in our original reply. We have also added a bold sentence stating the central result at the beginning of each weakness and question in our original reply (W1, W2, Q1 and Q2), and the key findings in all nine parts are now marked in bold. The nine parts of our original reply are now condensed as well. Long tables there are shortened to the rows that support the text, and each shortened table refers to its full version in the revised paper.

ConcernCentral resultIn the paperIn our reply
W1: increment over prior workAmong the benchmarks in Table 1, only RoboMARL-Bench varies role assignment and controller timing within a task at a matched budget. RoboFactory constrains timing during data generation but does not vary roles or phases.Sec. 2, Table 1 (new RoboFactory column)W1, Part 2/9
W1, Q1: whether the order effect reflects the share and accompanying propertiesAt approximately matched shares (0.2–4.2 pp apart), the purposeful controller acting first is better in 11 of the 12 native pool pairs at 25–45% share. With recomposed pools containing exactly 50% purposeful transitions on 12 tasks, the same holds on 11 (Guided) and 10 (PretrainedExpert) tasks. At 65–75% share, the difference is not significant.A.26, Tables 33–35Q1 (ii) and (iii), Part 5/9
W1, Q1: whether accompanying properties identify the better poolNone of the tested summary statistics identifies the better pool more reliably than the order itself (point 2 below).A.29, Table 45W1, Part 2/9, and Q1 (iv), Part 6/9
W2, Q2: whether the conclusions change with the original coordination mechanismsWithin-family rankings change little (Kendall τ = 0.81–1.00), OMIGA keeps its rank in 40 of 58 cells and its learning gain, and the per-task results of InSPO change substantially (point 3 below).A.28, Tables 40–41W2, Part 3/9, and Q2, Parts 8–9/9

2. Table 45 and the scope of the conclusion

We fully agree and have revised the paper as you suggest. The revised Sec. 5.2 and A.29 state the narrower conclusion. Because coverage and termination change together with the order in these pairs, we now describe the result as an effect of the controlled collection protocol: with the behavior sources, the budget and approximately the share held fixed, changing when each controller acts within an episode changes the training outcome.

3. InSPO and the wording of the conclusions

Overall, with CTDE InSPO the family-level rankings remain similar, but its per-task results change substantially, and on average the Independent InSPO performs better (by 5.2 pp over the 29 tasks, Table 40). Following your suggestion, we have revised Sec. 5.1 and A.18 accordingly. We also agree that an interval containing zero does not establish equivalence, and we use the intervals in Table 40 only to bound the average change. The continuous-action CTDE InSPO is our adaptation, and its results describe this adaptation only.

AuthorsOpenReview ↗

Concerns reply 2/2

Part 2/2

Our implementation of CTDE InSPO is as follows (goals are omitted for brevity). Unlike the Independent version, its critic conditions on the central state, and when agent ii is updated, the samples are resampled according to importance weights on the teammates' actions. This shifts the distribution of the teammates' actions from the behavior distribution of data collection to the teammates' current policies.

  • Model structure. Within each update, the agents are updated sequentially in a random order. Agent ii comprises a Gaussian policy πi\pi_i with mean μi\mu_i, a Gaussian behavior model βi\beta_i, a critic Qi(s,ai)Q_i(s, a_i) conditioned on the central state, and a teammate behavior model β^−i(a−i∣oi,ai)\hat\beta_{-i}(a_{-i} \mid o_i, a_i), which models the conditional distribution of the teammates' actions in the data given the observation and action of agent ii. Both behavior models are fitted by maximum likelihood before training (at most 2,000 steps, with early stopping after 100 steps without improvement) and are kept fixed thereafter.
  • Objective. The critic is regressed onto ri+γ(1−d)[Qˉi(s′,μi(oi′))−αtKL(πi(⋅∣oi′)∥βi(⋅∣oi′))]r_i + \gamma (1-d) [\bar Q_i(s', \mu_i(o_i')) - \alpha_t \mathrm{KL}(\pi_i(\cdot \mid o_i') \Vert \beta_i(\cdot \mid o_i'))], and the actor minimizes −exp⁡(min⁡(Ai/αt,5))log⁡πi(ai∣oi)-\exp(\min(A_i/\alpha_t, 5)) \log \pi_i(a_i \mid o_i) with Ai=Qˉi(s,ai)−Qˉi(s,μi(oi))A_i = \bar Q_i(s, a_i) - \bar Q_i(s, \mu_i(o_i)), where Qˉi\bar Q_i denotes the target critic, γ=0.98\gamma = 0.98 and αt=(10−3)t/T\alpha_t = (10^{-3})^{t/T} with TT = 50k (TT = 100k for PandaSlideHandover-Good, PandaSlideHandover-Medium and PandaStack-Good, identical in both versions).
  • Teammate importance correction and resampling. The teammates' actions in the data follow the behavior distribution of data collection, whereas agent ii should be updated against the teammates' current policies. Each sample is therefore weighted by the ratio of the probability of the teammates' actions under their current policies to their probability in the data, w=clip(exp⁡[(∑j≠ilog⁡πj(aj∣oj)−log⁡β^−i(a−i∣oi,ai))/(N−1)],0,100)w = \mathrm{clip}(\exp[(\sum_{j \neq i} \log \pi_j(a_j \mid o_j) - \log \hat\beta_{-i}(a_{-i} \mid o_i, a_i)) / (N-1)], 0, 100). Dividing by N−1N-1 amounts to the geometric mean of the per-teammate probability ratios, and clipping to [0, 100] bounds the weights. Since the weights are computed with the teammates' current policies, teammates that precede agent ii in the update order enter with the policies updated in the same pass. Both the critic and the actor losses are computed on a batch resampled with replacement in proportion to ww.
  • Training settings. Batch size 512, Adam optimizer, learning rate 8×10−48 \times 10^{-4}, soft target update rate 10−310^{-3}, gradient-norm clipping threshold 1 and hidden width 256, without entropy regularization or conservative penalty. The Independent version contains neither β^−i\hat\beta_{-i} nor ww, its critic conditions only on oio_i, and all other settings are identical.

The supplementary material has been updated with the code and configurations of CTDE InSPO (trains/offline/algorithms/inspo.py, where the CTDE version is selected with --paradigm ctde, and revision/ctde_comparison_runs.jsonl).

4. Corrections

Thank you for these careful observations. We have corrected the statements concerned.

  • In Q1 (iv), the original sentence placed the average over all 20 pairs (78.1% and 30.0%) directly after "at about 45% share", so that it read as the value at 45%. At about 45% share, the values are 77.7% for G→R and 21.1% for R→G (Table 36). We have corrected the original reply.
  • We wrote "matched" for brevity. The revision now refers to the shares as approximately matched and reports the difference between the two orders (0.2–4.2 pp), and we have corrected the corresponding statements in our original reply.
  • Sampling unit of the Wilcoxon test. The test in our original reply treated individual seed-matched runs as samples (4 tasks × 7 methods × 5 seeds = 140 pairs). Since the 35 comparisons from one pool pair all use the same two training pools, the pool pair is the more appropriate unit of comparison. We therefore repeated the same test with the pool pair as the unit, using for each pool the seven-method mean over five seeds. At 25–45% share, the purposeful controller acting first is better in 11 of the 12 pairs (Wilcoxon signed-rank test, p = 0.0015). As pairs from the same task still share the environment and the behavior sources, we additionally performed a cross-task test on the recomposed pools with exactly 50% purposeful transitions on 12 different tasks, which gives consistent results (11 of 12 tasks with Guided, p = 0.001, and 10 of 12 with PretrainedExpert, p = 0.003). The intervals in Table 33 are obtained by resampling tasks and then seeds and therefore also treat the pool as the unit. A.26 and our original reply have been updated accordingly.
AuthorsOpenReview ↗

Follow-up on our revised reply

Dear Reviewer 42ZM,

Thank you again for your follow-up comments. We have answered all four points in our two follow-up comments and revised the paper accordingly. We also shortened our earlier reply, and the table at the start of our follow-up gives the central result for each concern and where to find it in the paper.

We would really appreciate knowing whether this resolves your concerns. If it does, we would be very grateful if you could consider raising your score. If anything is still unclear, please tell us, and we will continue to revise the paper. The discussion period ends on November 18 (AoE).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.