acceptodds
Under review as a conference paper at ICLR 2027

Co-Design Gym: A Unified Benchmark for Embodiment-Policy Co-Optimization

Abstract

Finding an optimal behaviour policy within a given environment is a widely studied problem in domains as diverse as games, robotics, energy infrastructure, communication networks, and multi-agent systems. Numerous benchmarks have been developed to support such research, but the vast majority assume that the agent’s embodiment (design) is fixed, focusing instead on policy learning alone. Lifting this assumption gives rise to a broader class of problems in which optimizing embodiment and policy separately is highly suboptimal. An agent’s embodiment strongly shapes which control policies can be discovered, while the optimal embodiment is in turn defined by the policies it admits. To help the research community study this class of problems explicitly and systematically, we introduce *Co-Design Gym* – a suite of benchmark environments for jointly optimizing embodiment and policy. Our environments span domains such as robotic manipulation and locomotion, multi-robot cooperation, deformable and soft dynamics, video games, electricity grids, wireless networks, F1 racing, multi-agent warehouses, and optimal control, offering 20 environment families (domains), with over 85 distinct co-design presets in total. We further contribute a systematic evaluation of representative co-design algorithms, characterizing the current state of the art. Together, these contributions lay the groundwork for cumulative, comparable progress in co-design.

Primary area:
datasets and benchmarks
Keywords:
optimization, benchmark, robotics, video games, reinforcement learning, control, design

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ecMzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the authors introduce Co-Design Gym, which extends Gymnasium with an explicit embodiment space and design arguments, and introduces native, extended, and reframed co-design environments across 20 families and over 85 presets. Besides, four adapted optimizers are introduced and compared under a nominal four-hour budget. In my opinion, the contribution is principally hybrid benchmark infrastructure (which covers various robotic problems and some application problems), rather than a new optimization algorithm or substantive co-design theory.

Critical strengths
  1. This benchmark provides a potentially useful common interface for co-design, which includes explicit design spaces, batched validation, and separation of training-time and evaluation-time embodiment changes address practical obstacles to comparing co-design methods. This is a real infrastructure contribution beyond domain-specific platforms such as EvoGym and SoftZoo.

  2. This benchmark includes contact-rich manipulation and mixed discrete/continuous applications broaden evaluation beyond locomotion. Five-seed reporting is useful and convincing. It is more applicable than existing benchmarks.

Critical weaknesses
  1. The API and task taxonomy are understandable. However, the evaluated-preset count differs between Figure 8 (16) and Section 6 (18), Figure 28 mentions unfinished runs, and Appendix D retains “temperature of X”; Table 3 omits the LLM rows described in its caption. I don't think the author took the experiment seriously.

  2. The benchmark does not demonstrate the quantity it claims to measure: benefit from embodiment-policy co-optimization. Tables 1–2 compare joint optimizers but omit matched control-only training on the base embodiment, random-design controls, and design-only evaluation with a controlled controller. Consequently, poor task performance and changing method rankings cannot distinguish difficult co-design from difficult policy learning or infeasible design sampling. Appendix A provides task descriptions rather than complete bounds, constraints, rewards, and instance splits. The soft-body and shape-changing environments are prominent contributions but are absent from the main scoreboard. The elementary supremum statements in §3/Appendix F do not fill this empirical gap. A smaller, fully specified core suite with measured design gains would be more convincing than an unvalidated headline preset count.

I believe the true value of a co-design benchmark should be to provide a reliable standard for measuring exactly how much benefit is derived from the coordinated optimization of morphology and strategy, rather than simply accumulating the number and variety of tasks.

  1. Baseline modifications and evaluation choices undermine the claimed representative comparison.** LOKI loses its robot-specific tokenization and uses six rather than forty clusters; the original LOKI relies on a learned morphology space and clustered policy reuse. FastTD3 rejects every NeroGraspAll design, while PPO+NGOpt scores designs using a changing controller. There is no demonstrated validation of these adaptations or adequate disclosure of comparable model sizes, tuning, interactions, and actual preprocessing costs. Four-hour wall time is a useful engineering metric, but cannot alone establish an algorithmic ranking. FastTD3 also generates an embodiment per evaluation rollout, whereas other baselines evaluate selected designs: clarify whether all methods solve the same fixed-design or observation-conditioned design problem. These issues affect the central scoreboard, not merely presentation.

  2. The authors’ approach to modeling co-design problems is limited and unclear (whether it involves a two-layer cyclic optimization approach like Evogym or a single-layer joint optimization approach like Transform2Act), and they do not provide a detailed description of the design spaces and characterization methods for morphological and control parameters. This makes it quite difficult for users to adapt the method using their own characterizations. I encourage the authors to focus on robot co-design.

Questions for authors

For main concerns, please refer to the Critical Weaknesses section. Here are some minor questions:

  1. For Multi-Robot Truck Unloading, I don't think this approach to modeling integrated control strategies fully leverages the advantages of multi-agent co-design.

  2. For different type of robots, is there a unified approach to modeling design spaces—one that goes beyond simply vectorization and fixed topology settings?

6 comments
AuthorsOpenReview ↗

Response to Reviewer ecMz (1/3)

Thank you for your detailed review. We appreciate your recognition that the benchmark is "a real infrastructure contribution", and that it "broaden[s] evaluation beyond locomotion". We fully share your view that a Co-design benchmark should measure the benefit of coordinated optimization itself, and the revision now does so directly (W2 below).

W1: Reporting inconsistencies

You are right, and we are sorry for these oversights. All of them are corrected in the revised PDF:

IssueRevised PDF
Preset count in Fig. 8 (16) vs Section 6 (18)Both now refer to the same 35 presets (Table 1, Fig. 8)
Figure mentioning unfinished runsRectified
"temperature of X" in Appendix DRemoved - we ran with Gemma's default
Table 3 missing the LLM rowsCompleted (now Table 14: PPO+LLM and PPO+NGOpt)

The LLM table is now complete (Table 14). This exploratory experiment is intended to demonstrate what a very low-budget optimization approach with an LLM in the loop can achieve. We believe a low-cost experiment is still a useful investigation, because the cost of frontier models can make more ambitious comparisons financially prohibitive. Our benchmark focuses on wider participation and inclusion, as all of our environments and presets can be run on a single GPU. Our Gemma explorations were conducted within the free Google API tier, and as such represent an exploratory low-budget LLM experiment; we now label the study specifically as low-budget. We welcome suggestions on how to demonstrate the power of LLMs for co-design within reasonable budgets, to keep the benchmark open and accessible to a wide range of researchers.

W2: Measuring the benefit of embodiment-policy co-optimization

We fully agree, so we have run two control settings that isolate the value of policy-only and design-only optimization (App. C.2, Tables 6–7). For policy-only optimization, we employ PPO with the fixed base embodiment of the environment, and for design-only optimization we initialize a random policy and perform pure design-space exploration. A random policy cannot deliver competitive performance on some environments (e.g., robotic control), but there is a subset of our presets on which a random policy may do something meaningful (e.g., Pokémon). Hence, the design-only control was run on those 13 presets. The results measure the design gain directly:

  • design search alone never matches Co-design (13 of 13 presets);
  • co-design beats fixed-design PPO on 13 of 22 presets, by up to 32.5 points;
  • fixed-design PPO is ahead of every co-design method on the other 9.

Note that fixed-design PPO solves a strictly smaller problem with a mature method, while co-design methods must search the joint space of designs and policies under the same budget, using far less mature methods. Since fixed-design policy learning is a special case of co-design, the best achievable co-design return is never lower (Proposition 4, Section 3). Hence, a fixed-design policy coming out ahead most likely reflects how hard and how young co-optimization is, not a general lack of benefit from co-design. Against this, the wins on 13 of 22 presets are encouraging.

The soft-body and shape-changing environments, which the reviewer noted were missing from the scoreboard, are now part of both the main scoreboard and the controls. The table shows these four presets; the results for all 22 presets with controls are in Tables 6-7 (App. C.2). Controls for the remaining 13 presets of the scoreboard are currently running, and we will add them to the paper as soon as they finish rather than delay our response:

Preset (normalized %)Policy onlyBest Co-design
HandPickUpHammer9.527.5 (CMA-ES)
HandPickUpCube1.61.7 (CMA-ES)
SoftWalkerBeam3D4.34.5 (PPO+NGOpt)
RotateInHandCube2.51.6 (FastTD3)

These measured gains - positive on some presets and negative on others - provide the empirical counterpart to the existence results of Section 3 and Appendix F, and show that current methods, still young compared with policy-only RL, do not yet reliably convert design freedom into performance.

Furthermore, our released code contains documentation outlining the design space for every environment family (layout, bounds and how to build a design), the observations, actions and rewards, and the runnable examples. Appendix A now summarizes it in a table that gives, for every family, the design space, the rule for rejecting designs, the return range and the episode length (Table 2); the normalization anchors are in Table 3.

AuthorsOpenReview ↗

Response to Reviewer ecMz (2/3)

W3: Baseline adaptations and evaluation choices

We appreciate the close look at our baselines; each adaptation is discussed below, with new evidence where it was needed. All of our baseline algorithms were run under a matched budget to reflect practical computational choices.

LOKI. LOKI's clustering approach with 40 clusters and controllers in parallel made it infeasible to apply within the budget provided for the other baseline algorithms, so we reduced the number of clusters to 6, so that each individual controller receives sufficient training time. Its robot-specific tokenization is designed for the limb-based robots on which it was engineered (one token per limb [13]) and does not transfer to designs such as meshes, layouts or Pokémon teams, which further motivates the need for a common API and representation. To test the validity of this reduction, we ran LOKI with its full 40-cluster specification on two presets: Microgrid-Random, where 6-cluster LOKI achieves the highest percentage of the achievable reward, and SolarCleaner-Reach, where it achieves the lowest (App. C.5, Table 13):

Preset6 clusters (norm. %)40 clusters (norm. %)Raw return, 6 → 40 clusters
Microgrid-Random (LOKI's best)52.1 ± 0.251.8 ± 0.4−25,107 → −25,262
SolarCleaner-Reach (LOKI's worst)0.07 ± 0.020.17 ± 0.010.0061 → 0.0155

With 40 clusters, LOKI's standing is unchanged on both presets - Microgrid-Random is unchanged and SolarCleaner-Reach improves marginally, even though each of the 40 controllers receives roughly 7× fewer PPO updates within the same budget.

FastTD3 on NeroGraspAll. FastTD3 alone never learns to search the space: 99.8% of the designs it proposed during training, and all of those at evaluation, failed structural validation, compared with 0.7% for CMA-ES and 10% for PPO+NGOpt (Table 12). We report this as a finding about RL-only co-design on constrained design spaces.

PPO+NGOpt scoring designs with a changing controller. This indeed is a defining trait of alternating co-design: the PPO policy is conditioned on the design, so a single controller learns across all the designs NGOpt proposes, and since it keeps improving while designs are proposed, each design is scored by how well the current controller uses it. This lets the design search and the design-conditioned policy adapt to each other over time. We agree it is worth contrasting with scoring designs under a fixed controller, and the new design-only control (App. C.2) does that.

Model sizes, interactions and wall-clock budget. We agree that wall-clock time alone cannot establish an algorithmic ranking. We discussed the choice of budgeting metric at length internally; to our knowledge, there is no established standard for comparing algorithms as different as those in our study, and every candidate metric introduces its own bias (environment timesteps favor sample-efficient methods; FLOPs do not account for parallelization). We ultimately chose wall-clock time because it most closely reflects practical constraints, and we now make all relevant cost measures visible: trainable parameters and environment steps per method (Table 10), per-preset steps and sample efficiency (Table 11), and design-rejection rates and stability (Table 12). Compared at matched environment steps instead of matched time, the leader changes on 22 of 35 presets, and no method dominates in either view (App. C.4).

Fixed-design vs observation-conditioned design. The common denominator is that each method explores the same design space and is given the same wall-clock budget; we deliberately do not restrict other aspects of how the methods explore the design and policy space, as we aim to show approaches that employ a variety of strategies. To show how to use RL for co-design with minimal method modifications, we use FastTD3, which samples a new design at the start of each episode and conditions the subsequent steps on that design; the code change is minimal compared to policy search, so it indicates how well RL can handle co-design on its own. At evaluation, its deterministic policy chooses one design per evaluation world from that world's initial observation, and the design then stays fixed for the episode. PPO+NGOpt instead shows how RL can benefit from combination with global optimization and population-based sampling: NGOpt samples a design and passes it to PPO, which conditions the policy on the selected design but does not need to learn design-sampling choices itself. Hence off-policy FastTD3 reuses scarce design decisions through replay in a single-level optimization, while on-policy PPO gives NGOpt up-to-date design fitness in a two-level formulation - the former favoring formulation simplicity, the latter following the most commonly used two-level co-design setup. CMA-ES further simplifies the single-level setting, while LOKI tests a more complex single-level setup than RL-only FastTD3.

AuthorsOpenReview ↗

Response to Reviewer ecMz (3/3)

W4 / Q2: Modelling approach and design-space representations

Our goal is to accommodate both styles, so that method developers can choose the organization that suits their approach best. Section 3 treats co-design as a joint problem over embodiment and policy, which nested methods also fit. The API supports both two-level methods, in which an outer design optimizer proposes embodiments and an inner learner trains a controller for them, as in EvoGym [14], and single-level methods, in which one agent or optimizer chooses both the embodiment and the control, as in Transform2Act [15]. The baselines illustrate this range: PPO+NGOpt alternates an outer design optimizer with PPO training (two-level); FastTD3 treats embodiment parameters as actions at the start of each rollout (single-level); and CMA-ES searches over design and controller parameters as one vector (single-level). All baselines run on the same environments under the same evaluation protocol, making it possible to compare the styles directly without mandating a single way to organize the optimization loop(s). We now state this explicitly in Appendix E.

The design spaces are summarized for every environment family in Appendix A (Table 2) and described in full, including how to build a design, in the documentation released with the code.

A unified approach to modelling design spaces beyond vectors and fixed topologies (Q2) remains an open and interesting question. Graph-based representations have been explored before: encoding a robot's morphology as a graph whose nodes and edges carry link geometry, material properties and actuator parameters, then processing these representations with graph neural networks. It remains open whether such representations make co-design more efficient than flat vectors in general, or whether other representations would better support embodiment optimization and policy learning. Since the embodiment space is a standard Gymnasium Space, novel representations can be plugged in and tested; we provide flat vectors as the default because they are the most appropriate starting point now, given how difficult it is to foresee which structures users will want to try.

Regarding robot co-design: while robotics domains constitute a large part of the suite (the Native robotics families and the MuJoCo ports), we hope the non-robotic families add value by testing whether conclusions transfer beyond robots. Indeed, they already reveal some domain-specific winners (Section 6).

Q1: Multi-Robot Truck Unloadingz

That is a fair observation and our approach does not fully leverage the advantages of multi-agent co-design. The Multi-Robot Truck Unloading task allows for richer multi-agent co-design formulations (and the environment does not preclude these in principle) than the one we adopt as a baseline.

Our current formulation was a deliberate choice for the purposes of a benchmark, for three reasons: (i) it keeps the interface identical across all environments, so every co-design method in the benchmark runs without modification; (ii) it separates the co-design problem from well-known multi-agent RL difficulties, such as credit assignment, non-stationarity and communication, which would otherwise confound the comparison between co-design algorithms; and (iii) a centralized controller provides a natural reference against which decentralized formulations can be compared in future work.

More broadly, our primary contribution is a benchmark, not an optimal solution to each task. As in established RL benchmarks, such as the Arcade Learning Environment [6] and the DeepMind Control Suite [7], our baselines are intended as a starting point for comparison rather than a comprehensive documentation of the state of the art or an upper bound on what is achievable. Formulations that fully exploit multi-agent co-design in this task are precisely the kind of research we hope the benchmark will enable.

Thank you once more for the care you put into this review. Your challenge to measure the benefit of co-optimization carefully has shaped the large part of the revision; the new experiments based on your suggestions now make that benefit, and its current limits, more concrete and visible. We hope the new experiments and clarifications address your concerns, and we welcome any further discussion.

Reviewer_ynk6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Co-Design Gym, a benchmark suite for jointly optimizing an agent's embodiment and policy. The suite claims 20 environment families and more than 85 presets across robotics, deformable dynamics, games, energy, wireless networks, racing, logistics, and control. The paper formalizes the co-design problem and compares four representative optimization approaches on a subset of presets under a fixed wall-clock budget, finding that no method dominates across tasks and budgets.

Critical strengths

This paper addresses a meaningful gap: most policy-learning benchmarks fix the agent design, whereas co-design requires searching over coupled embodiment and policy choices. The proposed API makes this class of problems more accessible to the community.

Critical weaknesses
  1. The benchmark is broad, but the comparative scoreboard covers only a small subset of the claimed suite. This leaves much of the benchmark without reference results and makes it difficult to assess how representative the reported algorithm ranking is across the full range of design spaces.

  2. A matched four-hour wall-clock budget does not by itself ensure an algorithmically fair comparison. The results can depend on simulator throughput, implementation maturity, tuning effort, and algorithm-specific overhead. LOKI, for example, reserves one hour for preparation and has three hours of online optimization. The paper acknowledges this limitation, but should also report simulator transitions or environment interactions, tuning budgets, hardware details, and sample-efficiency curves so readers can distinguish engineering speed from optimization quality.

  3. The evaluation would benefit from controls that isolate the value of co-design itself. Across the reframed and extended tasks, fixed-embodiment policy optimization and simple design-search baselines would help establish whether improvements require joint design-policy search rather than reflecting differences in controller learning or reward structure.

  4. Returns use very different scales across environments, which is acceptable for within-task ranking but limits cross-task summaries; clearer per-task reference ranges or normalized scores would improve interpretation. There is also a reporting inconsistency worth resolving: the main results table covers 18 presets, while Figure 8 describes 16 presets with all four baselines.

Questions for authors

See weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer ynk6 (1/2)

We appreciated this clear and encouraging review. We appreciate your recognition that the paper "addresses a meaningful gap" and that the proposed API "makes this class of problems more accessible to the community." Each of your four suggestions led directly to an addition in the revision, and we are grateful for them.

W1: Coverage of the scoreboard

We followed this suggestion carefully and completed further baseline experiments: Table 1 now reports 35 presets (up from 18), with 4 baselines, 4 hours and 5 seeds each (2,800 hours of experiments total), with each of the 20 environment families represented. The 35 presets were selected as follows: one preset from each environment family to ensure variety; beyond that, all BallCatcher presets (to test locomotion and manipulation, with sparser rewards than the locomotion-only OpenAI Gym extensions), additional SolarCleaner presets (to test loco-manipulation with a robot based on the Boston Dynamics Spot), all MorphHand presets (to test rapid per-step shape updates), and all presets of the reframed families (except LQR, which as the simplest was limited to one preset, and the Pokémon presets with expert-curated strategies, which go well beyond the baselines).

Original submissionRevision
Presets with all four baselines1835
Environment families representedpartialall 20
Soft-body and shape-changing presetsnot in the scoreboardSoftWalkerBeam3D, RotateInHandCube, HandPickUpCube, HandPickUpHammer

With the wider coverage, the ranking remains mixed across design spaces - FastTD3 leads on 13 presets, CMA-ES and PPO+NGOpt on 9 each, and LOKI on 4 - and the winners cluster by domain (Section 6).

W2: Wall-clock budgets and cost measures

We agree: a matched wall-clock budget is a practical choice, not a guarantee of an algorithmically fair comparison.

We discussed the choice of budgeting metric at length internally when designing the experiments. To our knowledge, there is no established standard for comparing algorithms as different as those in our study, and every candidate metric introduces its own bias. For instance, environment timesteps favor sample-efficient methods, whereas FLOPs do not account for parallelization. We ultimately chose wall-clock time because it most closely reflects practical constraints.

We agree with the reviewer that the most constructive path forward is to make all relevant cost measures visible. We now report:

Suggested measureWhere in the revision
Environment interactionsTable 10 (per method), Table 11 (per preset)
Sample-efficiency curvesFig. 23
Model sizesTable 10
LOKI's preparation and online budgetSection 5, Table 10
Design validity and stabilityTable 12

Our findings suggest that the methods differ by more than an order of magnitude in environment interactions within the same budget (median 1.2 billion steps per run for CMA-ES versus 24 million for LOKI), but that the main conclusion that no method dominates does not depend on the metric: compared at matched environment steps instead of matched time, the leader changes on 22 of 35 presets, but no method dominates in either view (App. C.4). We would be happy to provide any further comparisons.

AuthorsOpenReview ↗

Response to Reviewer ynk6 (2/2)

W3: Controls isolating the value of co-design

This is an excellent point, and we fully agree. We have now run two control settings that isolate the value of policy-only and design-only optimization (App. B, App. C.2, Tables 6–7). For policy-only optimization, we employ PPO with the fixed base embodiment of the environment, and for design-only optimization we initialize a random policy and perform pure design-space exploration with NGOpt. A random policy can only do so much on many environments (e.g., robotic control), but there is a subset of our presets on which a random policy may do something meaningful (e.g., Pokémon), so we ran the design-only control on those 13 presets.

The policy-only control was run on 22 presets and the design-only control on 13 of them. To keep this response short, the table below shows the six presets where the difference between co-design and fixed-design PPO is largest: the three where co-design gains most, and the three where it falls furthest behind. The full results for all 22 presets are in Tables 6–7 (App. C.2). Controls for the remaining 13 presets of the scoreboard are currently running; rather than delay our response, we report the 22 completed presets here and will add the rest to the paper as soon as they finish.

Preset (normalized %)Policy onlyDesign onlyBest Co-designCo-design vs. policy only
MicrogridRandom63.873.596.3 (FastTD3)+32.5
TruckUnloadSingle0.5—22.3 (FastTD3)+21.7
HumanoidStandup10.8—31.4 (CMA-ES)+20.6
BallCatcher-Fielding51.4−0.130.9 (FastTD3)−20.5
Racing (Spielberg)44.4−0.421.9 (FastTD3)−22.5
BallCatcher-Pitch60.30.233.6 (FastTD3)−26.7

— : indicates design-only control not run, as the task requires behaviour a random policy never achieves (e.g. grasping or delivering an object).

The controls show that:

  • policy learning is indispensable - design search alone falls short of the best co-design method on all 13 presets where it was run;
  • the embodiment matters - the best co-design method beats fixed-design PPO on 13 of 22 presets, by up to 32.5 points;
  • on the other 9, fixed-design PPO is ahead of every co-design method, so current co-design methods do not yet reliably turn design freedom into better performance.

It is worth noting what exactly this comparison measures. Fixed-design PPO solves a strictly smaller problem with a mature policy optimization method, while co-design methods must search the joint space of designs and policies under the same budget, using far less mature co-design methods. Since fixed-design policy learning is a special case of co-design, the best achievable co-design return is never lower (Proposition 4, Section 3), so a fixed-design policy coming out ahead most likely reflects how hard and how young co-optimization is, not a general lack of benefit from co-design. Against this, it is encouraging that co-design already wins on 13 of 22 presets, by up to 32.5 points. The remaining presets are precisely the gap a co-design benchmark should expose, and the controls now let future methods measure their progress on it, preset by preset.

W4: Reward scales and the reporting inconsistency

Both points are now addressed in the paper. Our results now additionally report normalized scores, as suggested by the reviewer and as established in seminal work [3]; aggregating via normalization is the standard way to summarize performance on a heterogeneous suite [3, 4, 5]. The per-preset reference ranges are listed in Table 3 (App. C.1), and the aggregate is the interquartile mean over runs [9] (Table 1). We deliberately keep the raw rewards on their native scales, as in most RL benchmarks: real design problems do not come with standardized rewards, so robustness to reward scale is itself a property worth testing, and normalization is used only to compare across tasks. The inconsistency you noted in our results reporting has also been rectified — Table 1 and Figure 8 now refer to the same 35 presets. Thank you very much for pointing this out.

Thank you again for such a clear and constructive review. The broader scoreboard, the normalized scores and the controls came directly from your suggestions, and we believe they make the paper's message considerably sharper. We hope the revision addresses your concerns, and we would be very glad to continue the discussion if you have further critical remarks or suggestions.

AuthorsOpenReview ↗

References

[3] V. Mnih et al. Human-level control through deep reinforcement learning. Nature 518(7540):529–533, 2015.

[4] H. van Hasselt, A. Guez, D. Silver. Deep Reinforcement Learning with Double Q-learning. AAAI 2016.

[5] M. Hessel et al. Rainbow: Combining Improvements in Deep Reinforcement Learning. AAAI 2018.

[9] R. Agarwal, M. Schwarzer, P. S. Castro, A. Courville, M. G. Bellemare. Deep Reinforcement Learning at the Edge of the Statistical Precipice. NeurIPS 2021.

Reviewer_P2zcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Traditional reinforcement learning typically assumes a fixed agent morphology and focuses solely on optimizing the control policy. To address this limitation, this paper introduces Co-Design Gym, a unified benchmark for the joint optimization of agent embodiment and control policy. The platform brings together a diverse range of tasks spanning robotic manipulation and locomotion, soft robotics, energy systems, networking, racing, warehousing, and gaming, enabling systematic comparisons of co-optimization methods across different domains. The primary contribution of this work lies in establishing a cross-domain benchmark that supports reproducible and comparable evaluations of joint optimization methods. More broadly, it provides a common experimental framework for investigating a fundamental question: When a system’s design can itself be optimized, how can we identify effective combinations of embodiment and control policy that maximize task performance?

Critical strengths

1.This paper moves beyond the conventional reinforcement learning paradigm, in which an agent’s hardware configuration or morphology is fixed, by integrating design optimization with policy learning. It emphasizes that task performance depends not only on the control policy but also on the agent’s physical design, providing a unified framework for the co-optimization of robot morphology and control.

2.The platform encompasses 20 environment families and more than 85 task configurations, covering a broad range of domains, including continuous control, discrete combinatorial optimization, robotic manipulation, multi-robot collaboration, and resource scheduling. By extending beyond benchmarks tailored to individual robots or specific physical environments, Co-Design Gym enables a more comprehensive evaluation of joint optimization algorithms across diverse task settings.

3.Furthermore, by extending the Gymnasium interface to support explicit design spaces and design parameters, the paper lowers the engineering overhead associated with developing joint optimization tasks. This unified interface allows researchers to reuse existing environments, implement and evaluate new algorithms, and conduct reproducible experiments under a common framework.

Critical weaknesses

1.The paper primarily adopts a 4-hour wall-clock time budget for algorithmic comparisons. Although this choice reflects practical computational constraints, the methods may differ substantially in simulation throughput, the number of environment interactions, and pretraining overhead. Consequently, the observed performance differences may not be attributable solely to the underlying optimization algorithms, but may also reflect differences in implementation efficiency and computational resource utilization.

2.The evaluation considers four representative approaches to joint optimization; however, these methods do not encompass the full spectrum of potentially effective strategies. For example, more diverse gradient-based methods, surrogate-model-based optimization, hierarchical search, and alternative learning-based design generation approaches are not explored. Consequently, while the experimental results provide useful insights into the relative strengths and weaknesses of the selected methods, they are insufficient to establish a comprehensive picture of the performance potential of existing joint optimization techniques.

3.Furthermore, reward functions and numerical scales vary substantially across environments, making direct comparisons of raw returns across tasks difficult. Some tasks also exhibit considerable variability across random seeds, while the generation of invalid designs introduces additional challenges. Therefore, relying primarily on mean returns and standard deviations may be insufficient to comprehensively characterize algorithm performance. Additional metrics, such as sample efficiency, design validity rate, optimization stability, and cross-task generalization, would provide a more complete and informative assessment of the evaluated methods.

Questions for authors

The paper primarily reports mean evaluation returns and standard deviations across five random seeds. However, reward scales vary substantially across environments, making raw returns difficult to compare across tasks. How do the authors determine the overall relative effectiveness of different co-design algorithms across the benchmark?

The paper motivates Co-Design Gym by the limited domain coverage and physical modeling capabilities of existing benchmarks, such as SoftZoo and EvoGym. However, the proposed environments differ substantially in their design-space structures, simulation characteristics, and reward definitions. What principles guided the selection and construction of these environments?

The proposed benchmark provides a broad collection of simulated environments, including physically grounded robotics tasks and non-robotic problems. Given that simulation-to-reality transfer remains a challenge in co-design, what evidence supports the practical applicability of the benchmark findings to real-world systems?

The paper includes an exploratory study in which an LLM is used as an outer-loop design optimizer and reports poor performance in the evaluated setting. Could the authors clarify the extent to which the outcome depends on the selected model and whether the experiment supports a conclusion about LLM-based design optimization in general or only about the particular implementation studied here?

7 comments
AuthorsOpenReview ↗

Response to Reviewer P2zc (1/5)

Thank you for the thoughtful review. We are glad you found that Co-design Gym "moves beyond the conventional reinforcement learning paradigm", that it "enables a more comprehensive evaluation of joint optimization algorithms across diverse task settings", and that extending Gymnasium "lowers the engineering overhead" Your suggestions on metrics and baselines shaped several of the most useful additions to the revision.

W1: The 4-hour wall-clock budget

You are right that a wall-clock budget favors algorithms that use the available hardware efficiently.

We discussed the choice of budgeting metric at length internally when designing the experiments. To our knowledge, there is no established standard for comparing algorithms as different as those in our study, and every candidate metric introduces its own bias. For instance, environment timesteps favor sample-efficient methods, whereas FLOPs do not account for parallelization. We ultimately chose wall-clock time because it most closely reflects practical constraints.

Since no single metric is unbiased, we believe the most constructive path forward is to make all relevant cost measures visible. We now report, for every method and preset, the environment steps taken within the budget, throughput, trainable parameters, the steps needed to reach 90% of the final score, design-rejection rates and the drop from best to final score (App. C.4, Tables 10–12, Fig. 23). Summarized over all tested presets, 5 seeds each (175 runs per method; median, with the interquartile range in brackets):

MethodBudgetEnv. steps per run (M)Steps / sTrainable parameters, median [range over presets]
CMA-ES4 h1,176 [13–2,639]81,666 [920–183,254]4.4 k [390–32.7 k]: controller weights + design parameters, searched as one vector
FastTD34 h209 [11–799]14,530 [794–55,455]2.0 M [1.6 M–5.0 M]: actor + critics
PPO+NGOpt4 h250 [1.8–763]17,301 [134–52,690]25 k [9.7 k–135 k]: actor + critic (two 64-unit hidden layers each)
LOKI1 h preparation + 3 h online24 [5.4–42]2,197 [497–3,884]20.2 M [20.1 M–21.3 M]: six Transformer controllers (about 3.3 M each, actor + critic) + design VAE (0.25 M)

The parameter counts span a wide range because every network is sized by its preset's observation, action and design dimensions. The smallest models are on MicrogridCampus (a 5-value observation, 3 actions and 3 design parameters) and the largest on WarehouseCongested (8 robots with a 528-value observation and 438 design bits); LOKI's Transformer controllers have a fixed width, so its size barely changes.

MethodSteps to reach 90% of the final score (M)Evaluated designs rejectedBest-to-final drop (normalized points)
CMA-ES242 [4.8–1,295]none0.00 [0.00–0.62]
FastTD367 [4.6–197]9.3% on average; in 34 of 165 runs (NeroGrasp, Pokémon, two Network presets, SoftWalker, WarehouseCongested)0.26 [0.00–6.1]
PPO+NGOpt83 [1.0–440]none0.02 [0.00–2.7]
LOKI5.6 [2.4–22]none0.08 [0.00–1.5]

Our findings suggest that the methods differ by more than an order of magnitude in environment interactions within the same budget, yet the main conclusion that there is no single winner does not depend on the budgeting metric: compared at matched environment steps rather than at the end of the budget, the leading method changes on 22 of 35 presets, but no method dominates. Regarding pre-training: LOKI's preparation (data generation, VAE pre-training and clustering) is part of its 4-hour budget, which is why it receives 3 hours of online optimization (Section 5, Table 10). We would be happy to provide any further comparisons.

AuthorsOpenReview ↗

Response to Reviewer P2zc (2/5)

W2: Breadth of baselines

We agree that four methods cannot cover the full spectrum of joint optimization strategies, and we were happy to widen the set.

Following the reviewer's suggestion, we have also run two additional methods from distinct families, using the same API and evaluation protocol. These were: LA-MCTS [10], representing tree-based Monte Carlo search approaches that learn to partition the search space in a more hierarchical manner, and SPSA [11, 12], which uses simultaneous-perturbation finite differences to estimate gradients through the non-differentiable simulator interface. Both search the joint vector of embodiment and controller parameters (App. B, App. C.3, Tables 8–9). Together, the six baselines now span evolutionary, tree-search, stochastic-gradient, reinforcement-learning, alternating, and generative approaches. Interestingly, LA-MCTS and SPSA lead on NetworkUrban, where every other baseline scores below the random policy:

NetworkUrban (normalized %)CMA-ESFastTD3PPO+NGOptLOKILA-MCTSSPSA
Final evaluation−7.6−15.0−4.3−1.37.36.8

On the other 19 presets they do not lead, so the conclusion that no method dominates is unchanged. In the revised manuscript we also clarify that our empirical findings characterize the selected representative methods rather than the full space of co-design techniques. We would welcome the reviewer's recommendation of any specific further baseline they consider important, and we would be glad to include it in the benchmark.

We would additionally like to clarify the scope of our contribution. Our primary contribution is the benchmark itself: the environments, the common interface and the evaluation protocol. The representative baselines were selected to represent distinct paradigms of joint optimization: (i) they show that the benchmark is challenging and discriminative, i.e., that it separates methods meaningfully, and (ii) they give future work a starting point for comparison.

A comprehensive method comparison of the kind the reviewer describes is precisely the kind of community contribution a standardized benchmark is meant to enable. To support this, the benchmark is open-source and modular, so new methods can be added easily. This scope is consistent with established RL benchmark papers: the Arcade Learning Environment [6] reported SARSA(λ) with linear function approximation and two search methods as "a point of comparison with future, more advanced, approaches"; the DeepMind Control Suite [7] presented A3C, DDPG and D4PG "as a starting point for comparison"; and Gymnasium [8] reports no algorithm evaluation at all.

AuthorsOpenReview ↗

Response to Reviewer P2zc (3/5)

W3 / Q1: Reward scales, additional metrics and overall effectiveness

We added the additional metrics suggested in the review; together they now form App. C:

Suggested metricWhere in the revision
Normalized scores and their aggregate (IQM over runs)Table 1, App. C.1, Tables 3–5
Sample efficiency (steps within budget, steps to reach 90% of the final score, score vs. steps by category)Table 11, Fig. 23
Design validity (percentage of evaluated designs rejected)Table 12
Optimization stability (best-to-final drop, seed spread, bootstrap confidence intervals)Table 12, Tables 5 and 9, Fig. 23

We agree that reward magnitudes differ substantially across environments. However, this is the norm in essentially every widely used RL benchmark. For instance, in Atari, raw per-game scores range from roughly −22 to +16 in Double Dunk to several hundred thousand in Atlantis ([1], Table 7). In the classic MuJoCo environments, returns range from roughly 3,000 in Hopper to around 13,500–15,500 in HalfCheetah ([2], Table 1).

Hence, keeping the heterogeneous reward scales is a deliberate feature of the benchmark rather than a shortcoming. Real-world design problems do not come with standardized reward magnitudes, so an algorithm's robustness to reward scale is a practically relevant property. Forcing all environments onto a common scale would itself be a limitation, as it could systematically favor some methods over others; maximum-entropy methods, for instance, are known to be sensitive to reward scale.

Nonetheless, we do agree that for an overall assessment across the benchmark (Q1), normalization could be useful, and seems to be the established remedy [3, 4, 5]. Hence, we now report normalized scores following Mnih et al. [3]: 100×(R−Rrand)/(Rref−Rrand)100 \times (R - R_\text{rand}) / (R_\text{ref} - R_\text{rand}), where RrandR_\text{rand} is a uniform-random policy on the base embodiment and RrefR_\text{ref} the theoretical maximum return or a physics-based cap where returns are unbounded. We aggregate them with the interquartile mean over runs, as recommended by Agarwal et al. [9]: FastTD3 has the highest IQM (13.5), followed by CMA-ES (10.5), PPO+NGOpt (9.1) and LOKI (7.0) (Table 1). Because each method wins on a different cluster of domains (Section 6), we present this summary alongside the per-preset results rather than as a single ranking. Cross-task generalization is a direction the framework supports naturally, since sibling presets share design and action spaces (e.g., BallCatcher-Pitch, -Arc and -Fielding), and we see it as an exciting direction.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to Area Chair and Reviewers

We thank all reviewers for their careful and constructive reviews. We were encouraged that the reviewers found that Co-design Gym "addresses a meaningful gap", is "a real infrastructure contribution", and “makes this class of problems more accessible to the community.” We have run a number of experiments to improve the evaluation of the baselines and updated the scoreboard, tables, and text accordingly to address the reviewers’ points. We believe this has strengthened the paper, showing with further experiments how Co-design Gym serves as a benchmark that helps to discern the strengths and shortcomings of various families of co-design methods.

What we have added

Significantly expanded evaluation and updated the scoreboard. The main table covers 35 presets, up from 18, with at least 4 baselines each run for 4 hours and 5 seeds (for a total of 2,800 hours of experiments). Now every one of the 20 environment families is represented, including the soft-body and shape-changing hand tasks (Table 1).

Confirmed that Co-design Gym separates methods. No single method dominates: FastTD3 is best on 13 of 35 presets, CMA-ES and PPO+NGOpt on 9 each and LOKI on 4, and the leader changes during training on 24 presets (Table 1, Fig. 8).

Measured the value of co-design. Design search alone never matches the best co-design method, and co-design beats a policy trained on the fixed base design on 13 of 22 presets, by up to 32.5 points; on the remaining 9 the fixed-design policy is still ahead. PPO solves only the policy-learning half of the problem with a mature method, while co-design methods must also search over designs within the same budget. Since keeping the base design is always one of co-design's options, the best co-design result can never be worse (Section 3). The gap therefore shows that current co-design methods are still young, not that co-design has no benefit, and gives future methods a clear target (App. C.2).

Showed that Co-design Gym leaves substantial headroom. The best method reaches less than 10% of the achievable score on 11 presets, with a median of 22.7% across presets (Section 6).

Clarified that Co-design Gym supports both two-level and single-level Co-design. The API does not prescribe how design and control optimization are organized: it supports two-level methods, in which an outer design optimizer proposes embodiments for an inner policy learner, and single-level methods, in which one agent or optimizer chooses both, and our baselines cover both (App. E).

Clarified that wall-clock budgets reflect practical constraints, but also ensures that other types of costs are now reported as well. Alongside the budget, every method reports environment steps, throughput, model size, sample efficiency, design validity and stability, so results can also be compared at equal experience (App. C.4).

Showed how to plug in new benchmark families. We tested two new families of co-design methods, showing that the common interface provided by the benchmark ensures fast integration. Overall, the six baselines span evolutionary, tree-search, stochastic-gradient, reinforcement-learning, alternating and generative approaches.

Provided normalized scores for easy comparison. Scores are normalized between a uniform-random policy and the theoretical maximum return, with all anchors listed, and aggregated with the interquartile mean (Table 1, App. C.1).

We have updated the PDF noting the revisions in blue. A brief summary of the PDF updates:

New:

  • updated scoreboard with expanded experiments; normalized scores and IQM (Table 1, App. C.1, Tables 3–5);
  • policy-only and design-only controls (App. C.2, Tables 6–7);
  • two additional baselines, LA-MCTS and SPSA (App. B, App. C.3, Tables 8–9);
  • environment steps, throughput, model sizes, sample efficiency, design validity and stability (App. C.4, Tables 10–12, Fig. 23);
  • LOKI with 40 clusters (App. C.5, Table 13);
  • a specification table for every environment family: design space, rejection rule, return range and episode length (App. A, Table 2).

Clarified:

  • two-level and single-level Co-design in the API (App. E);
  • the scope of the findings and of sim-to-real transfer (Sec. 6);
  • the scope of the exploratory LLM study (App. D).

Corrected:

  • preset counts in Fig. 8 and Sec. 6, the placeholder in App. D, and the rows of the LLM table (Table 14).
AuthorsOpenReview ↗

Final Remarks and a Summary of Revisions/Discussions

We would like to sincerely thank all three reviewers for their time and engagement throughout the discussion period. Their suggestions led directly to the most valuable additions in the revision, and the paper is much stronger for it.

All three reviewers agree on the core contribution. Co-Design Gym “addresses a meaningful gap” (ynk6), “enables a more comprehensive evaluation of joint optimization algorithms across diverse task settings” (P2zc), and is “a real infrastructure contribution beyond domain-specific platforms” that is “more applicable than existing benchmarks” (ecMz).

The main concerns raised were addressed with new experiments:

  • Scoreboard coverage: expanded from 18 to 35 presets, covering all 20 environment families, including the soft-body and shape-changing tasks (2,800 GPU hours of experiments).

  • Measuring the value of co-design: added policy-only and design-only controls on 22 presets (App. C.2).

  • Breadth of baselines: added two new method families, for six baselines in total (App. C.3).

  • Comparing across tasks and costs: normalized scores with IQM, plus environment steps, sample efficiency, model sizes, design validity and stability (App. C.1, C.4).

  • Baseline fidelity: added LOKI with its original 40 clusters (App. C.5), and a full specification table for every environment family (Table 2).

The remaining suggestion, from Reviewer ecMz, is to focus on a single domain and add domain-specific evaluation measures such as morphological computation and empowerment. We see this as potential for valuable future extensions in domain-specific applications. Single-domain co-design benchmarks already exist, and what the field lacks is a common benchmark across domains, in the way OpenAI Gym and the ALE served RL. Our results show why this matters: no single method dominates, and the leading method changes across domains, which a single-domain benchmark cannot reveal.

We hope Co-Design Gym can give the co-design community the shared ground that helped RL progress so quickly. Thank you again to the reviewers and the Area Chair for your valuable time and care.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.