acceptodds
Under review as a conference paper at ICLR 2027

CDRL-Bench: Benchmarking Cross-Domain Transfer in Reinforcement Learning

Abstract

Cross-domain reinforcement learning (CDRL) aims to improve sample efficiency by reusing knowledge across tasks, environments, and embodiments. However, existing benchmarks largely cover specific subsets of CDRL, such as dynamics variation, while broader settings involving heterogeneous state and action spaces are evaluated less systematically and lack a unified framework. We introduce CDRL-Bench, a unified benchmark for evaluating transfer between heterogeneous source and target Markov decision processes. CDRL-Bench spans four transfer settings: same-task morphology transfer, cross-task sub-skill transfer, transfer from pretrained generalist models, and real-world transfer on physical robotic hardware. Across locomotion, navigation, and robotic manipulation, CDRL-Bench standardizes source–target pairs, interaction budgets, transfer protocols, and evaluation metrics, enabling fair and reproducible comparison across diverse transfer approaches. By broadening evaluation beyond morphology adaptation, CDRL-Bench provides a unified testbed for studying more general forms of cross-domain knowledge transfer.

Primary area:
reinforcement learning
Keywords:
Cross-domain reinforcement learning, benchmarking, cross-domain transfer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LheoOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed an unified benchmark environments for Cross-domain Reinforcement Learning (CDRL) - which includes 1). Same-task morphology transfer, 2). cross-task sub-skill transfer, 3). transfer from pretrained generalist models, and 4). real-world transfer on physical robotic hardware; Then the authors also provide comprehensive comparisons among existing work and make lots of analysis over it; Overall - this paper is well-written and easy to follow, and it definitely has the contributions to the community. It also covers lots of environments in the previous work and also include the real-world transfer which is more practicals in my opinion. I prefer to accept it; The only concern is should this paper be classified into Evaluations & Datasets Track?

Critical strengths
  • Provide an unified benchmark environments for the CDRL research community.
  • Have comprehensive comparison among several existing methods.
  • Provide the setting for the real-transfer in the real physical robotic, and the robot is also the open-sourced one which allow easy access for other researchers
Critical weaknesses
  • The user didn't report the success rate, only the accumulated reward is reported, which cannot reflect if the agent complete the task or not;
  • The number of tasks is not that lot, of course users has proposed a very new setting, which is real-transfer, however, for the other tasks/settings they're already exists in the previous literature reviews; Although it's not a big problem, it still recommend the authors can keep adding more meaning tasks into the environment.
  • This work is more suitable for Evaluations & Datasets Track?
Questions for authors

The only question is - will the authors keep maintaining the benchmark repo - for example - adding more robotic arm selections, adding more tasks etc, and keep adding more baselines. This can help other researchers to focus on the core algorithms developments and avoid repeating baseline training work.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Lheo

We sincerely thank the reviewer for the thoughtful and constructive feedback. We address each concern point by point below and have revised the manuscript accordingly, with all changes highlighted in blue.

Q1: Lack of task success rate evaluation.

A: Thank you for the suggestion. We agree that cumulative reward alone may not fully reflect whether an agent successfully completes a task. To address this, we additionally include success-rate learning curves alongside reward curves for all task-oriented environments. Due to space constraints, we only show the results of cross-morphology transfer on the Door/Lift tasks (see: https://imgur.com/a/Wlm6yOx, corresponding to Figures 17-18 in Appendix C). Evaluation rewards and success-rate curves for all remaining task-oriented environments are provided in Appendix C.

The success-rate trends closely follow the evaluation rewards: as returns approach expert-level performance, success rates approach 1.0. Conversely, for sub-skill and navigation tasks where rewards show limited improvement, success rates remain near zero. For locomotion tasks, where binary success criteria are not defined, we continue to report episodic returns.

Episodic return is particularly important for evaluating knowledge transfer in CDRL, as it captures incremental learning progress that binary success rates may overlook. Since the primary goal of CDRL is to assess how effectively transferred knowledge accelerates target-domain learning, relying solely on success rates could obscure meaningful improvements before task completion. We therefore report both episodic returns and success rates (where applicable) to evaluate transfer efficiency and task completion comprehensively.

Q2: The number of tasks is not that lot, of course users has proposed a very new setting, which is real-transfer, however, for the other tasks/settings they're already exists in the previous literature reviews; Although it's not a big problem, it still recommend the authors can keep adding more meaning tasks into the environment.

A: Thank you for the suggestion. CDRL-Bench covers 53 transfer configurations spanning locomotion, robotic manipulation, maze navigation, cross-task sub-skill transfer, and Real2Real transfer, varying in embodiment, task, source composition, and observation modality.

We agree that several individual settings have precedents. The contribution lies in bringing these previously fragmented settings into a unified evaluation framework, alongside Real2Real transfer. Shared protocols, baselines, and pretrained source models enable systematic comparisons that isolated evaluations cannot provide, revealing how transfer effectiveness changes across morphology families, target embodiments, and source combinations.

We have also added GR1 humanoid → Baxter transfer in Robosuite, expanding coverage to substantially different embodiments. We will continue adding meaningful tasks, embodiments, and baselines. In summary, CDRL-Bench combines established and new settings to test whether CDRL advances generalize beyond the individual tasks on which they were developed.

Q3: Will the authors keep maintaining the benchmark repo - for example - adding more robotic arm selections, adding more tasks etc, and keep adding more baselines. This can help other researchers to focus on the core algorithms developments and avoid repeating baseline training work.

Thanks for the suggestion! Yes, we are committed to maintaining CDRL-Bench as a shared resource for the CDRL community. Specifically:

  1. We will expand its robotic embodiments, transfer tasks, and baseline algorithms. Notably, the newly added GR1 → Baxter scenario and morphology-aware baselines are concrete steps in this direction.

  2. We will also maintain reusable baseline implementations and pretrained source models, with documented training and evaluation configurations, to support reproducible comparisons.

Our goal is to let researchers build on a common evaluation infrastructure and devote their effort to algorithmic advances rather than repeatedly implementing and training baselines.

Q4: This work is more suitable for Evaluations & Datasets Track?

A: Thank you for raising this question. We agree that the work has a strong evaluation and benchmarking focus. Our understanding is that ICLR 2027 does not offer a separate Datasets & Benchmarks track, while its Call for Papers explicitly includes “datasets and benchmarks” among the relevant topics. We therefore believe this submission aligns with the conference’s scope.

Beyond the benchmark itself, our unified evaluation reveals when transfer helps or hurts across embodiments, when additional sources fail to improve performance, and how inherited critics can impede fine-tuning.

We hope CDRL-Bench contributes both a useful community resource and empirical insights that support the development of more general CDRL methods.

Dear Reviewer Lheo,

Thank you for your positive and encouraging review. With about 24 hours left in the discussion, we kindly invite you to review our rebuttal and are happy to clarify any remaining concerns. We briefly summarize the main updates addressing your concerns:

Success rates (Q1): We added success-rate curves for all task-oriented environments, which closely track returns, while retaining returns to capture incremental transfer progress. Representative results for cross-morphology transfer on Door/Lift are shown in results, with the remaining results in Appendix C.

Benchmark coverage and expansion (Q2): We clarified that CDRL-Bench unifies 53 transfer configurations under shared protocols, baselines, and source models, and added a Baxter-to-GR1 humanoid transfer experiment.

Maintenance (Q3): We committed to continually expanding the benchmark with new tasks, embodiments, and baselines, and to maintaining reusable implementations and pretrained source models to support future research.

Venue fit (Q4): ICLR 2027 has no separate Datasets & Benchmarks track, and its Call for Papers explicitly includes datasets and benchmarks. Beyond the resource itself, our evaluation offers insights into when transfer helps or hurts across embodiments, when additional sources fail, and how inherited critics impede fine-tuning.

We hope these additional experiments and clarifications have addressed your concerns and would be grateful if you could take them into account in your final evaluation. We sincerely appreciate your time and valuable suggestions, and look forward to hearing from you.

Best regards,

Authors of Paper 18680

Reviewer_EKTLOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CDRL-Bench, a benchmark for cross-domain RL where source and target MDPs can have different state and action spaces. It has four groups of settings. The first is morphology transfer: Cheetah, Swimmer, Ant and Centipede variants with more legs or links, Robosuite Lift and Door from Panda to other arms, and PointMaze to AntMaze. The second is sub-skill transfer: Lift to harder manipulation tasks, PointMaze plus Ant to AntMaze, and four single-arm Lift policies to TwoArmLift. The third transfers from the pretrained 30-task TD-MPC2 model into a small SAC policy. The fourth is a real SO-ARM101 reaching task where the target locks one of the four source joints. Source models are frozen and shared, all methods use the same target interaction budget, and results are reported as expert-normalized best score and average normalized regret. Seven methods are evaluated: PPO, SAC, FT-PPO, FT-SAC, CAT-PPO, CAT-SAC and QAvatar-SAC. The paper concludes that transfer benefits depend strongly on the setting, that fine-tuning helps little, that maze and complementary-skill settings remain unsolved, and that no transfer method beats SAC from scratch on the real robot.

Critical strengths

There's a real gap here. CDRL papers each pick their own source–target pairs and protocols, and existing benchmarks mostly cover dynamics shift with matching state and action spaces. A shared suite that includes heterogeneous spaces, cross-task sub-skills and multi-source transfer would be useful, and the released single-file baselines and environment XMLs lower the barrier to entry. The real-robot setup is described in unusual detail (Appx. D), which I appreciated.

Critical weaknesses

A large part of the benchmark doesn't separate methods, and the main metric isn't defined. In Table 6, all 16 maze rows (Point→Ant morphology and PointMaze+Ant→AntMaze) have regret between 0.97 and 1.00 for every method, including SAC and PPO from scratch. Another 10 sub-skill rows (Lift→X, TwoArmLift) sit between 0.84 and 1.00. So close to half of the 53 configurations either can't be learned by anything at this budget or are near that point. They tell us nothing about transfer. On top of that, both metrics are normalized by an "expert policy in the target domain", but the paper never says what the expert is, how it was trained, or with what budget. Without that, I can't tell whether the maze tasks are solvable at all, or how to read any normalized number in the paper. For a benchmark paper, these two issues go to the core of the contribution.

The baselines are too few, and some results look like implementation problems rather than findings. Only two dedicated CDRL methods are evaluated (CAT, in two variants, and QAvatar). Morphology-aware methods such as NerveNet and TURRET, and hierarchical approaches like Hejna et al., are excluded because they need extra structure. But in a morphology-transfer benchmark that structure is available, and these are the methods people actually use. The generalist-model setting ends up evaluating QAvatar alone, because of a "compact target" constraint the authors chose themselves. FT-PPO has regret of 0.99–1.02 on all six Door/Lift configurations, meaning it stays at or below random for the whole run, while PPO from scratch learns (0.63–0.90). I'd expect a bug before I'd accept this as evidence that fine-tuning is unreliable. There are also consistency issues:

  • I could not reproduce Table 1 by averaging Table 6. For example, A+C CAT-SAC averages to 0.675 vs. 0.63 reported, and D+L FT-SAC to 0.70 vs. 0.67.
  • The single Real2Real row differs between the two tables.
  • The claim that QAvatar-SAC beats SAC "in most settings" isn't true for regret: it is worse in 3 of the 6 groups in Table 1.
  • The real robot uses 4 seeds, not the 5 stated in §5.2, and only CAT's hyperparameters were retuned for it.
Questions for authors
  1. What exactly is the "expert" for each target? How was it trained, and with what budget? Can it solve AntMaze? If SAC from scratch, all transfer methods and possibly the expert make no progress on the maze settings, what are those 16 configurations measuring?
  2. FT-PPO stays at or below random on every Door and Lift configuration, while PPO from scratch learns. Initializing from source weights shouldn't do that. Did you check whether this is an implementation issue, for example the log-std carried over from the source, or the reinitialized layers?
  3. Why leave morphology-aware methods out of a benchmark whose first and largest setting is morphology transfer? If the benchmark can't host the standard methods for that setting, I'm not sure what its morphology results tell a reader that prior papers didn't.
  4. In the generalist setting, how does QAvatar actually use TD-MPC2? It's a task-conditioned latent world model, not a per-task SAC critic. Also, why is "the target must be compact" part of the benchmark definition rather than one possible choice, given that it's what rules out FT and CAT?
  5. On the real robot, the source and target differ only in that one joint is locked, so the correspondence is known: drop that action dimension and the joint's state entries. Would simply running the source policy with that dimension removed beat every method in Fig. 7? More generally, have you tried oracle-mapping baselines in the simulated settings? Without them, it's hard to tell whether a failure comes from the methods or from the task.
  6. How is Table 1 computed from Table 6? Simple averaging doesn't reproduce it.
5 comments
AuthorsOpenReview ↗

Response to Reviewer EKTL (1/4)

We sincerely thank the reviewer for the thoughtful and constructive feedback. We address each concern point by point below and have revised the manuscript accordingly, with all changes highlighted in blue.

Q1: Unclear metric definitions, unspecified expert references, and task solvability.

A: Thanks for pointing this out.

1. Metric definitions. As defined in Section 4.5 of the paper, the expert-normalized score (NS) measures normalized return between random (=0) and expert (=1), while Average Normalized Regret Rˉ\bar{R} measures the average gap to expert performance over training (0 = immediate expert performance; 1 = random-level throughout).

2. Expert references. We clarify how to interpret the expert references in Appendix A.5 in the revised version. Specifically, JrandJ_{rand} denotes the return of a random policy, whereas JexpertJ_{expert} is obtained as follows:

  • Locomotion task: Highest evaluation return of SAC trained from scratch for 10610^6 steps across 5 seeds.

  • Robot Arm Manipulation task: Total return of human-level demonstrations collected via keyboard teleoperation.

  • Navigation task: Following the Minari [2] / D4RL [4] hierarchical protocol, we use a Q-Value Iteration (QIteration) waypoint planner paired with a SAC-trained low-level goal-reaching policy to control the Ant's joints.

  • Real2Real: Best 100-episode moving average return(~150) achieved by a 3-joint target-domain SAC policy trained from scratch, which averages out single-episode variance from random target spawn locations.

Crucially, all expert trajectories in Manipulation and Navigation successfully complete the tasks.

3. Task solvability and interpretation. Since each expert reference is computed as the total reward collected along successful expert trajectories, every expert return corresponds to an achievable behavior in the target domain. The low performance on the sub-skill and navigation tasks instead stems from their need for long-horizon exploration. For the sub-skill tasks, even the original Robosuite benchmark reports [1] only slow SAC progress on Stack, Nut Assembly, Pick-and-Place, and Two Arm Lifting. The gap between SAC from scratch and the expert is precisely what source-domain knowledge is expected to close, and QAvatar-SAC already narrows it consistently. Navigation is challenging for the same reason, and HDI [3] (see: https://imgur.com/a/qPxCUly and Figure 26 in the paper) confirms that knowledge transfer from Point to Ant is achievable in this setting. Therefore, a regret near to 1 indicates that current CDRL methods fail to exploit transferable knowledge within the given budget, rather than that the task itself is unsolvable.

Q2: Limited benchmark discriminability in high-regret maze and sub-skill tasks.

A: CDRL-Bench is designed to cover both tasks that clearly distinguish current methods and tasks that remain challenging and largely unsolved by existing CDRL methods. Tables 1 and 6 and Figure 4 illustrate this range. For example, on Ant transfer (4leg–5leg → 6leg), QAvatar-SAC achieves a regret of 0.35, compared with 0.64 for SAC and 0.59 for CAT-SAC. On robotic sub-skills, QAvatar-SAC achieves average regret of 0.89, compared with 0.95 for SAC. On the other hand, we acknowledge that the maze tasks offer limited discriminability under the current budget.

Sub-skill transfer. Existing CDRL baselines allow arbitrary source-target MDP differences, yet prior evaluations focus on morphology transfer with identical reward functions. Sub-skill transfer tests this broader formulation, and QAvatar-SAC’s improvement over training from scratch provides evidence of its potential.

Maze navigation. HDI [3] demonstrates Point-to-Ant transfer on UMaze using predefined hierarchical decompositions. We extend evaluation to Open, Medium, and Large layouts, which remain challenging without such structural priors. Following D4RL [4] and ODRL [5], we retain these difficult configurations as targets for future progress, while recognizing their limited ability to distinguish current methods.

In summary, CDRL-Bench measures both where transfer works today and where new capabilities are needed to advance the field.

[1] Zhu et al., “robosuite: A Modular Simulation Framework and Benchmark for Robot Learning.” arXiv:2009.12293, 2020.

[2] Younis et al., "Minari." Zenodo, 2024.

[3] Hejna et al., "Hierarchically Decoupled Imitation for Morphological Transfer." ICML 2020.

[4] Fu et al., "D4RL: Datasets for Deep Data-Driven Reinforcement Learning." arXiv preprint arXiv:2004.07219, 2020.

[5] Lyu et al., "ODRL: A Benchmark for Off-Dynamics Reinforcement Learning." NeurIPS 2024 (D&B Track).

AuthorsOpenReview ↗

Response to Reviewer EKTL (2/4)

Q3: Potential implementation issues behind FT-PPO's failure on Door and Lift tasks.

A: FT-PPO is implemented as intended, and the Door/Lift degradation arises from its critic-transfer initialization. We agree that these results should not be interpreted as evidence that fine-tuning is generally unreliable. Our additional experiments isolate the inherited critic as the main cause.

Implementation. FT-PPO transfers the actor’s hidden layer and the critic’s hidden and output layers. All dimension-dependent components, including the actor’s input/output layers and critic’s input layer, are reinitialized for the target domain, and log-standard deviation is initialized exactly as in PPO. Thus, FT-PPO uses a target-compatible architecture with the intended parameter transfer.

Why this initialization hurts Door/Lift. Early target policies rarely reach reward-bearing stages, yielding near-zero returns with little variation (0.050.15 in Lift; 0.280.21 in Door). However, the inherited critic initially predicts values on the scale of trained source-policy returns. This mismatch distorts advantage estimates and delays policy learning until the critic adapts. PPO’s freshly initialized critic starts near zero and adapts much faster. Locomotion is less affected because dense rewards provide substantially more variable returns and stronger learning signals.

Two controlled experiments support this diagnosis:

  • FT-PPO exhibits delayed learning rather than complete failure. When training is extended to 10610^6 steps (see https://imgur.com/a/0GBqa5r and also in Figure 24 of the revised appendix), some seeds begin to improve after about 5×1055\times10^5 steps, although they remain below PPO’s performance at 5×1055\times10^5 steps.

  • We further evaluate an actor-only variant, FT-PPO-AO, in which the entire critic network is reinitialized and only the actor's hidden layer is transferred from the source model. As shown in https://imgur.com/a/fVsVAqs and Figure 25 of the revised appendix, FT-PPO-AO performs comparably to PPO in all Door and Lift tasks, confirming that the failure of FT-PPO stems from the inherited critic rather than from the transferred actor features.

The failure is therefore a limitation of FT-PPO’s critic-transfer design in these tasks: inherited value estimates can obstruct learning even when the implementation is correct and the transferred actor features remain usable.

Q4: In the generalist setting, how does QAvatar use TD-MPC2, given that it is a task-conditioned latent world model rather than a per-task SAC critic? Why is the compact target part of the benchmark definition?

A:

Using TD-MPC2 as a source critic. Although TD-MPC2 is a task-conditioned latent world model, it also learns a latent Q-function that can serve as a task-specific critic when conditioned on a fixed task embedding. Specifically, for each pretrained task (i), fixing its learned embedding eie_i yields Qsrc,i(s,a)=Q(h(s,ei),a,ei)Q_{\mathrm{src},i}(s,a) = Q(h(s,e_i),a,e_i), where hh denotes the TD-MPC2 encoder. Since QAvatar only requires access to a source Q-function, we treat each of the 30 pretrained tasks as an individual source and learn a separate cross-domain mapping for each. This allows QAvatar to leverage the generalist model without requiring a per-task SAC critic.

Motivation for the compact-target benchmark. Fine-tuning pretrained generalist models on new target tasks has already been investigated in prior work, including the original TD-MPC2 paper [6] and Octo [7]. In contrast, transferring knowledge from a large generalist model into a compact, independently parameterized target policy is an important yet underexplored direction, offering a practical way to leverage pretrained knowledge while retaining the efficiency and flexibility of smaller models. We therefore deliberately define the benchmark around the compact-target regime to investigate this distinct transfer setting. Importantly, QAvatar's architecture-agnostic design enables such transfer without requiring architectural compatibility between the source and target networks.

[6] Hansen et al., "TD-MPC2: Scalable, Robust World Models for Continuous Control." ICLR 2024.

[7] Octo Model Team., "Octo: An Open-Source Generalist Robot Policy." RSS 2024.

AuthorsOpenReview ↗

Response to Reviewer EKTL (3/4)

Q5: Exclusion of morphology-aware baselines and the added value of the morphology transfer results.

A: Thank you for the suggestion. We have added morphology-aware baselines, and our unified evaluation reveals transfer patterns that narrower morphology studies can miss.

Added baselines. The revised locomotion suite includes:

  • TURRET [8]: A morphology-aware transfer method using a GNN.
  • GNN-PPO: PPO trained from scratch with the same GNN architecture, isolating the benefit of transfer from that of the backbone.

TURRET requires explicit morphology graphs, identical per-node feature dimensions, and a shared maximum node count across domains. These priors make it applicable to locomotion but limit its use in settings without a physical joint graph, such as image-based Real2Real, and with arbitrary pretrained source architectures.

Because the released TURRET repository provides neither an official implementation nor hyperparameter settings, we implement TURRET from the paper and tune its hyperparameters. We build on CAT’s lateral-connection mechanism, replacing its MLP backbone with a GNN and computing source weights from embedding distances.

In https://imgur.com/a/Gtq16Hi and Figures 13-16 in the revised paper, TURRET matches or outperforms GNN-PPO on all morphology pairs except Cheetah 3-leg → 4-leg, demonstrating gains from transfer. GNN-PPO matches or underperforms PPO, consistent with the original paper. TURRET does not outperform CAT-PPO in our suite. One plausible explanation is scale and budget: the original paper reports stronger gains on Centipede-16/20 and later in training with 10710^7 steps, whereas our agents have at most eight legs and budgets of 5×1055\times 10^5--10610^6 steps.

Structure-aware navigation reference. We also evaluate HDI [3] on Point → Ant under the same target-domain budget and evaluation protocol. Its required hierarchical source model is retrained to match our shared source models’ performance. As shown in https://imgur.com/a/qPxCUly and Figure 26 in the paper, HDI reaches nearly 100% success on dense-reward OpenMaze and makes partial progress on other layouts, but fails on LargeMaze. As a reference with predefined hierarchical structure, HDI demonstrates that useful transfer is possible and highlights the challenge of discovering that structure automatically.

Added value of the morphology track. Table 6 reveals three findings under a unified protocol:

1. Rankings depend on the morphology family. QAvatar-SAC beats SAC on all four Ant pairs, while SAC is best on all four Centipede pairs. Results from a few families therefore do not establish general superiority.

2. Changing only the target embodiment can reverse rankings. With the same Panda source and Door task, CAT-SAC is best on Jaco, whereas SAC is best on UR5e.

3. Multi-source benefits depend on both method and family. QAvatar-SAC benefits on Cheetah and Ant but not Swimmer or Centipede; CAT-SAC benefits on Ant and Centipede but degrades on Cheetah and Swimmer.

CDRL-Bench accommodates morphology-aware methods while revealing when transfer helps, when it hurts, and which conclusions fail to generalize across embodiments.

[8] Yang et al., “A Transfer Approach Using Graph Neural Networks in Deep Reinforcement Learning.” AAAI 2024.

Q6: Mismatch between Table 6 and Table 1.

A: We thank the reviewer for identifying this inconsistency. Before computing regret, we smooth each learning curve using an exponential moving average (EMA) to reduce evaluation noise. Table 1 used a coefficient of 0.9, consistent with our figures, while Table 6 mistakenly retained 0.95.

We have corrected Table 6 using 0.9, ensuring that each entry in Table 1 matches the corresponding average in Table 6. This correction does not affect our conclusions. We will release the raw evaluation logs and regret computation script to ensure full reproducibility.

Q7: The claim that QAvatar-SAC beats SAC "in most settings" isn't true for regret: it is worse in 3 of the 6 groups in Table 1.

A: We thank the reviewer for pointing this out. The statement was based on best performance in Figure 4, where QAvatar-SAC outperforms SAC in three of the six groups and underperforms in two. Even under this metric, “most” is not an appropriate description. Considering both Table 1 and Figure 4, we have revised the statement as follows:

“QAvatar-SAC shows a clear benefit over SAC on Ant+Centipede and Robotic Sub-skills under both regret and best performance, but this benefit does not generalize across all settings.”

Reviewer_1d6YOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a benchmark for cross-domain reinforcement learning (CDRL) including robotic locomotion and manipulation tasks. It covers four CD transfer settings: same-task cross-morphology transfer, cross-task sub-skill transfer, fine-tuning generalist models on substantial tasks, and sim-to-real transfer. Two evaluation metrics are proposed, and several baselines have been implemented within a unified framework for fair comparison.

Critical strengths

The idea of a unified benchmark for CDRL is meaningful and valuable.

The results cover sim-to-real and cross-embodiment transfer performance, which enhances the analysis.

Critical weaknesses

However, there are some concerns about this paper regarding the coverage of representative environments and baselines.

The main contribution of this paper is providing a benchmark for CDRL; nevertheless, the investigation of CDRL is insufficient. Many representative robotics benchmarks are ignored, such as HumanoidBench, LIBERO, and Maniskill, which are more challenging for CDRL.

Second, this paper lacks comparison and analysis of many ways of transferring prior knowledge from source tasks to the target task. For example, prior knowledge can be transferred through neural network weights, learned representations, or action guidance from source policies. These approaches have different assumptions and may perform differently depending on the similarity between source and target tasks, the available target-domain data, and the differences in observation and action spaces. The current evaluation does not seem to provide a systematic comparison of these transfer mechanisms. What knowledge should be transferred, and under which settings does it perform better?

It is unclear how to unify these environments and evaluate these baselines. Do the authors encapsulate a unified interface for all the environments, including observation spaces, action spaces, reward functions, and task configurations?

Questions for authors

Please see the pros and cons part.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1d6Y (1/2)

We sincerely thank the reviewer for the thoughtful and constructive feedback. We address each concern point by point below and have revised the manuscript accordingly, with all changes highlighted in blue.

Q1: Limited CDRL Evaluation on Challenging Robotics Benchmarks

A: We thank the reviewer for pointing out these important benchmarks. We would like to clarify that our proposed benchmark addresses the core challenges these environments represent within the scope of Cross-Domain Reinforcement Learning. Specifically, while ManiSkill emphasizes object-level generalization and LIBERO focuses on multi-task knowledge transfer, our task suite systematically mirrors these difficulties through rigorous settings of cross-arm manipulation, cross-task sub-skill transfer, multi-source transfer, and real-world adaptation. The majority of existing works in CDRL actively utilize MuJoCo-based robotic arm environments similar to those in our task suite [1, 2, 3]. Therefore, researchers focusing on CDRL can readily adopt and directly benefit from our benchmark. Furthermore, to directly tackle the high-DoF and whole-body control challenges posed by HumanoidBench, we have provided an additional evaluation demonstrating cross-embodiment transfer from a Baxter robot to a GR1 humanoid robot for the Door Opening task. As shown in the https://imgur.com/a/19ihkkf and Figure 19 in the paper, the SAC-based methods make little progress on the target GR1 domain. CAT-PPO and PPO achieve relatively higher rewards (over 200) with a stable learning trend, yet still fall far short of solving the task. This highlights that, compared with transfer between single-arm manipulators, transfer between humanoid robots poses a substantially more challenging setting, leaving significant room for future CDRL research. We believe our current setups, combined with the newly added humanoid experiment, thoroughly validate the capability of our approach and highlight the significant room for future improvement in extreme generalization scenarios. In the revision, we have added ManiSkill, LIBERO, and HumanoidBench to the benchmark discussion in the related work section.

[1] Chen et al., "Cross-domain policy optimization via bellman consistency and hybrid critics." ICLR 2026.

[2] You et al., "Cross-domain adaptive transfer reinforcement learning based on state-action correspondence." UAI, 2022.

[3] Zhu et al., "Cross domain policy transfer with effect cycle-consistency." ICRA, 2024.

Q2: Clarification on the unified environment interface and baseline evaluation.

A: Thank you for the suggestion. We highlight how CDRL-Bench unifies the environment interface and evaluation as follows:

  • A common command-line interface. CDRL-Bench provides a unified training interface that allows different baseline algorithms and source–target environment pairs to be configured through a common command-line interface:

python train.py --alg $baseline --src_env $src_envs --tar_env $tar_env

The source and target environment names (--src_env and --tar_env) used in this command-line interface are listed in Table 2 of the Appendix. Specifically, we implement a shared construct_env(env_name) function that supports environments from different frameworks, including MuJoCo, Robosuite, and maze navigation tasks.

  • A Gymnasium-compatible training interface. All supported simulation environments expose a consistent Gymnasium-compatible interface, with Robosuite environments adapted through GymWrapper.

  • Environment specifications. While the interaction interface is standardized, each environment retains its task-specific observation space, action space, and reward function, which allows CDRL-Bench to accommodate heterogeneous source and target MDPs. Specifically: (1) all simulated observations and actions are flat continuous (Box) vectors, whose dimensions are listed in Table 2. (2) actions are rescaled to [−1, 1] for all environments. (3) rewards are dense for locomotion, shaped dense rewards for Robosuite tasks, and provided in both dense and sparse variants for maze navigation. The Real2Real task additionally uses image observations, as detailed in Appendix D.

  • A standardized evaluation configuration. For evaluation, as described in Section 4.5, we standardize the pretrained source models, target-domain interaction budgets, and evaluation metrics across applicable baselines. We report expert-normalized scores and average normalized regret to enable comparisons across tasks with different reward scales.

AuthorsOpenReview ↗

Response to Reviewer 1d6Y (2/2)

Q3: Comparison and analysis of different knowledge-transfer mechanisms.

A: We thank the reviewer for this valuable suggestion. Following suggestions from Reviewer EKTL, we have also included TURRET [4], a CDRL approach that uses graph neural networks to learn shared semantic representations to enable policy transfer between tasks with distinct state and action spaces, in our comparison. To clarify how the evaluated methods differ in the prior knowledge and assumptions they rely on, we summarize them in the following table:

Property\BaselineCAT-SACQAvatar-SACFT-SACCAT-PPOFT-PPOTurret-PPO
Source model accessParameter-levelFunction-levelParameter-levelParameter-levelParameter-levelParameter-level
Action-guided learning✓✓✓
Value-guided learning✓✓✓✓
Src–tar shared architecture requirement✓✓✓✓✓
State dim mismatchLearned mappingLearned mappingRe-init I/O layersLearned mappingRe-init I/O layers0-padding
Action dim mismatchLearned mappingLearned mappingRe-init I/O layersLearned mappingRe-init I/O layers-
Network Architecture AssumptionGNN
Inter-source weightingSource returnBellman ErrorUniformSource returnUniformCommom latent space distance
Control the contribution of between the srcs-trgLearnedBellman Error-Learned-Learned

Combining the properties in this table with the results in Table 6, we can have the following observations:

  • (1) The advantage of value-only transfer grows with domain dimensionality. Comparing the two SAC-based transfer methods, transferring both action and value knowledge (CAT-SAC) is preferable in low-dimensional domains, outperforming value-only transfer (QAvatar-SAC) on all four Swimmer pairs. As dimensionality increases, the trend reverses: QAvatar-SAC wins five of the six robot-arm pairs and matches or outperforms CAT-SAC on all high-dimensional pairs in Ant, Centipede, and TwoArmLift. A plausible explanation is that feature-level transfer requires the learned mapping to reproduce the source network’s internal representations, which becomes harder in high-dimensional spaces.

  • (2) Parameter initialization without cross-domain alignment rarely improves over learning from scratch. FT-SAC, which only initializes the target networks with source weights and re-initializes the dimension-dependent layers, never clearly outperforms SAC across morphology pairs and underperforms it on all eight Ant and Centipede pairs. This suggests that source parameters provide little benefit unless the input and output spaces are explicitly aligned.

  • (3) Return-based source weighting is sensitive to estimation noise. CAT estimates source weights from the episodic returns of mapped source policies in the target domain, using only a few rollouts to avoid consuming the target budget. Such noisy estimates make the weighting unreliable when sources differ in relevance. In multi-source settings, CAT-SAC generally underperforms QAvatar-SAC, which instead weights sources by cross-domain Bellman errors computed on collected target data, on Cheetah, Ant, Centipede, and TwoArmLift, where neither CAT variant improves over its scratch baseline even though two of the four sources match the target embodiments.

[4] Yang et al., "A Transfer Approach Using Graph Neural Networks in Deep Reinforcement Learning." AAAI 2024.

Dear Reviewer 1d6Y,

We sincerely thank you for your valuable suggestions for reviewing the paper. With about 24 hours left in the discussion, we kindly invite you to review our rebuttal and are happy to clarify any remaining concerns. We briefly summarize the main updates addressing your concerns:

Challenging benchmarks (Q1): We clarified how our suite captures the challenges of ManiSkill and LIBERO, discussed both alongside HumanoidBench in related work, and added a challenging Baxter-to-GR1 humanoid experiment (results) where transfer is feasible yet far from solved.

Unified interface (Q2): We clarified the shared command-line and Gymnasium interfaces, environment specifications, and standardized evaluation protocol.

Comparison of transfer mechanisms (Q3): We tabulate all compared methods’ transfer mechanisms in several dimensions: source-model access, value- vs. action-guided learning, state/action mismatch handling, architecture assumption, and domain-contribution control. Together with our results, this table shows that value-only transfer scales better to high dimensions, unaligned initialization rarely helps, and return-based source weighting is noise-sensitive.

We hope these additions have filled the gaps you pointed out and further strengthened the paper, and we would be grateful if you could take them into account in your final assessment.

Thank you again, and we look forward to hearing from you.

Best regards,

Authors of Paper 18680

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.