acceptodds
Under review as a conference paper at ICLR 2027

Bridging Dynamics Gaps via Diffusion Schrödinger Bridge for Cross-Domain Reinforcement Learning

Abstract

Cross-domain Reinforcement Learning (CDRL) aims to learn transferable policies under dynamics shifts between source and target domains. A key challenge lies in the lack of target-domain environment interaction and reward supervision, which prevents direct policy learning. To address this challenge, we propose Bridging Dynamics Gaps for Cross-Domain Reinforcement Learning (BDGxRL), a novel framework that leverages Diffusion Schrödinger Bridge (DSB) to align source transitions with target-domain dynamics encoded in offline demonstrations. Moreover, we introduce a reward modulation mechanism that estimates rewards based on state transitions, applying to DSB-aligned samples to ensure consistency between rewards and target-domain dynamics. BDGxRL performs target-oriented policy learning entirely within the source domain, without access to the target environment or its rewards. Experiments on MuJoCo cross-domain benchmarks demonstrate that BDGxRL outperforms state-of-the-art baselines and shows strong adaptability under transition dynamics shifts.

Primary area:
reinforcement learning
Keywords:
Cross-domain Reinforcement Learning, Diffusion Schrödinger Bridge, Off-Dynamics Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5YxsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes BDGxRL, a novel framework that leverages Diffusion Schrödinger Bridge (DSB) to align source transitions with target-domain dynamics encoded in offline demonstrations. The method first transforms transtions collected from source domain to target domain, and then uses a reward estimator to calculate reward from transitions. The author claims that proposed method solves two problems: 1. the target domain data are usually difficult to collect and 2. the data do not need reward annotation.

Critical strengths
  1. This paper studies a setting where target domain data and reward is hard to collect, which is practical under real-world case.
  2. The simulation results show the proposed method outperforms the baselines, which proves the effectiveness of the proposed method. The ablation experiments show the different components contribute to the final reported results.
Critical weaknesses
  1. The problem formulation relies on unjustified assumption about reward model and state representations. The reward estimator takes the form of R^(s,s~′)\hat R(s,\tilde{s}'), which is not common in RL research where reward model is usually set to $R(s, a, s`).AsfarasIknow,thisisparticularlyproblematicforMuJoCo/D4RLrewardsthatcanincludeaction−dependentcontrolcosts,whereastheproposedrewardmodelignorestheaction.Also,thevalueboundprovedbyTheorem1contains. As far as I know, this is particularly problematic for MuJoCo/D4RL rewards that can include action-dependent control costs, whereas the proposed reward model ignores the action. Also, the value bound proved by Theorem 1 contains R_{max},whichisnotclearlydefinedthatthisistheboundforsourcedomainreward,targetdomainrewardorlearnedrewardmodel.Whatsˋmore,thepolicyinputseemstomistakestatefromsourcedomainandtargetdomain.Thepolicyoutputsactionsconditioningonstatefromsourcedomain, which is not clearly defined that this is the bound for source domain reward, target domain reward or learned reward model. What\`s more, the policy input seems to mistake state from source domain and target domain. The policy outputs actions conditioning on state from source domain s_t$, but the corresponding transitions are considered collected from target domain. This implies that only transition function is different from source domain and target domain, while the state distribution is the same. This hypothesis does not hold in most applications. For example, if the length of the robot body is different, the height of head can vary, which may probably affect the state representation.
  2. The motivation of applying DSB for distribution adaptation is not clearly claimed. DSB can be a good tool for distribution matching, but the paper does not explain its advantages over other alternatives like MLP, transformer or diffusion model. Futhermore, if the performance improvements simply come from DSB over other technics, the contribution of this paper is not sufficiently proved.
Questions for authors
  1. Can the author justify why in the same transition, the previous state comes from source domain while the next output state comes from target domain? Is there implicit assumption that the source domain and target domain shares the same state distribution and only differs in transition model?
  2. What exactly is Rmax⁡R_{\max} and is it a bound on the true target reward, the source reward, or the learned reward model? Please explain how action-dependent reward terms are handled and add the reward-model error to the value bound if the learned reward differs from the target reward.
  3. Can the author provide results on other environments rather than Walker2d and HalfCheetah? And it would be better if the method can be adapted to zero-shot generalization of unknown domain.
Reviewer_k3G3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a method that can transfer data from online source environment to a smaller offline data w/o sufficient reward information. Through the transfer, the further interaction for reward info can be saved. A Diffusion Schrödinger Bridge is proposed to be employed to map transition vectors from the source environment to the target distribution. Transformer successor is bundled with original state-action space by a reward model pre-trained under source environement. Those things make it possible for SAC to adopt behavior cloing and initilizaiton/regularization.

Critical strengths

S1: valid problem to work on. Idea about a translator also makes sense to me

S2: paper tested 18 different exp configs and provides lots of ablation study, although further verification needed about some figures (commented below)

Critical weaknesses

W1: Actual source steps and baseline target access are unclear to me. Across 4 baseline columns, almost half of mean-and-dispersion cells match xTED arXiv v3

W2: The study did not include control groups such as "source data-SAC," "target data-BC," and "SAC+BC" not convincing me by the method's marginal advantage over the current SOTA.

W3: I doubt that joint matching can really fix the target dynamics. It looks to me more like making the source data more alike target data. The missing part to me is the data collection policy. Is occupancy differences equivalent to dynamics differences? plz

  • Use data not included in the training set to compare the next-state prediction errors before and after the transformation.
  • Evaluate performance both in "regions frequently visited by the target data" and "regions actually visited by the learned policy" to ensure the model is not effective only in the vicinity of the training data.
  • Include a control case where the source and target environments are identical, to verify that the transformer does not inadvertently degrade the data when no transformation is actually required.

W4: I am thinking, for an RL changing the next state would also cause distribution changing of reward functiosn for consistency The original reward may no longer be valid following a state transition. For ex, if the reward is defined as "forward velocity minus action energy consumption" modifying s′s' could alter the forward velocity, rendering the original reward value inapplicable to the new sample. Consequently, it is necessary to compare 3 things: using R(s,s′)R(s, s') (based only on states), using R(s,a,s′)R(s, a, s') (incorporating the action), and calculating the "ground truth" directly from formula.

W5: The assumption on the theory looks too strong. The value bound assumes policy closeness and model-error conditions that are not shown for the trained method.

Questions for authors

for W3

  • Use data not included in the training set to compare the next-state prediction errors before and after the transformation.
  • Evaluate performance both in "regions frequently visited by the target data" and "regions actually visited by the learned policy" to ensure the model is not effective only in the vicinity of the training data.
  • Include a control case where the source and target environments are identical, to verify that the transformer does not inadvertently degrade the data when no transformation is actually required.

for W4 examine the successor states, rewards, and termination signals given identical full state/action pairs. Compare R(s,s′)R(s, s') versus R(s,a,s′)R(s, a, s'). Distinguish between exact tuple-based rewards and oracle rewards from ground-truth.

for W1 use xTED v3 for verification Explain the advantages of DSB over SimGAN/GARAT, DCC, CompFlow, FOOD. Specify the source, run, seed, and checkpoint for each baseline unit. Explain any modified entries and provide results obtained using the same protocol;

Reviewer_KqUoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The use of Schrodinger bridges as a control mechanism in diffusion based methods has been a topic of interest for me and for others for a few year now, as has been the desire to perform transfer of autonomy across domains. So, this paper really addresses an important problem with a timely solution approach, and is definitely a step in the right direction. The main achievement is the use of diffusion Schrödinger (auto-corrected by Apple's autocorrection system - not AI usage) that bridges target dynamics with source transitions -- not just source states. Experiments have been performed on the MuJoCo cross-domain benchmarks,

Critical strengths

Strengths:

  1. The idea of using diffusion Schrodinger bridges to match target dynamics with source transitions is interesting.
  2. This only uses offline target demos.
  3. This is definitely a good direction that is likely to have an impact.
Critical weaknesses

Weaknesses:

  1. The benchmarks could have been more interesting and perhaps 1-2 real-world demonstrations could have been really appealing.
  2. The theoretical proofs should have been checked in Lean and perhaps Lean certificate provided. This makes sense since AI helped with the proofs anyway.
Questions for authors

Question: How is the bound in Theorem 1 verified by your experiments? What happens if the input is pixel level and the space of actions are different?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.