acceptodds
Under review as a conference paper at ICLR 2027

Cross-Domain Energy-Guided Diffusion Generation for Off-Dynamics Reinforcement Learning

Abstract

Off-dynamics offline reinforcement learning seeks to learn a target-domain policy from a large source dataset and a limited target dataset under mismatched transition dynamics. Existing approaches such as reward augmentation and data filtering are constrained to the existing source dataset and thus have limited coverage beyond the collected dataset. Recent model-based methods address this issue by learning target-aware dynamics, but are limited to transition-level generation, which can lead to accumulated errors over long horizons. These limitations motivate trajectory-level generation for off-dynamics offline RL. We propose CEDGE, a Cross-domain Energy-guided Diffusion GEneration framework. CEDGE trains a trajectory diffusion model on source-domain trajectories and adapts the generated samples to the target domain through energy guidance. This guidance is derived from the mismatch between source and target trajectory distributions and decomposed into return, domain, and policy energies for high return, target-dynamics compatibility, and source behavior correction respectively. The resulting energy-guided trajectories are useful both for direct planning and as synthetic data for policy learning. Compared with previous methods, CEDGE performs target adaptation through energy guidance, and the source trained diffusion model can be reused across target dynamics without retraining. Our theoretical analysis studies generation error and downstream policy performance. Empirical results demonstrate that trajectory-level energy-guided generation improves diffusion planning under dynamics shifts and produces synthetic data that improves downstream target-domain policy learning.

Primary area:
reinforcement learning
Keywords:
Off-dynamics reinforcement learning, Offline reinforcement learning, Diffusion models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Yoe6OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies offline RL with a large source-domain dataset and a small target-domain dataset collected under different dynamics. CEDGE trains a trajectory diffusion model once on the source data, then adapts its samples with three target-specific guidance terms for dynamics compatibility, predicted return, and source-policy correction. The guided model is used either as a receding-horizon planner or as a generator of synthetic data for IQL.

The empirical study covers gravity and friction shifts on four MuJoCo agents, with additional kinematic and morphology shifts in the appendix. The paper also gives an error bound for guided generation and an idealized finite-sample analysis of policy learning with real and synthetic data.

Critical strengths
  1. The reuse setting is well motivated. Training one source trajectory model and adapting it through smaller target-specific modules is a reasonable design for settings with many target dynamics and little target data. The separation into domain, return, and policy guidance also makes the method easy to interpret.

  2. The evaluation is broad and includes several useful controls. In Table 12, the full CEDGE planner has the highest reported mean among the guidance ablations in all 16 HalfCheetah and Walker2d settings. In Table 14, CEDGE-Policy has a higher mean than the unguided Diffuser synthetic-data baseline in all 16 settings, and Table 15 reports higher means than target-only diffusion in all eight gravity-shift settings. These experiments give evidence for the guidance mechanism beyond the total score in Table 1, although the three-seed uncertainty should still be kept in mind.

  3. The paper is clear about the scope of its theory. Appendix A explicitly states that the analysis does not cover trajectory filtering, planner candidate selection, or the implemented IQL learner, and Assumption A.2 notes that the probability bound does not cover dependent transitions from the same trajectory. The submitted code is also substantial enough to inspect the main training and data-generation paths.

Critical weaknesses
  1. The submitted synthetic-data exporter appears to create an artificial final transition for every generated segment. In train_planner.py (lines 770-787), a segment contains H state-action entries and next_states is formed by shifting the states and repeating the last state. The last exported tuple is therefore of the form (s_{H-1}, a_{H-1}, s_{H-1}). It is marked as a timeout but not as terminal. The policy replay buffer then sets not_done = 1 - terminals and does not use the timeout field (policy_optimization/algo/utils.py, lines 40-45), so this tuple is included in bootstrapping. With the supplied horizons, this is one quarter of the exported HalfCheetah entries and one thirty-second of the Hopper and Walker2d entries.

This observation is limited to the submitted implementation. It does not show that the reported returns are incorrect, and it does not affect CEDGE-Planner. If the same exporter was used for the unguided synthetic-data controls, some relative comparisons may also remain informative. However, the issue is directly relevant to the claim that CEDGE-Policy benefits from correctly formed target-compatible transitions. A corrected rerun on the short-horizon HalfCheetah tasks would be particularly useful.

  1. The artifact does not currently support full reproduction of Table 1. The main table contains eight Ant gravity/friction settings, but the submitted task configurations and environment assets cover only HalfCheetah, Hopper, and Walker2d. Both dataset/call_dataset.py and train_iql.py explicitly reject other agents. There are also discrepancies between the paper and the provided defaults: Tables 5 and 7 report 500,000 diffusion updates and 200,000 energy updates, whereas mujoco.yaml sets both to 1,000,000; the task YAML files use guidance weights (0.01, 0.01, 0.01), while the paper lists search values beginning at 0.1 and the README example uses (1.0, 1.0, 0.1). These may be stale defaults or values overridden at launch, rather than the settings used for the experiments, but no run commands or manifests are supplied to resolve the discrepancy.

  2. The target used to train the return-energy model differs between the method description and the code. Equation (4.6) defines an H-step discounted segment return. The implementation passes batch["val"] to the return predictor (train_planner.py, lines 371-380), while the dataset constructs seq_val recursively to the end of the episode and returns its value at the segment start (cleandiffuser/dataset/d4rl_mujoco_dataset.py, lines 124-145). This is a return-to-go target rather than the segment return written in the paper. Both choices are plausible, but they yield different guidance signals and should not be treated as interchangeable.

  3. Some of the aggregate conclusions are difficult to calibrate from three seeds. Table 1 emphasizes the sum of task means, 1504.16 for CEDGE-Policy versus 1427.26 for MOBODY, without uncertainty for the aggregate. The per-task table contains both large gains and differences that are small relative to the reported standard deviations. It is also unclear how the domain and return guidance weights were selected and whether target-environment evaluations were used in that selection. This matters because Table 17 shows substantial sensitivity to the domain and policy guidance weights on HalfCheetah-Gravity-0.1. A stated selection protocol and a paired or seed-level aggregate analysis would make the main comparison easier to assess.

  4. The theory is useful as an error decomposition, but it should not be read as a guarantee for the implemented CEDGE-Policy pipeline. Theorem 4.2 uses independent samples, finite function classes, exact minimax optimization, realizability, completeness, and coverage assumptions. The actual learner uses correlated generated segments, filtering, learned rewards, and neural IQL. The appendix acknowledges this gap, so I do not regard it as a flaw in the proof; it is mainly a limitation on how strongly the theorem can support the empirical algorithm.

Minor Comments

  1. The trajectory notation should be made consistent. Section 3 includes a final state s_H, while Table 3 and the submitted dataset return tensors with H state-action entries and no separate s_H.

  2. The Windy Maze experiment is helpful as an illustration, but it uses one stated training seed and a small target reference set. I would avoid treating the difference between 0.269 and 0.236 as strong standalone distributional evidence.

  3. The runtime table contains only two tasks. It supports the claimed reuse mechanism in those cases, but not a general runtime comparison across all agents and shifts.

Questions for authors
  1. Were the CEDGE-Policy results produced with the submitted transition-export code? If so, what changes when the final tuple of each segment is dropped, or when an additional next state is generated and timeouts are handled consistently in IQL?

  2. Can the authors provide the Ant configurations and the exact commands or run manifests used for Table 1, including the training steps and guidance weights for each setting?

  3. Was the return predictor used in the reported runs trained on an H-step segment return or on the episode return-to-go implemented by seq_val?

  4. How were the guidance weights selected? In particular, was any target-environment interaction used for model selection?

  5. Can the authors report uncertainty for an aggregate comparison, for example paired seed-level differences within each task followed by a suite-level summary?

5 comments

We thank the reviewer for their valuable time and constructive feedback. We hope the responses below address all questions and concerns.


Q1: Were the CEDGE-Policy results produced with the submitted code? And what changes when the final tuple (s_{H-1}, a_{H-1}, s_{H-1}) of each segment, denoted as 'timeout ', is dropped?

A1: We clarify that the reported CEDGE-Policy results were produced using the submitted code. Dropping the final tuple has little effect on the overall performance and does not change our main empirical conclusions. In the submitted code, the final tuple (sH−1,aH−1,sH−1)(s_{H-1}, a_{H-1}, s_{H-1}) of each generated segment is included in IQL bootstrapping. To address this issue, we remove the final tuple from each segment. We keep only the H−1H-1 transitions between consecutive generated states. We then rerun CEDGE-Policy on all 32 tasks with three seeds.

As shown below, the total normalized score changes from 1504.16 to 1511.53 after dropping the final tuple. For HalfCheetah, the final tuple accounts for one-fourth of the original generated transitions. The total score across eight tasks changes only from 373.66 to 373.18. Overall, dropping the final tuple has little effect on the aggregate performance.

EnvironmentShiftRatioPaperDrop final tuple
HalfCheetahFriction0.164.31 ± 0.3963.74 ± 0.31
HalfCheetahFriction0.552.06 ± 6.4156.71 ± 4.58
HalfCheetahFriction2.049.15 ± 4.0347.86 ± 3.48
HalfCheetahFriction5.068.56 ± 1.9466.43 ± 2.85
HalfCheetahGravity0.120.04 ± 3.5217.26 ± 1.55
HalfCheetahGravity0.52.54 ± 0.593.37 ± 1.62
HalfCheetahGravity2.041.66 ± 5.8443.83 ± 8.49
HalfCheetahGravity5.075.34 ± 1.6373.98 ± 0.24
HopperFriction0.152.94 ± 7.7160.56 ± 5.98
HopperFriction0.541.97 ± 6.5143.47 ± 4.50
HopperFriction2.011.08 ± 0.0211.09 ± 0.01
HopperFriction5.08.24 ± 0.048.27 ± 0.02
HopperGravity0.136.02 ± 0.3336.75 ± 0.67
HopperGravity0.568.25 ± 3.0367.17 ± 3.29
HopperGravity2.026.94 ± 0.5925.68 ± 0.29
HopperGravity5.08.80 ± 0.028.81 ± 0.02
Walker2dFriction0.161.66 ± 4.7061.24 ± 1.23
Walker2dFriction0.570.16 ± 2.2570.16 ± 10.01
Walker2dFriction2.073.05 ± 0.0273.26 ± 0.08
Walker2dFriction5.036.44 ± 8.2333.90 ± 2.83
Walker2dGravity0.169.48 ± 1.5771.19 ± 1.88
Walker2dGravity0.544.36 ± 1.0742.74 ± 2.40
Walker2dGravity2.044.36 ± 2.3544.57 ± 2.56
Walker2dGravity5.054.01 ± 10.2550.83 ± 6.68
AntFriction0.161.85 ± 3.9458.12 ± 0.16
AntFriction0.554.06 ± 1.9352.35 ± 7.96
AntFriction2.057.49 ± 4.3855.01 ± 4.00
AntFriction5.060.31 ± 9.3259.27 ± 2.77
AntGravity0.137.99 ± 1.1325.83 ± 2.20
AntGravity0.535.16 ± 4.5136.54 ± 4.03
AntGravity2.044.42 ± 1.2670.51 ± 1.70
AntGravity5.071.46 ± 6.5671.03 ± 0.04
Total1504.161511.53

Q2: Missing Ant configurations and inconsistencies of hyperparameters in the supplementary code.

A2: Thanks for catching these mismatches. We clarify that, following convention, we only provided the submitted supplementary code with default hyperparameters. We provide the configurations and running configs in the updated supplementary code.

The reported results use 500,000 diffusion training updates and 200,000 energy training updates as shown in Tables 5 and 7. For guidance weights, we select task-specific values for domain and return guidance, while fixing the policy guidance weight at 0.1. The exact weights used for CEDGE-Policy in Table 1 are shown below. Each entry represents (λ1,λ2,λ3)(\lambda_1,\lambda_2,\lambda_3) for domain, return, and policy guidance, respectively.

EnvironmentShift0.10.52.05.0
HalfCheetahGravity(0.7, 5.0, 0.1)(0.7, 5.0, 0.1)(0.4, 2.0, 0.1)(5.0, 0.4, 0.1)
HalfCheetahFriction(0.7, 5.0, 0.1)(1.0, 5.0, 0.1)(1.0, 5.0, 0.1)(0.4, 5.0, 0.1)
AntGravity(2.0, 0.5, 0.1)(1.0, 0.5, 0.1)(2.0, 0.5, 0.1)(0.5, 2.0, 0.1)
AntFriction(1.0, 2.0, 0.1)(0.1, 0.5, 0.1)(0.1, 1.0, 0.1)(0.1, 1.0, 0.1)
Walker2dGravity(0.1, 5.0, 0.1)(0.7, 5.0, 0.1)(0.7, 5.0, 0.1)(5.0, 1.0, 0.1)
Walker2dFriction(0.4, 5.0, 0.1)(5.0, 2.0, 0.1)(5.0, 2.0, 0.1)(2.0, 2.0, 0.1)
HopperGravity(0.1, 5.0, 0.1)(0.4, 2.0, 0.1)(5.0, 0.1, 0.1)(5.0, 5.0, 0.1)
HopperFriction(0.1, 2.0, 0.1)(5.0, 5.0, 0.1)(0.4, 0.4, 0.1)(1.0, 2.0, 0.1)

Q3: Was the return predictor used in the reported runs trained on an H-step segment return or on the episode return-to-go implemented by seq_val?

A3: We thank the reviewer for pointing out this mismatch. We clarify that the return energy predictor in our reported experiments is trained using the discounted episode return-to-go as implemented in the submitted code.

The H-step segment return only considers rewards within the generated segment. In contrast, the episode return-to-go includes rewards from the start of the segment through the end of the episode. In our implementation, the return predictor takes an H-step trajectory segment as input and predicts the return-to-go from its starting state. We use return-to-go because it provides longer-term reward information beyond the generated segment. This choice follows the return prediction implementation in Diffuser [1]. We revised Eq. (4.6) and the corresponding description in our revision.

[1] Planning with Diffusion for Flexible Behavior Synthesis. ICML 2022.


Q4: Selection of guidance weights and evaluation checkpoints.

A4: We select the guidance weights and evaluation checkpoints based on performance evaluations in the target environments, which is a common practice in offline RL. It can be considered as selecting the guidance weights with the validation environments. Once the hyperparameters are fixed, CEDGE requires only offline source and target datasets without additional environment interactions.


Q5: Can the authors report uncertainty for an aggregate comparison?

A5: We provide additional analysis showing that CEDGE-Policy maintains its aggregate performance advantage over the baselines across seeds. For each seed, we sum the scores across all 32 tasks for each method. We then compute the paired differences between CEDGE-Policy and each baseline using matched seeds. We report the mean and standard deviation across three seeds. As shown below, CEDGE-Policy achieves an aggregate score of 1504.16 ± 23.08, compared with 1427.26 ± 27.14 for MOBODY. The paired improvement is 76.90 ± 39.75.

MethodAggregate ScorePaired Difference (CEDGE-Policy − Method)
MOBODY1427.26 ± 27.14+76.90 ± 39.75
OTDF1056.87 ± 7.64+447.29 ± 28.74
DmC1044.40 ± 43.66+459.76 ± 35.60
xTED998.75 ± 44.47+505.41 ± 60.16
Diffuser865.91 ± 10.37+638.25 ± 25.31
CEDGE-Planner908.92 ± 3.90+595.24 ± 23.41
CEDGE-Policy1504.16 ± 23.08–

Q6: Idealized assumptions in Theorem 4.2 compared to the implemented pipeline.

A6: We clarify that Theorem 4.2 provides theoretical insights into the benefits and limitations of using synthetic data for downstream policy learning. Our main goal is not to provide a theoretical guarantee specifically for IQL. Instead, we use an idealized offline RL learner to study how synthetic data affects downstream policy learning. The theorem shows the trade-off between increasing the amount of training data and introducing errors from generated transitions. Such idealized analysis is common in theoretical offline RL research, where simplifying assumptions are used to understand the main factors affecting policy performance. We follow this approach to provide theoretical insights rather than a direct guarantee for the implemented CEDGE-Policy.


Q7: Inconsistency between the trajectory definition for s_H in Section 3 and Table 3.

A7: We thank the reviewer for pointing this out. We remove the final state sHs_H from the trajectory definition in Section 3 to align the notation with the HH state-action pairs used in Table 3 and our implementation. We add this revision in our revised manuscript.


Q8: Limited training seeds in Windy Maze

A8: We provide additional results addressing the concern about single seed evaluation. We repeat the Windy Maze experiment with 10 independent training seeds, following the setup in Appendix F.1. For each seed, we average the distances over 32 generated trajectories and report the mean and standard deviation across the 10 seeds. CEDGE achieves a lower mean distance than MOBODY. We replace the original results with these multiple-seed results in our revised Table 2.

SettingCEDGEMOBODY
1 seed0.2360.269
10 seeds0.260 ± 0.0170.304 ± 0.031

Q9: The runtime table contains only two tasks.

A9: We extend the runtime evaluation to more settings, including HalfCheetah, Walker2d, and Hopper under gravity and friction shifts with shift level 2.0. Each entry reports the Full / Reuse runtime in hours. For CEDGE, Full includes source diffusion model training, while Reuse uses a pretrained source diffusion model. As shown below, reusing the source diffusion model reduces the runtime of both CEDGE variants across all six settings. We add these results to our revised Appendix H.4 Table 14.

MethodHalfCheetah-Gravity-2.0HalfCheetah-Friction-2.0Walker2d-Gravity-2.0Walker2d-Friction-2.0Hopper-Gravity-2.0Hopper-Friction-2.0
IQL~3.0h / ~3.0h~3.0h / ~3.0h~3.0h / ~3.0h~3.0h / ~3.0h~3.0h / ~3.0h~3.0h / ~3.0h
MOBODY~6.1h / ~6.1h~6.1h / ~6.1h~7.8h / ~7.8h~7.8h / ~7.8h~6.7h / ~6.7h~6.7h / ~6.7h
Diffuser~5.5h / ~5.5h~3.0h / ~3.0h~7.6h / ~7.6h~4.1h / ~4.1h~4.1h / ~4.1h~4.1h / ~4.1h
CEDGE-Planner~5.6h / ~1.9h~4.4h / ~1.6h~7.6h / ~3.3h~6.2h / ~2.5h~6.3h / ~2.5h~6.3h / ~2.5h
CEDGE-Policy~7.4h / ~3.7h~5.4h / ~2.5h~9.1h / ~4.8h~5.2h / ~2.4h~5.2h / ~2.3h~5.2h / ~2.4h

We hope these additions address the reviewer's concerns. We would be happy to answer any remaining questions.

Reviewer_J5aqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces an off-dynamics offline reinforcement learning method named CEDGE to address two problems that limited coverage beyond the collected dataset and transition-level generation leading to accumulated errors over long horizons. The CEDGE is a Cross-domain Energy-guided Diffusion Generation framework. It contains return, domain, and policy energies of three guidances from source and target trajectory distributions. The results show that the proposed method improves planning under dynamics shifts and downstream target-domain policy learning.

Critical strengths
  1. The research of Off-dynamics offline reinforcement learning is an important and practical problem in the real world.
  2. The writing and structure of the paper are clear.
Critical weaknesses
  1. The performance of CEDGE mainly depends on learned guidance modules and more target data.
  2. The experiments are limited to MuJoCo tasks.
  3. The core energy-guided diffusion generation is a well-established method. CEDGE is combined with these components to use the off-dynamics offline reinforcement learning setting.
Questions for authors
  1. What is different between CEDGE and EDIS [1], beyond task setting?
  2. What are advantages of Energy decomposition compared to existing guidance diffusion model methods?

[1] Energy-guided diffusion sampling for offline-to-online reinforcement learning. ICML 2024.

7 comments

We thank the reviewer for their valuable time and constructive feedback. We hope the responses below address all questions and concerns.


Q1: Clarification on the novelty of CEDGE.

A1: We disagree that CEDGE is simply an application of existing energy-guided diffusion methods. Applying energy-guided diffusion to off-dynamics offline RL is not straightforward. The source and target environments can have very different transition dynamics. Vanilla transition dynamics-based guidance is not sufficient to guarantee good performance, even though the main difference between these two environments is the dynamics.

This motivates us to design different energy terms to compensate for the performance. We derive the guidance from the difference between the source trajectory distribution and the desired target trajectory distribution. This leads to three energy terms. Domain energy corrects the dynamics mismatch. Return energy encourages high-return trajectories. Policy energy corrects the source behavior-policy bias. These terms are derived from the trajectory distributions rather than combined heuristically.

Our ablation results support this design. As shown in Table 13, the full CEDGE-Planner outperforms No Guidance, Domain Only, Return Only, and Policy Only in all 16 evaluated settings. For example, on Walker2d-Gravity-0.5, Domain Only achieves 38.76, while the full CEDGE-Planner achieves 47.84. This shows that dynamics guidance alone cannot achieve the performance of the full method.

Beyond energy guidance, we also design CEDGE for both planning and offline policy optimization. CEDGE-Planner generates trajectories conditioned on the current state and selects actions based on predicted returns. CEDGE-Policy generates synthetic trajectories and assigns rewards to them. It then filters the trajectories based on target dynamics compatibility and predicted returns before policy optimization. Both methods can reuse the source-trained diffusion model across different target dynamics without retraining the diffusion model. Table 15 also shows that CEDGE-Policy outperforms vanilla Diffuser in all 16 evaluated settings.

Therefore, CEDGE is not a simple extension of existing energy-guided diffusion methods. Our contributions include the cross-domain energy derivation and the design of trajectory generation for planning and offline policy optimization. Making these components work together under dynamic shifts is non-trivial and novel.


Q2: Difference between CEDGE and EDIS.

A2: We emphasize that CEDGE is not simply EDIS applied to a different setting. CEDGE differs from EDIS in its target distribution, energy design, and trajectory generation, which are specifically designed to address dynamics mismatch in off-dynamics offline RL.

EDIS addresses offline-to-online RL where the transition dynamics are the same. In contrast, CEDGE addresses off-dynamics offline RL, where source and target environments have different dynamics. CEDGE uses only fixed offline datasets without additional environment interactions.

EDIS uses state, action, and transition energies to align generated data with the current online policy distribution. In contrast, CEDGE derives its energy functions from the difference between the source trajectory distribution and the desired target trajectory distribution. Specifically, domain energy estimates the source-target dynamics density ratio, return energy favors high-return trajectories, and policy energy corrects source behavior-policy bias. Thus, CEDGE addresses cross-domain dynamics mismatch via a compositional energy function rather than matching an online policy with the same transition dynamics in EDIS.

EDIS generates individual transitions (s,a,s’)(s, a, s’) for online policy fine-tuning, while CEDGE jointly generates state-action trajectories for both diffusion planning and offline data augmentation to eliminate the effect of compounding error in transition-level generation.


Q3: Advantages of the energy decomposition in this paper.

A3: The main motivation for energy decomposition is that directly learning a single energy function for the desired target trajectory distribution is impractical in our setting. This is because the desired energy involves three coupled factors, including transition dynamics, returns, and behavior policy. Learning these factors jointly is challenging when only limited target data is available. Therefore, we decompose the energy into three terms based on the source and desired target trajectory distributions. Each energy term has a specific purpose and can be learned separately, making the guidance more flexible and easier to estimate.

Our decomposition is specifically derived to correct the dynamics mismatch between two environments while encouraging high-return trajectories. Furthermore, the ablation results in Table 12 show that combining all three energy terms outperforms using any individual energy term across the evaluated tasks, supporting the benefit of our decomposition.


Q4: The experiments are limited to MuJoCo tasks.

A4: To evaluate CEDGE beyond locomotion, we conducted additional experiments on the Adroit Pen and Door dexterous manipulation tasks. Pen requires reorienting a pen to match a target orientation, while Door requires opening a latched door. We consider kinematic (kin) and morphology (morph) shifts at medium (M) and hard (H) levels from MOBODY.

As shown below, CEDGE-Policy achieves the highest mean score in seven of the eight settings. Its aggregate score of 446.83 is also the highest among all baselines. These results show that CEDGE-Policy also performs well on dexterous manipulation tasks under both types of dynamics shifts. We add these results to our revised Appendix H.2.

EnvTypeLevelBOSAIQLTD3-BCDARAMOBODYOTDFDmCxTEDDiffuserCEDGE Policy
PenkinM30.63 ± 9.0124.34 ± 15.496.86 ± 6.6338.60 ± 3.4437.67 ± 4.5453.87 ± 4.2450.10 ± 5.7315.38 ± 4.107.57 ± 2.3672.14 ± 9.25
PenkinH7.18 ± 2.027.74 ± 3.481.31 ± 1.299.41 ± 6.0613.73 ± 6.3222.13 ± 9.9321.17 ± 2.023.24 ± 1.616.78 ± 3.5037.40 ± 22.56
PenmorphM10.72 ± 6.6513.75 ± 4.912.20 ± 1.718.72 ± 3.1216.48 ± 10.4617.78 ± 6.2114.61 ± 2.416.33 ± 3.0110.22 ± 2.5330.75 ± 10.52
PenmorphH11.78 ± 6.5732.16 ± 1.149.12 ± 9.0322.17 ± 3.9037.80 ± 1.1819.67 ± 2.0219.68 ± 1.629.73 ± 8.210.52 ± 0.1636.59 ± 5.83
DoorkinM25.42 ± 22.0437.43 ± 12.76-0.23 ± 0.0120.18 ± 5.2939.26 ± 3.7242.03 ± 4.0536.68 ± 4.8311.39 ± 7.580.00 ± 0.0054.93 ± 0.92
DoorkinH30.64 ± 26.8756.02 ± 7.74-0.12 ± 0.0258.22 ± 9.9161.61 ± 9.8461.84 ± 1.6150.51 ± 5.493.46 ± 4.49-0.04 ± 0.0068.56 ± 5.37
DoormorphM41.59 ± 5.9560.74 ± 12.83-0.19 ± 0.0150.32 ± 4.7863.67 ± 9.5260.30 ± 5.7658.33 ± 3.563.52 ± 4.93-0.04 ± 0.0073.21 ± 5.67
DoormorphH26.97 ± 8.6268.64 ± 8.34-0.20 ± 0.0244.22 ± 7.1962.88 ± 5.2563.12 ± 3.3248.40 ± 7.747.93 ± 11.53-0.02 ± 0.0073.25 ± 1.76
Total184.93300.8218.75251.84333.10340.74299.4860.9824.99446.83

We hope these additions address the reviewer's concerns. We would be happy to answer any remaining questions.

Reviewer_J5aqOpenReview ↗

Response to Authors 27706.

Thanks for your responses. They address most of my concerns. I have one remaining concern: CEDGE depends on learned-well guidance modules and more target data. Could you quantify its sensitivity to (i) the amount of target data and (ii) guidance quality, and clarify whether all comparison baselines used the same target-data budget and comparable tuning?

Reviewer_bYMPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes CEDGE for off-dynamics offline RL. A diffusion model trained on source trajectories is adapted using three energy functions corresponding to dynamics correction, return maximization, and source behavior-policy correction. The generated trajectories are used for planning or synthetic-data augmentation with IQL. Experiments across many MuJoCo dynamics shifts show strong aggregate results, supported by ablations and theoretical bounds on generation error and policy performance.

Critical strengths
  1. Reusable diffusion prior. Training the trajectory model once and adapting it through target-specific guidance is a practical contribution, particularly when transferring to multiple dynamics configurations.

  2. Broad empirical evaluation. The method achieves strong aggregate results across 64 shifted tasks. The comparisons against target-only training, unguided diffusion, and filtered source data provide useful evidence that the synthetic-data pipeline offers benefits beyond simply reusing or filtering source data.

Critical weaknesses
  1. The theoretical guidance construction is not sufficiently justified for the actual setting.

The clean trajectory-density ratio assumes overlapping source and target distributions, which is problematic for deterministic dynamics shifts. Furthermore, the exact noisy-time guidance involves the log of a conditional exponential expectation, whereas the proposed implementation uses independently learned noisy-time energy functions. Even perfectly trained individual predictors need not recover the required guidance.

Although Proposition 4.1 includes a guidance-error term, the paper does not establish that this error is small. Theorem 4.2 also concerns an idealized independent-transition learner rather than the implemented IQL and filtering pipeline.

  1. The main claim concerning trajectory-level generation is not convincingly isolated experimentally.

The paper argues that trajectory diffusion mitigates compounding errors, but does not evaluate how target-dynamics consistency changes with horizon. HalfCheetah and Ant use trajectories of only four steps.

The Windy Maze experiment is also not a controlled comparison of dynamics prediction: MOBODY uses controller-provided actions, while CEDGE generates both states and actions. Distance to a mean target trajectory does not directly establish whether the generated transitions are dynamically feasible.

The policy gains are heterogeneous. In HalfCheetah-Gravity-0.5, CEDGE-Policy scores 2.54, despite CEDGE-Planner reaching 49.84 and MOBODY reaching 47.18. This raises questions about the reliability of generated transitions and their use for downstream policy learning.

Questions for authors

Q1. Dynamics support and validity. How is the trajectory-density ratio justified when source and target transitions are deterministic and may have non-overlapping support? Does adding diffusion noise provide a principled approximation, and what happens as the noise level approaches zero?

Q2. Exact versus practical guidance. The exact noisy-time energy in Appendix A differs from the sum of the independently learned guidance terms. How large is this approximation error empirically? Moreover, given that the selected energy weights do not generally correspond to the ideal distribution in Eq. 4.1, what distribution is the practical sampler intended to approximate?

Q3. Trajectory feasibility and horizon. Can the authors report the target-simulator consistency error of generated state-action trajectories over varying horizons? In particular, can they replay generated actions under the actual target dynamics and compare the resulting states with the generated states? A comparison with transition-level models under matched actions would more directly support the proposed motivation.

Q4. Behavior guidance and downstream failures. The negative behavior-policy score is closely related to the guidance correction developed in STITCH-OPE (NeurIPS 2025). Could the authors clarify the relationship and explain the choice of a fixed behavior-guidance weight of 0.1? Additionally, are synthetic actions constrained to the valid action range, and what explains the large discrepancy between planner and policy performance in some tasks?

Q5. Model selection. How are the guidance weights and evaluation checkpoints selected without additional target-environment interactions? This is important given the sensitivity reported in Table 17.

7 comments

We thank the reviewer for their valuable time and constructive feedback. We hope the responses below address all questions and concerns.


Q1: Dynamics support and validity: Deterministic transition dynamics, source–target support overlap, and the role of diffusion noise in density-ratio estimation.

A1: We clarify that deterministic transition is not a major concern in our formulation. Deterministic transitions are often used in practical settings such as MuJoCo and can also be modeled within a probabilistic framework. Deterministic dynamics can be considered a special case of probabilistic dynamics. Thus, it is reasonable to use probabilistic modeling in our algorithm design. In our implementation, we train domain classifiers to distinguish transitions collected from the source and target environments and use their predictions to estimate the dynamics discrepancy between the two domains' transition distributions.

The requirement for overlap between the source and target domains is not a limitation specific to CEDGE. In CEDGE, we need sufficient support overlap to reliably estimate the density ratio between two domains. In the literature, reward augmentation methods such as DARC [1] and DARA [2] face similar difficulties in estimating dynamics discrepancies. Data filtering methods such as OTDF [3] and LoDADA [4] require sufficient overlap to identify and estimate scores for source transitions relevant to the target domain. Model-based methods such as MOBODY [5] also depend on sufficient data coverage to learn target dynamics. Support overlap between the two domains is a general requirement in offline off-dynamics RL. This is usually not a problem in practice. DARC and DARA use target data upsampling to ensure effective classifier training. OTDF and LoDADA use optimal transport and localized clustering, respectively, to measure the distance between source and target data and then filter samples. MOBODY combines source and target datasets when training the dynamics model to ensure good modeling of the target dynamics. Our CEDGE also leverages upsampling when training the energy function to ensure a good estimation of the difference between source and target domains.

Regarding diffusion noise, its purpose is not to increase the overlap between the source and target distributions. Instead, we train timestep-conditioned domain classifiers on noisy transitions so that they can provide energy guidance for intermediate trajectories during the reverse diffusion process. Diffusion noise enables guidance during the reverse diffusion process but does not eliminate the need for sufficient support overlap between the source and target distributions.

[1] Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers. ICLR 2021.

[2] DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning. ICLR 2022.

[3] Cross-Domain Offline Policy Adaptation with Optimal Transport and Dataset Constraint. ICLR 2025.

[4] Localized Dynamics-Aware Domain Adaptation for Off-Dynamics Offline Reinforcement Learning. NeurIPS 2026.

[5] Model-Based Off-Dynamics Offline Reinforcement Learning. ICLR 2026.


Q2: Exact versus practical guidance: Approximation error between the independently learned energy functions and their quantity. What is the intended distribution under guidance weights?

A2: We clarify that our energy decomposition is exact at the clean trajectory level and does not introduce additional approximation error at this level. The approximation error arises when implementing the guidance at intermediate diffusion timesteps. Specifically, there are two main sources of error. First, we use neural networks to learn the energy functions. Since these predictors are learned from finite data, they introduce estimation errors. Second, we approximate the exact noisy-time guidance by combining independently learned energy functions. At the clean trajectory level, the three energy terms can be exactly combined to recover the desired clean energy. Therefore, the decomposition itself does not introduce additional approximation error at this level. However, training the classifiers at intermediate diffusion timesteps with noisy data introduces additional approximation error [6]. As a result, the sum of the individual noisy-time energies may differ from the exact noisy-time energy. We expect it to be relatively small in our evaluated settings. A precise quantitative characterization is beyond the scope of this work. In practice, we train timestep-conditioned predictors on noisy transitions and combine their guidance during the reverse diffusion process, following the compositional guidance approach used in prior diffusion methods [7]. This is a commonly used approach in classifier- and energy-guided diffusion methods.

We further clarify that using guidance weights does not invalidate our practical guidance framework. In the ideal formulation, unit weights recover the desired target distribution. In practice, the learned energy functions have different magnitudes, so we tune their weights to adjust the strength of each guidance term. Consequently, the practical sampler is guided toward a weighted energy distribution. This does not affect our empirical conclusions.

[6] Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC. ICML 2023.

[7] Compositional Visual Generation with Composable Diffusion Models. ECCV 2022.


Q3: Guidance-error term in Proposition 4.1 and idealized learner in Theorem 4.2.

A3: Regarding the guidance-error term, we clarify that it captures the difference between the learned guidance and the exact noisy-time guidance. This error comes from both neural network estimation and the approximation used to combine the noisy-time energy functions. We believe this error is reasonably small in our evaluated settings, as the empirical results show strong performance.

We emphasize that Theorem 4.2 aims to explain the trade-off between the benefits of additional synthetic data and the errors introduced by generated transitions, rather than provide a direct guarantee for the implemented IQL pipeline. To study this trade-off, we use an idealized offline RL learner. Such idealized analysis is common in theoretical offline RL research, where simplifying assumptions are used to understand the main factors affecting policy performance.


Q4: Trajectory feasibility and horizon: Consistency error of generated state-action trajectories over varying horizons.

A4: We provide additional experiments showing that trajectory-level generation mitigates compounding errors compared with transition-level modeling. Specifically, we evaluate the target-dynamics consistency of generated trajectories across different horizons and compare CEDGE with MOBODY under matched initial states and actions. We consider Hopper and Walker2d under Gravity-0.5 and Gravity-2.0 shifts. For each setting, CEDGE generates 1024 state-action trajectories with three seeds using a source-trained diffusion model with horizon 32. We evaluate H∈{4,8,12}H\in \{4,8,12 \} steps of the generated trajectories and measure the normalized RMSE between the generated states and the states obtained by replaying the same actions in the actual target simulator. MOBODY uses the same initial states and action sequences for comparison. CEDGE achieves lower consistency errors than MOBODY in all settings. As the horizon increases, MOBODY's errors grow rapidly, while CEDGE's errors remain relatively low. This supports the advantage of trajectory-level generation in reducing compounding errors. We add these results to our revised Appendix H.9.

EnvironmentH=4 CEDGEH=4 MOBODYH=8 CEDGEH=8 MOBODYH=12 CEDGEH=12 MOBODY
Hopper-Gravity-0.50.399 ± 0.0060.557 ± 0.0070.490 ± 0.0081.502 ± 0.1150.617 ± 0.00739.316 ± 9.765
Hopper-Gravity-2.00.454 ± 0.0080.482 ± 0.0140.599 ± 0.00713.099 ± 15.5890.889 ± 0.0107.6e3 ± 1.1e3
Walker2d-Gravity-0.50.918 ± 0.0071.061 ± 0.0041.035 ± 0.0141.915 ± 0.2491.144 ± 0.01725.784 ± 4.067
Walker2d-Gravity-2.00.834 ± 0.0121.186 ± 0.0310.894 ± 0.0116.964 ± 1.3861.031 ± 0.020389.985 ± 50.393

Q5: Differences between MOBODY's use of controller-provided actions and CEDGE's joint generation of states and actions.

A5: We clarify that the experimental design is reasonable. MOBODY uses a learned dynamics model and requires a policy to provide actions, while CEDGE directly generates states and actions together. Therefore, we need to provide a policy for MOBODY when generating trajectories. To make a fair comparison, we use the optimal policy in Windy Maze, which provides optimal actions for the current state, to illustrate the compounding errors and trajectory deviations in MOBODY and CEDGE.

Regarding dynamics feasibility, we clarify that we generate states and actions using a generative model that learns the trajectory distribution. The generated states are not always feasible, but in most cases, they follow the learned distribution and can still be useful for policy learning, as demonstrated by our results against existing baselines. Probabilistic generative methods cannot guarantee that generated states are feasible without explicit constraints. These constraints are environment-dependent and are beyond the scope of general algorithms.


Q6: Performance discrepancy between CEDGE-Policy and CEDGE-Planner and the heterogeneous results in HalfCheetah-Gravity-0.5.

A6: The performance differences between CEDGE-Planner and CEDGE-Policy arise from how they use generated trajectories. CEDGE-Planner interacts with the target environment and generates new actions based on the current observed state at each step. Therefore, its performance depends heavily on the quality of generated actions. For some states, the generated actions may still be biased toward the source domain, leading to poor performance under the target dynamics. In contrast, CEDGE-Policy generates a synthetic dataset and filters the trajectories before training an offline policy using IQL. This filtering process can reduce the influence of low-quality trajectories, and offline RL can further combine useful transitions across different trajectories through its stitching ability. As a result, CEDGE-Policy can achieve better performance in most tasks.

We also clarify that HalfCheetah-Gravity-0.5 is an outlier in our experiments. In this task, methods that rely heavily on the source dataset, such as source-only IQL, DARA, and Diffuser, achieve good performance. In contrast, target-only IQL performs poorly, suggesting that the target dataset may not provide sufficient information for effective policy learning. As CEDGE relies on target data to guide trajectory generation, the generated trajectories may not accurately reflect the target distribution in this case. This can negatively affect downstream policy learning and explains why CEDGE-Policy performs similarly to target-only IQL and worse than source-only IQL.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.