- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies offline RL with a large source-domain dataset and a small target-domain dataset collected under different dynamics. CEDGE trains a trajectory diffusion model once on the source data, then adapts its samples with three target-specific guidance terms for dynamics compatibility, predicted return, and source-policy correction. The guided model is used either as a receding-horizon planner or as a generator of synthetic data for IQL.
The empirical study covers gravity and friction shifts on four MuJoCo agents, with additional kinematic and morphology shifts in the appendix. The paper also gives an error bound for guided generation and an idealized finite-sample analysis of policy learning with real and synthetic data.
Critical strengths
-
The reuse setting is well motivated. Training one source trajectory model and adapting it through smaller target-specific modules is a reasonable design for settings with many target dynamics and little target data. The separation into domain, return, and policy guidance also makes the method easy to interpret.
-
The evaluation is broad and includes several useful controls. In Table 12, the full CEDGE planner has the highest reported mean among the guidance ablations in all 16 HalfCheetah and Walker2d settings. In Table 14, CEDGE-Policy has a higher mean than the unguided Diffuser synthetic-data baseline in all 16 settings, and Table 15 reports higher means than target-only diffusion in all eight gravity-shift settings. These experiments give evidence for the guidance mechanism beyond the total score in Table 1, although the three-seed uncertainty should still be kept in mind.
-
The paper is clear about the scope of its theory. Appendix A explicitly states that the analysis does not cover trajectory filtering, planner candidate selection, or the implemented IQL learner, and Assumption A.2 notes that the probability bound does not cover dependent transitions from the same trajectory. The submitted code is also substantial enough to inspect the main training and data-generation paths.
Critical weaknesses
- The submitted synthetic-data exporter appears to create an artificial final transition for every generated segment. In
train_planner.py(lines 770-787), a segment containsHstate-action entries andnext_statesis formed by shifting the states and repeating the last state. The last exported tuple is therefore of the form(s_{H-1}, a_{H-1}, s_{H-1}). It is marked as a timeout but not as terminal. The policy replay buffer then setsnot_done = 1 - terminalsand does not use the timeout field (policy_optimization/algo/utils.py, lines 40-45), so this tuple is included in bootstrapping. With the supplied horizons, this is one quarter of the exported HalfCheetah entries and one thirty-second of the Hopper and Walker2d entries.
This observation is limited to the submitted implementation. It does not show that the reported returns are incorrect, and it does not affect CEDGE-Planner. If the same exporter was used for the unguided synthetic-data controls, some relative comparisons may also remain informative. However, the issue is directly relevant to the claim that CEDGE-Policy benefits from correctly formed target-compatible transitions. A corrected rerun on the short-horizon HalfCheetah tasks would be particularly useful.
-
The artifact does not currently support full reproduction of Table 1. The main table contains eight Ant gravity/friction settings, but the submitted task configurations and environment assets cover only HalfCheetah, Hopper, and Walker2d. Both
dataset/call_dataset.pyandtrain_iql.pyexplicitly reject other agents. There are also discrepancies between the paper and the provided defaults: Tables 5 and 7 report 500,000 diffusion updates and 200,000 energy updates, whereasmujoco.yamlsets both to 1,000,000; the task YAML files use guidance weights(0.01, 0.01, 0.01), while the paper lists search values beginning at 0.1 and the README example uses(1.0, 1.0, 0.1). These may be stale defaults or values overridden at launch, rather than the settings used for the experiments, but no run commands or manifests are supplied to resolve the discrepancy. -
The target used to train the return-energy model differs between the method description and the code. Equation (4.6) defines an
H-step discounted segment return. The implementation passesbatch["val"]to the return predictor (train_planner.py, lines 371-380), while the dataset constructsseq_valrecursively to the end of the episode and returns its value at the segment start (cleandiffuser/dataset/d4rl_mujoco_dataset.py, lines 124-145). This is a return-to-go target rather than the segment return written in the paper. Both choices are plausible, but they yield different guidance signals and should not be treated as interchangeable. -
Some of the aggregate conclusions are difficult to calibrate from three seeds. Table 1 emphasizes the sum of task means, 1504.16 for CEDGE-Policy versus 1427.26 for MOBODY, without uncertainty for the aggregate. The per-task table contains both large gains and differences that are small relative to the reported standard deviations. It is also unclear how the domain and return guidance weights were selected and whether target-environment evaluations were used in that selection. This matters because Table 17 shows substantial sensitivity to the domain and policy guidance weights on HalfCheetah-Gravity-0.1. A stated selection protocol and a paired or seed-level aggregate analysis would make the main comparison easier to assess.
-
The theory is useful as an error decomposition, but it should not be read as a guarantee for the implemented CEDGE-Policy pipeline. Theorem 4.2 uses independent samples, finite function classes, exact minimax optimization, realizability, completeness, and coverage assumptions. The actual learner uses correlated generated segments, filtering, learned rewards, and neural IQL. The appendix acknowledges this gap, so I do not regard it as a flaw in the proof; it is mainly a limitation on how strongly the theorem can support the empirical algorithm.
Minor Comments
-
The trajectory notation should be made consistent. Section 3 includes a final state
s_H, while Table 3 and the submitted dataset return tensors withHstate-action entries and no separates_H. -
The Windy Maze experiment is helpful as an illustration, but it uses one stated training seed and a small target reference set. I would avoid treating the difference between 0.269 and 0.236 as strong standalone distributional evidence.
-
The runtime table contains only two tasks. It supports the claimed reuse mechanism in those cases, but not a general runtime comparison across all agents and shifts.
Questions for authors
-
Were the CEDGE-Policy results produced with the submitted transition-export code? If so, what changes when the final tuple of each segment is dropped, or when an additional next state is generated and timeouts are handled consistently in IQL?
-
Can the authors provide the Ant configurations and the exact commands or run manifests used for Table 1, including the training steps and guidance weights for each setting?
-
Was the return predictor used in the reported runs trained on an
H-step segment return or on the episode return-to-go implemented byseq_val? -
How were the guidance weights selected? In particular, was any target-environment interaction used for model selection?
-
Can the authors report uncertainty for an aggregate comparison, for example paired seed-level differences within each task followed by a suite-level summary?
5 comments
Q3: Was the return predictor used in the reported runs trained on an H-step segment return or on the episode return-to-go implemented by seq_val?
A3: We thank the reviewer for pointing out this mismatch. We clarify that the return energy predictor in our reported experiments is trained using the discounted episode return-to-go as implemented in the submitted code.
The H-step segment return only considers rewards within the generated segment. In contrast, the episode return-to-go includes rewards from the start of the segment through the end of the episode. In our implementation, the return predictor takes an H-step trajectory segment as input and predicts the return-to-go from its starting state. We use return-to-go because it provides longer-term reward information beyond the generated segment. This choice follows the return prediction implementation in Diffuser [1]. We revised Eq. (4.6) and the corresponding description in our revision.
[1] Planning with Diffusion for Flexible Behavior Synthesis. ICML 2022.
Q4: Selection of guidance weights and evaluation checkpoints.
A4: We select the guidance weights and evaluation checkpoints based on performance evaluations in the target environments, which is a common practice in offline RL. It can be considered as selecting the guidance weights with the validation environments. Once the hyperparameters are fixed, CEDGE requires only offline source and target datasets without additional environment interactions.
Q5: Can the authors report uncertainty for an aggregate comparison?
A5: We provide additional analysis showing that CEDGE-Policy maintains its aggregate performance advantage over the baselines across seeds. For each seed, we sum the scores across all 32 tasks for each method. We then compute the paired differences between CEDGE-Policy and each baseline using matched seeds. We report the mean and standard deviation across three seeds. As shown below, CEDGE-Policy achieves an aggregate score of 1504.16 ± 23.08, compared with 1427.26 ± 27.14 for MOBODY. The paired improvement is 76.90 ± 39.75.
| Method | Aggregate Score | Paired Difference (CEDGE-Policy − Method) |
|---|---|---|
| MOBODY | 1427.26 ± 27.14 | +76.90 ± 39.75 |
| OTDF | 1056.87 ± 7.64 | +447.29 ± 28.74 |
| DmC | 1044.40 ± 43.66 | +459.76 ± 35.60 |
| xTED | 998.75 ± 44.47 | +505.41 ± 60.16 |
| Diffuser | 865.91 ± 10.37 | +638.25 ± 25.31 |
| CEDGE-Planner | 908.92 ± 3.90 | +595.24 ± 23.41 |
| CEDGE-Policy | 1504.16 ± 23.08 | – |
Q6: Idealized assumptions in Theorem 4.2 compared to the implemented pipeline.
A6: We clarify that Theorem 4.2 provides theoretical insights into the benefits and limitations of using synthetic data for downstream policy learning. Our main goal is not to provide a theoretical guarantee specifically for IQL. Instead, we use an idealized offline RL learner to study how synthetic data affects downstream policy learning. The theorem shows the trade-off between increasing the amount of training data and introducing errors from generated transitions. Such idealized analysis is common in theoretical offline RL research, where simplifying assumptions are used to understand the main factors affecting policy performance. We follow this approach to provide theoretical insights rather than a direct guarantee for the implemented CEDGE-Policy.
Q7: Inconsistency between the trajectory definition for s_H in Section 3 and Table 3.
A7: We thank the reviewer for pointing this out. We remove the final state from the trajectory definition in Section 3 to align the notation with the state-action pairs used in Table 3 and our implementation. We add this revision in our revised manuscript.
Q8: Limited training seeds in Windy Maze
A8: We provide additional results addressing the concern about single seed evaluation. We repeat the Windy Maze experiment with 10 independent training seeds, following the setup in Appendix F.1. For each seed, we average the distances over 32 generated trajectories and report the mean and standard deviation across the 10 seeds. CEDGE achieves a lower mean distance than MOBODY. We replace the original results with these multiple-seed results in our revised Table 2.
| Setting | CEDGE | MOBODY |
|---|---|---|
| 1 seed | 0.236 | 0.269 |
| 10 seeds | 0.260 ± 0.017 | 0.304 ± 0.031 |
Q9: The runtime table contains only two tasks.
A9: We extend the runtime evaluation to more settings, including HalfCheetah, Walker2d, and Hopper under gravity and friction shifts with shift level 2.0. Each entry reports the Full / Reuse runtime in hours. For CEDGE, Full includes source diffusion model training, while Reuse uses a pretrained source diffusion model. As shown below, reusing the source diffusion model reduces the runtime of both CEDGE variants across all six settings. We add these results to our revised Appendix H.4 Table 14.
| Method | HalfCheetah-Gravity-2.0 | HalfCheetah-Friction-2.0 | Walker2d-Gravity-2.0 | Walker2d-Friction-2.0 | Hopper-Gravity-2.0 | Hopper-Friction-2.0 |
|---|---|---|---|---|---|---|
| IQL | ~3.0h / ~3.0h | ~3.0h / ~3.0h | ~3.0h / ~3.0h | ~3.0h / ~3.0h | ~3.0h / ~3.0h | ~3.0h / ~3.0h |
| MOBODY | ~6.1h / ~6.1h | ~6.1h / ~6.1h | ~7.8h / ~7.8h | ~7.8h / ~7.8h | ~6.7h / ~6.7h | ~6.7h / ~6.7h |
| Diffuser | ~5.5h / ~5.5h | ~3.0h / ~3.0h | ~7.6h / ~7.6h | ~4.1h / ~4.1h | ~4.1h / ~4.1h | ~4.1h / ~4.1h |
| CEDGE-Planner | ~5.6h / ~1.9h | ~4.4h / ~1.6h | ~7.6h / ~3.3h | ~6.2h / ~2.5h | ~6.3h / ~2.5h | ~6.3h / ~2.5h |
| CEDGE-Policy | ~7.4h / ~3.7h | ~5.4h / ~2.5h | ~9.1h / ~4.8h | ~5.2h / ~2.4h | ~5.2h / ~2.3h | ~5.2h / ~2.4h |
We hope these additions address the reviewer's concerns. We would be happy to answer any remaining questions.
We thank the reviewer for their valuable time and constructive feedback. We hope the responses below address all questions and concerns.
Q1: Were the CEDGE-Policy results produced with the submitted code? And what changes when the final tuple
(s_{H-1}, a_{H-1}, s_{H-1})of each segment, denoted as 'timeout ', is dropped?A1: We clarify that the reported CEDGE-Policy results were produced using the submitted code. Dropping the final tuple has little effect on the overall performance and does not change our main empirical conclusions. In the submitted code, the final tuple (sH−1,aH−1,sH−1) of each generated segment is included in IQL bootstrapping. To address this issue, we remove the final tuple from each segment. We keep only the H−1 transitions between consecutive generated states. We then rerun CEDGE-Policy on all 32 tasks with three seeds.
As shown below, the total normalized score changes from 1504.16 to 1511.53 after dropping the final tuple. For HalfCheetah, the final tuple accounts for one-fourth of the original generated transitions. The total score across eight tasks changes only from 373.66 to 373.18. Overall, dropping the final tuple has little effect on the aggregate performance.
Q2: Missing Ant configurations and inconsistencies of hyperparameters in the supplementary code.
A2: Thanks for catching these mismatches. We clarify that, following convention, we only provided the submitted supplementary code with default hyperparameters. We provide the configurations and running configs in the updated supplementary code.
The reported results use 500,000 diffusion training updates and 200,000 energy training updates as shown in Tables 5 and 7. For guidance weights, we select task-specific values for domain and return guidance, while fixing the policy guidance weight at 0.1. The exact weights used for CEDGE-Policy in Table 1 are shown below. Each entry represents (λ1,λ2,λ3) for domain, return, and policy guidance, respectively.