- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes TCE for cross-domain offline RL with abundant source data and limited target data. Two score models expand state coverage using nearby source states and generate next states with target-like dynamics. Learned inverse-dynamics and reward models label the transitions for KL-regularized IQL training, optionally mixed with nearby source data. A theoretical bound decomposes the performance gap into dynamics mismatch and generation error. Experiments on 108 MuJoCo tasks and 8 Adroit settings show substantial gains over baselines.
Critical strengths
- Consistent empirical gains. TCE achieves best on most of the 108 MuJoCo tasks and on all Adroit settings.
- Simple design with useful diagnostics. Separating where to generate (state support) from how transitions behave (target-only model) is clean. The analyses against the true target simulator and the gravity sweep show when direct source reuse helps and when it hurts.
Critical weaknesses
- Most of the gain does not come from coverage expansion. Target-only augmentation (η = 0) already scores 54.6 on the morphology suite, against 57.4 for the default η = 0.2. Source-guided expansion, the paper's central idea, adds only about 3 points.
- Key baselines are missing. There is no comparison to target-only generative augmentation (e.g., SynthER [1]) or to target-only BC/IQL with the same KL term. Simple Aug baseline is also absent from the main tables.
- Results seem to track target data quality. On Hopper morphology, TCE(Gen) scores about 98 to 100 with expert target data but only 16 to 19 with medium-expert target data, for every source dataset. This suggests the KL term cloning the target data drives much of the performance.
- The theory gives limited guidance on coverage. Thm 4.1 follows directly from the telescoping lemma and bounds J_mix(π) − J_tar(π) for a given policy. No term in it rewards broader coverage, so the coverage/error trade-off in Sec. 5.3 is supported only empirically.
- The comparison protocol favors TCE. Baseline numbers are copied from previous work, while TCE uses per-environment η and β. TCE(Mix) reports the best λ per setting, chosen on evaluation results.
[1] Synthetic Experience Replay. NeurIPS 2023.
Questions for authors
- Please report Simple Aug. (η = λ = 0) for all tasks in Tables 1, 2, 11 and 12. Does TCE(Gen) clearly beat it, especially under kinematic shifts and on Adroit?
- How does TCE compare to SynthER-style target-only augmentation and to target-only BC/IQL with the same KL term?
- What explains the gap between expert and medium-expert targets on Hopper? How does performance change with β = 0?
- Beyond Thm 4.1, is there any formal argument for why expanding coverage (larger η) should improve the learned policy's target performance?
- Were the copied baselines evaluated on the same datasets and episodes? How is λ chosen without target returns, and how does TCE(Mix) perform with a single fixed λ?
4 comments
W4 & Q4 (Formal Argument for Coverage Expansion):
We thank the reviewer for highlighting that Theorem 1 bounds the dynamics gap and generation error but does not directly explain the role of coverage expansion. We therefore introduced Proposition 1 in Section 4.2 (proof in Appendix B.2), extending Theorem 1 to state-wise selection of generated versus source transitions. For the set of states covered by generation, the bound becomes
where measures the generation-induced value discrepancy and measures the discrepancy due to the source-target dynamics gap. The bound identifies a state-wise criterion for expansion: generated transitions are preferable where their discrepancy is smaller than that of direct source reuse, while generation error must remain controlled. This is a criterion on contributions to the bound, rather than a guarantee of monotonic policy improvement.
To examine this criterion, we added to Table 3 in Section 5.3 (Table R.1). Under morphology shifts, the difference is positive on average at every tested , helping explain why TCE(Gen) outperforms TCE(Mix). However, generation-induced discrepancy increases and the difference decreases as grows. The best average performance at therefore supports moderate coverage expansion that balances the benefit of additional states against increasing generation error.
Table R.1. Value discrepancies and average normalized scores of TCE(Gen) on 36 morphology-shift tasks, reproduced from Table 3 in Section 5.3.
| 0 | 0.1 | 0.2 | 0.5 | 0.9 | |
|---|---|---|---|---|---|
| $\lvert\hat\Delta_{\mathrm{tar}}\rvert $ | 1.47 ± 0.21 | 2.56 ± 0.20 | 3.01 ± 0.24 | 3.71 ± 0.29 | 3.85 ± 0.27 |
| $\lvert\Delta_{\mathrm{src}}\rvert -\lvert\hat\Delta_{\mathrm{tar}}\rvert $ | 3.40 ± 0.32 | 3.26 ± 0.46 | 3.01 ± 0.36 | 2.70 ± 0.26 | 2.52 ± 0.22 |
| Score | 54.6 ± 3.0 | 55.8 ± 1.8 | 57.4 ± 1.9 | 52.1 ± 2.7 | 50.9 ± 2.9 |
W5 & Q5 (Comparison Protocol):
We thank the reviewer for questioning the comparison protocol and the selection of . The reported baseline results from [2] were tuned on the same benchmarks, and TCE uses the same datasets, evaluation episodes, and normalization with evaluation-based selection over a narrower range. Most settings use the same default and , and separate-seed validation in Appendix D.3 produces results consistent with those reported. We clarified the comparison protocol in Section 5.1.
For TCE(Mix), a fixed reproduces the selected results in all settings except HalfCheetah gravity, since is already the selected value in those settings. On HalfCheetah gravity, which we additionally evaluated, its total score over nine tasks is 88.8, below 297.3 with the selected but above 50.6 for TCE(Gen), as stronger source reuse performs better because generation is inaccurate with the limited target data. Without target returns or a reliable discrepancy estimate, selecting remains difficult [3]. A small target validation set would help compare source and generation discrepancies; otherwise, small and offer a practical default, though not a guaranteed optimum. We discuss this limitation in Appendix A.
References
[2] Lyu et al., "Cross-Domain Offline Policy Adaptation with Optimal Transport and Dataset Constraint", ICLR 2025.
[3] Liu et al., "When is Offline Policy Selection Sample Efficient for Reinforcement Learning?", AAMAS 2026.
Once again, we thank the reviewer for the careful review and feedback. We hope the additional experiments and clarifications adequately address the concerns raised.
Thank you for the additional experiments. The target-only comparisons show that coverage expansion provides gains beyond generation alone, particularly on kinematic shifts and Adroit. The KL ablation also addresses my concern that regularization explains most of the improvement. I remain concerned about selecting hyperparameters using evaluation returns, but the new results address my main objections. I will raise my rating to Weak Accept.
We sincerely thank the reviewer for their thoughtful reassessment and constructive feedback throughout the discussion. We are pleased that the additional experiments have addressed the main concerns. We acknowledge that hyperparameter selection without evaluation returns remains an important challenge in offline RL and will consider more principled offline selection criteria in future work. We greatly appreciate the reviewer's time and consideration.
We thank the reviewer for the comments on the contribution of coverage expansion, the target-only baselines, the role of the KL term, the theoretical justification, and the comparison protocol. Below, we respond to each point and describe the corresponding revisions to the manuscript.
W1, Q1 & Q2 (Contribution of Coverage Expansion):
We thank the reviewer for the suggestion and agree that target-oriented augmentation comparisons are needed to isolate the contribution of coverage expansion. We added target-only IQL, target-only generation with SynthER [1], Gaussian-noise augmentation with Noisy, Interpolation between target states, and Simple Aug. (TCE with η=λ=0), alongside TCE(Gen). Section 5.4 (Table 5) reports averages across all MuJoCo shifts and Adroit, with per-task results in Appendix E.5. All methods use the same KL-regularized IQL, and Noisy and Interpolation change only the starting states while retaining TCE's remaining generation pipeline.
Simple Aug. outperforms the other target-only baselines across all benchmarks, supporting the effectiveness of TCE's two-stage, state-pair-first generation even without coverage expansion. TCE(Gen) further improves upon Simple Aug., as shown in Table R.3. Although the morphology gain is moderate, the gains are larger under kinematic shifts and on Adroit, with improvements in all 12 kinematic target configurations and all 8 Adroit settings. These comparisons show that both the generation design and source-guided coverage expansion contribute to TCE's performance.
Table R.3. Average normalized score (mean ± std over 5 seeds), reproduced from Table 5 in Section 5.4.
References
[1] Lu et al., "Synthetic Experience Replay", NeurIPS 2023.
W2 (Target-only Baselines):
We thank the reviewer for identifying these missing target-only comparisons. Section 5.4 (Table 5) now compares target-only IQL, SynthER [1], Noisy, Interpolation, Simple Aug., and TCE(Gen) under the same KL-regularized IQL learner. Table R.3 summarizes the aggregate results, and Appendix E.5 provides per-task results for all MuJoCo shifts and Adroit due to space constraints. These comparisons evaluate both the benefit of TCE's state-pair-first generation over other target-oriented augmentation methods and the additional contribution of source-guided expansion.
W3 & Q3 (Role of the KL Term):
We thank the reviewer for questioning whether the KL term drives the difference between expert and medium-expert target datasets. The regularization coefficient is already small (β=0.001) in nearly all settings, and removing regularization changes the morphology average only modestly (57.4 to 55.8, Table 4). To examine the Hopper results directly, we additionally evaluated TCE(Gen) with β=0, as reported in Table R.5. Performance changes little across target-data qualities, while the large gap between expert and medium-expert targets persists. These results show that the KL term is not the primary driver of the observed performance gap, which is instead associated with differences in the target datasets.
Table R.5. TCE(Gen) with and without policy regularization on Hopper morphology shifts, averaged over the three source datasets (mean ± std over 5 seeds).