acceptodds
Under review as a conference paper at ICLR 2027

Generative Support Realignment for Cross-Domain Offline Reinforcement Learning

Abstract

Cross-domain offline reinforcement learning learns a target policy from pre-collected source and target datasets with different dynamics. When target data are scarce, effectively compensating for their limited coverage using source data remains challenging due to the discrepancy between domains. We propose *Target-aligned Coverage Expansion (TCE)*, which leverages source states to generatively realign and expand the limited target support while controlling the generation error induced by this expansion. We further derive a performance gap bound that characterizes the interplay between generation error and source–target dynamics gap, providing theoretical guidance for effective coverage expansion and source utilization. Across diverse cross-domain environments, TCE consistently outperforms state-of-the-art baselines, while further analysis empirically supports the guidance provided by our theoretical findings.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Offline Learning, Cross-domain

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HgsyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TCE for cross-domain offline RL with abundant source data and limited target data. Two score models expand state coverage using nearby source states and generate next states with target-like dynamics. Learned inverse-dynamics and reward models label the transitions for KL-regularized IQL training, optionally mixed with nearby source data. A theoretical bound decomposes the performance gap into dynamics mismatch and generation error. Experiments on 108 MuJoCo tasks and 8 Adroit settings show substantial gains over baselines.

Critical strengths
  1. Consistent empirical gains. TCE achieves best on most of the 108 MuJoCo tasks and on all Adroit settings.
  2. Simple design with useful diagnostics. Separating where to generate (state support) from how transitions behave (target-only model) is clean. The analyses against the true target simulator and the gravity sweep show when direct source reuse helps and when it hurts.
Critical weaknesses
  1. Most of the gain does not come from coverage expansion. Target-only augmentation (η = 0) already scores 54.6 on the morphology suite, against 57.4 for the default η = 0.2. Source-guided expansion, the paper's central idea, adds only about 3 points.
  2. Key baselines are missing. There is no comparison to target-only generative augmentation (e.g., SynthER [1]) or to target-only BC/IQL with the same KL term. Simple Aug baseline is also absent from the main tables.
  3. Results seem to track target data quality. On Hopper morphology, TCE(Gen) scores about 98 to 100 with expert target data but only 16 to 19 with medium-expert target data, for every source dataset. This suggests the KL term cloning the target data drives much of the performance.
  4. The theory gives limited guidance on coverage. Thm 4.1 follows directly from the telescoping lemma and bounds J_mix(π) − J_tar(π) for a given policy. No term in it rewards broader coverage, so the coverage/error trade-off in Sec. 5.3 is supported only empirically.
  5. The comparison protocol favors TCE. Baseline numbers are copied from previous work, while TCE uses per-environment η and β. TCE(Mix) reports the best λ per setting, chosen on evaluation results.

[1] Synthetic Experience Replay. NeurIPS 2023.

Questions for authors
  1. Please report Simple Aug. (η = λ = 0) for all tasks in Tables 1, 2, 11 and 12. Does TCE(Gen) clearly beat it, especially under kinematic shifts and on Adroit?
  2. How does TCE compare to SynthER-style target-only augmentation and to target-only BC/IQL with the same KL term?
  3. What explains the gap between expert and medium-expert targets on Hopper? How does performance change with β = 0?
  4. Beyond Thm 4.1, is there any formal argument for why expanding coverage (larger η) should improve the learned policy's target performance?
  5. Were the copied baselines evaluated on the same datasets and episodes? How is λ chosen without target returns, and how does TCE(Mix) perform with a single fixed λ?
4 comments

We thank the reviewer for the comments on the contribution of coverage expansion, the target-only baselines, the role of the KL term, the theoretical justification, and the comparison protocol. Below, we respond to each point and describe the corresponding revisions to the manuscript.


W1, Q1 & Q2 (Contribution of Coverage Expansion):

We thank the reviewer for the suggestion and agree that target-oriented augmentation comparisons are needed to isolate the contribution of coverage expansion. We added target-only IQL, target-only generation with SynthER [1], Gaussian-noise augmentation with Noisy, Interpolation between target states, and Simple Aug. (TCE with η=λ=0\eta=\lambda=0), alongside TCE(Gen). Section 5.4 (Table 5) reports averages across all MuJoCo shifts and Adroit, with per-task results in Appendix E.5. All methods use the same KL-regularized IQL, and Noisy and Interpolation change only the starting states while retaining TCE's remaining generation pipeline.

Simple Aug. outperforms the other target-only baselines across all benchmarks, supporting the effectiveness of TCE's two-stage, state-pair-first generation even without coverage expansion. TCE(Gen) further improves upon Simple Aug., as shown in Table R.3. Although the morphology gain is moderate, the gains are larger under kinematic shifts and on Adroit, with improvements in all 12 kinematic target configurations and all 8 Adroit settings. These comparisons show that both the generation design and source-guided coverage expansion contribute to TCE's performance.

Table R.3. Average normalized score (mean ± std over 5 seeds), reproduced from Table 5 in Section 5.4.

BenchmarkTarget-only IQLSynthERNoisyInterpolationSimple Aug.TCE(Gen)
Morphology36.8 ± 1.043.0 ± 1.449.0 ± 1.249.3 ± 1.554.6 ± 3.057.4 ± 1.9
Kinematic35.2 ± 1.142.2 ± 1.349.8 ± 1.647.0 ± 1.651.0 ± 1.757.3 ± 1.9
Gravity17.5 ± 1.324.5 ± 1.628.5 ± 1.726.3 ± 1.431.3 ± 1.835.3 ± 2.0
Adroit28.8 ± 1.237.8 ± 1.632.8 ± 1.128.0 ± 1.446.5 ± 1.164.2 ± 1.0

References

[1] Lu et al., "Synthetic Experience Replay", NeurIPS 2023.


W2 (Target-only Baselines):

We thank the reviewer for identifying these missing target-only comparisons. Section 5.4 (Table 5) now compares target-only IQL, SynthER [1], Noisy, Interpolation, Simple Aug., and TCE(Gen) under the same KL-regularized IQL learner. Table R.3 summarizes the aggregate results, and Appendix E.5 provides per-task results for all MuJoCo shifts and Adroit due to space constraints. These comparisons evaluate both the benefit of TCE's state-pair-first generation over other target-oriented augmentation methods and the additional contribution of source-guided expansion.


W3 & Q3 (Role of the KL Term):

We thank the reviewer for questioning whether the KL term drives the difference between expert and medium-expert target datasets. The regularization coefficient is already small (β=0.001\beta=0.001) in nearly all settings, and removing regularization changes the morphology average only modestly (57.4 to 55.8, Table 4). To examine the Hopper results directly, we additionally evaluated TCE(Gen) with β=0\beta=0, as reported in Table R.5. Performance changes little across target-data qualities, while the large gap between expert and medium-expert targets persists. These results show that the KL term is not the primary driver of the observed performance gap, which is instead associated with differences in the target datasets.

Table R.5. TCE(Gen) with and without policy regularization on Hopper morphology shifts, averaged over the three source datasets (mean ± std over 5 seeds).

Target dataTCE(Gen), β=0.001\beta=0.001TCE(Gen), β=0\beta=0
medium43.9 ± 0.941.2 ± 5.1
medium-expert16.9 ± 4.816.8 ± 3.6
expert99.2 ± 0.199.8 ± 0.1

W4 & Q4 (Formal Argument for Coverage Expansion):

We thank the reviewer for highlighting that Theorem 1 bounds the dynamics gap and generation error but does not directly explain the role of coverage expansion. We therefore introduced Proposition 1 in Section 4.2 (proof in Appendix B.2), extending Theorem 1 to state-wise selection of generated versus source transitions. For the set Scov\mathcal S_{\mathrm{cov}} of states covered by generation, the bound becomes

∣Jmix(π)−Jtar(π)∣≤γ1−γEρmixπ[∣Δ^tar∣+1{s∉Scov}(∣Δsrc∣−∣Δ^tar∣)],|J_{\mathrm{mix}}(\pi)-J_{\mathrm{tar}}(\pi)|\le\frac{\gamma}{1-\gamma}\mathbb E_{\rho^\pi_{\mathrm{mix}}}\big[|\hat\Delta_{\mathrm{tar}}|+\mathbf 1\{s\notin\mathcal S_{\mathrm{cov}}\}(|\Delta_{\mathrm{src}}|-|\hat\Delta_{\mathrm{tar}}|)\big],

where Δ^tar\hat\Delta_{\mathrm{tar}} measures the generation-induced value discrepancy and Δsrc\Delta_{\mathrm{src}} measures the discrepancy due to the source-target dynamics gap. The bound identifies a state-wise criterion for expansion: generated transitions are preferable where their discrepancy is smaller than that of direct source reuse, while generation error must remain controlled. This is a criterion on contributions to the bound, rather than a guarantee of monotonic policy improvement.

To examine this criterion, we added ∣Δsrc∣−∣Δ^tar∣|\Delta_{\mathrm{src}}|-|\hat\Delta_{\mathrm{tar}}| to Table 3 in Section 5.3 (Table R.1). Under morphology shifts, the difference is positive on average at every tested η\eta, helping explain why TCE(Gen) outperforms TCE(Mix). However, generation-induced discrepancy increases and the difference decreases as η\eta grows. The best average performance at η=0.2\eta=0.2 therefore supports moderate coverage expansion that balances the benefit of additional states against increasing generation error.

Table R.1. Value discrepancies and average normalized scores of TCE(Gen) on 36 morphology-shift tasks, reproduced from Table 3 in Section 5.3.

η\eta00.10.20.50.9
$\lvert\hat\Delta_{\mathrm{tar}}\rvert $1.47 ± 0.212.56 ± 0.203.01 ± 0.243.71 ± 0.293.85 ± 0.27
$\lvert\Delta_{\mathrm{src}}\rvert -\lvert\hat\Delta_{\mathrm{tar}}\rvert $3.40 ± 0.323.26 ± 0.463.01 ± 0.362.70 ± 0.262.52 ± 0.22
Score54.6 ± 3.055.8 ± 1.857.4 ± 1.952.1 ± 2.750.9 ± 2.9

W5 & Q5 (Comparison Protocol):

We thank the reviewer for questioning the comparison protocol and the selection of λ\lambda. The reported baseline results from [2] were tuned on the same benchmarks, and TCE uses the same datasets, evaluation episodes, and normalization with evaluation-based selection over a narrower range. Most settings use the same default η\eta and β\beta, and separate-seed validation in Appendix D.3 produces results consistent with those reported. We clarified the comparison protocol in Section 5.1.

For TCE(Mix), a fixed λ=0.1\lambda=0.1 reproduces the selected results in all settings except HalfCheetah gravity, since λ=0.1\lambda=0.1 is already the selected value in those settings. On HalfCheetah gravity, which we additionally evaluated, its total score over nine tasks is 88.8, below 297.3 with the selected λ=0.9\lambda=0.9 but above 50.6 for TCE(Gen), as stronger source reuse performs better because generation is inaccurate with the limited target data. Without target returns or a reliable discrepancy estimate, selecting λ\lambda remains difficult [3]. A small target validation set would help compare source and generation discrepancies; otherwise, small η\eta and λ\lambda offer a practical default, though not a guaranteed optimum. We discuss this limitation in Appendix A.

References

[2] Lyu et al., "Cross-Domain Offline Policy Adaptation with Optimal Transport and Dataset Constraint", ICLR 2025.
[3] Liu et al., "When is Offline Policy Selection Sample Efficient for Reinforcement Learning?", AAMAS 2026.


Once again, we thank the reviewer for the careful review and feedback. We hope the additional experiments and clarifications adequately address the concerns raised.

Thank you for the additional experiments. The target-only comparisons show that coverage expansion provides gains beyond generation alone, particularly on kinematic shifts and Adroit. The KL ablation also addresses my concern that regularization explains most of the improvement. I remain concerned about selecting hyperparameters using evaluation returns, but the new results address my main objections. I will raise my rating to Weak Accept.

We sincerely thank the reviewer for their thoughtful reassessment and constructive feedback throughout the discussion. We are pleased that the additional experiments have addressed the main concerns. We acknowledge that hyperparameter selection without evaluation returns remains an important challenge in offline RL and will consider more principled offline selection criteria in future work. We greatly appreciate the reviewer's time and consideration.

Reviewer_yB86OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Given source domain (very accessible) and a target domain (scarce access, having morphological differences or differences in state transition probability from the source domain), the authors develop target-aligned coverage expansion (TCE), which uses score matching-based generation to produce data consistent with the target. In addition to the generated target-aligned data, source data which meets a distance threshold is included in the new dataset. Then, an offline reinforcement learning (ORL) method, implicit Q-learning, is used to develop a policy which is deployed in the target domain.

Two implementations of TCE (-mix and -gen) are evaluated across a variety of MuJoCo continuous control tasks. TCE-mix incorporates data from the source domain which is close to the target domain, while TCE-gen solely expands the target dataset through generation. In domains where the gap between the source domain and the target domain is minor (e.g., where a gravity coefficient is changed, but the agent retains all morphological characteristics from the target domain), TCE-mix is shown to perform well. However, when the gap between the source and target domains is pronounced, TCE-gen significantly outperforms TCE-mix. Both developed methods are compared against baseline ORL methods, and the developed methods demonstrate superior performance in the majority of evaluations.

Critical strengths
  1. The problem is relevant and the main design choice is well motivated. Source states may provide useful coverage even when source transitions are unreliable under target dynamics. Separating these two uses of source data gives a clear rationale for the proposed approach.

  2. The empirical experiments are broad. The paper evaluates several agents, three types of dynamics shifts, and Adroit manipulation tasks. The MuJoCo comparisons report results over five seeds. Additional analyses examine source selection, policy regularization, target data size, sample fidelity, and computational cost.

Critical weaknesses
  1. The incremental benefit of source-state expansion is not established. The η=0\eta=0 control appears only as a 36-task average in Table 3 (54.6 vs. 57.4) and as one Ant data pairing in Appendix H.1. Given the large across-task variance in Table 4, this does not show that expansion helps consistently across individual tasks or Adroit.

  2. Failures of expanded coverage are not diagnosable offline. Theorem 4.1 bounds errors for a fixed policy but does not guarantee generation accuracy in newly covered states. Table 12 shows TCE(Gen) at 4.1 versus 40.7 for TCE(Mix)/OTDF on HalfCheetah under a gravity shift, yet the paper provides no offline-computable indicator of when expansion is reliable.

Questions for authors
  1. How are η,λ\eta,\lambda, and the policy-regularization strength selected? Are target-environment returns used for selection? If so, are they obtained from separate validation runs or from the evaluation used for the final reported results? If not, what is the validation source, and what validation access and selection budgets are provided to the baselines?

  2. For the HalfCheetah gravity failures in Table 12, how do the default expansion setting and η=0\eta=0 compare in return, transition error, and action-reconstruction error? Can these measurements identify whether expansion causes the failure or amplifies an existing limitation of the generation pipeline?

  3. Which quantities computable from the fixed offline datasets, if any, indicate when coverage expansion is reliable? Can the authors relate such quantities to the reported generation errors and failure cases, so that practitioners could detect unreliable expansion without access to target returns?

I would consider raising my score if the authors address these concerns with convincing evidence and clarify the central claims accordingly.

4 comments

We thank the reviewer for the comments on the consistency of the expansion gain and on diagnosing unreliable expansion without target returns, as well as the questions regarding hyperparameter selection and the HalfCheetah gravity failure. Below, we respond to each point and describe the corresponding revisions to the manuscript.


W1 (Consistency of the Expansion Gain):

We thank the reviewer for questioning whether source-guided expansion improves performance consistently beyond target-only augmentation. We added comparisons with target-only IQL, SynthER [1], Noisy, Interpolation, Simple Aug. (η=λ=0\eta=\lambda=0), and TCE(Gen) across all MuJoCo shifts and Adroit. Section 5.4 (Table 5) reports aggregate results, and Appendix E.5 provides all per-task results. Simple Aug. outperforms the other target-only baselines, supporting TCE's two-stage, state-pair-first generation, while TCE(Gen) further improves upon Simple Aug. in every benchmark, particularly under kinematic shifts and on Adroit. Although the morphology gain is moderate, the gains are larger under kinematic shifts and on Adroit, with improvements in all 12 kinematic target configurations and all 8 Adroit settings. These comparisons isolate the additional contribution of source-guided coverage expansion.

We also corrected the ±\pm values in Tables 3 and 4 from across-task standard deviations to standard deviations over five seeds, consistent with the other result tables. The corrected variation is smaller, making the reported performance differences clearer.

Table R.3. Average normalized score (mean ± std over 5 seeds), reproduced from Table 5 in Section 5.4.

BenchmarkTarget-only IQLSynthERNoisyInterpolationSimple Aug.TCE(Gen)
Morphology36.8 ± 1.043.0 ± 1.449.0 ± 1.249.3 ± 1.554.6 ± 3.057.4 ± 1.9
Kinematic35.2 ± 1.142.2 ± 1.349.8 ± 1.647.0 ± 1.651.0 ± 1.757.3 ± 1.9
Gravity17.5 ± 1.324.5 ± 1.628.5 ± 1.726.3 ± 1.431.3 ± 1.835.3 ± 2.0
Adroit28.8 ± 1.237.8 ± 1.632.8 ± 1.128.0 ± 1.446.5 ± 1.164.2 ± 1.0

References

[1] Lu et al., "Synthetic Experience Replay", NeurIPS 2023.


W2, Q1 & Q3 (Selection and Reliability without Target Returns):

We thank the reviewer for raising the important challenge of hyperparameter selection in offline RL. Regarding comparison fairness, the baseline results reported from [2] were also selected through return-based hyperparameter tuning on the same benchmarks. TCE follows the same evaluation-based protocol over a narrower range, using common default settings for most environments. To assess the stability of this selection, we additionally evaluated the selected configurations on independent seeds in Appendix D.3; the resulting scores remain within one standard deviation of the reported results. We clarified this comparison protocol in Section 5.1.

Selecting hyperparameters or detecting unreliable expansion without target returns remains difficult [3]. Our theoretical analysis suggests comparing generation-induced value discrepancy with the source-target dynamics gap, but estimating these quantities reliably would require a small target validation set, such as transitions obtained by executing inferred actions at source states, together with a reliable target value estimate. The NN distance is available from the fixed datasets and helps restrict expansion, but does not by itself certify generation accuracy. If no validation data are available, TCE(Mix) with small η\eta and λ\lambda is a practical starting point, limiting extrapolation and direct source reuse. This is a heuristic rather than a fully offline selection rule. We discuss this general limitation of offline model selection and possible future directions in Appendix A.

References

[2] Lyu et al., "Cross-Domain Offline Policy Adaptation with Optimal Transport and Dataset Constraint", ICLR 2025.
[3] Liu et al., "When is Offline Policy Selection Sample Efficient for Reinforcement Learning?", AAMAS 2026.

Q2 (Cause of the HalfCheetah Gravity Failure):

We thank the reviewer for this question. The failure is primarily associated with insufficient target data for accurate generation rather than coverage expansion. We retained the prior benchmark's target-data budget for comparison and added the requested medium-to-medium analysis in Appendix E.2 (Table R.4). With 5K target transitions, η=0\eta=0 and the default η=0.1\eta=0.1 yield similar transition errors, action-reconstruction errors, and returns, indicating that moderate expansion does not noticeably worsen the failure. Increasing the same-quality target data to 10K substantially reduces generation errors and raises the score from 4.1 to 47.2. This identifies the limited target-data budget, rather than expansion, as the main bottleneck in this setting.

Table R.4. TCE(Gen) on the HalfCheetah gravity shift (medium-to-medium) under different target-data sizes (5 seeds), reproduced from Table 14 in Appendix E.2.

Target dataη\etaTransition MSEAction MSEReward MAEScore
5K (default)00.652 ± 0.1680.074 ± 0.0160.172 ± 0.0503.8 ± 0.4
5K (default)0.10.594 ± 0.1620.076 ± 0.0160.173 ± 0.0444.1 ± 0.4
10K0.10.127 ± 0.0570.005 ± 0.0010.019 ± 0.00347.2 ± 0.9
20K0.10.065 ± 0.0340.002 ± 0.0010.009 ± 0.00247.9 ± 0.5

Once again, we thank the reviewer for the careful review and feedback. We hope the additional results and clarifications adequately address the concerns raised.

Thank you for the additional experiments. My remaining concern (W2 & Q3) is whether the offline data alone provide a reliable basis for choosing between generated transitions and source transitions.

In Table 13, TCE(Gen) scores 4.1 on the HalfCheetah gravity task with medium data in both domains. At the same expansion coefficient, TCE(Mix) scores 40.7 with a mixing coefficient of 0.9 selected using target returns. The evaluation on separate seeds in Appendix D.3 covers only TCE(Gen) under morphology and kinematic shifts.

Nearest neighbor distance can be computed offline, but the paper does not show whether it can flag cases like this one before target returns are observed. Given how strongly the choice between generation and source reuse affects performance in this setting, I maintain my original score

We thank the reviewer for clarifying the remaining concern. We share this concern and have acknowledged the difficulty of selecting the best configuration using only offline data in Appendix A. In addition, we would like to clarify that this difficulty reflects a broader challenge in offline RL: proximity measures used by source-reuse methods likewise cannot directly predict policy performance, which depends on the underlying MDP. Recent work has explored estimated returns for model selection [1], although reliably applying such approaches remains challenging in settings like ours, where target data are extremely limited.

We would also like to highlight the methodological contribution of TCE. Rather than assuming source transitions can be reused reliably, TCE identifies how source-target dynamics mismatch limits direct reuse and proposes source-guided, target-aligned generation as an alternative. Our theoretical analysis characterizes when generation can reduce the discrepancy associated with source reuse, and the experiments support this motivation while demonstrating the additional benefit of source-guided coverage expansion beyond target-only augmentation. These findings suggest a useful direction for cross-domain offline RL: source data can guide where target-aligned transitions should be generated, even when their original transitions are unreliable for direct reuse. We appreciate the reviewer's constructive feedback and recognize fully offline configuration selection as an important direction for future work.

[1] Pai Liu et al. Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol. NeurIPS, 2025.

Reviewer_penuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigated the cross-domain offline RL, the challenge is that, the limited target data makes it challenging to learn a reliable policy under different source-target dynamics. This paper proposes the Target-aligned Coverage Expansion, it uses source states to expand target coverage while generating target-aligned transitions through score-based models. After that, the generated state pairs are further combined with inverse dynamics and reward models for offline policy learning. This work also tries to provide a theoretical analysis to characterize the trade-off between generation error and dynamics mismatch, which motivates the use of source data. The experiments on MuJoCo and Adroit show substantial improvements over existing methods across different domain shifts. I think the paper presents an interesting approach with relatively good empirical results.

Critical strengths
  1. Using source states to expand target coverage is interesting and well motivated. It provides a reasonable way to utilize source data when the dynamics gap is large, which is an important challenge in cross-domain offline RL.

  2. The analysis in Table 3 (investigate how different coverage expansion strengths affect generation error and policy performance). The results show that more coverage is not always better, which helps explain the motivation behind controlling the expansion strength.

  3. The proposed method achieves good improvements across different types of dynamics shifts, which reveals the effectiveness of the proposed method.

Critical weaknesses
  1. First is whether source states can actually be reliably used to expand the target state distribution when the two domains have substantially different dynamics? Although the source and target domains share the same state space, this does not necessarily mean that they share the same reachable states. E.g., morphology changes or broken joints may make certain source states physically unreachable in the target environment. The proposed nearest-neighbor selection and coverage coefficient control how far the distribution expands, but they do not guarantee that the expanded states are feasible under the target dynamics.

  2. Theorem 4.1 characterizes the performance gap in terms of source-target dynamics mismatch and generation error. But the bound mainly describes the error introduced by different transition kernels and does not explicitly characterize the benefit of expanding state coverage. In particular, it does not establish when additional coverage is beneficial or how the coverage coefficient should be selected. The empirical results in Table 3 show that moderate expansion improves performance despite increasing generation error, but this trade-off is not directly explained by the theoretical bound.

Questions for authors
  1. The method first generates state pairs and then predicts the actions using an inverse dynamics model. Could the authors discuss whether the inferred actions can actually produce the generated next states under the target dynamics? I think this would help clarify the reliability of the synthetic transitions.

  2. Could the authors compare the proposed coverage expansion with simpler alternatives, such as perturbing target states or interpolating between target states? This way, it can help clarify whether the improvements specifically come from useful source-domain coverage rather than general data augmentation.

  3. TCE(Gen) performs better under larger dynamics gaps, while TCE(Mix) can be beneficial under smaller gaps. How to select between these variants in practice, when the true target dynamics and corresponding dynamics gap are unknown?

  4. Table 4 shows that removing policy regularization causes only a modest performance decrease. I wonder when this regularization becomes important and whether it may restrict learning better policies from the expanded dataset?

  5. Could the authors provide a more detailed discussion of recent studies on sim-to-real transfer and cross-domain offline RL, including [1], [2], and [3]? In particular, [2] also explores generating target-aligned data using source-domain information, which is closely related to this paper. I would be interested in understanding how TCE differs from these approaches, especially regarding the assumptions about source-target transferability and the reliability of generated data.

[1] Latent Adaptation of Foundation Policies for Sim-to-Real Transfer - ICLR'26

[2] Cross-Domain Energy-Guided Diffusion Generation for Off-Dynamics Reinforcement Learning

[3] Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasets - ICML'26

2 comments

We sincerely thank the reviewer for the thoughtful comments and constructive suggestions. Below, we respond to each point and describe the corresponding revisions to the manuscript.


W2 (Theoretical Characterization of Coverage Expansion):

We thank the reviewer for highlighting that Theorem 1 bounds the dynamics gap and generation error but does not directly explain the role of coverage expansion. We therefore introduced Proposition 1 in Section 4.2 (proof in Appendix B.2), extending Theorem 1 to state-wise selection of generated versus source transitions. For the set Scov\mathcal S_{\mathrm{cov}} of states covered by generation, the bound becomes

∣Jmix(π)−Jtar(π)∣≤γ1−γEρmixπ[∣Δ^tar∣+1{s∉Scov}(∣Δsrc∣−∣Δ^tar∣)],|J_{\mathrm{mix}}(\pi)-J_{\mathrm{tar}}(\pi)|\le\frac{\gamma}{1-\gamma}\mathbb E_{\rho^\pi_{\mathrm{mix}}}\big[|\hat\Delta_{\mathrm{tar}}|+\mathbf 1\{s\notin\mathcal S_{\mathrm{cov}}\}(|\Delta_{\mathrm{src}}|-|\hat\Delta_{\mathrm{tar}}|)\big],

where Δ^tar\hat\Delta_{\mathrm{tar}} measures the generation-induced value discrepancy and Δsrc\Delta_{\mathrm{src}} measures the discrepancy due to the source-target dynamics gap. The bound identifies a state-wise criterion for expansion: generated transitions are preferable where their discrepancy is smaller than that of direct source reuse, while generation error must remain controlled. This is a criterion on contributions to the bound, rather than a guarantee of monotonic policy improvement.

To examine this criterion, we added ∣Δsrc∣−∣Δ^tar∣|\Delta_{\mathrm{src}}|-|\hat\Delta_{\mathrm{tar}}| to Table 3 in Section 5.3 (Table R.1). Under morphology shifts, the difference is positive on average at every tested η\eta, helping explain why TCE(Gen) outperforms TCE(Mix). However, generation-induced discrepancy increases and the difference decreases as η\eta grows. The best average performance at η=0.2\eta=0.2 therefore supports moderate coverage expansion that balances the benefit of additional states against increasing generation error.

Table R.1. Value discrepancies and average normalized scores of TCE(Gen) on 36 morphology-shift tasks, reproduced from Table 3 in Section 5.3.

η\eta00.10.20.50.9
$\lvert\hat\Delta_{\mathrm{tar}}\rvert $1.47 ± 0.212.56 ± 0.203.01 ± 0.243.71 ± 0.293.85 ± 0.27
$\lvert\Delta_{\mathrm{src}}\rvert -\lvert\hat\Delta_{\mathrm{tar}}\rvert $3.40 ± 0.323.26 ± 0.463.01 ± 0.362.70 ± 0.262.52 ± 0.22
Score54.6 ± 3.055.8 ± 1.857.4 ± 1.952.1 ± 2.750.9 ± 2.9

W1 (Reachability of Source States):

We understand the concern and agree that sharing a state space does not guarantee that every source state is reachable in the target. TCE considers domains with a shared physical state representation and sufficient target-relevant overlap to support transfer. In MuJoCo, the shared coordinates primarily describe joint configurations and velocities rather than domain-specific body lengths, allowing corresponding configurations to be represented across morphologies and initialized in the target simulator. Adroit introduces stronger morphology and joint constraints but retains a shared representation. States based on raw images or domain-specific geometry may lack this correspondence and require additional alignment.

The goal is therefore to identify useful regions within the shared state space, not to reuse every source state or transition. TCE restricts expansion to source states close to the target data to limit generation error, although proximity alone does not guarantee reachability. We clarified these setting assumptions in Section 5.1 and their limitations in Appendix A.


Q1 (Reliability of Inferred Actions):

We thank the reviewer for this suggestion. Table 3 already compares generated transitions with target-simulator outcomes and shows limited errors at moderate expansion levels (Table R.2). We added a direct simulator-based verification in Appendix F.2: we initialize the target MuJoCo simulator at a source state, execute the inferred action, and compare the resulting next state with the generated one through renderings and error measurements. Fig. 8 shows closely matching states, and within η≤0.2\eta\leq0.2, the normalized next-state error is 0.050 for generated transitions versus 0.106 for source transitions. These results further support the reliability of inferred actions and generated transitions within TCE's default expansion range.

Table R.2. Generation errors of TCE(Gen) across coverage coefficients on 36 morphology-shift tasks, reproduced from Table 3 in Section 5.3.

η\etaActionRewardTransition
00.038 ± 0.0090.024 ± 0.0100.016 ± 0.005
0.10.062 ± 0.0110.025 ± 0.0120.024 ± 0.013
0.20.066 ± 0.0140.025 ± 0.0130.029 ± 0.012
0.50.085 ± 0.0210.052 ± 0.0140.046 ± 0.015
0.90.118 ± 0.0200.069 ± 0.0170.052 ± 0.019

Q2 (Comparison with Perturbation and Interpolation):

We thank the reviewer for this suggestion. We added target-only IQL, SynthER [1], Noisy with Gaussian noise on target states, Interpolation between target states, and Simple Aug. (TCE with η=λ=0\eta=\lambda=0). Section 5.4 (Table 5) reports averages across all MuJoCo shifts and Adroit, with per-task results in Appendix E.5. All methods use the same KL-regularized IQL, and Noisy and Interpolation change only the starting states and retain TCE's remaining generation pipeline.

Simple Aug. outperforms the other target-only baselines across all benchmarks, supporting the effectiveness of TCE's two-stage, state-pair-first generation even without coverage expansion. TCE(Gen) further improves upon Simple Aug., as shown in Table R.3. Although the morphology gain is moderate, the gains are larger under kinematic shifts and on Adroit, with improvements in all 12 kinematic target configurations and all 8 Adroit settings. Thus, both the generation design and source-guided coverage expansion contribute to TCE's performance.

Table R.3. Average normalized score (mean ± std over 5 seeds), reproduced from Table 5 in Section 5.4.

BenchmarkTarget-only IQLSynthERNoisyInterpolationSimple Aug.TCE(Gen)
Morphology36.8 ± 1.043.0 ± 1.449.0 ± 1.249.3 ± 1.554.6 ± 3.057.4 ± 1.9
Kinematic35.2 ± 1.142.2 ± 1.349.8 ± 1.647.0 ± 1.651.0 ± 1.757.3 ± 1.9
Gravity17.5 ± 1.324.5 ± 1.628.5 ± 1.726.3 ± 1.431.3 ± 1.835.3 ± 2.0
Adroit28.8 ± 1.237.8 ± 1.632.8 ± 1.128.0 ± 1.446.5 ± 1.164.2 ± 1.0

References

[1] Lu et al., "Synthetic Experience Replay", NeurIPS 2023.


Q3 (Selecting between TCE(Gen) and TCE(Mix)):

We thank the reviewer for this question. Without target returns or reliable value-discrepancy estimates, selecting the better variant is difficult. A small target validation set, such as transitions obtained by executing inferred actions at source states, with a reliable target value estimate, would help compare the generation error with the dynamics gap. When such validation is unavailable, TCE(Mix) with small η\eta and λ\lambda is a practical starting point: the former limits extrapolation and the latter limits direct source reuse. This choice follows most of our settings but remains a heuristic rather than discrepancy-based selection. We discuss this limitation in Appendix A.


Q4 (Role of Policy Regularization):

We agree that policy regularization has only a modest effect on TCE's performance: removing it changes the morphology average from 57.4 to 55.8 (Table 4). In source-reuse methods, regularization encourages target behavior despite mismatched source transitions. TCE's target-trained generator reduces this need. When generation error is small, strong regularization may unnecessarily restrict policy improvement beyond the target behavior, motivating our small coefficient in most settings. When generated transitions are less reliable, however, regularization can help limit their influence. Its role is therefore supplementary and depends on transition reliability.


Q5 (Distinction from Related Work):

We thank the reviewer for these references, which we now discuss in Section 2. Like TCE, they use source information under limited target data but differ in what they transfer:

  • Da et al. [2] adapt a source-pretrained policy by tuning its latent input with limited target data.
  • Yang et al. [3] also generate target-oriented data, but steer a source-trained diffusion model using energy guidance learned from target data. Generation reliability therefore depends on correcting the source dynamics.
  • Qiao et al. [4] select target-consistent transitions from heterogeneous source datasets through dynamics and value alignment, requiring suitable transitions to exist in them.

TCE instead uses source states to expand generation support and trains its transition generator solely on target data. Its generated transitions need not inherit the original source dynamics; their reliability depends on generation error, which TCE limits through controlled expansion and examines through simulator verification. TCE still requires target-relevant state overlap, while TCE(Mix) additionally reuses selected source transitions.

References

[2] Da et al., "Latent Adaptation of Foundation Policies for Sim-to-Real Transfer", ICLR 2026.
[3] Yang et al., "Cross-Domain Energy-Guided Diffusion Generation for Off-Dynamics Reinforcement Learning", arXiv:2605.24810, 2026.
[4] Qiao et al., "Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasets", ICML 2026.


Once again, we thank the reviewer for the careful review and feedback. We hope the additional analyses and clarifications adequately address the concerns raised.

Comments on the paper

2 comments

We sincerely thank the Area Chair for coordinating the review process and all reviewers for their careful reviews and constructive feedback. During the rebuttal period, we strengthened TCE's theoretical foundation, clarified its evaluation protocol, and conducted additional experiments on coverage expansion and generation reliability. The main revisions, highlighted in blue in the revised manuscript, are summarized below.

(i) Theoretical Justification of Coverage Expansion (Reviewers penu, Hgsy; Sections 4.2 and 5.3, Appendix B.2)

Reviewers penu and Hgsy noted that Theorem 1 does not explicitly explain when coverage expansion is beneficial. We therefore added Proposition 1 in Section 4.2 (proof in Appendix B.2), derived from Theorem 1, which decomposes the performance-gap bound under state-wise selection into a generation-induced value discrepancy term and a term involving its difference from the source-induced discrepancy. It shows that using generated transitions at states where their discrepancy is smaller than that of source transitions reduces the corresponding contribution to the bound. This provides a theoretical guideline for expanding coverage while controlling generation error. We also added the discrepancy difference to Table 3 in Section 5.3. It remains positive on average but decreases with expansion, explaining both the advantage of generation over source reuse and its diminishing benefit as coverage grows.

(ii) Hyperparameter Selection and Comparison Protocol (Reviewers yB86, Hgsy, penu; Section 5.1, Appendix A)

Reviewers raised concerns about offline hyperparameter and variant selection, as well as the fairness of baseline comparisons. We clarified in Section 5.1 that the reported baselines were tuned on the same benchmarks and that TCE follows the same evaluation-based protocol over a narrower range. TCE uses common default hyperparameters in nearly all settings, and separate-seed evaluations support the stability of the selected configurations. We also clarified that TCE(Gen) is preferable under large dynamics gaps, whereas without validation data, TCE(Mix) with small η\eta and λ\lambda offers a practical starting point by limiting both extrapolation and direct source reuse. We acknowledge that selection without target returns or reliable discrepancy estimates remains challenging in strictly offline settings and discuss this limitation in Appendix A.

(iii) Comparison with Target-Oriented Augmentation (Reviewers yB86, Hgsy, penu; Section 5.4, Appendix E.5)

A central concern was whether source-guided coverage expansion improves performance beyond target-only augmentation. We added comparisons with target-only IQL, SynthER, Noisy, Interpolation, and Simple Aug. (TCE without source-guided expansion) across all MuJoCo shifts and Adroit, with aggregate results in Section 5.4 (Table 5) and per-task results in Appendix E.5. Simple Aug. outperforms the other target-only baselines, supporting the effectiveness of TCE's state-pair-first generation. TCE(Gen) further improves upon Simple Aug. across all benchmarks, particularly under kinematic shifts and on Adroit. These results distinguish the contributions of the generation mechanism and source-guided coverage expansion.

(iv) Shared State Space and Reliability of Generated Transitions (Reviewers penu, yB86; Section 5.1, Appendices A, E.2, and F.2)

Reviewer penu questioned the reachability of source states and the consistency of inferred actions with target dynamics, while Reviewer yB86 raised concerns about the HalfCheetah gravity failure. We clarified that shared state representations do not guarantee reachability and that TCE restricts expansion to source states close to the target data (Section 5.1, Appendix A). Target-simulator verification and visualizations in Appendix F.2 show that inferred actions closely reproduce generated next states within the default expansion range. Appendix E.2 further shows that the HalfCheetah gravity failure persists without expansion but is substantially alleviated by additional target data. These results support generation reliability in the evaluated range and identify limited target data, rather than expansion, as the primary cause of this failure.

We also corrected the reported standard deviations, expanded the related-work discussion, and added policy-regularization analyses. Detailed reviewer-specific responses and results follow.

Dear Reviewers,

We appreciate the time and effort you have devoted to evaluating our work. Your feedback has helped us strengthen the manuscript through additional experiments, theoretical analysis, and clarifications.

With the discussion period approaching its conclusion, we would welcome any remaining questions or concerns regarding our responses and revisions. We would be happy to address them.

Thank you again for your consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.