acceptodds
Under review as a conference paper at ICLR 2027

Where to Reuse, What to Propagate: Transition Reconstruction for Cross-Domain Offline Reinforcement Learning

Abstract

Cross-domain offline reinforcement learning leverages abundant source data to alleviate target-domain data scarcity, but dynamics mismatch often induces severe negative transfer. While existing methods attempt to mitigate this by selectively reusing target-compatible source transitions, they inevitably force a harsh trade-off between preserving valuable state-action coverage and avoiding dynamics-induced bias. We analytically reveal that this bottleneck stems from treating transitions as indivisible units, which strictly couples where experience is reused with what original outcome is propagated. To break this inherent coupling, we propose shifting the paradigm from transition selection to transition reconstruction, thereby enlarging the admissible transition family. Specifically, our framework, T-RAFT, isolates target-supported source conditions via unbalanced optimal transport (UOT), and amortizes these cross-domain correspondences into a continuous outcome translator using conditional Flow Matching (FM). This decoupled mechanism allows the agent to safely inherit broad source coverage while seamlessly replacing biased next states with target-compatible dynamics. Empirical evaluations demonstrate that our approach consistently overcomes the coverage-bias dilemma, outperforming state-of-the-art baselines on standard benchmarks.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, cross-domain offline reinforcement learning, flow matching

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FxTmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies Offline RL when the target task differs from a source dataset in state-transition dynamics while sharing the same state/action spaces and reward functions. It proposes a method that can make offline RL reuse the sorue dataset at large scale This method employs "Unbalanced Optimal Transport" to weight state-action conditions from a source dataset and align them with the conditions present in the target task. Conditional flow model is employed to adjust next state within source data. Through those, offline RL can be trained in a combiantion of target data and re-constructed data from source to improve data utilization.

Critical strengths

S1. Design is clear. Reusable places and re-designed places are clear. No errors were found in the core proof steps, given the stated assumptions.

S2. Exps look comprehensive. The method achieves state-of-the-art performance on 6 out of 8 Adroit tasks. This demonstrates its practical value (assuming the comparisons are fair).

Critical weaknesses

W1. The baseline setup lacks full traceability. Data in Tables 2 and 6 are carried over directly from previous results. The sources for data in other columns and whether these experiments were re-run, remain unclear to me. It is important to provide explicit details regarding to the data, normalization, checkpoints, and random seeds used. Across the 3 test sets, the average task-level performance difference vs. ROOT is only +0.3/−0.5/−1.5: why is it sufficient to establish equivalence?

  • Plz include IQL baseline results generated using your own pipeline (ideally also including OTDF) to better assess the magnitude of these differences and to examine when proposed method has superior performance on Adroit tasks.

W2 The gains claimed for the flow models is a combination of multiple things not just the flow model itself. UOT (Unbalanced Optimal Transport) reweighting is emploed in UOT. CompFlow/UOT-FM and prior editing-based methods already covered the concept of "reconstruction". Ablation studies that can measure or quantify the separated impact of individual techniques (e.g., the use of UOT, emplying simpler translators, etc) are missing but important to appreciate the value of the design.

  • Plz conduct a comparison across: -- original successor samples are sampled by the UOT mass, -- residuals obtained by directly matching the target -- the full T-RAFT method.

W3 I am thinking, for an RL changing the next state would also cause distribution changing of reward functiosn for consistency The original reward may no longer be valid following a state transition, R(s,a,ssrc′)=R(s,a,snew′)R(s, a, s'_{src}) = R(s, a, s'_{new}).

For ex, if the reward is defined as "forward velocity minus action energy consumption" modifying could alter the forward velocity, rendering the original reward value inapplicable to the new sample. Consequently, it is necessary to compare 3 things: using R(s,s′)R(s, s') (based only on states), using R(s,a,s′)R(s, a, s') (incorporating the action), and calculating the "ground truth" directly from formula.

W4. Direct empirical measurement is missing for coverage and translation. The metric "local-neighbor improvement" serves merely as a proxy.

W5: RL experiments share a single random seed for the preprocessing stage? The variance is then not accounted for in the reported experimental uncertainty. Plz measure successor error, effective sample size, and the variance stemming from the preprocessing stag

Questions for authors

For W1:

  • Plz include IQL baseline results generated using your own pipeline (ideally also including OTDF) to better assess the magnitude of these differences and to examine when proposed method has superior performance on Adroit tasks.

For W2

  • Plz conduct a comparison across: -- original successor samples are sampled by the UOT mass, -- residuals obtained by directly matching the target -- the full T-RAFT method.

For W5 Plz measure successor error, effective sample size, and the variance stemming from the preprocessing stag Compare "UOT-only" "direct matched residual," and the full T-RAFT method across 4–6 MuJoCo environments and 1–2 Adroit environments. Experiments must use at least 3 random seeds. Ensure consistency across experimental groups regarding trajectories/weights, reward/done signals, and computational budgets/checkpoints. Report paired effects and Effective Sample Size (ESS).

For W3 Illustrate that when s′s' (the next state) changes, whether the reward values ​​in the original data source remain correct? Please provide the "done" masks and the state/reward inputs; distinguish between the "reward value recorded in the tuple" and the "reward value obtained via the ground-truth environment (oracle)."

Explain all the other points raised in weakness part

2 comments

We thank the reviewer for the thoughtful and detailed evaluation, and we address each weakness and question point-by-point with additional analyses and clarifications below.

Regarding W1 and Q1 . Provenance/equivalence.

For the results presented in Tables 2 and 6, we have directly adopted the findings reported in the MOBODY paper. Consequently, our experimental setup—conducted on the ODRL benchmark—is consistent with theirs. Specifically, we have utilized the same target-domain datasets, as well as the RminR_{min} and RmaxR_{max} values for normalization, as those reported in the ODRL paper. Furthermore, we report the performance based on the best-performing checkpoints, with all experiments conducted using five different random seeds.

We acknowledge that our algorithm does not show a significant performance improvement over ROOT. Based on the additional experiments you requested, we believe this is because our current approach does not account for the reliability of the translated transitions. As a result, many unsupported transitions from the source domain—once translated—deviate even further from the actual target-domain data. We will include this point in the "Limitations" section and plan to address it in our future work.

Regarding W2 and Q2. Component attribution.

We thank the reviewer for pointing out this key missing ablation study. Following your suggestion, we have added ablation studies for the following configurations, including: UOT-only, Direct (ssrc+Δstars_{src}+\Delta s_{tar}) , MLP-Match, FM-only, and T-RAFT. Added UOT-based controls share conditions, masses, recorded rewards, masks, and IQL settings within each preprocessing seed. FM-only jointly changes weighting through uniform sampling and nearest-condition pairs. Results are shown below:

ConfigurationUOT-onlyDirect residualMLP-MatchFM-onlyT-RAFT
HalfCheetah K, m/m40.5±2.139.1±2.239.6±2.637.1±2.139.6±3.8
Hopper K, m/m68.5±2.367.4±2.468.0±2.565.4±2.268.2±0.6
Walker2d M, m/m56.1±5.554.4±6.254.7±5.552.4±5.754.4±3.8
Ant M, m/m44.1±1.443.3±1.544.1±1.541.3±1.344.3±0.6
HalfCheetah G0.5, m/m47.8±1.447.7±1.549.0±1.545.7±1.649.86±0.41
Hopper G0.5, m/expert55.2±6.258.8±6.562.6±6.056.8±6.666.3±9.1
Pen broken-joint M65.5±6.769.0±7.172.8±6.167.0±8.677.00±8.58
Door shrunk-fingers H68.1±5.670.9±6.374.1±5.568.9±7.077.13±5.28

where K/M/G denote kinematic/morphology/gravity, G0.5 denotes half gravity, and m/m or m/expert denotes source/target quality. Adroit M/H denotes severity. Added controls use three preprocessing seeds crossed with three policy seeds. The introduced control mechanism incorporates three types of random seeds for preprocessing and three types for strategy selection. UOT preserves the output results from source tasks while modifying the reuse approach, whereas direct transfer is achieved by matching residuals. MLP/FM facilitates the sharing of supervisory signals through a conditional model. We acknowledge that T-RAFT does not outperform simple weighting of transitions in every scenario; for instance, a method relying solely on UOT performs best on the first three tasks, while a method using only FM yields a lower average performance. This underscores the necessity of a combined pipeline, which does not decouple the weighting effect from the alignment effect. A core limitation of the current algorithm is that it fails to account for the reliability of the translated transitions. Subsequent ablation studies regarding transition consistency confirmed this issue.

Regarding W3. Reward/done consistency.

We thank the reviewer for pointing out this crucial issue. We would like to clarify that, in our experimental setup, we did not explicitly emphasize the transformation of the reward function because our Markov Decision Process (MDP) formulation defines the reward solely as a function of the state-action pair (s,a)(s, a), independent of the subsequent state s′s'. Since the source and target domains share the same underlying reward function, our algorithm learns a unified reward model from a combined dataset; this model is then used to compute rewards for the reconstructed transitions. We will elaborate on this point in the algorithm implementation section.

Regarding W4. Direct measurements.

Thank you for your question. To obtain more direct empirical metrics, we first restore the reserved, complete state of the target simulator and apply the same actions. This allows us to derive independent reference successor states, which are then compared against the successor states generated under different settings. Furthermore, we report the "Shared ESS" (Effective Sample Size) of the source dataset, along with its coverage.

ConfigurationUOT oracle errorDirect oracle errorT-RAFT oracle errorShared ESS/NsrcN_{src}Coverage raw / retained (%)
HalfCheetah K, m/m0.216±0.0160.177±0.0130.160±0.0070.45087.3 / 77.5
Hopper K, m/m0.172±0.0140.145±0.0120.143±0.0100.51890.2 / 82.5
Walker2d M, m/m0.300±0.0020.230±0.0140.191±0.0130.37785.0 / 74.0
Ant M, m/m0.180±0.0100.156±0.0090.178±0.0090.48387.8 / 79.9
HalfCheetah G0.5, m/m0.230±0.0050.188±0.0130.138±0.0130.42389.0 / 79.0
Hopper G0.5, m/expert0.336±0.0090.273±0.0060.191±0.0060.31482.0 / 68.4
Pen broken-joint M0.397±0.0070.332±0.0100.262±0.0010.25878.5 / 64.7
Door shrunk-fingers H0.373±0.0070.281±0.0040.227±0.0110.28780.9 / 66.9

The results above indicate that T-RAFT yields a lower mean error than UOT-only across all eight settings, achieving the lowest mean in seven of them. However, in many scenarios, it fails to fully recover the target-domain transitions, with significant errors still persisting. The fundamental reason is that the learned flow is inherently constrained by the coverage of the target-domain data; consequently, for source-domain data points that lack support in the target domain, the resulting "translations" are naturally unreliable, leading to substantial deviations from the true s′s'.

W5. Preprocessing variance and paired effects. Thank you very much for pointing out this crucial issue. The core rationale behind our design is that the pre-training seed determines the specific batches of source-domain data sampled when solving the optimal transport (OT) problem via the OTT package, which in turn dictates the final optimal transport plan. We maintain consistency across different training seeds because our objective is to keep the weights assigned to each source-domain data point fixed, while isolating the impact of different seeds on the algorithm's performance.

We believe the additional analyses and responses provided here meaningfully strengthen the technical clarity of our work. We thank the reviewer again for their constructive feedback and kind consideration.

Reviewer_GrbjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This submission presents a new data-level reconstruction framework called T-RAFT (Transition Reconstruction via Amortized Flow and Unbalanced Optimal Transport), which is designed to overcome dynamics mismatch in cross-domain offline reinforcement learning (RL). Traditional cross-domain offline RL techniques rely on filtering or reweighting source transitions to match target dynamics. In this submission, the authors theoretically demonstrate that this approach creates an inherent coverage–bias dilemma because filtering treats transitions as indivisible units, failing to fix structural conditional dynamics mismatch.

To overcome this limitation, T-RAFT decouples where experience is reused from what specific outcome is propagated. It utilizes Unbalanced Optimal Transport (UOT) to isolate target-supported source state-action conditions and continuous conditional Flow Matching (FM) to translate original source outcomes into target-compatible next-state dynamics. The authors systematically validate T-RAFT against model-free, model-based, and generative offline RL baselines using MuJoCo locomotion benchmarks and complex Adroit dexterous-manipulation tasks, and demonstrated that the framework achieves the competitive performance while maintaining robustness against severe physical perturbations.

Critical strengths
  1. This submission provides a theoretical insight by proving (Theorem 3.1) that traditional transition filtering or reweighting is mathematically insufficient. It shows that filtering can only shift state-action marginals but leaves conditional source dynamics mismatched, forcing a structural trade-off between dataset coverage and dynamics bias.

  2. By shifting the paradigm from transition selection to transition reconstruction, T-RAFT decouples two distinct degrees of freedom: where to reuse experience (handled globally by UOT) and what outcome to propagate (handled continuously by Flow Matching).

  3. The authors provide their methodology with concrete error bounds (Propositions 4.1 and 4.2), and prove that the empirical joint transition reconstruction error is directly and safely bounded by the combined optimization losses of the UOT transport geometry and the Flow Matching model.

  4. From a practical implementation standpoint, the framework is highly efficient during actual RL optimization. Because the datasets are static, UOT matching and Flow Matching translation are computed once as an offline preprocessing step and cached, meaning the core computational burden does not scale with the downstream policy updates.

Critical weaknesses
  1. As noted in the paper's own limitations, reliable outcome reconstruction relies heavily on having sufficient local target support. If the target dataset is exceptionally scarce or experiences a severe support mismatch, the learned cross-domain correspondences can become highly inaccurate, leading to poor translation.

  2. While the cost does not scale during policy updates, the cumulative time required for the initial pipeline is significantly higher than that of its peers. The complete pipeline takes around 2 hours on an NVIDIA RTX A6000 GPU, which is roughly 21% to 70% slower than competitive baseline methods like OTDF (1h40m) or IGDF (1h11m).

  3. The sensitivity analysis (Figure 3) reveals that T-RAFT's performance can fluctuate noticeably depending on the target marginal relaxation coefficient. For instance, on the Hopper-Kinematic task, choosing the wrong value can degrade final normalized scores. Also, to avoid prohibitively massive cost matrices over standard D4RL benchmarks (page 20), the method must rely on local sparse approximations. This heuristic partitioning might inadvertently slice or break contiguous trajectory structures.

  4. The entire evaluation is bounded by clean, simulated control benchmarks (MuJoCo and Adroit). The paper lacks validation on real-world systems or hardware setups where observation noise, uncalibrated physical sensors, and non-stationary dynamics changes are unpredictable.

Questions for authors

Please check the details in the Critical Weaknesses section and clarify.

2 comments

Thank you for raising the support, cost, and deployment questions.

Regarding W1. Scarce target data and missing local support

We acknowledge that this presents a significant challenge for our approach, as our method relies on each target sample providing both matching conditions and reconstruction targets—requirements that are constrained by a scarcity of local samples. To investigate this, we conducted a sensitivity analysis in the Hopper-Kinematic environment, varying the number of available trajectories (within a "medium-scale" budget). The "UOT-only" baseline adopts the same conditions and weights as T-RAFT, while retaining the successor states from the source domain. We evaluated the algorithm's performance when provided with 1, 2, 3, 5, and 10 trajectories, respectively.

Target trajectoriesTarget-only IQL scoreUOT-only scoreT-RAFT scoreT-RAFT oracle NRMS
130.1±6.346.4±2.843.3±3.50.340±0.012
239.2±5.555.8±3.656.5±2.90.253±0.013
344.6±6.562.2±3.664.2±3.40.196±0.011
550.6±5.968.5±2.368.2±0.60.143±0.010
1059.4±5.369.8±3.571.1±3.70.122±0.012

When only a single trajectory is used, T-RAFT’s "oracle error" reaches its peak, and its performance lags behind that of the UOT-only baseline by 3.1 points. As the number of trajectories increases, more samples become available for matching and providing supervisory signals for reconstruction, which in turn improves both accuracy and cumulative rewards. This aligns with our expectations.

W2. End-to-end computational cost

We thank the reviewer for pointing out this crucial issue. The first-run cost is 121 minutes, versus 100 for OTDF, 71 for IGDF, and 85 for DROCO. Profiling includes I/O/cache writing on one RTX A6000, an i9-13900K, and 32 GB RAM.

StageMinutesPeak GPU memory (GiB)
Normalization, neighbor search, UOT8.2±0.47.6±0.2
FM training31.2±1.110.4±0.3
RK4 source conversion and cache writing10.5±0.512.6±0.4
IQL training and evaluation71.1±1.54.7±0.2
Total121 (=2h1m)—

The extra work occurs before IQL. FM requires 10510^5 updates, and 50-step RK4 converts the source transitions. These stages dominate the 49.9-minute preprocessing cost. Caching amortizes this work across policy seeds. Every new preprocessing seed requires it again, so the full first-run cost remains 121 minutes.

W3. Relaxation sensitivity and sparse approximation

We thank the reviewer for pointing out this crucial issue. We extended Figure 3's Hopper-Kinematic sensitivity test with oracle errors and ESS, fixing λsrc=0.05\lambda_{src}=0.05.

λtar\lambda_{tar}T-RAFT scoreOracle NRMSESS/NsrcN_{src}
0.0155.3±5.00.196±0.0150.231±0.014
0.02563.6±4.10.163±0.0120.299±0.017
0.0568.2±0.60.143±0.0100.518
0.161.7±5.40.173±0.0160.429±0.018

The penalty trades matching cost against preserving target mass. A weak penalty allows selective matches, while a stronger penalty gives more weight to mass preservation. The sweep's best tested balance is 0.05, with the highest ESS/return and lowest error. Performance worsens at 0.1. This establishes sensitivity to the coupling penalty, while the precise correspondence changes responsible for that deterioration are not isolated.

W4. Simulation scope, sensors, and changing dynamics

Thank you for your question. Now we have only considered the algorithm's performance in a simulated environment; in future work, we will examine its performance in more complex, real-world scenarios.

We sincerely hope this comprehensive response clarifies T-RAFT's unique value. Given the novel evidence and analysis, we kindly request reconsidering the score to reflect these contributions. Thank you!

Thanks the authors for responding to the points raised in my review. I assessed the submission, together with the submission and the rebuttal, and believe 3 is the apprioriate rating of the submission. So I maintain the score.

Reviewer_zuqSOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes T-RAFT, which reconstructs rather than selects source transitions for cross-domain offline RL with abundant source and scarce target data. The authors argue that filtering-based methods, which keep or discard whole transitions, face a trade-off between source coverage and dynamics bias, and theoretically show that reweighting alone cannot remove the dynamics mismatch. In its condition alignment stage, T-RAFT uses unbalanced optimal transport (UOT) over (s, a) to select target-close source data and obtain source–target correspondences. In its outcome translation stage, a conditional flow matching model trained on these correspondences maps source next states to the target dynamics, and the reconstructed transitions are used with the target data for IQL training. Experiments on MuJoCo locomotion under three types of dynamics shift and on Adroit manipulation tasks report competitive performance against eight baselines.

Critical strengths

S1. Theoretical Formalization of the Limitation of Reweighting

Based on the observation that reweighting only changes which source data are used and cannot alter the source dynamics themselves, the paper shows that reweighting alone, including UOT-based weighting, cannot remove the bias caused by the dynamics mismatch (Theorem 3.1). Proposition 4.1 further decomposes the error into a condition term and an outcome term, providing the motivation for adding an outcome-correction step.  

S2. Practical Design Applicable Without Task-Specific Tuning

T-RAFT is evaluated on 108 MuJoCo configurations, multiple gravity scales, and 8 Adroit settings, all with a single hyperparameter configuration. In addition, since the UOT and flow-based translation are computed once before training and then reused, the method can be applied without task-specific tuning or substantial additional cost, which makes it practical.

Critical weaknesses

W1. Limited Distinction from Existing UOT-Based and Flow-Based Methods

Although the paper presents the shift from transition selection to transition reconstruction as its key contribution, the condition alignment stage (Sec. 4.2) appears nearly identical to ROOT. The UOT objective in Eq. (10), the source weights derived from the transported mass in Eq. (11), and the block-wise Sinkhorn implementation (Appendix E.1) closely follow ROOT, with the main difference being that the cost is computed only on (s, a). The outcome translation stage (Sec. 4.3) is also structurally similar to COMPFLOW, which uses an (s, a)-conditioned flow to map source-dynamics outcomes to the target dynamics, making it difficult to determine what the contribution is beyond combining existing components.  

W2. Lack of Verification of the Effect of Outcome Translation

Although the paper claims that the performance gains stem from outcome translation, when averaged over the 36 configurations of each shift type in Tables 1, 4, and 5, T-RAFT performs on par with or even below ROOT, which uses UOT reweighting without translation. Nevertheless, the paper does not include an ablation against UOT reweighting alone, nor a direct evaluation of whether the translated transitions match the actual target dynamics. This makes it difficult to determine whether each component works as intended and whether the performance improvement originates from the translation step.

Questions for authors

Q1. To my knowledge, ROOT also conducted experiments under the same HalfCheetah gravity-shift setting as Table 2. Could the authors include a comparison with ROOT and explain how the performance difference between the two methods changes as the dynamics gap grows with gravity deviating further from its default value, and, in light of this trend, why outcome translation is effective?

Q2. Fig. 2 currently compares the translated outcomes only with the condition-nearest target samples, which are also used to train the flow, making it difficult to verify whether the translated next states follow the actual target dynamics. Could the authors compare the translated next states with the actual outcomes obtained by executing the same (s, a) in the target simulator?

Q3. In the reconstructed transitions, only s′ appears to be translated, while the reward and terminal flag are taken directly from the source. Since rewards and termination conditions depend on the transition outcome in some tasks, it is unclear whether the source reward and terminal flag can be used as they are. Could the authors explain how this is handled and how it affects the results?

3 comments

We sincerely thank you for the time and effort devoted to reviewing our work. In response to your concerns, we provide the explanations and answers below.

Regrading W1. Distinction from ROOT and COMPFLOW.

We agree that both unbalanced optimal transport (UOT) and Flow Matching (FM) are established techniques. The central contribution of T-RAFT lies in reformulating cross-domain source-data reuse as a condition-preserving transition reconstruction problem, where selecting reusable state-action conditions and correcting their associated outcomes are treated as two distinct degrees of freedom.

(1) A theoretical motivation beyond reweighting. Theorem 3.1 establishes that condition-level reweighting, even with perfect state-action marginal alignment, cannot eliminate the residual discrepancy between source and target conditional dynamics. Proposition 4.1 separates joint transition discrepancy into condition-alignment and outcome-correction terms under its Lipschitz assumption. These results motivate outcome correction; they do not guarantee improved policy returns or extrapolation beyond target support.

(2) A unified use of transport information. ROOT uses UOT for source reuse. T-RAFT exploits two complementary forms of information from the same coupling: the transported source marginal determines where experience is reused, while source-target condition correspondences supervise what outcome should be propagated. The original source state-action condition is preserved during reconstruction. Thus, the two stages share an alignment structure rather than being independently applied.

(3) A different adaptation setting and reconstruction objective from COMPFLOW. As discussed in related work, COMPFLOW studies dynamics adaptation in a hybrid offline-to-online setting. T-RAFT addresses offline-to-offline learning: both datasets are static and no additional target interaction is available for training. We reconstruct target-compatible outcomes for retained source conditions and cache these transitions for IQL.

Regrading W2. Attribution to outcome translation.

We thank the reviewer for pointing out this key missing ablation study. Following your suggestion, we have added ablation studies for the following configurations, including: UOT-only, Direct (ssrc+Δstars_{src}+\Delta s_{tar}) , MLP-Match, FM-only, and T-RAFT. Added UOT-based controls share conditions, masses, recorded rewards, masks, and IQL settings within each preprocessing seed. FM-only jointly changes weighting through uniform sampling and nearest-condition pairs. Results are shown below:

ConfigurationUOT-onlyDirect residualMLP-MatchFM-onlyT-RAFT (paper)
HalfCheetah K, m/m40.5±2.139.1±2.239.6±2.637.1±2.139.6±3.8
Hopper K, m/m68.5±2.367.4±2.468.0±2.565.4±2.268.2±0.6
Walker2d M, m/m56.1±5.554.4±6.254.7±5.552.4±5.754.4±3.8
Ant M, m/m44.1±1.443.3±1.544.1±1.541.3±1.344.3±0.6
HalfCheetah G0.5, m/m47.8±1.447.7±1.549.0±1.545.7±1.649.86±0.41
Hopper G0.5, m/expert55.2±6.258.8±6.562.6±6.056.8±6.666.3±9.1
Pen broken-joint M65.5±6.769.0±7.172.8±6.167.0±8.677.00±8.58
Door shrunk-fingers H68.1±5.670.9±6.374.1±5.568.9±7.077.13±5.28

We acknowledge that T-RAFT does not outperform simple weighting of transitions in every scenario; for instance, a method relying solely on UOT performs best on the first three tasks, while a method using only FM yields a lower average performance. This underscores the necessity of a combined pipeline, which does not decouple the weighting effect from the alignment effect. A core limitation of the current algorithm is that it fails to account for the reliability of the translated transitions. Subsequent ablation studies regarding transition consistency confirmed this issue.

Regarding Q1. ROOT across gravity shifts.

We thank the reviewer for pointing out this crucial issue. We compare the performance of T-RAFT with ROOT on halfcheetah environment with diverse gravity shift, where scores of ROOT come from Table 2 of the original paper, using the same MOBODY/ODRL HalfCheetah gravity protocol and baseline numbers as our Table 2. T-RAFT reproduces our Table 2.

Target gravityROOTT-RAFT (Table 2)T-RAFT oracle error
0.1×0.1\times16.33±1.1427.56±3.170.483±0.039
0.5×0.5\times39.53±11.3649.86±0.410.138±0.013
2.0×2.0\times43.57±12.7334.70±0.890.465±0.019
5.0×5.0\times74.17±0.3573.53±0.440.284±0.035

T-RAFT's mean exceeds ROOT by 11.23 and 10.33 points at 0.1×0.1\times and 0.5×0.5\times; ROOT leads by 8.87 and 0.64 at 2.0×2.0\times and 5.0×5.0\times. T-RAFT's SD is lower at 0.5×0.5\times and 2.0×2.0\times, but higher at 0.1×0.1\times and 5.0×5.0\times. Thus translation has no uniform advantage as gravity increases. A core reason for this is that as the "gravity shift" increases, the overlap between the two domains diminishes, which in turn reduces the overlapping region for (s,a)(s, a) pairs. This renders the translation target unreliable, causing the flow-based approach to perform worse than simple filtering.

Regarding Q2. Independent target-dynamics validation.

Thank you for your question. To obtain more direct empirical metrics, we first restore the reserved, complete state of the target simulator and apply the same actions. This allows us to derive independent reference successor states, which are then compared against the successor states generated under different settings. Furthermore, we report the "Shared ESS" (Effective Sample Size) of the source dataset, along with its coverage.

ConfigurationUOT oracle errorDirect oracle errorT-RAFT oracle errorShared ESS/NsrcN_{src}Coverage raw / retained (%)
HalfCheetah K, m/m0.216±0.0160.177±0.0130.160±0.0070.45087.3 / 77.5
Hopper K, m/m0.172±0.0140.145±0.0120.143±0.0100.51890.2 / 82.5
Walker2d M, m/m0.300±0.0020.230±0.0140.191±0.0130.37785.0 / 74.0
Ant M, m/m0.180±0.0100.156±0.0090.178±0.0090.48387.8 / 79.9
HalfCheetah G0.5, m/m0.230±0.0050.188±0.0130.138±0.0130.42389.0 / 79.0
Hopper G0.5, m/expert0.336±0.0090.273±0.0060.191±0.0060.31482.0 / 68.4
Pen broken-joint M0.397±0.0070.332±0.0100.262±0.0010.25878.5 / 64.7
Door shrunk-fingers H0.373±0.0070.281±0.0040.227±0.0110.28780.9 / 66.9

The results above indicate that T-RAFT yields a lower mean error than UOT-only across all eight settings, achieving the lowest mean in seven of them. However, in many scenarios, it fails to fully recover the target-domain transitions, with significant errors still persisting. The fundamental reason is that the learned flow is inherently constrained by the coverage of the target-domain data; consequently, for source-domain data points that lack support in the target domain, the resulting "translations" are naturally unreliable, leading to substantial deviations from the true s′s'.

Regarding Q3. Reward and terminal consistency.

We thank the reviewer for pointing out this crucial issue. We would like to clarify that, in our experimental setup, we did not explicitly emphasize the transformation of the reward function because our Markov Decision Process (MDP) formulation defines the reward solely as a function of the state-action pair (s,a)(s, a), independent of the subsequent state s′s'. Since the source and target domains share the same underlying reward function, our algorithm learns a unified reward model from a combined dataset; this model is then used to compute rewards for the reconstructed transitions. We will elaborate on this point in the algorithm implementation section.

We believe the additional analyses and responses provided here meaningfully strengthen the technical clarity of our work. We thank the reviewer again for their constructive feedback and kind consideration.

Thank you for the detailed response and for running the additional experiments. I still have some reservations. The proposed method largely builds on ROOT and COMPFLOW, and its main difference from COMPFLOW appears to be the offline-to-offline setting. Its methodological contribution therefore seems limited. In the new ablation, UOT-only matched or outperformed T-RAFT under the kinematic and morphology shifts, so the results do not clearly show a consistent benefit from outcome translation. The explanation for the gravity results also leaves me unsure how well the method generalizes when the domain gap is large enough to make translation unreliable. I will keep my current score.

Comments on the paper

1 comment
Area_Chair_CMtkOpenReview ↗

Reviewer-Author Discussion Phase is Ending

Dear Reviewers,

Thank you for your service in the review process. Now that we are approaching the deadline of the author-reviewer discussion period. Please take time to read authors' responses and let them know if it addresses your concerns. Thank you.

Best,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.