- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies Offline RL when the target task differs from a source dataset in state-transition dynamics while sharing the same state/action spaces and reward functions. It proposes a method that can make offline RL reuse the sorue dataset at large scale This method employs "Unbalanced Optimal Transport" to weight state-action conditions from a source dataset and align them with the conditions present in the target task. Conditional flow model is employed to adjust next state within source data. Through those, offline RL can be trained in a combiantion of target data and re-constructed data from source to improve data utilization.
Critical strengths
S1. Design is clear. Reusable places and re-designed places are clear. No errors were found in the core proof steps, given the stated assumptions.
S2. Exps look comprehensive. The method achieves state-of-the-art performance on 6 out of 8 Adroit tasks. This demonstrates its practical value (assuming the comparisons are fair).
Critical weaknesses
W1. The baseline setup lacks full traceability. Data in Tables 2 and 6 are carried over directly from previous results. The sources for data in other columns and whether these experiments were re-run, remain unclear to me. It is important to provide explicit details regarding to the data, normalization, checkpoints, and random seeds used. Across the 3 test sets, the average task-level performance difference vs. ROOT is only +0.3/−0.5/−1.5: why is it sufficient to establish equivalence?
- Plz include IQL baseline results generated using your own pipeline (ideally also including OTDF) to better assess the magnitude of these differences and to examine when proposed method has superior performance on Adroit tasks.
W2 The gains claimed for the flow models is a combination of multiple things not just the flow model itself. UOT (Unbalanced Optimal Transport) reweighting is emploed in UOT. CompFlow/UOT-FM and prior editing-based methods already covered the concept of "reconstruction". Ablation studies that can measure or quantify the separated impact of individual techniques (e.g., the use of UOT, emplying simpler translators, etc) are missing but important to appreciate the value of the design.
- Plz conduct a comparison across: -- original successor samples are sampled by the UOT mass, -- residuals obtained by directly matching the target -- the full T-RAFT method.
W3 I am thinking, for an RL changing the next state would also cause distribution changing of reward functiosn for consistency The original reward may no longer be valid following a state transition, .
For ex, if the reward is defined as "forward velocity minus action energy consumption" modifying could alter the forward velocity, rendering the original reward value inapplicable to the new sample. Consequently, it is necessary to compare 3 things: using (based only on states), using (incorporating the action), and calculating the "ground truth" directly from formula.
W4. Direct empirical measurement is missing for coverage and translation. The metric "local-neighbor improvement" serves merely as a proxy.
W5: RL experiments share a single random seed for the preprocessing stage? The variance is then not accounted for in the reported experimental uncertainty. Plz measure successor error, effective sample size, and the variance stemming from the preprocessing stag
Questions for authors
For W1:
- Plz include IQL baseline results generated using your own pipeline (ideally also including OTDF) to better assess the magnitude of these differences and to examine when proposed method has superior performance on Adroit tasks.
For W2
- Plz conduct a comparison across: -- original successor samples are sampled by the UOT mass, -- residuals obtained by directly matching the target -- the full T-RAFT method.
For W5 Plz measure successor error, effective sample size, and the variance stemming from the preprocessing stag Compare "UOT-only" "direct matched residual," and the full T-RAFT method across 4–6 MuJoCo environments and 1–2 Adroit environments. Experiments must use at least 3 random seeds. Ensure consistency across experimental groups regarding trajectories/weights, reward/done signals, and computational budgets/checkpoints. Report paired effects and Effective Sample Size (ESS).
For W3 Illustrate that when (the next state) changes, whether the reward values in the original data source remain correct? Please provide the "done" masks and the state/reward inputs; distinguish between the "reward value recorded in the tuple" and the "reward value obtained via the ground-truth environment (oracle)."
Explain all the other points raised in weakness part
2 comments
Regarding W4. Direct measurements.
Thank you for your question. To obtain more direct empirical metrics, we first restore the reserved, complete state of the target simulator and apply the same actions. This allows us to derive independent reference successor states, which are then compared against the successor states generated under different settings. Furthermore, we report the "Shared ESS" (Effective Sample Size) of the source dataset, along with its coverage.
| Configuration | UOT oracle error | Direct oracle error | T-RAFT oracle error | Shared ESS/ | Coverage raw / retained (%) |
|---|---|---|---|---|---|
| HalfCheetah K, m/m | 0.216±0.016 | 0.177±0.013 | 0.160±0.007 | 0.450 | 87.3 / 77.5 |
| Hopper K, m/m | 0.172±0.014 | 0.145±0.012 | 0.143±0.010 | 0.518 | 90.2 / 82.5 |
| Walker2d M, m/m | 0.300±0.002 | 0.230±0.014 | 0.191±0.013 | 0.377 | 85.0 / 74.0 |
| Ant M, m/m | 0.180±0.010 | 0.156±0.009 | 0.178±0.009 | 0.483 | 87.8 / 79.9 |
| HalfCheetah G0.5, m/m | 0.230±0.005 | 0.188±0.013 | 0.138±0.013 | 0.423 | 89.0 / 79.0 |
| Hopper G0.5, m/expert | 0.336±0.009 | 0.273±0.006 | 0.191±0.006 | 0.314 | 82.0 / 68.4 |
| Pen broken-joint M | 0.397±0.007 | 0.332±0.010 | 0.262±0.001 | 0.258 | 78.5 / 64.7 |
| Door shrunk-fingers H | 0.373±0.007 | 0.281±0.004 | 0.227±0.011 | 0.287 | 80.9 / 66.9 |
The results above indicate that T-RAFT yields a lower mean error than UOT-only across all eight settings, achieving the lowest mean in seven of them. However, in many scenarios, it fails to fully recover the target-domain transitions, with significant errors still persisting. The fundamental reason is that the learned flow is inherently constrained by the coverage of the target-domain data; consequently, for source-domain data points that lack support in the target domain, the resulting "translations" are naturally unreliable, leading to substantial deviations from the true .
W5. Preprocessing variance and paired effects. Thank you very much for pointing out this crucial issue. The core rationale behind our design is that the pre-training seed determines the specific batches of source-domain data sampled when solving the optimal transport (OT) problem via the OTT package, which in turn dictates the final optimal transport plan. We maintain consistency across different training seeds because our objective is to keep the weights assigned to each source-domain data point fixed, while isolating the impact of different seeds on the algorithm's performance.
We believe the additional analyses and responses provided here meaningfully strengthen the technical clarity of our work. We thank the reviewer again for their constructive feedback and kind consideration.
We thank the reviewer for the thoughtful and detailed evaluation, and we address each weakness and question point-by-point with additional analyses and clarifications below.
Regarding W1 and Q1 . Provenance/equivalence.
For the results presented in Tables 2 and 6, we have directly adopted the findings reported in the MOBODY paper. Consequently, our experimental setup—conducted on the ODRL benchmark—is consistent with theirs. Specifically, we have utilized the same target-domain datasets, as well as the Rmin and Rmax values for normalization, as those reported in the ODRL paper. Furthermore, we report the performance based on the best-performing checkpoints, with all experiments conducted using five different random seeds.
We acknowledge that our algorithm does not show a significant performance improvement over ROOT. Based on the additional experiments you requested, we believe this is because our current approach does not account for the reliability of the translated transitions. As a result, many unsupported transitions from the source domain—once translated—deviate even further from the actual target-domain data. We will include this point in the "Limitations" section and plan to address it in our future work.
Regarding W2 and Q2. Component attribution.
We thank the reviewer for pointing out this key missing ablation study. Following your suggestion, we have added ablation studies for the following configurations, including: UOT-only, Direct (ssrc+Δstar) , MLP-Match, FM-only, and T-RAFT. Added UOT-based controls share conditions, masses, recorded rewards, masks, and IQL settings within each preprocessing seed. FM-only jointly changes weighting through uniform sampling and nearest-condition pairs. Results are shown below:
where K/M/G denote kinematic/morphology/gravity, G0.5 denotes half gravity, and m/m or m/expert denotes source/target quality. Adroit M/H denotes severity. Added controls use three preprocessing seeds crossed with three policy seeds. The introduced control mechanism incorporates three types of random seeds for preprocessing and three types for strategy selection. UOT preserves the output results from source tasks while modifying the reuse approach, whereas direct transfer is achieved by matching residuals. MLP/FM facilitates the sharing of supervisory signals through a conditional model. We acknowledge that T-RAFT does not outperform simple weighting of transitions in every scenario; for instance, a method relying solely on UOT performs best on the first three tasks, while a method using only FM yields a lower average performance. This underscores the necessity of a combined pipeline, which does not decouple the weighting effect from the alignment effect. A core limitation of the current algorithm is that it fails to account for the reliability of the translated transitions. Subsequent ablation studies regarding transition consistency confirmed this issue.
Regarding W3. Reward/done consistency.
We thank the reviewer for pointing out this crucial issue. We would like to clarify that, in our experimental setup, we did not explicitly emphasize the transformation of the reward function because our Markov Decision Process (MDP) formulation defines the reward solely as a function of the state-action pair (s,a), independent of the subsequent state s′. Since the source and target domains share the same underlying reward function, our algorithm learns a unified reward model from a combined dataset; this model is then used to compute rewards for the reconstructed transitions. We will elaborate on this point in the algorithm implementation section.