- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies cross-domain offline reinforcement learning with abundant source data and limited target data under dynamics shift. The proposed RIFT method transfers a reward-improvement structure rather than directly reusing source transitions, by learning low-to-high reward displacements in a shared latent space and applying them to target-domain anchors. Concretely, RIFT combines a reward-regularized VAE, optimal-transport matching between low- and high-reward source transitions, and an affine displacement field to synthesize additional target-domain data. Experiments on locomotion tasks with gravity, morphology, and friction shifts show improved aggregate performance over existing cross-domain offline RL baselines.
Critical strengths
- The paper introduces a novel transfer perspective: transferring reward-improvement structure instead of source transitions themselves.
- The target-anchored augmentation strategy is conceptually appealing for mitigating direct source--target dynamics mismatch.
- The paper provides a theoretical analysis of synthesis error and conditions under which target anchoring can outperform direct source-high transplant.
- The method achieves strong aggregate performance across multiple dynamics shifts, with useful ablations on augmentation and representation design.
Critical weaknesses
- The central notion of reward improvement is insufficiently justified: the paper directly operationalizes it as an OT-induced displacement between bottom- and top-reward transition distributions in Section 4.1, without explaining why this is the appropriate or transferable notion of behavioral improvement.
- The method relies critically on shared reward-improvement geometry across domains, but the reward-conditioned latent regularization only encourages such alignment and does not establish that the learned improvement field is actually invariant across domains.
- The theoretical results are largely conditional on a fixed representation, coupling, and improvement field, leaving a gap between the guarantees and the end-to-end learned algorithm.
- The empirical evaluation demonstrates strong downstream performance, but provides limited direct evidence that the gains arise from transferring the proposed improvement geometry rather than from the particular representation and augmentation pipeline.
- It also lacks citations for key papers and fails to discuss or compare recent work; for instance, in Section 5.1, you introduce the experimental setup involving ODRL but do not cite the corresponding paper. Furthermore, you do not discuss or compare recent cross-domain ORL studies, such as MOBODY[1], DROCO[2], REAG[3], ROOT[4], and V2A[5].
- It lacks experiments in Adroit or Antmaze environments to further validate the algorithm's effectiveness in more complex scenarios.
references:
[1] MOBODY- Model-based Off-Dynamics Offline Reinforcement Learning, ICLR, 2026
[2] Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shifts, ICLR, 2026
[3] Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learning, TMLR, 2026
[4] Transport or Discard: Robust Unbalanced Optimal Transport for Cross-Domain Policy Adaptation, ICML, 2026
[5] Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasets, ICML, 2026
Questions for authors
- Why is transport from the bottom to the top immediate-reward quantiles an appropriate definition of behavioral improvement, and under what task or reward structures should this notion be expected to transfer across domains?
- Can the authors provide a more direct empirical measurement of source--target improvement-field compatibility, beyond PCA visualization and downstream return?
- How much of RIFT's performance depends on the shared-geometry assumption being approximately satisfied, and can the method detect when this assumption fails?
- How well does RIFT extend beyond dense-reward locomotion tasks, e.g., to sparse or delayed rewards or larger dynamics shifts where immediate reward may not reflect long-horizon behavioral quality?
We thank the reviewer for the thoughtful feedback. We address the concerns and questions below.
W1 / Q1: Why use low-to-high reward transport?
Our goal is to exploit abundant source data without directly importing transitions that may violate target dynamics. RIFT instead learns how low- and high-reward source transitions differ and applies this relationship to target anchors. OT provides a principled construction of this relationship: among couplings with the prescribed low/high marginals, it minimizes transport cost under the learned latent metric. Transferability is not implied by OT optimality: Theorem 1 bounds synthesis error under shared class geometry and explicitly accounts for incompatibility of the source improvement field with the target context.
Immediate reward provides a local signal for the transition being augmented, while downstream IQL estimates long-horizon values through Bellman backups. The augmentation signal therefore works alongside the long-horizon policy objective by guiding new data generation.
We compare immediate-reward conditioning with 5-, 10-, and 20-step-return conditioning, keeping the VAE fixed and retaining immediate rewards in the generated transitions:
The aggregate scores show no consistent improvement as the conditioning horizon increases. These results support immediate reward as an effective, simple choice in the evaluated tasks and show that RIFT can also use longer-horizon conditioning within the same representation. We will clarify that “reward improvement” denotes the low-to-high distributional augmentation target, not a guaranteed policy-improvement operator.
W2, W4 / Q2–Q3: Does the source-derived field contribute beyond the representation and augmentation pipeline?
We examine this contribution by holding the VAE and normalization fixed and adding two controls:
RIFT exceeds the target-only field by 27.5 points and high-reward resampling by 66.8 points. In particular, replacing the source-derived field with a target-only field reduces performance while retaining the shared representation. These results support the contribution of learning the improvement relationship from abundant source data, beyond representation alone or simply emphasizing high-reward target examples.
Further, we note that the reward-conditioned regularizer encourages shared structure; it does not guarantee field invariance. Theorem 1 explicitly accounts for residual class misalignment and field incompatibility.
W3: What does the theory establish?
Our theory motivates target anchoring by showing when it can yield less synthesis error than directly transferring high-reward source transitions. Proposition A.1 also bounds the error of the field learned from minibatch OT pairs, which feeds into Theorem 1 alongside field mismatch and other approximation errors.
We use these results to motivate our approach, but recognize that they do not prove the full learned pipeline will satisfy the assumptions. The representation and population coupling remain fixed in the analysis; we will make this scope clearer.
W5: Related work and benchmark attribution.
We thank the reviewer for bringing these papers to our attention. We will discuss MOBODY, DROCO, REAG, ROOT, and V2A in the revised related work and clarify how RIFT relates to these approaches. We will also explicitly cite ODRL where we introduce the experimental setup.
W6 / Q4: Broader environments and reward structures.
Our experiments cover gravity, morphology, and friction shifts in dense-reward locomotion tasks. We will state the limitations more clearly: sparse or delayed rewards may give weak low/high class labels, and larger dynamics shifts may make the source field unsuitable for the target. The n-step results show that RIFT can use other conditioning signals, but testing these settings, including AntMaze and Adroit, remains future work.
We hope this addresses your concerns, and if so, would appreciate it if you could reconsider your score!