acceptodds
Withdrawn from ICLR 2027

RIFT: Reward Improvement Field Transfer for Cross-Domain Offline RL

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Learning from limited target data is a central challenge in cross-domain offline reinforcement learning. Although source data can provide additional experience, differences in dynamics can make these transitions unsuitable for the target domain. We propose Reward Improvement Field Transfer (RIFT), a method that instead uses source data to learn how behavior changes from low reward to high reward. RIFT encodes source and target transitions in a shared latent space, pairs low- and high-reward source transitions using optimal transport, and fits an affine reward-improvement field to their displacements. We apply this transformation to target transitions and decode the results to augment the target dataset. Our key assumption is that related domains may differ in their dynamics while still sharing a similar reward-improvement structure. We bound synthesis error under shared improvement geometry, identify conditions favoring target anchoring over source-data transplant, and derive a conditional connection to downstream return. Experiments on locomotion tasks with changes in gravity, morphology, and friction show that RIFT improves aggregate performance over the existing baselines.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, cross-domain transfer, synthetic data

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eTrLOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies cross-domain offline reinforcement learning with abundant source data and limited target data under dynamics shift. The proposed RIFT method transfers a reward-improvement structure rather than directly reusing source transitions, by learning low-to-high reward displacements in a shared latent space and applying them to target-domain anchors. Concretely, RIFT combines a reward-regularized VAE, optimal-transport matching between low- and high-reward source transitions, and an affine displacement field to synthesize additional target-domain data. Experiments on locomotion tasks with gravity, morphology, and friction shifts show improved aggregate performance over existing cross-domain offline RL baselines.

Critical strengths
  1. The paper introduces a novel transfer perspective: transferring reward-improvement structure instead of source transitions themselves.
  2. The target-anchored augmentation strategy is conceptually appealing for mitigating direct source--target dynamics mismatch.
  3. The paper provides a theoretical analysis of synthesis error and conditions under which target anchoring can outperform direct source-high transplant.
  4. The method achieves strong aggregate performance across multiple dynamics shifts, with useful ablations on augmentation and representation design.
Critical weaknesses
  1. The central notion of reward improvement is insufficiently justified: the paper directly operationalizes it as an OT-induced displacement between bottom- and top-reward transition distributions in Section 4.1, without explaining why this is the appropriate or transferable notion of behavioral improvement.
  2. The method relies critically on shared reward-improvement geometry across domains, but the reward-conditioned latent regularization only encourages such alignment and does not establish that the learned improvement field is actually invariant across domains.
  3. The theoretical results are largely conditional on a fixed representation, coupling, and improvement field, leaving a gap between the guarantees and the end-to-end learned algorithm.
  4. The empirical evaluation demonstrates strong downstream performance, but provides limited direct evidence that the gains arise from transferring the proposed improvement geometry rather than from the particular representation and augmentation pipeline.
  5. It also lacks citations for key papers and fails to discuss or compare recent work; for instance, in Section 5.1, you introduce the experimental setup involving ODRL but do not cite the corresponding paper. Furthermore, you do not discuss or compare recent cross-domain ORL studies, such as MOBODY[1], DROCO[2], REAG[3], ROOT[4], and V2A[5].
  6. It lacks experiments in Adroit or Antmaze environments to further validate the algorithm's effectiveness in more complex scenarios.

references:

[1] MOBODY- Model-based Off-Dynamics Offline Reinforcement Learning, ICLR, 2026

[2] Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shifts, ICLR, 2026

[3] Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learning, TMLR, 2026

[4] Transport or Discard: Robust Unbalanced Optimal Transport for Cross-Domain Policy Adaptation, ICML, 2026

[5] Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasets, ICML, 2026

Questions for authors
  1. Why is transport from the bottom to the top immediate-reward quantiles an appropriate definition of behavioral improvement, and under what task or reward structures should this notion be expected to transfer across domains?
  2. Can the authors provide a more direct empirical measurement of source--target improvement-field compatibility, beyond PCA visualization and downstream return?
  3. How much of RIFT's performance depends on the shared-geometry assumption being approximately satisfied, and can the method detect when this assumption fails?
  4. How well does RIFT extend beyond dense-reward locomotion tasks, e.g., to sparse or delayed rewards or larger dynamics shifts where immediate reward may not reflect long-horizon behavioral quality?
1 comment

We thank the reviewer for the thoughtful feedback. We address the concerns and questions below.

W1 / Q1: Why use low-to-high reward transport?

Our goal is to exploit abundant source data without directly importing transitions that may violate target dynamics. RIFT instead learns how low- and high-reward source transitions differ and applies this relationship to target anchors. OT provides a principled construction of this relationship: among couplings with the prescribed low/high marginals, it minimizes transport cost under the learned latent metric. Transferability is not implied by OT optimality: Theorem 1 bounds synthesis error under shared class geometry and explicitly accounts for incompatibility of the source improvement field with the target context.

Immediate reward provides a local signal for the transition being augmented, while downstream IQL estimates long-horizon values through Bellman backups. The augmentation signal therefore works alongside the long-horizon policy objective by guiding new data generation.

We compare immediate-reward conditioning with 5-, 10-, and 20-step-return conditioning, keeping the VAE fixed and retaining immediate rewards in the generated transitions:

Conditioning horizon1 step5 steps10 steps20 steps
Aggregate score451.3452.0438.1451.0

The aggregate scores show no consistent improvement as the conditioning horizon increases. These results support immediate reward as an effective, simple choice in the evaluated tasks and show that RIFT can also use longer-horizon conditioning within the same representation. We will clarify that “reward improvement” denotes the low-to-high distributional augmentation target, not a guaranteed policy-improvement operator.

W2, W4 / Q2–Q3: Does the source-derived field contribute beyond the representation and augmentation pipeline?

We examine this contribution by holding the VAE and normalization fixed and adding two controls:

  • Target-only field: Learn the field from target transitions and apply it to the same target anchors.
  • High-reward resampling: Repeat real high-reward target transitions instead of generating transitions.
MethodAggregate score
IQL*359.5
High-reward target resampling384.5
Target-only field423.8
RIFT451.3

RIFT exceeds the target-only field by 27.5 points and high-reward resampling by 66.8 points. In particular, replacing the source-derived field with a target-only field reduces performance while retaining the shared representation. These results support the contribution of learning the improvement relationship from abundant source data, beyond representation alone or simply emphasizing high-reward target examples.

Further, we note that the reward-conditioned regularizer encourages shared structure; it does not guarantee field invariance. Theorem 1 explicitly accounts for residual class misalignment and field incompatibility.

W3: What does the theory establish?

Our theory motivates target anchoring by showing when it can yield less synthesis error than directly transferring high-reward source transitions. Proposition A.1 also bounds the error of the field learned from minibatch OT pairs, which feeds into Theorem 1 alongside field mismatch and other approximation errors.

We use these results to motivate our approach, but recognize that they do not prove the full learned pipeline will satisfy the assumptions. The representation and population coupling remain fixed in the analysis; we will make this scope clearer.

W5: Related work and benchmark attribution.

We thank the reviewer for bringing these papers to our attention. We will discuss MOBODY, DROCO, REAG, ROOT, and V2A in the revised related work and clarify how RIFT relates to these approaches. We will also explicitly cite ODRL where we introduce the experimental setup.

W6 / Q4: Broader environments and reward structures.

Our experiments cover gravity, morphology, and friction shifts in dense-reward locomotion tasks. We will state the limitations more clearly: sparse or delayed rewards may give weak low/high class labels, and larger dynamics shifts may make the source field unsuitable for the target. The n-step results show that RIFT can use other conditioning signals, but testing these settings, including AntMaze and Adroit, remains future work.

We hope this addresses your concerns, and if so, would appreciate it if you could reconsider your score!

Reviewer_4tWvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a cross-environemnet data-augmentation method for offline RL with IQL empowered to overcome the data sparsity problem in target environment. Specifically, it tackles three challenges:

  1. Scarcity of target data: It is not feasible to collect more data through further interaction with the target environment. For example, in the paper’s experiments, there are approximately 1 million source-environment samples, compared to only 5,000 target-environment samples.

  2. Different environmental dynamics: Differences in gravity, friction, or the robot’s physical embodiment mean that the same state-action pair can yield different outcomes. Simply pooling the data could lead the policy to learn incorrect dynamics for the target environment.

  3. Different environmental dynamics: Differences in factors such as gravity, friction, or the robot’s physical embodiment mean that the same state-action pair can yield different outcomes. Simply pooling the data could lead the policy to learn incorrect dynamics for the target environment.

The core idea is to avoid directly transferring samples from the source environment. Instead, it learns the underlying pattern of "how to transition from lower-reward to higher-reward behavior" within the source environment, and then applies this transformation to the target-environment samples to generate additional training data.

The core assumtpion also the restrcition is although the dynamics of two environments may differ, the structural pattern of "how to improve" remains similar.

Critical strengths

S1: Data augmentation is indeed helpful. Under identical implementation conditions, IQL achieves 451.3 vs. 359.5 w/o data augmentation. Even without using Morph-HC, performance improves by 56.7

S2: Experiments are verifialbe. The sources for the baselines are included in the paper. Within the framework of the authors' stated assumptions, no errors were found in the proof

Critical weaknesses

W1: Concerns center around the validity of synthetic transitions and rewards and the claim is built on top of unvetted data quality. The decoding process may alter states, actions, and subsequent states and rewards are obtained from inference. Although it already filtered terminal states, there is no guarantee that the remaining tuples conform to the target dynamics, nor does it ensure the accuracy of the associated reward values. Plz

  • use hold out data to examine the errors in the subsequent states and rewards derived from the decoded states and actions
  • specify waht full state variables are available what are not, and distinguish the data used for simulator diagnostics from the data employed during the training process.

W2: It is unclear to me the performance improvement is due to the better direction learned from the source environment, or is it simply a result of having more training data. If we look at the results: RIFT 393.4, SMOTE-B (target-data interpolation augmentation) 374.3, and IQL* 336.7. That means, standard data augmentation can also help; then how to separate the contribution from individual deisng modules? Plz add

  • A control test arm: one that learns the improvement using only the target data. To show whether a similar level of performance can be achieved without using the source data to learn the improvement.
  • A control baseline arm: reusing actual, high-reward target samples. By adding an equivalent number of training samples, determine whether similar performance gains can be achieved without relying on synthetic data.
  • These arms must use the same VAE.
  • Ablation study of the delta benefits of the proposed method for each individual module in the method.

W3: How parameters and models are selected for each task are missing but those are important to assess how the outperforming in performance is solid empirically. Plz add

  • How was λ\lambda determined? Were they predetermined, or were they selected post-hoc based on which yielded better results? - Clarify the origin of IQL results. Present both of these baselines "mixed-data IQL" and "target-data-only IQL." for comparison.
  • Report performance for all tasks with λ fixed at 0.5 and λ fixed at 1. Benchmark those against the better-performing baseline—either STC or "target-only" IQL—and clarify whether the reported results are based on any specific validation/selection criterion.
  • λ=1 does not mean that source data is entirely not used. The policy is just not trained directly on the raw source samples but since the VAE is still learned from the source data, the source information still remains. Discuss how to rule this out in exps.
  • Seems there is much bigger perf lifting for Morph-HalfCheetah - 118.3 vs. 273.4 (43%). Explain the bigger advantage.
  • Independently extract samples of the target data—each consisting of 5,000 records—and repeat the entire process to see whether the observed advantage persists and to assess the degree of varaince.

RL-seed variation on a fixed 5,000-transition target set still misses target-sampling data uncertainty. Changing the random seed for training is not the same as changing the target dataset. Training multiple times on the same set of 5,000 target samples primarily tests the volatility of the training process; it does not reveal whether that specific dataset happens to be an exceptionally good sample. Per-seed VAE/OT rebuilding is unclear.

Again, the point here is not to accuse the authors of "cherry-picking" results, but rather to determinine whether the observed performance gains are robust by knowing the selection critera and simple baseline

Questions for authors

For each point mentioned in weakness, reviewer provided suggestions/questions that hope the authors to answer Would like to also hear author's rebuttal for the other challenging points made there w/o clear follow-ups

1 comment

We thank the reviewer for the helpful feedback! We address the three concerns below.

W1: Validity of synthetic transitions and rewards.

We agree that filtering terminal transitions alone does not establish consistency with target dynamics. We add held-out reconstruction and target-simulator replay diagnostics.

Held-out reconstruction and replay. We exclude 500 target transitions (10%) and 5,000 source transitions from VAE fitting. On held-out target transitions, per-dimension reconstruction MSE ranges from 0.001 to 0.07 times the corresponding dimension’s variance across the eight settings. We also encode and decode each held-out transition, restore the decoded state in the target simulator, apply the decoded action, and compare the simulator’s successor and reward with the decoded values.

Generated-transition replay. We apply the same diagnostic to RIFT’s 1,000 generated transitions per setting and to mixup and SMOTE transitions constructed from the same anchors. Successor errors are normalized such that an error of 1 corresponds to a typical real-step displacement. Reward errors are normalized by the target’s high-minus-low class mean reward.

SettingHeld-out decoded successor error (real steps)Generated successor error (real steps)Generated reward MAE (class gaps)Generated realized reward
Gravity: Ant0.250.570.180.87
Gravity: HalfCheetah0.050.110.120.99
Gravity: Hopper0.331.460.051.06
Gravity: Walker2d0.657.070.080.80
Morphology: Ant0.160.290.140.91
Morphology: HalfCheetah0.050.080.051.08
Morphology: Hopper0.190.600.190.87
Morphology: Walker2d0.592.200.050.88

The reward labels are reasonably accurate, but successor fidelity varies and does not consistently track downstream gains. We also further acknowledge that generated transitions are not always dynamically exact.

W2: Source-derived improvement versus additional training data.

We added the two requested controls, keeping the VAE and normalization fixed and adding 1,000 transitions in each arm:

  • Target-only field: Learn the low-to-high field from target transitions and apply it to the same target anchors.
  • High-reward resampling: Resample real high-reward target transitions with replacement.
SettingIQL*High-reward resamplingTarget-only fieldRIFT
Gravity: Ant29.8 ± 4.936.1 ± 6.727.3 ± 9.740.8 ± 10.3
Gravity: HalfCheetah42.0 ± 2.347.3 ± 7.148.4 ± 4.447.3 ± 6.6
Gravity: Hopper52.7 ± 12.552.2 ± 8.561.1 ± 10.260.4 ± 7.3
Gravity: Walker2d30.6 ± 4.342.5 ± 6.738.3 ± 7.946.5 ± 5.5
Morphology: Ant76.9 ± 0.377.0 ± 0.477.6 ± 0.576.2 ± 2.4
Morphology: HalfCheetah22.8 ± 2.419.2 ± 2.456.7 ± 5.157.9 ± 4.4
Morphology: Hopper55.5 ± 6.453.2 ± 7.163.5 ± 6.769.4 ± 13.2
Morphology: Walker2d49.2 ± 5.157.0 ± 7.050.9 ± 8.952.8 ± 8.2
Aggregate359.5384.5423.8451.3

W3: Selection, source use, and robustness.

Mixing and selection. We considered only λ ∈ {0.5, 1}, corresponding to half-source minibatches and no raw source transitions in policy training, respectively. Unlike methods that filter source transitions, RIFT tries to keep the protocol simpler by controlling raw source exposure through this mixing choice. The candidate hyperparameters values were predetermined, but the submission reported the better arm per task using mean evaluation return across seed. From our runs, we note

Methodλ = 0.5λ = 1
IQL234.0312.4
SMOTE262.1337.2
RIFT280.5392.3

RIFT retains an aggregate advantage over both comparisons under either fixed mixing choice.

Source use at λ = 1. This removes raw source transitions from policy training, but retains source information in the VAE and field, as intended in our transfer setting.

Dependence on HalfCheetah. The improvement is not confined to Morph-HalfCheetah. Excluding that setting, RIFT scores 393.4 versus 336.7 for IQL*; and excluding both for grav & morph, the scores are 346.1 vs 294.7. Thus, although HalfCheetah contributes to the aggregate improvement, RIFT retains an advantage across the remaining environments (of around ~50pts).

Target-data uncertainty. Only 5,000 target transitions are available per setting. We therefore test four 4,000-transition subsets, rebuilding the full pipeline for each. RIFT scores 429.0 versus 353.5 for IQL*, with higher means in six of eight settings (revised Appendix, Table 11).

Finally, we clarify that each training seed refits the VAE, recomputes the couplings and field, and retrains the learner.

If this addresses your concerns, we would appreciate it if you could reconsider your score!

Reviewer_zqJHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed a novel approach called reward Improvement Field Transfer (RIFT) which aims to resolve the challenge that learning from limited target data in cross-domain offline reinforcement learning. The approach aims to learn the change from low-to-high reward changes from the source data then apply it to generate additional target data for better training. A comprehensive experiment is conducted to evaluate the proposed method, and it achieves improvements over previous state-of-the-art methods. Overall, the proposed method is interesting and novel, and the paper is well-organised.

Critical strengths
  • The paper proposed a very interesting approach, instead of learning dynamics and transfer between source and target data source. This method is trying to learn the structure or pattern that the transition between low-reward and high-reward, and then use this to extend the target source data for training.
  • Sufficient experiments demonstrate the improvement of the performance over existing state-of-the-art approaches.
Critical weaknesses
  • The transition between the low-reward to high-reward doesn't mean a good policy / behaviour, so the motivation is still a bit weak. If the author can try n-steps transitions might be more interesting.
  • The ablation study of using different size of target dataset should also be conducted.
Questions for authors

The questions has been listed in the above weakness part.

1 comment

We thank the reviewer for the positive assessment of RIFT’s novelty and empirical results. We address both concerns below with additional experiments.

W1: Higher immediate reward does not necessarily imply better long-horizon behavior.

We agree with this distinction. RIFT uses immediate reward to organize augmentation, rather than as a policy objective. Its low-to-high mapping generates complete transition tuples, and we leave downstream IQL to estimate long-horizon values through Bellman backups.

Immediate reward offers a directly observed, transition-level conditioning signal in the dense-reward locomotion tasks studied. An n-step return incorporates longer-term outcomes, but can also depend on subsequent actions taken by the data-collecting policy. Thus, longer-horizon labels offer a potential alternative for modeling the transfer signal. We will clarify this motivation and explicitly acknowledge that immediate-reward conditioning may be insufficient for sparse or delayed rewards.

Further, following the reviewer’s suggestion, we tested 5-, 10-, and 20-step-return conditioning, retaining one-step transition generation.

Setting1 step (paper)5 steps10 steps20 steps
Gravity: Ant40.8 ± 10.335.9 ± 2.244.9 ± 8.940.3 ± 9.4
Gravity: HalfCheetah47.3 ± 6.648.3 ± 2.951.3 ± 2.150.7 ± 2.1
Gravity: Hopper60.4 ± 7.354.4 ± 6.853.2 ± 12.560.5 ± 6.6
Gravity: Walker2d46.5 ± 5.546.1 ± 3.736.5 ± 4.639.9 ± 11.3
Morphology: Ant76.2 ± 2.476.6 ± 2.876.3 ± 1.874.2 ± 1.1
Morphology: HalfCheetah57.9 ± 4.459.0 ± 2.157.2 ± 2.857.2 ± 4.4
Morphology: Hopper69.4 ± 13.277.7 ± 5.362.6 ± 7.579.4 ± 4.1
Morphology: Walker2d52.8 ± 8.254.0 ± 9.556.1 ± 10.648.8 ± 5.8
Sum of displayed means451.3452.0438.1451.0

The longer-horizon variants differ from the original aggregate by +0.2%, −2.9%, and −0.1%, respectively, with no consistent improvement as the horizon increases. These results support immediate reward as an effective, simple conditioning signal in the evaluated dense-reward tasks and demonstrate that RIFT also accommodates longer-horizon signals.

W2. Sensitivity to target-data size

We additionally evaluated 500, 1k, 2.5k, and 5k target transitions across eight gravity and morphology settings.

Target transitionsIQL* aggregateRIFT aggregateRelative difference
500257.6245.6−4.7%
1k288.9293.6+1.6%
2.5k336.8398.6+17.8%
5k359.5451.3+25.5%

RIFT improves aggregate performance at three of the four target-data sizes. At 500 transitions, RIFT underperforms the baseline, indicating that its benefit does not extend uniformly to extreme target-data scarcity. This is consistent with the need for sufficient target information to learn the representation and anchor synthesis.

If these satisfy your concerns, we would really appreciate it if you would reconsider the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.