acceptodds
Under review as a conference paper at ICLR 2027

Disentangling Task-Relevant Latent Dynamics for Reward Modeling

Abstract

Reward design remains a major bottleneck for reinforcement learning in robotic manipulation. Handcrafted dense rewards can compensate for the limited guidance provided by sparse rewards, but require substantial task-specific reward engineering. Learning rewards from videos provides a scalable, data-driven alternative to manual reward design. However, existing video-based rewards are often computed in entangled visual spaces where task-relevant state changes are confounded with extraneous visual variation. As a result, the learned reward may respond to visual distractors rather than actual task execution. To address this limitation, we propose DiTaR (Disentangling Task-Relevant Latent Dynamics for Reward Modeling), a framework for learning dense rewards from heterogeneous videos. DiTaR uses an asymmetric two-stage representation learning procedure to separate task-relevant state information from task-irrelevant visual variation. In the task-relevant latent space, we train a history- and language-conditioned rectified-flow model to capture task dynamics and derive a directional dense reward from temporal changes in dynamics compatibility. Together, these components enable reward modeling directly over task-relevant dynamics rather than generic visual variation. DiTaR substantially improves policy learning on MetaWorld-v3 and ManiSkill over video-based reward baselines, and further supports real-world offline policy learning through reward relabeling. Reward analyses further show that DiTaR produces a more reliable reward signal for robotic policy learning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Reward Modeling, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xvsAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DiTaR is a reward model for robotic reinforcement learning that addresses spurious correlations between task progress and irrelevant visual changes, such as lighting. It first uses two-stage representation learning to disentangle task-relevant information from visual distractions. It then trains a dynamics model conditioned on visual history and task instructions to predict task-relevant representations. Rewards reflect improvements in how well observed transitions align with the learned dynamics.

DiTaR achieves success rates of 94.3% on MetaWorld, 87.5% on ManiSkill, and 83.3% in real-world offline RL, outperforming the evaluated baselines. However, simulated training still uses sparse environment rewards, and the method does not guarantee complete separation of task-relevant and irrelevant information.

Critical strengths
  1. Performance on MetaWorld and ManiSkill is substantially better than that of prior methods, suggesting that the learned rewards effectively support RL training.
  2. The authors provide detailed ablations and analyses supporting the effectiveness of their temporal-difference rewards and task-relevant representations.
  3. This paper addresses an important problem in robotic RL: using dense rewards to improve sample efficiency while reducing the influence of task-irrelevant visual information on reward prediction.
Critical weaknesses
  1. The paper is somewhat difficult to follow.
  2. As acknowledged in the limitations section, the method still relies on sparse environment rewards during policy training.
  3. Disentanglement is not guaranteed. Language does not fully describe task progress, so the “task-irrelevant” latent may still encode important motion information.
  4. The real-world evaluation uses only 10 trials per task, which may produce noisy success-rate estimates. More evaluation trials would strengthen the results.
Questions for authors
  1. How well does RL perform using only the reward signal provided by DiTaR? Alternatively, could a learned sparse reward model replace the environment reward when combined with DiTaR?
  2. What direct evidence shows that the two latents separate task-relevant information from visual distractions?
  3. How well does the method generalize to unseen tasks, objects, and viewpoints?
4 comments
AuthorsOpenReview ↗

Response to xvsA (1/2)

Thank you for the constructive feedback. Your review raises four concerns and three questions:

  • W1: readability

  • W2/Q1: reliance on the sparse environment reward

  • W3/Q2: evidence that the two latents separate

  • W4: number of real-world trials

  • Q3: generalization to unseen tasks, objects, and viewpoints

We hope the responses below address them.

W1. Readability

We agree and have revised Sec. 3 (highlighted in blue) to clarify the motivation and roles of the two representation-learning stages, as well as how the dynamics residual is used to construct the temporal-difference reward. We welcome any further specific suggestions.

W2/Q1. Reliance on the sparse environment reward

We tested both settings you suggested on the 12 MetaWorld tasks (Appendix D.6, Table 12).

Reward settingFinal SR ↑SR≥80 ↑[email protected] ↓
Env-only56.850.00.68
DiTaR + sparse env. reward94.391.70.24
DiTaR w/o sparse env. reward12.70.00.93
DiTaR + learned sparse reward60.241.70.64

Without the sparse reward, final success drops from 94.3% to 12.7%. DiTaR's dense reward measures temporal changes in dynamics compatibility (Sec. 3.4) but does not explicitly indicate task completion. For the learned sparse reward, we train a CNN success classifier with final frames of successful demonstrations as positives, and initial demonstration frames and failed random-action rollouts as negatives. Its average F1 on observations encountered during RL is 0.57. On the six tasks with F1 ≥ 0.79, success reaches 98.5%, close to the 99.8% with the environment reward on the same tasks. Across all 12 tasks, F1 correlates strongly with RL success (Spearman ρ=0.93\rho=0.93). In conclusion, DiTaR supplies dense guidance while a success signal identifies task completion, and sufficiently accurate learned success detection can replace the environment's sparse reward.

W3/Q2. Evidence of disentanglement

We agree that the two-stage procedure encourages but does not guarantee complete separation. zirrz^{\mathrm{irr}} is encouraged to capture information complementary to the instruction, so it may retain some motion information, as you note; the reward is derived from the dynamics of zrelz^{\mathrm{rel}}. We added direct comparisons (Appendix D.4; full tables in our response to R2). Linear probes on 12 MetaWorld tasks predict task identity (72.3% vs. 58.1%) and progress (R2R^2 0.247 vs. 0.140) better from zrelz^{\mathrm{rel}} than from zirrz^{\mathrm{irr}}; the nonzero progress R2R^2 of zirrz^{\mathrm{irr}} also indicates that separation is not complete. Under four visual perturbations, zrelz^{\mathrm{rel}} changes 3.2–17.2× less than zirrz^{\mathrm{irr}} and changes less on 94–99% of paired clips. On real-world videos, task predictions from zrelz^{\mathrm{rel}} remain more consistent under all four tested perturbations (65.3–79.0% vs. 49.4–57.7%). Thus, the separation is partial, and zrelz^{\mathrm{rel}}, the latent used for reward modeling, retains more task-relevant information and is more invariant to visual perturbations.

AuthorsOpenReview ↗

Response to xvsA (2/2)

W4. Real-world trials

We conducted a new evaluation with 20 trials per task for every method, doubling the number of trials in the original submission.

TaskVIPERDiffusion RewardReWiNDDiTaR
Object-to-Basket2/204/207/2013/20
Bottle-to-Plate3/207/2010/2016/20
Knife-to-Glove6/208/2013/2015/20
Overall18.3%31.7%50.0%73.3%

DiTaR has the highest success rate on all three tasks and exceeds ReWiND by 23.3 percentage points overall (p = 0.014, two-sided Fisher's exact test pooled across tasks). These results are reported in the revised Table 3, replacing the original evaluation with 10 trials per task (83.3% success). Thus, with twice as many trials, DiTaR's advantage holds on every task.

Q3. Generalization

Thanks for pointing this out. For viewpoints, DiTaR's reward is less sensitive to camera and lighting changes than all baselines (Table 2), and switching between two ManiSkill camera views changes it by 3.2%, versus 18.5% for ReWiND (Appendix C.3).  Beyond viewpoint changes, we also test DiTaR in an environment it has never seen. SimplerEnv is a simulated benchmark that replicates the real-world BridgeData V2 robot setups. DiTaR was trained on real Bridge videos but on no SimplerEnv videos, and we apply the reward model without fine-tuning. Even so, it outperforms all baselines on three WidowX tasks (Put Carrot on Plate, Put Spoon on Towel, Put Eggplant in Basket). All methods use the same DrQ-v2 settings for 1M steps per task.

MethodFinal SR ↑SR≥80 ↑Reach 0.8 ↑[email protected] ↓
Env-only77.033.366.70.78
VIPER71.333.366.70.83
Diffusion Reward45.60.033.30.97
ReWiND55.40.066.70.90
DiTaR85.966.7100.00.49

DiTaR achieves the best results on every metric (Appendix D.7, Table 13, Fig. 16) and reaches 80% success on all three tasks during training. In short, DiTaR's reward generalizes across viewpoints and transfers to SimplerEnv without fine-tuning.

Thank you for the detailed responses and additional experiments. The authors have addressed my concerns, and I maintain my positive assessment and score of 3

We sincerely appreciate your continued support and positive assessment of our work. We are glad that our responses and additional experiments have addressed your concerns. Thank you again for your time and consideration.

Reviewer_48Q3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DiTaR, which is a framework for learning dense rewards from actionless videos. It proposes an asymmetric two-stage approach to learn task-relevant representations, then trains a rectified-flow model to capture task dynamics, and finally formulates a directional dense reward function via dynamics-compatibility temporal changes for reinforcement learning. The experiments include two simulation benchmarks and real robots.

Critical strengths
  1. The proposed approach shows strong empirical performance in both simulation benchmarks MetaWorld and ManiSkill, as well as in real-robot experiments of three manipulation tasks. Results show that DiTaR substantially outperforms the baselines in aggregate performance across both simulation benchmarks and real-world tasks.
  2. The ablation study shows strong evidence of the method design. History conditioning for task-relevant dynamics clearly improves learning speed and final performance on 3 of the 4 tested tasks, and the temporal difference design of the reward formulation also shows gains, though not substantially, compared to using absolute compatibility.
Critical weaknesses
  1. One key argument of the paper is that the proposed method disentangles task-relevant and -irrelevant representations via the asymmetric two-stage representation learning procedure. However, the main results only show the representations' efficacy in the subsequent reinforcement learning and their robustness, but do not show the actual disentanglement. There lacks more direct analysis or study of the representations. The analysis in App.D.1 is still indirect via policy learning results, and that in App.D.3 provides some evidence that the representations capture shared task information but does not really compare zirrz^{irr} vs zrelz^{rel}.
  2. Two ablations use two different sets of MetaWorld tasks. It is unclear whether the tasks were picked to show more obvious advantages in this study. In other words, how necessary the ablated components are when coming to other tasks.
Questions for authors

The main question is whether the authors could provide a more direct comparison of zirrz^{irr} and zrelz^{rel}? It could be studying whether zrelz^{rel} shows stronger task-relevant semantic consistency and invariance to induced task-irrelevant visual distractions than zirrz^{irr}, via linear probing analysis on some task-relevant and -irrelevant attributes, and representation analysis under visual perturbations.

The second question is whether more tasks on ablation can be provided to see whether the ablation results persist or whether there are clusters of tasks that are and are not affected by the ablated components.

4 comments
AuthorsOpenReview ↗

Response to 48Q3

Thank you for the constructive feedback. Your review raises two concerns, each with a matching question:

  • W1/Q1: direct evidence that zrel and zirr separate task-relevant information from visual variation

  • W2/Q2: whether the ablation results hold beyond the selected tasks

We hope the responses below address them.

W1/Q1. Direct comparison of the two latents

Thanks for pointing this out. We added linear probing and perturbation analyses of zirrz^{\mathrm{irr}} and zrelz^{\mathrm{rel}} extracted from the same video input (Appendix D.4).

Task-relevant information (Table 10). Linear probes on frozen representations predict task identity and progress more accurately from zrelz^{\mathrm{rel}} on 12 MetaWorld tasks. On real-world videos, we also measure task-prediction consistency using probes trained on clean clips, defined as the percentage of clips whose predicted task remains unchanged after perturbation.

Probezirrz^{\mathrm{irr}}zrelz^{\mathrm{rel}}
MetaWorld task accuracy (%) ↑58.172.3
MetaWorld task progress R2R^2 ↑0.1400.247
Real-world consistency (%) ↑, heavy lighting55.865.3
Real-world consistency (%) ↑, local shuffling53.079.0
Real-world consistency (%) ↑, temporal jitter ±257.773.9
Real-world consistency (%) ↑, shuffled chunks49.466.8

Invariance to visual perturbations (Table 9). We measure the cosine distance between clean and perturbed representations of the same clip, normalized by the median distance between clean clips from different tasks (lower is better).

Perturbationzirrz^{\mathrm{irr}} ↓zrelz^{\mathrm{rel}} ↓Reduction
Camera pose shift197.7%60.9%3.2×
Camera yaw 8°216.9%63.8%3.4×
Heavy lighting265.8%15.4%17.2×
Camera zoom 1.25×96.5%18.9%5.1×

Across all four perturbations, zrelz^{\mathrm{rel}} changes 3.2–17.2× less than zirrz^{\mathrm{irr}} and changes less on 94–99% of paired clips; task-level bootstrap confidence intervals are in Appendix Table 9. In short, these results provide direct evidence that zrelz^{\mathrm{rel}} preserves more task-relevant information while being substantially less sensitive to task-irrelevant visual variation.

W2/Q2. Ablations on more tasks

We extended both ablations to the same ten MetaWorld tasks (Appendix D.5, Table 11): Coffee Pull, Drawer Open, Faucet Open, Dial Turn, Coffee Push, Window Close, Drawer Close, Handle Press, Faucet Close, and Hammer.

AblationMethod     Final SR ↑  SR≥80 ↑Reach 0.8 ↑[email protected] ↓
History w/o history      79.8    70.0        90.0        0.39
         DiTaR    92.790.0        90.0    0.24
TD      w/o TD           91.1    90.0        90.0    0.21
         w/ TD    92.7    90.0        90.0        0.24

History conditioning substantially improves average final success from 79.8% to 92.7% and reduces [email protected] from 0.39 to 0.24. TD reward further improves average final success from 91.1% to 92.7%, with unchanged threshold coverage and comparable [email protected] (0.24 vs. 0.21). Therefore, both components improve final success across the ten-task evaluation, with larger gains from history conditioning.

Dear Reviewer 48Q3,

Thank you again for your positive assessment and constructive feedback. Following your suggestions, we conducted additional experiments, including a direct comparison of the two latent representations and an extended ablation study across ten MetaWorld tasks.

As the discussion period is coming to a close, we sincerely hope you may have an opportunity to review our rebuttal and share your thoughts on the additional results. We would also be grateful if you could consider these new findings in your overall evaluation.

We greatly appreciate your time and support, and would be happy to provide any further clarifications.

Thank you very much.

I appreciate the additional experiments on representation analysis and ablation. They have addressed my questions. I will maintain my positive rating: Weak accept.

Thank you very much for your positive response. We are pleased that our additional representation analyses and experiments have fully addressed your concerns and provided stronger evidence for our key claims. We sincerely appreciate your constructive feedback and continued support for our work.

Reviewer_D9C9OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a method for learning rewards from video demonstrations for robot reinforcement learning. The challenge with learning rewards from videos is that the videos contain background information that may not be relevant to the task. This paper presents a way of disentangle task-relevant latent dynamics from the video for more effective reward learning. Specifically, they learn a dense reward function using a history and language conditioned flow model. The method is compared with three other reward learning baselines through simulation and see physical experiments.

Critical strengths

The paper addresses a major challenge with robot RL, which is scalable learning. Learning from videos would be great since we have an abudnace of videos but not nearly the same robot data. As such, the paper is focussed on solving an important problem. The overall approach seems sound and the presented numerical results are promising.

Critical weaknesses

My main concern is the limited experiments. The simulation experiments are fine but at this point, the field is very mature that showing improvements over other video reward RL baselines is not enough. These simulated tasks don’t need video demonstrations as a hand-designed (without significant reward engineering; most of these simulations come with a reward function) would perform reasonably well. The real benefit of this method would be in tasks where designing a reward function is challenging. But none of the simulated or physical tasks seem like that.

The baselines also seem to do worse than sparse rewards which is inconsistent with the numbers reported in the original baseline papers.

Finally, the hardware results are very limited. Just one success rate metric (with no mention of how many iterations or any variations in the number of demonstartions, no hand-crafted reward baseline, or a sparse reward task, or even an off-the-shelf VLA comparison). Further, no real robustness experiments conducted in hardware.

Questions for authors

I would have also liked to seen what reward function the model actually learns. Is there something one could say about the “correctness” of the reward function beyond its performance in terms of RL? How different is it from a hand-crafted reward function?

It would also be interesting to see this in a setting where the videos are not just directly from the same setup as what the robot is operating in. To me, the real benefit of video demonstrations is when there is some mismatch (embodiment used for videos and the one used for the actual RL are not the same..). Else, one can argue that instead of video demonstrations —> RL, one should really do teleop data + videos —> imitation learning. In any case, it would be important to compare how the two approaches stack against each other, not just in terms of learning efficiency but also the “effort” required in providing demonstrations.

5 comments
AuthorsOpenReview ↗

Response to D9C9 (1/2)

Thank you for the constructive feedback. Your review raises three concerns and two questions:

  • W1: rewards already available in simulation

  • W2: baselines below the sparse-reward agent

  • W3: limited real-world evaluation

  • Q1: what reward DiTaR learns

  • Q2: other setups and imitation learning

We hope the responses below address them.

W1. Rewards already available in simulation

We agree that learned rewards matter most where designing a reward is costly, and we respectfully note that two of our settings do not provide handcrafted dense rewards. Neither the real-robot tasks nor SimplerEnv (Q2) provides dense environment rewards, and on SimplerEnv the same reward model is applied without fine-tuning or task-specific design. The 16 simulated tasks are a controlled testbed with ground-truth success, as in Diffusion Reward (ECCV 2024), ReWiND (CoRL 2025), GenReward (CVPR 2026), and TimeRewarder (ICML 2026). Although these benchmarks provide handcrafted dense rewards, we only use the environment's sparse binary success reward, together with DiTaR's learned dense reward. In short, these simulated benchmarks are the standard testbed in prior work on learned rewards, and our results are not limited to them: DiTaR also outperforms all baselines on SimplerEnv and on the real robot, where no handcrafted dense reward is available.

W2. Baselines below the sparse-reward agent

We clarify the comparison. Env-only already adds RND exploration to the sparse reward, and Diffusion Reward[1] likewise reports VIPER below its RND baseline. The original papers also evaluate different task suites, so their reported results are not directly comparable to ours. We therefore retrained each baseline with its official implementation on the same expert videos, changing only the learned reward in one DrQ-v2 protocol (Appendix F). Our Diffusion Reward reproduction learns both tasks shared with its original main evaluation (Coffee Push and Dial Turn, Fig. 3), and ReWiND exceeds Env-only on MetaWorld (65.2% vs. 56.8%, Table 1). Therefore, the baselines are evaluated fairly under our unified protocol, and only DiTaR improves over Env-only across all three benchmarks.

W3. Real-world evaluation

We conducted a new real-world evaluation with 20 trials per task for each method. Using the same 150 demonstrations and IQL pipeline, DiTaR achieves 73.3% overall success (44/60), compared with 50.0% for ReWiND, 31.7% for Diffusion Reward, and 18.3% for VIPER. DiTaR has the highest success rate on all three tasks, and its overall advantage over ReWiND is statistically significant (p = 0.014, two-sided Fisher's exact test on pooled trials). These results are updated in Table 3 in the revision. On real-world videos, camera and lighting changes alter DiTaR's reward by 1.6–4.3%, versus 10.2–60.8% for the baselines (Table 2). We address the VLA comparison in Q2. Thus, with twice as many trials, DiTaR's advantage holds on every real-world task.

References

[1] Diffusion Reward: Learning Rewards via Conditional Video Diffusion. ECCV 2024.

[2] ReWiND: Language-Guided Rewards Teach Robot Policies without New Demonstrations. CoRL 2025.

[3] Goal-Driven Reward by Video Diffusion Models for Reinforcement Learning. CVPR 2026.

[4] TimeRewarder: Learning Dense Reward from Passive Videos via Frame-wise Temporal Distance. ICML 2026.

AuthorsOpenReview ↗

Response to D9C9 (2/2)

Q1. What reward DiTaR learns

Thanks for this question. DiTaR scores each clip's consistency with the task dynamics learned from videos, given the history and instruction (ϕt=exp⁡(−et)\phi_t=\exp(-e_t)), and rewards its temporal change, rtTD=ϕt−ϕt−Δtr_t^{\mathrm{TD}}=\phi_t-\phi_{t-\Delta t}, which is positive when execution becomes more consistent with these dynamics (Sec. 3.4). Beyond RL success rates, we also test directly what the reward responds to. It drops when the temporal order of a video is corrupted, and the drop occurs at the corrupted segment (Table 2; Appendix C.1, C.2). It changes little under camera and lighting changes (Table 2; Appendix C.3). It also generally increases as the policy improves during training (Appendix C.4). DiTaR does not rely on handcrafted goal distances or privileged simulator states, and a single model serves all 16 simulated tasks. In conclusion, DiTaR rewards increasing consistency with learned task dynamics, without privileged state or per-task reward design.

Q2. Other setups and imitation learning

Thanks for pointing this out. VLAs and imitation learning are policy learners trained on action-labeled demonstrations, which typically require teleoperation on the target robot, whereas DiTaR is a reward model learned from action-free videos. We therefore fix the RL algorithm so that only the reward changes; a VLA or imitation-learning comparison would not isolate the reward. Our videos already come largely from other setups: 95% are from AgiBotWorld, DROID, and Bridge, and only 5% from MetaWorld and ManiSkill (Table 4). We further test DiTaR on an environment it has never seen. SimplerEnv provides simulated counterparts of the BridgeData V2 setups. DiTaR was trained on real Bridge videos but on no SimplerEnv videos, and we apply it to three SimplerEnv WidowX tasks without fine-tuning. With the same DrQ-v2 setup and 1M steps per task, DiTaR reaches 85.9% mean final success, versus 77.0% for Env-only, and outperforms all three baselines (Appendix D.7; see our response to R3). Thus, DiTaR learns from videos of other robots and cameras, and its reward transfers from real videos to simulation.

Dear Reviewer D9C9,

Thank you again for your time and constructive feedback. Following your comments, we conducted additional experiments, including an evaluation on SimplerEnv and a new real-world evaluation with 20 trials per task. We also clarified that our experiments do not use handcrafted dense rewards, along with the evaluation protocol and what the learned reward measures.

As the discussion period is coming to a close, we sincerely hope you may have an opportunity to review our responses and additional results. We would greatly appreciate any further feedback and would be happy to address any remaining concerns.

Thank you very much!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank Reviewers D9C9 (R1), 48Q3 (R2), and xvsA (R3) for their time and feedback. We appreciate the strengths you recognized:

  1. Strong empirical performance on MetaWorld, ManiSkill, and real-robot experiments (R2), "substantially better than that of prior methods" (R3).

  2. Method design supported by ablations and analyses: "strong evidence of the method design" (R2) and "detailed ablations and analyses" (R3).

  3. An important problem (R1, R3), with an approach that "seems sound" and results that are "promising" (R1).

We added the following results, detailed in the individual responses:

  • Direct latent comparison (R2, R3). Under camera and lighting perturbations, zrel changes 3.2-17.2× less than zirr, and linear probes predict task identity more accurately from zrel (72.3% vs. 58.1%).

  • Extended ablations (R2). On the same ten MetaWorld tasks, history conditioning raises final success from 79.8% to 92.7%, and the TD reward from 91.1% to 92.7%.

  • Real-world evaluation with 20 trials per task (R1, R3). DiTaR reaches 73.3% success, versus 50.0% for the strongest baseline (p = 0.014).

  • Real-to-sim transfer to SimplerEnv (R1, R3). Without fine-tuning, DiTaR reaches 85.9% final success, versus 77.0% for Env-only and lower values for all learned-reward baselines.

  • RL without the sparse success reward (R3). DiTaR still needs a success signal, and a learned success detector can provide it when accurate (98.5% success on tasks with detection F1 ≥ 0.79).

We uploaded a revised PDF with these results, marked in blue. We hope these results address the reviewers' concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.