- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DiTaR is a reward model for robotic reinforcement learning that addresses spurious correlations between task progress and irrelevant visual changes, such as lighting. It first uses two-stage representation learning to disentangle task-relevant information from visual distractions. It then trains a dynamics model conditioned on visual history and task instructions to predict task-relevant representations. Rewards reflect improvements in how well observed transitions align with the learned dynamics.
DiTaR achieves success rates of 94.3% on MetaWorld, 87.5% on ManiSkill, and 83.3% in real-world offline RL, outperforming the evaluated baselines. However, simulated training still uses sparse environment rewards, and the method does not guarantee complete separation of task-relevant and irrelevant information.
Critical strengths
- Performance on MetaWorld and ManiSkill is substantially better than that of prior methods, suggesting that the learned rewards effectively support RL training.
- The authors provide detailed ablations and analyses supporting the effectiveness of their temporal-difference rewards and task-relevant representations.
- This paper addresses an important problem in robotic RL: using dense rewards to improve sample efficiency while reducing the influence of task-irrelevant visual information on reward prediction.
Critical weaknesses
- The paper is somewhat difficult to follow.
- As acknowledged in the limitations section, the method still relies on sparse environment rewards during policy training.
- Disentanglement is not guaranteed. Language does not fully describe task progress, so the “task-irrelevant” latent may still encode important motion information.
- The real-world evaluation uses only 10 trials per task, which may produce noisy success-rate estimates. More evaluation trials would strengthen the results.
Questions for authors
- How well does RL perform using only the reward signal provided by DiTaR? Alternatively, could a learned sparse reward model replace the environment reward when combined with DiTaR?
- What direct evidence shows that the two latents separate task-relevant information from visual distractions?
- How well does the method generalize to unseen tasks, objects, and viewpoints?
4 comments
Response to xvsA (2/2)
W4. Real-world trials
We conducted a new evaluation with 20 trials per task for every method, doubling the number of trials in the original submission.
| Task | VIPER | Diffusion Reward | ReWiND | DiTaR |
|---|---|---|---|---|
| Object-to-Basket | 2/20 | 4/20 | 7/20 | 13/20 |
| Bottle-to-Plate | 3/20 | 7/20 | 10/20 | 16/20 |
| Knife-to-Glove | 6/20 | 8/20 | 13/20 | 15/20 |
| Overall | 18.3% | 31.7% | 50.0% | 73.3% |
DiTaR has the highest success rate on all three tasks and exceeds ReWiND by 23.3 percentage points overall (p = 0.014, two-sided Fisher's exact test pooled across tasks). These results are reported in the revised Table 3, replacing the original evaluation with 10 trials per task (83.3% success). Thus, with twice as many trials, DiTaR's advantage holds on every task.
Q3. Generalization
Thanks for pointing this out. For viewpoints, DiTaR's reward is less sensitive to camera and lighting changes than all baselines (Table 2), and switching between two ManiSkill camera views changes it by 3.2%, versus 18.5% for ReWiND (Appendix C.3). Beyond viewpoint changes, we also test DiTaR in an environment it has never seen. SimplerEnv is a simulated benchmark that replicates the real-world BridgeData V2 robot setups. DiTaR was trained on real Bridge videos but on no SimplerEnv videos, and we apply the reward model without fine-tuning. Even so, it outperforms all baselines on three WidowX tasks (Put Carrot on Plate, Put Spoon on Towel, Put Eggplant in Basket). All methods use the same DrQ-v2 settings for 1M steps per task.
| Method | Final SR ↑ | SR≥80 ↑ | Reach 0.8 ↑ | [email protected] ↓ |
|---|---|---|---|---|
| Env-only | 77.0 | 33.3 | 66.7 | 0.78 |
| VIPER | 71.3 | 33.3 | 66.7 | 0.83 |
| Diffusion Reward | 45.6 | 0.0 | 33.3 | 0.97 |
| ReWiND | 55.4 | 0.0 | 66.7 | 0.90 |
| DiTaR | 85.9 | 66.7 | 100.0 | 0.49 |
DiTaR achieves the best results on every metric (Appendix D.7, Table 13, Fig. 16) and reaches 80% success on all three tasks during training. In short, DiTaR's reward generalizes across viewpoints and transfers to SimplerEnv without fine-tuning.
Thank you for the detailed responses and additional experiments. The authors have addressed my concerns, and I maintain my positive assessment and score of 3
We sincerely appreciate your continued support and positive assessment of our work. We are glad that our responses and additional experiments have addressed your concerns. Thank you again for your time and consideration.
Response to xvsA (1/2)
Thank you for the constructive feedback. Your review raises four concerns and three questions:
W1: readability
W2/Q1: reliance on the sparse environment reward
W3/Q2: evidence that the two latents separate
W4: number of real-world trials
Q3: generalization to unseen tasks, objects, and viewpoints
We hope the responses below address them.
W1. Readability
We agree and have revised Sec. 3 (highlighted in blue) to clarify the motivation and roles of the two representation-learning stages, as well as how the dynamics residual is used to construct the temporal-difference reward. We welcome any further specific suggestions.
W2/Q1. Reliance on the sparse environment reward
We tested both settings you suggested on the 12 MetaWorld tasks (Appendix D.6, Table 12).
Without the sparse reward, final success drops from 94.3% to 12.7%. DiTaR's dense reward measures temporal changes in dynamics compatibility (Sec. 3.4) but does not explicitly indicate task completion. For the learned sparse reward, we train a CNN success classifier with final frames of successful demonstrations as positives, and initial demonstration frames and failed random-action rollouts as negatives. Its average F1 on observations encountered during RL is 0.57. On the six tasks with F1 ≥ 0.79, success reaches 98.5%, close to the 99.8% with the environment reward on the same tasks. Across all 12 tasks, F1 correlates strongly with RL success (Spearman ρ=0.93). In conclusion, DiTaR supplies dense guidance while a success signal identifies task completion, and sufficiently accurate learned success detection can replace the environment's sparse reward.
W3/Q2. Evidence of disentanglement
We agree that the two-stage procedure encourages but does not guarantee complete separation. zirr is encouraged to capture information complementary to the instruction, so it may retain some motion information, as you note; the reward is derived from the dynamics of zrel. We added direct comparisons (Appendix D.4; full tables in our response to R2). Linear probes on 12 MetaWorld tasks predict task identity (72.3% vs. 58.1%) and progress (R2 0.247 vs. 0.140) better from zrel than from zirr; the nonzero progress R2 of zirr also indicates that separation is not complete. Under four visual perturbations, zrel changes 3.2–17.2× less than zirr and changes less on 94–99% of paired clips. On real-world videos, task predictions from zrel remain more consistent under all four tested perturbations (65.3–79.0% vs. 49.4–57.7%). Thus, the separation is partial, and zrel, the latent used for reward modeling, retains more task-relevant information and is more invariant to visual perturbations.