- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents an empirical study of how different sources of counterfactual failure data affect reward model learning and downstream reinforcement learning for robotic manipulation. The authors investigate four sources: teleoperated robot failures, policy-generated rollouts, human videos, and Real2Sim-generated failures.
The study evaluates two reward-learning approaches: LoRA adaptation of Robometer-4B and few-shot prompting of a frozen Generative Value Learning (GVL) model. Experiments are conducted across five real-world manipulation tasks using a held-out benchmark of 250 annotated trajectories. Reward quality is evaluated through trajectory ranking, partial-completion estimation, and success detection.
The authors further investigate how reward quality changes with the amount of counterfactual data and the visual fidelity of simulated trajectories. Finally, selected reward models are used to guide DSRL post-training of a π0 manipulation policy in a real-world pouring task.
The results suggest that teleoperated failures provide strong and consistent reward supervision, while cheaper human-video and Real2Sim data can become competitive with additional examples. However, increasing the data scale does not consistently improve reward quality. The downstream RL experiments show that teleoperation- and Real2Sim-adapted rewards can improve real-world task success, although the relationship between offline reward metrics and downstream policy performance is not always consistent.
Critical strengths
1. A practically important and well-structured empirical study of counterfactual data sources.
The paper investigates an important but relatively underexplored question: which types of failure data are most useful for learning reward models under limited data-collection resources? Comparing teleoperation, on-policy rollouts, human videos, and Real2Sim under a common reward-learning framework provides useful insights for robotic RL practitioners. The matched-data comparison and scaling analysis further reveal that the quantity of failure data alone does not determine reward quality.
2. The combination of offline reward evaluation and real-world RL validation strengthens the practical relevance of the findings.
The study evaluates learned rewards using trajectory ranking, partial-completion error, and success detection, and further investigates their effectiveness in downstream robot-policy optimization. In particular, the real-world pouring experiment shows that teleoperation- and Real2Sim-adapted rewards improve success rates from 35% with the base Robometer reward to 65% and 60%, respectively. These experiments provide evidence that counterfactual source selection can meaningfully influence downstream robotic behavior.
Critical weaknesses
1. A practically important and well-structured empirical study of counterfactual data sources.
The paper investigates an important but relatively underexplored question: which types of failure data are most useful for learning reward models under limited data-collection resources? Comparing teleoperation, on-policy rollouts, human videos, and Real2Sim under a common reward-learning framework provides useful insights for robotic RL practitioners. The matched-data comparison and scaling analysis further reveal that the quantity of failure data alone does not determine reward quality.
2. The combination of offline reward evaluation and real-world RL validation strengthens the practical relevance of the findings.
The study evaluates learned rewards using trajectory ranking, partial-completion error, and success detection, and further investigates their effectiveness in downstream robot-policy optimization. In particular, the real-world pouring experiment shows that teleoperation- and Real2Sim-adapted rewards improve success rates from 35% with the base Robometer reward to 65% and 60%, respectively. These experiments provide evidence that counterfactual source selection can meaningfully influence downstream robotic behavior.
Questions for authors
1. How does counterfactual source selection change under a genuinely fixed collection budget?
The main motivation is to identify cost-effective counterfactual sources, yet the experiments primarily compare equal numbers of trajectories.
Could the authors provide measured operator time, annotation time, simulator construction cost, and computational overhead for each source? In particular, at what number of generated trajectories does the amortized cost of Real2Sim become lower than teleoperated or policy-rollout data?
A cost-normalized comparison, such as reward-model performance versus total collection time or total human effort, would substantially strengthen the paper's central conclusion.
2. Why does competitive offline reward quality not consistently translate into better downstream RL performance?
Figure 2 suggests that human-video counterfactuals can provide competitive reward-model performance, yet Figure 5 shows that human-video adaptation achieves only 30% downstream task success, compared with 35% using the unadapted Robometer reward.
Can the authors clarify whether this discrepancy is caused by embodiment mismatch, poor calibration of intermediate rewards, or reward errors on the states visited by the updated policy?
More fundamentally, which offline metric is most predictive of actual RL improvement, and could a reward model with strong trajectory-level ranking still provide misleading local optimization signals?
3. How are failure diversity, trajectory difficulty, and source identity disentangled?
The results show that increasing the number of counterfactual trajectories can eventually degrade reward quality. The paper suggests that redundant or insufficiently diverse failure behaviors may explain this effect.
However, different data sources also vary in their failure-mode distributions, trajectory difficulty, and partial-completion labels. In particular, scripted Real2Sim failures and intentionally teleoperated failures may cover different parts of the task-progress distribution.
Could the authors clarify whether the comparisons control for failure type and partial-credit distribution, rather than only trajectory count? Furthermore, does the observed scaling degradation persist after balancing failure categories or explicitly selecting diverse trajectories?
This distinction is important because it determines whether the observed effect reflects an inherent limitation of inexpensive counterfactual data or primarily a dataset-selection problem.
3 comments
Author Response Part 2
Question 2 — Offline vs. RL gap:
We ran a formal correlation analysis across all 15 (reward, task) pairs from the RL experiments. Trajectory ranking (τ_mean) shows moderate correlation with RL success (r=0.40, p=0.139), while point-wise success detection (ROC-AUC_last) shows weaker correlation (r=0.25, p=0.359). This highlights a nuance: a reward model can rank full trajectories correctly yet still provide misleading dense per-step signals during online RL optimization. The base Robometer on Stack Cups is the clearest case, where both metrics are poor and RL drops below zero-shot. For human video, the gap is task-specific rather than a broad failure of the source: human-video adapted rewards still achieve 60% average success versus 40% for the base reward, but underperform on Pour (30%) where embodiment mismatch between third-person human demonstrations and the robot's egocentric view is most pronounced.
| Offline Metric | r (Pearson) | p-value | Interpretation |
|---|---|---|---|
| τ_mean (trajectory ranking) | 0.40 | 0.139 | Moderate correlation |
| ROC-AUC_last (success detection) | 0.25 | 0.359 | Weak correlation |
Correlation across 15 (reward, task) pairs. Trajectory ranking catches broken rewards but does not reliably predict which adapted reward will perform best in RL.
Question 3 — Failure diversity and scaling degradation:
This is part of the study design: we study counterfactual sources as naturally available failure data, not as controlled failure categories. The intent behind each failure varies by source: human-collected failures have the highest intent, rollouts give us no control over what failures occur, and real2sim failures are restricted to a set of stage-wise failure modes. We do not explicitly balance failure types because the point is to compare sources as a practitioner would actually use them. The scaling degradation we observe occurs under our reported joint data-and-training scaling protocol; we have not tested whether it persists after balancing failure categories or explicitly selecting diverse trajectories, and we note this as a direction for future work. The benchmark itself is built on behaviors from two different policies, so there is no data leakage from training sources.
Reminder
We would like to kindly follow up on our rebuttal. We would appreciate it if you could take a look at our responses and let us know if you have any remaining questions or concerns.
Thank you for your time and consideration.
Author Response Part 1
We thank the reviewer for the positive assessment and structured questions.
Question 1 — Fixed collection budget:
Our study compares sample efficiency of sources with differing collection requirements, rather than making absolute cost claims. We use symbolic cost categories because absolute time measurements are setup-dependent and not representative across labs. The equal-trajectory comparison isolates source quality from quantity, and the scaling experiments then test whether sources with lower per-episode collection requirements can compensate with more data. We establish a per-episode cost ordering: KT (kinesthetic + replay) > RO (rollout on hardware) >> HV (no robot needed) > R2S (amortized asset setup / N), where all sources except R2S also incur human annotation time. For R2S, the per-episode marginal cost is near zero once assets are built, but the fixed setup cost is non-trivial and task-dependent. At 10 trajectories per task, policy rollouts match 5 kinesthetic-teaching trajectories on offline metrics, illustrating that less costly sources can reach comparable quality with moderately more data.
From Table 1. *R2S uses robot embodiment in sim but not physical hardware. Cost ordering: KT > RO >> HV > R2S (amortized).
Scaling summary from Figure 5 (RQ3). Cheap sources help up to ~10/task; more data can hurt. At 10/task, RO ≈ 5 KT on offline metrics.
Detailed scaling and sim-fidelity (Robometer, avg. over 5 tasks). Mean ± std over seeds 42–44. Rollout and R2S degrade past 20 traj./task; human video is stable up to 20.