acceptodds
Under review as a conference paper at ICLR 2027

Can Cheap Failures Teach Good Rewards? Studying Counterfactual Sources for Reward Learning in Manipulation

Abstract

Reinforcement learning (RL) can improve robot policies beyond imitation learning, but it requires rewards that capture task progress and failure. Learning such rewards requires not only successful demonstrations, but also informative counterfactual failures that can guide the model on behaviors to penalize. However, large-scale robot datasets are dominated by successful behavior, and collecting failures on real hardware is expensive. This raises a practical question: given a limited data-collection budget, which sources of counterfactual experience are worth investing in? We compare four sources of failure data: teleoperated failures, policy rollouts, human videos, and real-to-sim rollouts, across five real-world manipulation tasks. Holding the task suite, successful demonstrations, and evaluation fixed, we study these sources through LoRA adaptation of a reward model and few-shot prompting of a frozen model, using matched-data and scaling settings. We evaluate the resulting reward models on a custom held-out benchmark of 250 human-annotated policy rollouts, measuring trajectory ranking, partial-completion calibration, and success detection. Finally, we use selected reward models to guide RL post-training of and test whether improvements in offline reward quality translate into better robot behavior. Together, our study examines whether inexpensive counterfactual sources can match or surpass smaller amounts of faithful real-robot failure data when scaled.

Primary area:
applications to robotics, autonomy, planning
Keywords:
reward models, robot manipulation, real2sim

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xVPUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an empirical study of how different sources of counterfactual failure data affect reward model learning and downstream reinforcement learning for robotic manipulation. The authors investigate four sources: teleoperated robot failures, policy-generated rollouts, human videos, and Real2Sim-generated failures.

The study evaluates two reward-learning approaches: LoRA adaptation of Robometer-4B and few-shot prompting of a frozen Generative Value Learning (GVL) model. Experiments are conducted across five real-world manipulation tasks using a held-out benchmark of 250 annotated trajectories. Reward quality is evaluated through trajectory ranking, partial-completion estimation, and success detection.

The authors further investigate how reward quality changes with the amount of counterfactual data and the visual fidelity of simulated trajectories. Finally, selected reward models are used to guide DSRL post-training of a π0 manipulation policy in a real-world pouring task.

The results suggest that teleoperated failures provide strong and consistent reward supervision, while cheaper human-video and Real2Sim data can become competitive with additional examples. However, increasing the data scale does not consistently improve reward quality. The downstream RL experiments show that teleoperation- and Real2Sim-adapted rewards can improve real-world task success, although the relationship between offline reward metrics and downstream policy performance is not always consistent.

Critical strengths

1. A practically important and well-structured empirical study of counterfactual data sources.

The paper investigates an important but relatively underexplored question: which types of failure data are most useful for learning reward models under limited data-collection resources? Comparing teleoperation, on-policy rollouts, human videos, and Real2Sim under a common reward-learning framework provides useful insights for robotic RL practitioners. The matched-data comparison and scaling analysis further reveal that the quantity of failure data alone does not determine reward quality.

2. The combination of offline reward evaluation and real-world RL validation strengthens the practical relevance of the findings.

The study evaluates learned rewards using trajectory ranking, partial-completion error, and success detection, and further investigates their effectiveness in downstream robot-policy optimization. In particular, the real-world pouring experiment shows that teleoperation- and Real2Sim-adapted rewards improve success rates from 35% with the base Robometer reward to 65% and 60%, respectively. These experiments provide evidence that counterfactual source selection can meaningfully influence downstream robotic behavior.

Critical weaknesses

1. A practically important and well-structured empirical study of counterfactual data sources.

The paper investigates an important but relatively underexplored question: which types of failure data are most useful for learning reward models under limited data-collection resources? Comparing teleoperation, on-policy rollouts, human videos, and Real2Sim under a common reward-learning framework provides useful insights for robotic RL practitioners. The matched-data comparison and scaling analysis further reveal that the quantity of failure data alone does not determine reward quality.

2. The combination of offline reward evaluation and real-world RL validation strengthens the practical relevance of the findings.

The study evaluates learned rewards using trajectory ranking, partial-completion error, and success detection, and further investigates their effectiveness in downstream robot-policy optimization. In particular, the real-world pouring experiment shows that teleoperation- and Real2Sim-adapted rewards improve success rates from 35% with the base Robometer reward to 65% and 60%, respectively. These experiments provide evidence that counterfactual source selection can meaningfully influence downstream robotic behavior.

Questions for authors

1. How does counterfactual source selection change under a genuinely fixed collection budget?

The main motivation is to identify cost-effective counterfactual sources, yet the experiments primarily compare equal numbers of trajectories.

Could the authors provide measured operator time, annotation time, simulator construction cost, and computational overhead for each source? In particular, at what number of generated trajectories does the amortized cost of Real2Sim become lower than teleoperated or policy-rollout data?

A cost-normalized comparison, such as reward-model performance versus total collection time or total human effort, would substantially strengthen the paper's central conclusion.

2. Why does competitive offline reward quality not consistently translate into better downstream RL performance?

Figure 2 suggests that human-video counterfactuals can provide competitive reward-model performance, yet Figure 5 shows that human-video adaptation achieves only 30% downstream task success, compared with 35% using the unadapted Robometer reward.

Can the authors clarify whether this discrepancy is caused by embodiment mismatch, poor calibration of intermediate rewards, or reward errors on the states visited by the updated policy?

More fundamentally, which offline metric is most predictive of actual RL improvement, and could a reward model with strong trajectory-level ranking still provide misleading local optimization signals?

3. How are failure diversity, trajectory difficulty, and source identity disentangled?

The results show that increasing the number of counterfactual trajectories can eventually degrade reward quality. The paper suggests that redundant or insufficiently diverse failure behaviors may explain this effect.

However, different data sources also vary in their failure-mode distributions, trajectory difficulty, and partial-completion labels. In particular, scripted Real2Sim failures and intentionally teleoperated failures may cover different parts of the task-progress distribution.

Could the authors clarify whether the comparisons control for failure type and partial-credit distribution, rather than only trajectory count? Furthermore, does the observed scaling degradation persist after balancing failure categories or explicitly selecting diverse trajectories?

This distinction is important because it determines whether the observed effect reflects an inherent limitation of inexpensive counterfactual data or primarily a dataset-selection problem.

3 comments
AuthorsOpenReview ↗

Author Response Part 1

We thank the reviewer for the positive assessment and structured questions.

Question 1 — Fixed collection budget:

Our study compares sample efficiency of sources with differing collection requirements, rather than making absolute cost claims. We use symbolic cost categories because absolute time measurements are setup-dependent and not representative across labs. The equal-trajectory comparison isolates source quality from quantity, and the scaling experiments then test whether sources with lower per-episode collection requirements can compensate with more data. We establish a per-episode cost ordering: KT (kinesthetic + replay) > RO (rollout on hardware) >> HV (no robot needed) > R2S (amortized asset setup / N), where all sources except R2S also incur human annotation time. For R2S, the per-episode marginal cost is near zero once assets are built, but the fixed setup cost is non-trivial and task-dependent. At 10 trajectories per task, policy rollouts match 5 kinesthetic-teaching trajectories on offline metrics, illustrating that less costly sources can reach comparable quality with moderately more data.

SourceNeeds RobotCollection CostAnnotationData/task
Kinesthetic (KT)Yest_collect + t_replay (highest)Human5
Rollout (RO)Yest_rollout (high)Human20
Human Video (HV)Not_human (low)Human20
Real2sim (R2S)No*t_asset / N (lowest)Automatic30

From Table 1. *R2S uses robot embodiment in sim but not physical hardware. Cost ordering: KT > RO >> HV > R2S (amortized).

Source5/task10/task20/task25–30/task
KTbest baseline———
ROhelps≈ 5 KTworse than base—
HVhelpsstablestable—
R2Shelpsimproves—degrades below base

Scaling summary from Figure 5 (RQ3). Cheap sources help up to ~10/task; more data can hurt. At 10/task, RO ≈ 5 KT on offline metrics.

SourceTraj./taskτ_mean (↑)MAE_last (↓)ROC-AUC_last (↑)
Rollout50.489±0.0190.174±0.0270.888±0.056
Rollout100.613±0.0420.142±0.0190.925±0.050
Rollout200.243±0.0400.308±0.0010.878±0.003
Human50.564±0.0240.165±0.0160.893±0.033
Human100.562±0.0250.146±0.0190.956±0.031
Human200.550±0.0650.166±0.0070.930±0.018
R2S HiFi50.472±0.0270.186±0.0140.870±0.007
R2S HiFi100.539±0.0280.153±0.0090.885±0.019
R2S HiFi200.486±0.0580.158±0.0070.855±0.019
R2S HiFi250.242±0.0140.311±0.0040.879±0.005
R2S HiFi300.253±0.0230.310±0.0010.880±0.002
R2S LoFi50.491±0.0480.165±0.0050.885±0.016
R2S LoFi100.258±0.0170.311±0.0020.870±0.002

Detailed scaling and sim-fidelity (Robometer, avg. over 5 tasks). Mean ± std over seeds 42–44. Rollout and R2S degrade past 20 traj./task; human video is stable up to 20.

AuthorsOpenReview ↗

Author Response Part 2

Question 2 — Offline vs. RL gap:

We ran a formal correlation analysis across all 15 (reward, task) pairs from the RL experiments. Trajectory ranking (τ_mean) shows moderate correlation with RL success (r=0.40, p=0.139), while point-wise success detection (ROC-AUC_last) shows weaker correlation (r=0.25, p=0.359). This highlights a nuance: a reward model can rank full trajectories correctly yet still provide misleading dense per-step signals during online RL optimization. The base Robometer on Stack Cups is the clearest case, where both metrics are poor and RL drops below zero-shot. For human video, the gap is task-specific rather than a broad failure of the source: human-video adapted rewards still achieve 60% average success versus 40% for the base reward, but underperform on Pour (30%) where embodiment mismatch between third-person human demonstrations and the robot's egocentric view is most pronounced.

Offline Metricr (Pearson)p-valueInterpretation
τ_mean (trajectory ranking)0.400.139Moderate correlation
ROC-AUC_last (success detection)0.250.359Weak correlation

Correlation across 15 (reward, task) pairs. Trajectory ranking catches broken rewards but does not reliably predict which adapted reward will perform best in RL.

Question 3 — Failure diversity and scaling degradation:

This is part of the study design: we study counterfactual sources as naturally available failure data, not as controlled failure categories. The intent behind each failure varies by source: human-collected failures have the highest intent, rollouts give us no control over what failures occur, and real2sim failures are restricted to a set of stage-wise failure modes. We do not explicitly balance failure types because the point is to compare sources as a practitioner would actually use them. The scaling degradation we observe occurs under our reported joint data-and-training scaling protocol; we have not tested whether it persists after balancing failure categories or explicitly selecting diverse trajectories, and we note this as a direction for future work. The benchmark itself is built on behaviors from two different policies, so there is no data leakage from training sources.

We would like to kindly follow up on our rebuttal. We would appreciate it if you could take a look at our responses and let us know if you have any remaining questions or concerns.

Thank you for your time and consideration.

Reviewer_FzuCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates what sources of counterfactual failure data provide the best cost-benefit trade-off. The authors compare 4 failure sources across 5 tasks using LoRA-tuned Robometer and few-shot Generative Value Learning (GVL). The authors main findings are that: for Robometer teleoperated failures provide the highest improvement, but not for GVL (Research Question 1); cheaper sources can become competitive with more data but often saturate or degrade (Research Question 2); high-fidelity real2sim outperforms low-fidelity real2sim at larger scale (Research Question 3); and teleoperated failures and real2sim-adapted rewards provide the best RL success on the "Pour" task (Research Question 4).

Critical strengths

1- The contributions of the paper are useful for the robotics community. The paper is not a mere study, it provides actionable insights for the community: teleoperated failures are most effective but expensive, high-fidelity real2sim is a great trade-off of cost and performance, human videos are surprisingly competitive despite embodiment mismatch, and scaling cheaper sources often saturates or degrades beyond 10–20 trajectories. Also, the study aligns with growing interest in reward modeling for robotics (e.g. Robometer, GVL, RoboReward, etc.).

2- The authors provide a solid benchmark that consists in 250 episodes with human-annotated rollouts from two evaluation policies not used for counterfactual collection, and evaluates trajectory ranking, partial-completion calibration, and success detection. The study also includes real robot validation, which strenghtens the practical relevance of the offline findings (even though only one task is evaluated in the real robot).

Critical weaknesses

1- Tasks are not very diverse and relatively simple. For example, a task such as peg insertion, which requires precision, is much more sensitive to the design of the reward and it would have been interesting to see the effect of the different counterfactual sources. Without such a task, I believe several conclusions are overstated, as the findings may not extend to more complex tasks. If adding such an experiment is infeasible, the authors should explicitly condition their claims to the studied tasks and to the robot they use. Furthermore, contribution claims seem too strong, as they only hold for Robometer.

Questions for authors
  • Why was "Pour" the only task used for the real robot, and do you expect the source ranking to generalize to other tasks that require more precise manipulation (e.g. peg insertion)?
  • Why was Robometer selected for all scaling, fidelity, and RL experiments, while GVL was used only in the matched-data comparison? Would the conclusions change for another trainable reward model? Or, in other words, how generalizable are the findings?
  • It is not very clear from the main text why GVL is not sensitive to the choice of the sources and why Robometer is. Do you have any insight on why that happens? Also, in my opinion if GVL is truly insensitive to the choice of the source, this is a valuable negative result and should be highlighted provided you explain why that happens.
  • Considering GVL is not sensitive to the source, the conclusion should be re-written. Currently it makes claims such "Teleoperated failures provide strong supervision but require substantial robot and operator time", when in reality that is only true for Robometer.
3 comments

We thank the reviewer for the detailed feedback.

Weakness 1 / Question 1 — Task diversity and precision:

We now run RL on three tasks spanning a range of difficulty: Remove Lid (easy, single contact-and-lift), Pour (medium, controlled tilting), and Stack Cups (hard, precise alignment and release). Averaged over three tasks and 20 blind trials each, every adapted reward improves over the base, from 40% to between 60% and 73.3%. On Stack Cups, the base Robometer drops RL below zero-shot (5%), while all adapted rewards recover to 50%+ success. We scope our conclusions to contact-rich manipulation at easy-to-moderate difficulty: our task selection is driven by the practical requirement of having naturally available counterfactual sources and feasible RL evaluation. We agree that high-precision tasks like peg insertion would be a valuable extension, and we state this explicitly as a limitation in the revised paper.

SourceTraj./task for RLTasksTrials/cond.
Kinesthetic (KT)5Remove Lid, Pour, Stack Cups20
Rollout (RO)10Remove Lid, Pour, Stack Cups20
Human Video (HV)10Remove Lid, Pour, Stack Cups20
R2S-HiFi10Remove Lid, Pour, Stack Cups20

RL setup: best offline scale per source. Success improves from 40% (base) to 60–73.3% (adapted) across 3 tasks. See RL results table in jxjf response for the full per-task breakdown.

Weakness 2 / Questions 2, 4 — Contribution claims scoped to Robometer:

We agree the original framing was too broad. Our source-level findings (which source helps most, scaling behavior, RL gains) are specific to Robometer since it is the only trainable reward model in our study. GVL showing weak differentiation between sources is itself a finding worth reporting, but it means the contribution claims about source selection do not extend to it. In the discussion we now explicitly state: "Our trainable reward model is Robometer throughout, which already has good priors; training a reward model from scratch or adapting other VLM-based architectures may shift the source rankings." All source-specific claims in the conclusion are scoped to Robometer. GVL is a few-shot model, and one possible mechanism for its weak source differentiation is that the visual prior from pretraining dominates the few-shot signal; we cannot confirm saturation without a dedicated scaling experiment on GVL. Also, for RL, GVL would be really expensive to run across tasks and sources due to the use of an API model. Since Robometer has a good prior and is easily trainable, we pick that. Robometer is currently a strong baseline and easy to see results with in RL given the limited training budget. There might be a similar trend for other trainable reward models, but we are not sure if they work as well within the RL budget. On Q4 specifically, we accept this critique. Our source-specific training and scaling conclusions are dependent on Robometer. The discussion now states this explicitly, and the conclusion reads: "a small budget of counterfactual data combined with strong priors can improve the reward regardless of the source, and cheaper sources, when scaled carefully, can match expensive ones." All source-ranking claims are scoped to Robometer.

Question 3 — GVL insensitivity to source:

GVL shows weak source differentiation: its trajectory-ranking metric (τ_mean) varies only modestly across sources (0.35-0.41 at 5 traj./task), and its success detection (ROC-AUC_last) does not consistently improve over its own zero-shot baseline for every source. One possible mechanism is that the visual prior from pretraining dominates the few-shot signal, but we cannot confirm this without a dedicated GVL scaling study. We agree this is a valuable finding and highlight it in the discussion. It also motivates why all scaling, fidelity, and RL experiments use Robometer: GVL would be prohibitively expensive to run across tasks and sources through an API model, and the weak differentiation means source selection matters less for it.

We would like to kindly follow up on our rebuttal. We would appreciate it if you could take a look at our responses and let us know if you have any remaining questions or concerns.

Thank you for your time and consideration.

1- The authors have done enough to address the concern about insufficient task diversity for the current paper and explicitly limit their conclusions to easy-to-moderate contact-rich manipulation. 2- The authors acknowledge that the empirical contribution is narrower than originally suggested. 3- I understand the computational argument, however, I would expect the discussion to explicitly distinguish between GVL being insensitive to the choice of source and GVL not benefiting from counterfactual supervision in general. The current results support the former, but do not establish the latter.

In conclusion, if the revised paper consistently frames itself as an empirical study of counterfactual supervision for Robometer, I change my recommendation to weak accept.

Reviewer_jxjfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares four sources of failure data for robot reward learning, i.e., deliberate robot failures collected by an operator, policy rollouts, human videos, and trajectories generated in a reconstructed simulation. The paper's contribution is practical guidance on data sources, the underlying value estimation, reward model, and policy optimization come from GVL, Robometer, and DSRL. Across five tabletop tasks, the paper adapts Robometer-4B with LoRA and test frozen GPT-4o with GVL-style prompting. Evaluation uses 250 held-out trajectories collected by two policies distinct from the training-rollout policy.

Critical strengths

This paper asks a practical question that could make it valuable to the robotics community… which kinds of failure experience are worth collecting for reward learning?

The design has several thoughtful controls. Successful demonstrations are shared across source comparisons…

The study examines both a trainable reward model and in-context value prediction and the offline benchmark includes trajectories from policies distinct from the rollout-data collector.

The presented downstream experiment is useful because it complicates the offline results.

Human-video data can appear competitive on reward metrics yet produce weak policy improvement in the tested RL setting. That observation in the paper must be emphasized.

The blind, randomized policy evaluation with matched starting conditions is to me a genuine strength...

Critical weaknesses

First, the paper is motivated by deciding what to collect under a limited budget, but Tab. 1 and Section 3.3 provide symbolic costs rather than measured operator time, robot occupancy, annotation effort, or simulation setup time. Appendix A.1.4 requires scene reconstruction, collision preparation, milestone definition, and candidate validation. Low marginal simulation cost does not establish low total cost at the tested scale of roughly 5-30 trajectories per task.

A second issue with the allocation argument is the missing success-only adaptation control, i.e., the same successful demonstrations, optimizer, and synthetic temporal supervision, without newly collected failures. Comparing adapted models with a frozen model combines the effect of task adaptation with the effect of collecting new failures. ReWiND already uses synthetic temporal supervision and RoboReward constructs negatives and near-misses from successful videos. Those methods make the inexpensive control consequential.

The main offline figures need reconciliation. In Fig. 2, the average zero-shot Robometer Kendall correlation is approximately 0.40. In Fig. 3, it is approximately 0.22. Several trajectories adapted values also differ. These are approximate readings from the visible plots, but the difference is large, and the captions and Section 3.5 describe the same benchmark and metric without explaining an aggregation change.

The downstream experiment is a useful pilot… but it uses one task,, one reinforcement-learning seed, and selected reward checkpoints with different data counts and seeds (Tab 5-6). Teleoperation and simulation differ by 1 successful trial. Appendix A.3.4 also states that all conditions retain authoritative human sparse rewards and add predicted progress. In my view, a sparse-reward-only DSRL control is still needed to identify their added benefit….

The cross-policy benchmark and blinded, randomized trials are strengths, but they do not resolve the numerical discrepancy or establish a stable ranking from these small downstream differences.

Questions for authors
  1. Please provide the per-task, per-seed values behind Fig. 2-4 and explain why the zero-shot and five-trajectory results differ between Fig. 2 and 3. What Kendall statistic, aggregation weights, and bootstrap unit were used?

  2. What does partial credit mean across sources? Simulation uses a sticky milestone prefix, Robometer receives terminal-frame supervision, and the GVL prompt describes average behavior. How are these targets reconciled for a trajectory that makes progress and then loses it?

  3. What are the total measured costs at the tested dataset sizes, including simulation setup, labeling, discarded candidates, and resets? How do the source-adapted models compare with success-only adaptation using the same optimizer and synthetic supervision?

  4. What exact reward formula combines progress with human outcomes, and how were Tab. 6’s checkpoints selected? Can a sparse-only DSRL control and independent repetitions establish the incremental benefit and stability of the strongest source conditions?

3 comments
AuthorsOpenReview ↗

Author Response Part 1

We thank the reviewer for their thorough assessment.

Weakness 3 / Question 1 — Figure mismatch:

The discrepancy was due to a plotting error: Figure 2 used a different aggregation weighting than Figure 3, which led to inconsistent zero-shot and adapted baselines between the two figures. We have corrected both figures in the revision so they use the same per-task, three-seed-averaged values consistently. The corrected figures do not change any source-level conclusions. We provide the underlying per-task metrics here for full transparency.

τ_mean (↑) for the control experiment (5 traj./task). Mean ± std over seeds 42–44; Robometer zero-shot is a single run:

ModelSourceSpoonTapeRm. LidPourSt. CupsAvg.
RobometerZero-shot0.5870.5070.2930.5320.0610.396
RobometerKT0.688±0.0190.670±0.0680.583±0.0290.623±0.0190.578±0.1080.628±0.038
RobometerRollout0.653±0.0130.595±0.0070.460±0.1210.358±0.0990.377±0.0990.489±0.019
RobometerHuman0.686±0.0130.698±0.0070.529±0.1110.316±0.0860.590±0.0940.564±0.024
RobometerR2S HiFi0.649±0.0110.625±0.0310.318±0.0590.456±0.0770.311±0.1250.472±0.027
GVLZero-shot0.455±0.0150.538±0.067-0.011±0.0830.359±0.0680.402±0.0730.349±0.037
GVLKT0.588±0.0400.619±0.0240.192±0.0750.162±0.1030.486±0.0670.409±0.038
GVLRollout0.530±0.0280.602±0.0550.105±0.0610.408±0.1440.415±0.0760.412±0.047
GVLHuman0.535±0.0400.536±0.0420.145±0.0880.238±0.1880.425±0.0780.376±0.029
GVLR2S HiFi0.544±0.0580.597±0.0270.251±0.0650.165±0.0600.342±0.0250.380±0.029

Success ROC-AUC_last (↑) for the control experiment (5 traj./task):

ModelSourceSpoonTapeRm. LidPourSt. CupsAvg.
RobometerZero-shot0.9931.0000.9550.5710.9960.903
RobometerKT0.987±0.0051.000±0.0000.962±0.0050.784±0.0580.993±0.0090.945±0.013
RobometerRollout0.991±0.0041.000±0.0000.953±0.0080.502±0.2690.993±0.0070.888±0.056
RobometerHuman0.992±0.0060.999±0.0010.950±0.0110.531±0.1610.993±0.0030.893±0.033
RobometerR2S HiFi0.991±0.0040.999±0.0020.949±0.0120.425±0.0480.987±0.0140.870±0.007
GVLZero-shot0.962±0.0260.952±0.0170.884±0.0680.416±0.1380.941±0.0410.831±0.036
GVLKT0.950±0.0140.945±0.0300.929±0.0110.311±0.0630.990±0.0080.825±0.008
GVLRollout0.949±0.0200.954±0.0160.896±0.0560.440±0.0540.972±0.0240.842±0.021
GVLHuman0.924±0.0450.964±0.0170.888±0.0580.305±0.1350.937±0.0440.804±0.034
GVLR2S HiFi0.956±0.0170.968±0.0140.934±0.0210.311±0.0350.987±0.0050.831±0.004

Scaling and sim-fidelity (Robometer, avg. over 5 tasks). τ_mean (↑), MAE_last (↓), ROC-AUC_last (↑):

SourceTraj./taskτ_mean (↑)MAE_last (↓)ROC-AUC_last (↑)
Rollout50.489±0.0190.174±0.0270.888±0.056
Rollout100.613±0.0420.142±0.0190.925±0.050
Rollout200.243±0.0400.308±0.0010.878±0.003
Human50.564±0.0240.165±0.0160.893±0.033
Human100.562±0.0250.146±0.0190.956±0.031
Human200.550±0.0650.166±0.0070.930±0.018
R2S HiFi50.472±0.0270.186±0.0140.870±0.007
R2S HiFi100.539±0.0280.153±0.0090.885±0.019
R2S HiFi200.486±0.0580.158±0.0070.855±0.019
R2S HiFi250.242±0.0140.311±0.0040.879±0.005
R2S HiFi300.253±0.0230.310±0.0010.880±0.002
R2S LoFi50.491±0.0480.165±0.0050.885±0.016
R2S LoFi100.258±0.0170.311±0.0020.870±0.002
AuthorsOpenReview ↗

Author Response Part 2

Weakness 1 / Question 3 — Symbolic costs vs. measured operator time:

We deliberately compare sources at equal trajectory counts to isolate the effect of source quality from quantity, and then use scaling experiments to test whether sources with lower per-episode collection requirements can compensate with more data. Our cost ordering in Tab. 1 uses symbolic cost categories because absolute time measurements are setup-dependent: annotation time varies across annotators, and asset generation time changes with sim2real frameworks. The core claim is about sample efficiency of sources with differing collection requirements, not about which source is cheapest under a fixed budget. For R2S specifically, the per-episode marginal cost is near zero once assets are built, but the fixed setup cost (scene reconstruction, collision mesh, milestone definition) is non-trivial and task-dependent.

Weakness 2 / Question 3 — Success-only adaptation control:

Every condition shares the same successful demonstrations (one per task); the only variable is the failure data source, so the comparison isolates relative source quality. We acknowledge that we do not include a success-only adaptation control in our study. The original Robometer paper ablates the success-only baseline and shows substantial improvement from adding failures, but that ablation uses a different data distribution than ours. Our contribution is therefore about which failure source to collect given that one will collect failures, not about the marginal value of failures over success-only adaptation.

Weakness 4 / Question 4 — RL scope, reward formula, and sparse baseline:

We have expanded the RL experiments from one task to three tasks of increasing difficulty.

ConditionTraj./taskRemove LidPourStack CupsAvg.
Zero-shot π0—50%10%45%35%
DSRL + Base Robometer—80%35%5%40%
DSRL + KT-adapted5100%65%55%73.3%
DSRL + RO-adapted1085%45%70%66.7%
DSRL + HV-adapted1085%30%65%60%
DSRL + R2S-adapted1070%60%50%60%

RL results: success rate over 20 blind trials per condition. Every adapted reward improves over the base; on the hardest task (Stack Cups) the base reward drops RL below zero-shot, while all adapted rewards recover.

The per-step reward is r_t = r_sparse(t) + reward_model(o_t), where r_sparse is −1 per step and 0 at the terminal success timestep; reward_model(o_t) is the predicted absolute task progress from the current observation, clipped to [0, 1]. Checkpoints are selected per source by choosing the training epoch with the best average τ_mean on the held-out validation set. The 20 trials per condition are evaluation repetitions with randomized initial states under a single RL training run per condition, not independent training seeds; we acknowledge this limits our ability to assess variance from the RL optimization itself. Regarding the sparse-only baseline: we use a fixed compute budget of warmup and training steps per task across all sources. We compare against the zero-shot (unadapted) Robometer rather than a sparse-only DSRL control because the sparse-only baseline would need its own hyperparameter search to converge within the same budget, introducing a confound. We acknowledge this as a limitation: the current comparison isolates the effect of source-adapted dense rewards relative to an unadapted dense reward, not relative to sparse-only training.

Question 2 — Partial credit across sources:

Partial credit is P = {0.0, 0.2, 0.4, 0.6, 0.8, 1.0}, graded by humans based on average task progress. For R2S, labels are assigned automatically from sim-state milestones. While we use the same numerical scale across all sources, there is a semantic difference: human annotations reflect average observed progress (a trajectory that reaches 80% and falls back might receive a lower label), whereas R2S milestones are sticky (once a stage is reached the label does not decrease). We acknowledge this as a potential confound when comparing source quality and note it in the revised discussion.

We would like to kindly follow up on our rebuttal. We would appreciate it if you could take a look at our responses and let us know if you have any remaining questions or concerns.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.