acceptodds
Under review as a conference paper at ICLR 2027

What If? Counterfactual Simulation for Sample-Efficient Visual Reward Learning

Abstract

Pretrained visual robot reward models capture generic notions of task success, not necessarily a person's specific preferences for how their task should be done. The robot can learn these preferences from user demonstrations, but with only a few demonstrations, a reward model may latch onto spurious correlations without learning what the person actually wants. To better express their preferences, a person could directly specify a preference axis—or an aspect of the task that they care about—in language. But a language-conditioned reward model may still reward the wrong behavior when training demonstrations don't fully cover the entire preference axis. Our key idea is to use the person's instruction to cover this axis by imagining what they don't want. We introduce COVER, which uses a video world model to generate counterfactual trajectories: alternatives that are not preferred along the preference axis but keep the rest of the behavior and scene the same. Comparing the original demonstration against its counterfactual isolates the quality that should make the reward different, without asking the person to demonstrate both versions. We train the reward to score each demonstration higher than its corresponding counterfactual, alongside learning from the demonstrations directly. Across four simulated and two real-robot tasks, COVER learns rewards that capture user preferences better than language-conditioned baselines under the same demonstration budgets, matching the strongest baseline's accuracy with up to 10× fewer demonstrations. On a real robot, selecting training demonstrations with these rewards yields policies that follow both stated preferences of each task more closely than training on unfiltered demonstrations.

Primary area:
applications to robotics, autonomy, planning
Keywords:
reward learning, inverse reinforcement learning, preference learning, personalization, world models, counterfactual generation, robot manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KP37OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how to learn a specific visual reward model using only a small number of demonstrations and a single natural language instruction specifying the preference axis. The authors argue that demonstrations alone are insufficient to determine preferences, as numerous irrelevant features can distinguish demonstrations from background trajectories. To address this, the authors propose COVER: first, a video world model is fine-tuned on background trajectories, with trajectory captions derived from pairwise rankings generated by a Vision-Language Model (VLM); next, a counterfactual video is generated starting from the initial frame of each demonstration. The reward model is trained using a combination of a MaxEnt IRL loss and a hinge-based sensitivity loss, the latter requiring the score of each demonstration to exceed that of its counterfactual counterpart. Across four RoboCasa tasks, COVER achieved the highest pairwise ranking accuracy when using N=10 demonstrations per preference. On the CoffeeServeMug task, COVER matched the performance of the strongest baseline while using ten times fewer demonstrations. Ablation studies demonstrate that the "losing" trajectory must differ from the "winning" trajectory along the target preference axis. In experiments involving two real-world Franka tasks, behavior cloning performed on trajectories filtered by the learned reward model yielded higher returns—relative to the ground-truth preference—than training directly on unfiltered data.

Critical strengths

Strengths:

  1. The paper accurately identifies the identifiability gap in few-shot MaxEnt IRL: when only positive examples are available, many incidental features can distinguish demonstrations from the background. The use of "controlled counterfactuals that alter only the specified preference axis" to provide ranking supervision is conceptually elegant and powerful. Appendix B also establishes the correspondence with constrained maximum entropy problems.
  2. All baselines share the same reward architecture, optimizer, and instruction conditioning. Ablation studies on "loser" construction clearly demonstrate that losers must take "opposite values ​​along the target axis" to be effective; random backgrounds or cross-preference demonstrations are actually detrimental, a phenomenon explained by correlations present in the training data. Appendix E provides additional comparisons, including VLM pairwise comparisons, RoboReward (zero-shot/fine-tuned), and hinge versus BT losses. Furthermore, the paper reports reward ranking accuracy and the true returns of BC policies trained using the learned rewards. It also transparently presents less favorable results, such as those regarding generation fidelity and human evaluations.
Critical weaknesses

Weaknesses: For each task, COVER requires: (i) using a VLM to perform pairwise comparisons across the background trajectory pool to generate captions; (ii) fully fine-tuning a 2B-parameter video world model (Cosmos-Predict-2.5-2B); and (iii) generating a four-view counterfactual video for each demonstration. The paper only specifies that the world model was fine-tuned for 1,000 steps on 8×H100 GPUs (§4.1, App. D); it reports neither the actual duration of each stage nor the generation time per counterfactual video, nor does it disclose the number of VLM queries (K in §3.3) or the specific "frontier VLM" model employed.

Questions for authors

Questions:

  1. The authors claim that on the CoffeeServeMug task, COVER achieves accuracy comparable to the strongest baseline (MaxEnt IRL) at N=20 using only N=2 demonstrations (0.872 vs. 0.875), representing a tenfold reduction in required demonstrations (Fig. 5). However, the experiment analyzing performance relative to the number of demonstrations was conducted only on this specific task—which happens to be the one where COVER performs best (0.986 at N=10). Please provide similar plots including all baselines for the remaining three simulation tasks, particularly for CloseFridge, where COVER performs the weakest (0.817 at N=10). Additionally, the maximum N in Fig. 5 is 20. Please extend the baseline comparisons to larger numbers of demonstrations (e.g., N=50, 100) to determine whether COVER's advantages persist when ample demonstrations are available or if they are limited to scenarios with very few demonstrations.
  2. The paper lacks an "ideal counterfactual" upper bound: performing kinematic perturbations directly on demonstrations in the simulation—altering only the target preference axis while keeping other factors constant—to serve as a substitute for counterfactuals generated by the world model. This would allow for decoupling the value of counterfactual supervision itself from the impact of video generation quality on the results.
  3. The real-robot policy experiments (Fig. 8) report only the ground-truth preference reward. This metric measures trajectory attributes—such as carrying height, minimum horizontal distance to the laptop, and the height/tilt angle at the peak of the pouring motion (App. F.4)—but does not verify whether the task was actually completed. A failed execution could still yield a "better" preference score.
  4. In the trajectory pairs provided to the reward model, the world-model-generated trajectory is always designated as the "loser." How would the results differ if the world model were instead tasked with generating the "winner" trajectory for the reward model?
  5. The results in Fig. 6 and Appendix E.2 are difficult to reconcile; there are significant discrepancies between the results for identical configurations.
3 comments

Thank you for the review. New or changed text is in blue in the revised manuscript. The figures and tables referred to below (Rebuttal Figures and Tables) are in the supplementary response PDF.

R3-W1: compute cost transparency

We now report all four, in App. D and in brief here:

  • VLM and KK. The caption VLM is OpenAI Codex CLI (gpt-5.6-terra), used frozen; it answers K=8K{=}8 pairwise comparisons per clip per axis (1616 on the KettleBoiling lift axis), i.e. 1.7k–14k queries per task. Now also described in §4.1.
  • Generation time per counterfactual video. 1.3–2.7 min per four-view clip per GPU (H100 or RTX 5090); the pools are 78–624 clips per task, 22–98 min on an 8×8\timesH100 node.
  • Duration of each stage (range over the four tasks): VLM captioning 0.9–7.2 h of API wall-clock; world-model fine-tuning 50–255 min on 8×8\timesH100; counterfactual generation 22–98 min; reward training ~2–3 min for 5 seeds on one GPU. End to end, 3–11 h serial wall-clock and 14–47 H100 GPU-hours per task.

Every stage is one-time per task and label-free. Adapting to a new preference afterwards reuses the captions, the fine-tuned world model, and the counterfactual pool, so it costs only reward training: about 2525 s per run on one consumer GPU.

R3-Q1: demonstration-budget sweeps on the other tasks and larger NN

Added (revised Fig. 5, Rebuttal Fig. 6): all five methods on all four tasks at N∈{2,5,10,20,50}N \in \{2,5,10,20,50\}. Both variants of our method rank first on every task at every NN. With five demonstrations per preference, Demo Off-target Axis already matches or exceeds the best baseline at N=20N{=}20 on all RoboCasa tasks, thereby improving data efficiency 44x. More demonstrations do not help every baseline: MaxEnt IRL and Masking fall to 0.480.48–0.490.49 on OpenCabinet at N=50N{=}50, and Masking and Pixel-Mask decline on KettleBoiling beyond N=10N{=}10. The gap narrows at large NN on CloseFridge (Pixel-Mask 0.8490.849 vs. 0.8990.899 for ours at N=50N{=}50) and CoffeeServeMug (ours saturates at 1.0001.000, MaxEnt IRL reaches 0.9110.911), but ours still outperform baselines.

R3-Q2: an "ideal counterfactual" upper bound

We agree that this is the oracle upper bound, since it takes video generation out of the loop. We generate counterfactuals kinematically in simulation as the reviewer describes: each is a kinematic deformation of a demonstration, where its ground truth reward along the target preference becomes lowest while maintaining others same. We train the identical pipeline (same sensitivity loss, reward model, N=10N{=}10 demonstrations, and 55 seeds) on these kinematically generated counterfactuals instead of world-model-generated ones.

Kinematic counterfactuals on all four tasks (new Fig. 14 of the revised PDF; Rebuttal Fig. 7). Winners and losers are kinematically generated clips ranked by the ground-truth score on the target axis. They are at or above ours on every task. The gap is small on KettleBoiling (0.980.98 vs. 0.960.96), CoffeeServeMug (0.9950.995 vs. 0.980.98) and OpenCabinet (0.9980.998 vs. 0.990.99). The largest gap is on CloseFridge (0.9980.998 vs. 0.910.91), because contact_mid_height is the hardest preference to generate counterfactuals using world models since the cabinet handle is too small. As the reviewer suggested, this experiment allows us to know the effectiveness of our counterfactual simulation decoupled from video generation quality.

R3-Q3: task success on the real-robot policy experiments

(1) Task success. Mug transport: a rollout succeeds if the mug is placed within 10 cm of the target location. Each policy is evaluated on 10 start poses, and we train three independent policies with three random seeds per method and preference (see (3)), so each bar in Fig. 8(a) pools 30 rollouts. Wine pour: a rollout succeeds if the gripper holds the bottle and the bottle is positioned and tilted such that the liquid would clearly enter the glass (the nozzle was disabled during evaluation to avoid spilling on the table), evaluated over 7 start poses. We initially planned to evaluate wine pour on 10 start poses, but 3 of them make the robot hit kinematic limits that prevent it from completing the task, so we evaluate on the remaining 7 poses; this is unrelated to the policies. Mug success is 87–100% for every method (Sensitivity 30/30 on both avoid and high), so all policies have similar success rates. On wine pour, Sensitivity succeeds on 6/7 poses for both preferences, above MaxEnt IRL (5/7 and 3/7 on pour gently and pour low), Pixel-Mask (3/7, 3/7) and Uniform (3/7, 3/7), and at or below SFT (6/7, 7/7) and Masking (6/7, 6/7).

(2) Preference return conditional on success. Fig. 8 now averages the ground-truth return over successful rollouts only. On wine pour, Sensitivity has the highest return on pour low, whereas Masking and Pixel-Mask fall below Uniform. On mug transport it improves both preferences with the largest margins: the average planar distance to the laptop increases from 0.030 m (Uniform) to 0.108 m, and the average carry height from 0.141 m to 0.316 m (pooled over the three seeds). Task success alone does not show whether a policy follows the preference: baselines that succeed as often as Sensitivity can still obtain a lower preference return. Overall, Sensitivity achieves the highest preference return on mug transport and on wine pour low while keeping its success rate comparable to the baselines (30/30 on mug transport; 6/7 on wine pour). On wine pour gently, Masking and SFT reach a preference return similar to ours, but Masking and SFT fail to improve pour low (at or below Uniform), which suggests that their reward models are biased toward learning the pour gently preference.

(3) Variability across independently trained policies. In the initial submission, we trained one policy per method and preference. For mug transport, we have now trained three independent policies (random seeds 0, 1, 2) per method and preference, each a π0.5\pi_{0.5} LoRA fine-tune on the same reward-filtered episodes, and evaluated each on the same 10 start poses. Revised Fig. 8(a) pools the 30 rollouts of each method and preference, and the table below gives the mean ± SE per method (per-seed breakdown in the updated response PDF). Sensitivity is the best method within every seed on carry high and in two of the three seeds on avoid laptop (seed 2: SFT 0.095 m vs. 0.089 m). The standard error of its mean return across the three seeds is 0.011 m on avoid laptop and 0.023 m on carry high, small relative to its margin over Uniform (0.078 m and 0.175 m). Task success is 87–100% for every method when pooled over seeds (no method-seed cell below 8/10). Due to time constraints we ran this multi-seed study on mug transport only; we will repeat it for wine pour in the camera-ready version.

Methodavoid laptopcarry high
Uniform0.030 ± 0.0100.141 ± 0.009
SFT0.094 ± 0.0120.126 ± 0.007
MaxEnt IRL0.054 ± 0.0080.217 ± 0.006
Pixel-Mask + Raw RGB0.060 ± 0.0100.268 ± 0.007
Masking0.027 ± 0.0070.203 ± 0.006
Sensitivity (ours)0.108 ± 0.0080.316 ± 0.008

Rebuttal Table 2: Mug transport, success-conditioned return (m), mean ± SE over 30 rollouts (3 seeds × 10 poses); higher is better.

R3-Q4: what if the world model generated the winner?

If the generation were a second winner (or both sides were generated and equally preferred), the pair's correct Bradley–Terry label is 12\tfrac12, and a tied ranking pair reduces exactly to an invariance constraint, which is the Masking-loss baseline already reported: it trails the sensitivity loss on every task in Fig. 4. Keeping the real demonstration on the winning side anchors the ordering to verified preferred behavior and places any generation artifact on the side that is dispreferred by construction.

Additionally, from the user's perspective, generating the loser rather than the winner is more practical, since asking users to execute dispreferred trajectories in the physical world is often undesirable. First, physically executing dispreferred behavior on a robot can be dangerous, whereas generating it as a video poses no physical risk. Second, people are not motivated to show dispreferred trajectories because they are dispreferred.

R3-Q5: Fig. 6 vs. App. E.2

We apologize that the App. E.2 / Fig. 12 numbers were outdated results from an older implementation, which is why they did not match Fig. 6. We redid all of the experiments in App. E.2 with the current implementation of our method (sensitivity loss and counterfactuals generated with demo off-target axis), under the same protocol and evaluation pairs as Fig. 6, and now report the correct results (Rebuttal Fig. 2). The numbers are now consistent with revised Figs. 4 and 6. Hinge and Bradley–Terry, trained on identical sensitivity pairs, demonstrations, and reward model, perform similarly for lift_high and obstacle_dist, but the hinge is clearly better on the subtle upright axis, and thus outperforms Bradley–Terry in the average performance.

One explanation consistent with this result is that the hinge loss of a pair is exactly zero once the winner outscores the loser by the margin, so pairs that are already separated stop contributing gradient. The visually salient axes (lift_high, obstacle_dist) likely reach the margin early in training, after which the gradient comes mostly from the pairs not yet separated, which are the upright pairs. The logistic loss has a nonzero gradient on every pair, so salient-axis pairs may keep updating the shared reward model and interfere with learning the subtle upright readout. We keep the hinge, which also admits the constrained maximum-entropy reading of App. B. The revised App. E.2 text states this conclusion.

Reviewer_CEqCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper finetunes a video model on carefully captioned demos to generate counterfactual trajectories. These are then used to train a reward model using an Inverse-RL formulation, which are then used to improve policies.

Critical strengths

Good ablations: The paper properly ablates alternative masking/augmentation schemes for data generation along with other methods to generate negative data. Additional ablations provided for using Hinge v/s Bradly-terry for preference losses are also appreciated.

Critical weaknesses
  1. Tasks used both in simulation and real are very primitive. We, as a field, can now do much more contact-rich and long-horizon tasks and evaluation on such tasks is missing. Some results on real-world tasks like folding cloth, sorting blocks etc would improve the paper.
  2. Since the video model is also trained on "negative" data, one important baseline emerges: Rather than prompting the world model to generate reference trajectories, just retrieve trajectories from the data (whenever it is there in the dataset) and use them downstream. This is needed to establish that the generalization coming from the video model training is actually helping. Note that retrieval will always give some videos because the method uses the same structured caption for training and generation.
Questions for authors

Please address all questions in the weakness section

1 comment

Thank you for the review. New or changed text is in blue in the revised manuscript. The figures and tables referred to below (Rebuttal Figures and Tables) are in the supplementary response PDF.

R2-W2, R2-Q1: retrieval instead of generation

We thank the reviewer for suggesting a promising baseline. We implemented the strongest version of this baseline (App. E.4; Counter Background Retrieval in the revised Fig. 6). For each demonstration and target caption, retrieval candidates are every held-in pool clip whose caption flips the prescribed bucket on the target axis and keeps the demonstration's off-target-axis buckets, and we retrieve the clip nearest to the demonstration's first frame in the reward encoder's embedding space. Retrieved losers replace generated ones using the same sensitivity loss.

TaskMaxEnt IRLRetrievalGeneration
KettleBoiling0.880 ± 0.0180.849 ± 0.0050.962 ± 0.006
OpenCabinet0.678 ± 0.0110.906 ± 0.0160.993 ± 0.003
CloseFridge0.417 ± 0.0020.571 ± 0.0040.905 ± 0.005
CoffeeServeMug0.843 ± 0.0260.950 ± 0.0020.979 ± 0.002

Rebuttal Table 2: Retrieved vs. generated counterfactuals (Demo Off-target Axis) on the four RoboCasa tasks. We report average pairwise ranking accuracy (mean ± 1 SE over 5 seeds). Experiment setup is same with the revised Fig. 4.

Retrieval reaches 0.8490.849 vs. 0.9620.962 for generation on KettleBoiling, below even MaxEnt IRL, and drops sharply on the subtle upright axis (0.5950.595 vs. 0.9640.964). A retrieved pair simply differs in many uncontrolled properties such as the initial pose and scene configurations. In contrast, generation is seeded on the demonstration's first frame and prompted to change only the target axis, so the pair differs mainly along that axis. Generation shows the best performance across all four tasks. The margin is small on OpenCabinet and CoffeeServeMug because those tasks used smaller space thus background trajectories were covering the scene more densely, so retrieval therefore returns a near-counterfactual.

R2-W1: task scope

We thank the reviewer for encouraging us to broaden the task scope. Our method is task-agnostic, and we first wanted to show preference learning on tasks that the base policies complete reliably, so that capability failures do not confound preference adherence.

The difficulty in these tasks lies in the preferences rather than the manipulation: zero-shot, RoboReward-4B ranks our preference pairs at chance (0.44–0.54), and a frontier VLM's pairwise judgments agree with ground truth at or below chance on cabinet handle-grasp position (0.40) and fridge door order (0.49) (App. E.1). Following the suggestion, we add a contact-rich, deformable-object task, ClothFold (App. E.5, Rebuttal Fig. 5). ClothFold is a Franka folding a T-shirt simulated in Newton [1], an open-source GPU-accelerated physics engine built on NVIDIA Warp. Due to the time constraint, we add it as a simulation experiment, and we plan to apply our method to real robots in the future.

Preferences. The three ClothFold preferences concern how the pinched hem is carried from the grasp to the release, and are drawn in Rebuttal Fig. 5a (solid: preferred behavior; dashed: less preferred alternative). lift_high: the hem is lifted high above the table while it is carried (the vertical arrows from the current end-effector position; measured as the peak carry height). fold_deep: the hem is carried far across the shirt, all the way toward the collar, rather than stopping short (the two arcs along the shirt's mid-line; measured as the horizontal carry distance). gentle_release: the hem is lowered close to the shirt before the gripper opens, rather than dropped from above (the two release markers; measured as the release height).

Dataset. The unlabeled pool consists of 476476 simulated clips (428428 held-in and 4848 held-out), rendered from two camera views. The reward model is trained on N=10N{=}10 demonstrations per preference with the held-in pool as background, and ranking accuracy is evaluated on 400400 pairs per preference.

With N=10N{=}10 demonstrations per preference, both variants of our method reach 0.980.98 average pairwise ranking accuracy, compared with 0.930.93 for SFT, 0.910.91 for the masking loss, and 0.880.88–0.890.89 for MaxEnt IRL and Pixel Mask + RGB (Rebuttal Fig. 5b). Each baseline fails on one preference: MaxEnt IRL, Pixel Mask + RGB, and the masking loss drop to 0.700.70–0.780.78 on gentle_release when the lift height opposes it, and SFT drops to 0.830.83 on fold_deep, whereas the sensitivity pairs reach 0.990.99 on gentle_release and our worst preference is 0.950.95.

[1] The Newton Developers. Newton: a GPU-accelerated physics engine for robotics simulation. https://github.com/newton-physics/newton, 2025.

Reviewer_XSHNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces COVER, which learns visual rewards from a few demonstrations and generated counterfactual trajectories. Targeted negatives reduce ambiguity about the intended preference. Experiments evaluate reward ranking and downstream policies in simulation and on real robots.

Critical strengths
  1. Personalized reward learning from limited demonstrations is important, and targeted negatives are reasonably motivated.
  2. Comparisons with random negatives, cross-preference demonstrations, and VLM comparisons help examine the source of improvement.
  3. Real-robot experiments evaluate both rewards and downstream policies.
Critical weaknesses
  1. I have concerns about whether counterfactuals isolate the intended factor. Section 3.3 sets other axes to “moderate,” potentially changing the original demonstration. Appendix D, Table 2 reports only 0.40 off-axis preservation for CLOSEFRIDGE contact height. This weakens the single-factor explanation; a control preserving the demonstration's other axes is needed.
  2. The core results contain unexplained differences. Figure 6 reports upright accuracy of 0.89, whereas Appendix E.2 and Figure 12 report approximately 0.62–0.66 under the full objective. If these use different configurations, the data splits, generation settings, and training conditions should be specified to reconcile the main results and ablations.
  3. Policy improvements need task-success evidence. Section 4.4, Figure 8 mainly reports height, clearance, and tilt; smaller tilt does not necessarily indicate successful pouring. Appendix F.4 also specifies one trained policy per condition. Please report task success, preference return conditional on success, and variability across independently trained policies.
Questions for authors
  1. What explains the difference between Figures 6 and 12?
  2. Do reward gains persist when preserving the other axes?
  3. Are preference improvements compatible with task success and reproducible across policy-training runs?
3 comments

Thank you for the review. New or changed text is in blue in the revised manuscript. The figures and tables referred to below (Rebuttal Figures and Tables) are in the supplementary response PDF.

R1-W1, R1-Q2: Do the counterfactuals isolate the intended factor? Do gains persist when the other axes are preserved?

We agree that captioning the off-target axes as "moderate" can change the generated behavior along the other preference axes whenever the demonstration sits at an extreme on one of them. Table 2's 0.400.40 off-target axis preservation on CloseFridge contact_mid_height partly reflects that effect, since the rater judged preservation against the demonstration.

We revised the construction in §3.3 so that the off-target axes are kept as in the demonstration: a VLM watches the demonstration and names its level on every off-target axis, the counterfactual generation uses those levels and negates only the target axis. This controlled counterfactual construction is what Fig. 4 and §4.2 now report as ours (Sensitivity Loss (Ours, Demo Off-target Axis)); the initial submission's moderate setting is kept as an ablation (Sensitivity Loss (Ours, Moderate Off-target Axis)).

Compared to the original method, the performance improves the most on open_left_first in OpenCabinet from 0.8330.833 to 0.9880.988 and contact_mid_height in CloseFridge from 0.5830.583 to 0.7300.730. For KettleBoiling and CoffeeServeMug, the performance stays similar within one standard error (see revised Fig. 4, or Rebuttal Fig. 1).

Gains also persist when we preserve the other axes exactly in simulation. Kinematic counterfactuals in simulation, which change only the target axis by construction and represent an upper-bound on performance, reach 0.98-1.00 on all four tasks, and our generated counterfactuals are within 0.02 of them on KettleBoiling, CoffeeServeMug, and OpenCabinet (new Fig. 14; details in our response to R3-Q2).

Off-target Axis preservation (Table 2). On the preference axis the reviewer pointed out, CloseFridge contact_mid _height off-target axis preservation improves from 0.400.40 to 0.700.70 under the revised construction, with edit success 0.900.90 on the target axis (Table below). We are re-running the human evaluation of Table 2 on the controlled counterfactuals and will report the fully revised table in the final paper.

TaskAxisEdit successOff-target Axis preservation
CloseFridgecontact_mid_height0.900.70

Rebuttal Table 1: Human evaluation with the Demo Off-target Axis construction on 10 demonstration–counterfactual pairs: edit success and off-target axis preservation (Table 2 of the submission: 0.80 / 0.40).

R1-W2, R1-Q1: Fig. 6 vs. Fig. 12 / App. E.2

We apologize that the App. E.2 / Fig. 12 numbers were outdated results from an older implementation, which is why they did not match Fig. 6. We redid all of the experiments in App. E.2 with the current implementation of our method (sensitivity loss and counterfactuals generated with demo off-target axis), under the same protocol and evaluation pairs as Fig. 6, and now report the correct results (Rebuttal Fig. 2). The numbers are now consistent with revised Figs. 4 and 6. Hinge and Bradley–Terry, trained on identical sensitivity pairs, demonstrations, and reward model, perform similarly for lift_high and obstacle_dist (BT marginally ahead), but the hinge is clearly better on the subtle upright axis, and thus outperforms Bradley–Terry in the average performance.

One explanation consistent with this result is that the hinge loss of a pair is exactly zero once the winner outscores the loser by the margin, so pairs that are already separated stop contributing gradient. The visually salient axes (lift_high, obstacle_dist) likely reach the margin early in training, after which the gradient comes mostly from the pairs not yet separated, which are the upright pairs. The logistic loss has a nonzero gradient on every pair, so salient-axis pairs may keep updating the shared reward model and interfere with learning the subtle upright readout. We keep the hinge, which also admits the constrained maximum-entropy reading of App. B. The revised App. E.2 text states this conclusion.

R1-W3, R1-Q3: task success and variability across policy-training runs

(1) Task success. Mug transport: a rollout succeeds if the mug is placed within 1010 cm of the target location, evaluated over 17 start poses. Wine pour: a rollout succeeds if the gripper holds the bottle and the bottle is positioned and tilted such that the liquid would clearly enter the glass (the nozzle was disabled during evaluation to avoid spilling on the table), evaluated over 7 start poses. We initially planned to evaluate on 2020 start poses for mug transport and 1010 for wine pour, but 33 of the 2020 and 33 of the 1010 start poses, respectively, resulted in the robot hitting kinematic limits that prevent it from completing the task, so we evaluate on 1717 and 77 poses; this is unrelated to the policies. Mug success is 8888–9494% for every method (Sensitivity 15/1715/17 on avoid, 16/1716/17 on high), so all policies have similar success rates. On wine pour, Sensitivity succeeds on 6/76/7 poses for both preferences, above MaxEnt IRL (5/75/7 and 3/73/7 on pour gently and pour low), Pixel-Mask (3/73/7, 3/73/7) and Uniform (3/73/7, 3/73/7), and at or below SFT (6/76/7, 7/77/7) and Masking (6/76/7, 6/76/7).

(2) Preference return conditional on success. Fig. 8 now averages the ground-truth return over successful rollouts only. On wine pour, Sensitivity has the highest return on pour low, whereas Masking and Pixel-Mask fall below Uniform. On mug transport it improves both preferences with the largest margins: the average planar distance to the laptop increases from 0.0420.042 m (Uniform) to 0.0850.085 m, and the average carry height from 0.1160.116 m to 0.2890.289 m. Task success alone does not show whether a policy follows the preference: baselines that succeed as often as Sensitivity can still obtain a lower preference return, i.e., they complete the task without following the preference. Overall, Sensitivity achieves the highest preference return on mug transport and on wine pour low while keeping its success rate comparable to the baselines (8888–9494% on mug transport, 6/76/7 on wine pour). On wine pour gently, Masking and SFT reach a preference return similar to ours, but Masking and SFT fail to improve pour low (at or below Uniform), which suggests that their reward models are biased toward learning the pour gently preference.

(3) Variability across independently trained policies. In the initial submission, we trained one policy per method and preference using a single seed due to time constraints. We will train 3 independent seeds per method and report success and preference return with standard errors across seeds; we will share these real-robot results as soon as they are available and include them in the final paper.

Thank you for your detailed reply. Most of my concerns have been addressed, therefore, I raise my score to 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers: summary of revisions

We thank all reviewers for their careful and constructive reviews, which helped us strengthen the paper. We have uploaded a revised manuscript that adds new results and clarifications; new or changed text is shown in blue in the revised PDF. We have also uploaded our full author response (PDF) as supplementary material, which contains the per-reviewer answers together with the new figures and tables. Below is a summary of the main additions to the paper:

  • [R1-W1] Controlled counterfactuals: For a given preference, the target axis is the axis the reward should be sensitive to, and the off-target axes are all the other axes, which a counterfactual should leave unchanged. The off-target axes are kept as in the demonstration's levels, and this is our main method; the original moderate setting is kept as an ablation.
  • [R1-W2, R3-Q5] Hinge (ours) vs. Bradley-Terry: experiments in App. E.2 are redone and are now consistent with Fig. 6.
  • [R1-W3, R3-Q3] Real-robot policies: we added a report of task success and return on successful rollouts.
  • [R2-W2] Retrieval baseline: caption-matched retrieval on all four tasks, compared with generation.
  • [R2-W1] New task: we present a contact-rich, deformable-object cloth-folding task in simulation.
  • [R3-W1] Compute cost: we report VLM latency, number of queries KK, generation time, and stage durations.
  • [R3-Q1] Demonstration budget: we report sweeps on all four tasks up to N=50N{=}50.
  • [R3-Q2] Upper bound: we present results with kinematically generated ground-truth counterfactual trajectories in joint space and rendering them in simulation; we answer the winner-generation question in R3-Q4.

For the final version, we will train three independent seeds per method and preference for the real-robot policies and report mean and standard error across seeds, and we will complete the human evaluation of off-target axis preservation under the controlled construction. Detailed point-by-point answers are in each reviewer's thread and in the supplementary response PDF.

Update (Nov 18). For mug transport, we have now trained three independent policies per method and preference (random seeds 0, 1, 2; π0.5\pi_{0.5} fine-tuned with LoRA on the same reward-filtered episodes) and evaluated each on the same 10 start poses, so every bar of Fig. 8(a) pools 30 rollouts; per-seed success and return are in Rebuttal Table 2 of the updated response PDF. Sensitivity is the best method within every seed on carry high and in two of three seeds on avoid laptop, with task success of 87–100% for every method and seed (Sensitivity 30/30 on both preferences). Due to time constraints we ran this multi-seed study on mug transport only; we will repeat it for wine pour, and complete the human evaluation of off-target axis preservation under the controlled construction, for the camera-ready version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.