- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how to learn a specific visual reward model using only a small number of demonstrations and a single natural language instruction specifying the preference axis. The authors argue that demonstrations alone are insufficient to determine preferences, as numerous irrelevant features can distinguish demonstrations from background trajectories. To address this, the authors propose COVER: first, a video world model is fine-tuned on background trajectories, with trajectory captions derived from pairwise rankings generated by a Vision-Language Model (VLM); next, a counterfactual video is generated starting from the initial frame of each demonstration. The reward model is trained using a combination of a MaxEnt IRL loss and a hinge-based sensitivity loss, the latter requiring the score of each demonstration to exceed that of its counterfactual counterpart. Across four RoboCasa tasks, COVER achieved the highest pairwise ranking accuracy when using N=10 demonstrations per preference. On the CoffeeServeMug task, COVER matched the performance of the strongest baseline while using ten times fewer demonstrations. Ablation studies demonstrate that the "losing" trajectory must differ from the "winning" trajectory along the target preference axis. In experiments involving two real-world Franka tasks, behavior cloning performed on trajectories filtered by the learned reward model yielded higher returns—relative to the ground-truth preference—than training directly on unfiltered data.
Critical strengths
Strengths:
- The paper accurately identifies the identifiability gap in few-shot MaxEnt IRL: when only positive examples are available, many incidental features can distinguish demonstrations from the background. The use of "controlled counterfactuals that alter only the specified preference axis" to provide ranking supervision is conceptually elegant and powerful. Appendix B also establishes the correspondence with constrained maximum entropy problems.
- All baselines share the same reward architecture, optimizer, and instruction conditioning. Ablation studies on "loser" construction clearly demonstrate that losers must take "opposite values along the target axis" to be effective; random backgrounds or cross-preference demonstrations are actually detrimental, a phenomenon explained by correlations present in the training data. Appendix E provides additional comparisons, including VLM pairwise comparisons, RoboReward (zero-shot/fine-tuned), and hinge versus BT losses. Furthermore, the paper reports reward ranking accuracy and the true returns of BC policies trained using the learned rewards. It also transparently presents less favorable results, such as those regarding generation fidelity and human evaluations.
Critical weaknesses
Weaknesses: For each task, COVER requires: (i) using a VLM to perform pairwise comparisons across the background trajectory pool to generate captions; (ii) fully fine-tuning a 2B-parameter video world model (Cosmos-Predict-2.5-2B); and (iii) generating a four-view counterfactual video for each demonstration. The paper only specifies that the world model was fine-tuned for 1,000 steps on 8×H100 GPUs (§4.1, App. D); it reports neither the actual duration of each stage nor the generation time per counterfactual video, nor does it disclose the number of VLM queries (K in §3.3) or the specific "frontier VLM" model employed.
Questions for authors
Questions:
- The authors claim that on the CoffeeServeMug task, COVER achieves accuracy comparable to the strongest baseline (MaxEnt IRL) at N=20 using only N=2 demonstrations (0.872 vs. 0.875), representing a tenfold reduction in required demonstrations (Fig. 5). However, the experiment analyzing performance relative to the number of demonstrations was conducted only on this specific task—which happens to be the one where COVER performs best (0.986 at N=10). Please provide similar plots including all baselines for the remaining three simulation tasks, particularly for CloseFridge, where COVER performs the weakest (0.817 at N=10). Additionally, the maximum N in Fig. 5 is 20. Please extend the baseline comparisons to larger numbers of demonstrations (e.g., N=50, 100) to determine whether COVER's advantages persist when ample demonstrations are available or if they are limited to scenarios with very few demonstrations.
- The paper lacks an "ideal counterfactual" upper bound: performing kinematic perturbations directly on demonstrations in the simulation—altering only the target preference axis while keeping other factors constant—to serve as a substitute for counterfactuals generated by the world model. This would allow for decoupling the value of counterfactual supervision itself from the impact of video generation quality on the results.
- The real-robot policy experiments (Fig. 8) report only the ground-truth preference reward. This metric measures trajectory attributes—such as carrying height, minimum horizontal distance to the laptop, and the height/tilt angle at the peak of the pouring motion (App. F.4)—but does not verify whether the task was actually completed. A failed execution could still yield a "better" preference score.
- In the trajectory pairs provided to the reward model, the world-model-generated trajectory is always designated as the "loser." How would the results differ if the world model were instead tasked with generating the "winner" trajectory for the reward model?
- The results in Fig. 6 and Appendix E.2 are difficult to reconcile; there are significant discrepancies between the results for identical configurations.
3 comments
R3-Q3: task success on the real-robot policy experiments
(1) Task success. Mug transport: a rollout succeeds if the mug is placed within 10 cm of the target location. Each policy is evaluated on 10 start poses, and we train three independent policies with three random seeds per method and preference (see (3)), so each bar in Fig. 8(a) pools 30 rollouts. Wine pour: a rollout succeeds if the gripper holds the bottle and the bottle is positioned and tilted such that the liquid would clearly enter the glass (the nozzle was disabled during evaluation to avoid spilling on the table), evaluated over 7 start poses. We initially planned to evaluate wine pour on 10 start poses, but 3 of them make the robot hit kinematic limits that prevent it from completing the task, so we evaluate on the remaining 7 poses; this is unrelated to the policies. Mug success is 87–100% for every method (Sensitivity 30/30 on both avoid and high), so all policies have similar success rates. On wine pour, Sensitivity succeeds on 6/7 poses for both preferences, above MaxEnt IRL (5/7 and 3/7 on pour gently and pour low), Pixel-Mask (3/7, 3/7) and Uniform (3/7, 3/7), and at or below SFT (6/7, 7/7) and Masking (6/7, 6/7).
(2) Preference return conditional on success. Fig. 8 now averages the ground-truth return over successful rollouts only. On wine pour, Sensitivity has the highest return on pour low, whereas Masking and Pixel-Mask fall below Uniform. On mug transport it improves both preferences with the largest margins: the average planar distance to the laptop increases from 0.030 m (Uniform) to 0.108 m, and the average carry height from 0.141 m to 0.316 m (pooled over the three seeds). Task success alone does not show whether a policy follows the preference: baselines that succeed as often as Sensitivity can still obtain a lower preference return. Overall, Sensitivity achieves the highest preference return on mug transport and on wine pour low while keeping its success rate comparable to the baselines (30/30 on mug transport; 6/7 on wine pour). On wine pour gently, Masking and SFT reach a preference return similar to ours, but Masking and SFT fail to improve pour low (at or below Uniform), which suggests that their reward models are biased toward learning the pour gently preference.
(3) Variability across independently trained policies. In the initial submission, we trained one policy per method and preference. For mug transport, we have now trained three independent policies (random seeds 0, 1, 2) per method and preference, each a LoRA fine-tune on the same reward-filtered episodes, and evaluated each on the same 10 start poses. Revised Fig. 8(a) pools the 30 rollouts of each method and preference, and the table below gives the mean ± SE per method (per-seed breakdown in the updated response PDF). Sensitivity is the best method within every seed on carry high and in two of the three seeds on avoid laptop (seed 2: SFT 0.095 m vs. 0.089 m). The standard error of its mean return across the three seeds is 0.011 m on avoid laptop and 0.023 m on carry high, small relative to its margin over Uniform (0.078 m and 0.175 m). Task success is 87–100% for every method when pooled over seeds (no method-seed cell below 8/10). Due to time constraints we ran this multi-seed study on mug transport only; we will repeat it for wine pour in the camera-ready version.
| Method | avoid laptop | carry high |
|---|---|---|
| Uniform | 0.030 ± 0.010 | 0.141 ± 0.009 |
| SFT | 0.094 ± 0.012 | 0.126 ± 0.007 |
| MaxEnt IRL | 0.054 ± 0.008 | 0.217 ± 0.006 |
| Pixel-Mask + Raw RGB | 0.060 ± 0.010 | 0.268 ± 0.007 |
| Masking | 0.027 ± 0.007 | 0.203 ± 0.006 |
| Sensitivity (ours) | 0.108 ± 0.008 | 0.316 ± 0.008 |
Rebuttal Table 2: Mug transport, success-conditioned return (m), mean ± SE over 30 rollouts (3 seeds × 10 poses); higher is better.
R3-Q4: what if the world model generated the winner?
If the generation were a second winner (or both sides were generated and equally preferred), the pair's correct Bradley–Terry label is , and a tied ranking pair reduces exactly to an invariance constraint, which is the Masking-loss baseline already reported: it trails the sensitivity loss on every task in Fig. 4. Keeping the real demonstration on the winning side anchors the ordering to verified preferred behavior and places any generation artifact on the side that is dispreferred by construction.
Additionally, from the user's perspective, generating the loser rather than the winner is more practical, since asking users to execute dispreferred trajectories in the physical world is often undesirable. First, physically executing dispreferred behavior on a robot can be dangerous, whereas generating it as a video poses no physical risk. Second, people are not motivated to show dispreferred trajectories because they are dispreferred.
R3-Q5: Fig. 6 vs. App. E.2
We apologize that the App. E.2 / Fig. 12 numbers were outdated results from an older implementation, which is why they did not match Fig. 6. We redid all of the experiments in App. E.2 with the current implementation of our method (sensitivity loss and counterfactuals generated with demo off-target axis), under the same protocol and evaluation pairs as Fig. 6, and now report the correct results (Rebuttal Fig. 2). The numbers are now consistent with revised Figs. 4 and 6. Hinge and Bradley–Terry, trained on identical sensitivity pairs, demonstrations, and reward model, perform similarly for lift_high and obstacle_dist, but the hinge is clearly better on the subtle upright axis, and thus outperforms Bradley–Terry in the average performance.
One explanation consistent with this result is that the hinge loss of a pair is exactly zero once the winner outscores the loser by the margin, so pairs that are already separated stop contributing gradient. The visually salient axes (lift_high, obstacle_dist) likely reach the margin early in training, after which the gradient comes mostly from the pairs not yet separated, which are the upright pairs. The logistic loss has a nonzero gradient on every pair, so salient-axis pairs may keep updating the shared reward model and interfere with learning the subtle upright readout. We keep the hinge, which also admits the constrained maximum-entropy reading of App. B. The revised App. E.2 text states this conclusion.
Thank you for the review. New or changed text is in blue in the revised manuscript. The figures and tables referred to below (Rebuttal Figures and Tables) are in the supplementary response PDF.
R3-W1: compute cost transparency
We now report all four, in App. D and in brief here:
Every stage is one-time per task and label-free. Adapting to a new preference afterwards reuses the captions, the fine-tuned world model, and the counterfactual pool, so it costs only reward training: about 25 s per run on one consumer GPU.
R3-Q1: demonstration-budget sweeps on the other tasks and larger N
Added (revised Fig. 5, Rebuttal Fig. 6): all five methods on all four tasks at N∈{2,5,10,20,50}. Both variants of our method rank first on every task at every N. With five demonstrations per preference, Demo Off-target Axis already matches or exceeds the best baseline at N=20 on all RoboCasa tasks, thereby improving data efficiency 4x. More demonstrations do not help every baseline: MaxEnt IRL and Masking fall to 0.48–0.49 on OpenCabinet at N=50, and Masking and Pixel-Mask decline on KettleBoiling beyond N=10. The gap narrows at large N on CloseFridge (Pixel-Mask 0.849 vs. 0.899 for ours at N=50) and CoffeeServeMug (ours saturates at 1.000, MaxEnt IRL reaches 0.911), but ours still outperform baselines.
R3-Q2: an "ideal counterfactual" upper bound
We agree that this is the oracle upper bound, since it takes video generation out of the loop. We generate counterfactuals kinematically in simulation as the reviewer describes: each is a kinematic deformation of a demonstration, where its ground truth reward along the target preference becomes lowest while maintaining others same. We train the identical pipeline (same sensitivity loss, reward model, N=10 demonstrations, and 5 seeds) on these kinematically generated counterfactuals instead of world-model-generated ones.
Kinematic counterfactuals on all four tasks (new Fig. 14 of the revised PDF; Rebuttal Fig. 7). Winners and losers are kinematically generated clips ranked by the ground-truth score on the target axis. They are at or above ours on every task. The gap is small on KettleBoiling (0.98 vs. 0.96), CoffeeServeMug (0.995 vs. 0.98) and OpenCabinet (0.998 vs. 0.99). The largest gap is on CloseFridge (0.998 vs. 0.91), because
contact_mid_heightis the hardest preference to generate counterfactuals using world models since the cabinet handle is too small. As the reviewer suggested, this experiment allows us to know the effectiveness of our counterfactual simulation decoupled from video generation quality.