- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces STEAM, a self-supervised advantage model. Unlike previous robotics value reward model that takes the value as learning objective, STEAM directly learns the advantage for a robotics trajectory and use this label for downstream policy RL.
Critical strengths
-
Directly learning advantage is a well motivated and well formulized problem
-
Using model ensample to provide more stable estimation is an interesting design and from the experiment makes great sense.
-
Paper’s presentation including reward plots, task plots, as well as text is very good and clear.
Critical weaknesses
-
Directly estimating advantage is not brand new idea, authors should compare with [1]
-
Table 1 seems only include 1 task, as the main contribution of the paper, this seems less persuasive, please include at least 1 more task or split the table by task
-
In eq. 4, the baseline of advantage for every frame is the corresponding frame of the longest expert trajectory. I feel little confused here, longest expert traj is bad in average, but does not mean it’s bad at every frame or portion, for example, it can progress very fast at early stage than get stuck in middle and take very long to finish, under this circumstance, the model receive wrong learning signals for the first half portion of every trajectory.
Reference [1] Mao, Yiming, et al. "Arm: Advantage reward modeling for long-horizon manipulation." arXiv preprint arXiv:2604.03037 (2026).
Questions for authors
In table 4, performance drop when ensemble size keep increasing from 3 to 5, this is little anti-intuitive to the results in RL’s critic ensample, where performance usually stable improve with ensemble size to around 10. Can you explain or share some observations here?
3 comments
W3. In eq. 4, the baseline of advantage for every frame is the corresponding frame of the longest expert trajectory. I feel little confused here, longest expert traj is bad in average, but does not mean it’s bad at every frame or portion, for example, it can progress very fast at early stage than get stuck in middle and take very long to finish, under this circumstance, the model receive wrong learning signals for the first half portion of every trajectory.
We need a clarification first: STEAM does not compare a trajectory with the longest one segment by segment. The lengths of the expert demonstrations are used only for speed scaling: Eq. (2) rescales the frame offsets of every demonstration by to a common time scale. The predictor is trained on frame pairs from all expert demonstrations (Eq. 3), and the predictor gives the advantage of each frame from that frame and the frame H steps later.
Uneven pace within a trajectory does add noise to that demonstration's own targets, because Eq. (2) assigns targets by the demonstration's average pace. However, the predictor learns the mapping from visual change to offset, and the same visual change appears in many demonstrations, so this noise averages out (L256–258). In the appendix, the advantage of expert demonstrations drops clearly at pauses and retries and then recovers (Figures 8–11): these segments receive positive targets within their own demonstrations, yet the model scores them low, showing that such outliers are averaged out rather than learned as progress.
Q1. In table 4, performance drop when ensemble size keep increasing from 3 to 5, this is little anti-intuitive to the results in RL’s critic ensample, where performance usually stable improve with ensemble size to around 10. Can you explain or share some observations here?
We thank the reviewer for this observation. We believe the phenomenon is not a performance drop but saturation at M = 3, for the following reasons.
1. M = 5 is not worse; performance saturates. M = 3 and M = 5 reach 92.3% and 90.9% on towel folding (Table 4), with 95% Wilson confidence intervals of 75.9–97.9% and 72.2–97.5% (Fisher's exact test, p = 1.00), so the two are statistically indistinguishable. Table 4 therefore shows that performance saturates after three members rather than drops off.
2. Why it saturates earlier than critic ensembles in RL. In RL, the actor keeps searching during optimization for actions that some critic overestimates, so adding members keeps reducing the errors it can exploit. STEAM's ensemble labels a fixed dataset only once, offline, and the policy is never optimized against it. Moreover, Eq. (6) uses only the rank of each frame within its data pool; once the ensemble minimum pushes overestimated rollout frames below the threshold (Figure 4), additional members rarely change the labels. We measured this: on towel folding, going from M = 3 to M = 5 changes the optimality labels of only 4.87% of the samples. Hence M = 3 already suffices to suppress overestimation on rollouts; adding members barely changes the labels used to train the policy, so policy performance does not improve further.
Thank authors for the clarification, I will maintain my score.
We thank the reviewer for recognizing the motivation for learning the advantage directly and the ensemble design, and for the specific questions. We answer each point below.
ARM requires human-annotated tri-state labels (progress / stall / regression). Using ARM's official implementation, we faithfully reproduced its full pipeline, including reward model training and policy training, and replaced only the human annotations with labels generated by Claude Opus 5.5; we denote this ARM-VLM. We manually checked 50 intervals stratified by progress / stall / regression (20 / 20 / 10), and 47 of them agree with human judgment (94%). Due to the limited time of the discussion period, we have so far completed the experiment on one task, towel folding:
ARM-VLM reaches 55.0% on towel folding, below STEAM's 92.3%. Since ARM-VLM keeps ARM's original AW-BC for policy learning whereas STEAM uses CFGRL, this gap also reflects the policy learning method; on clustering accuracy and I-AUROC, the offline metrics unaffected by it, STEAM is again clearly better. This shows that, when neither method relies on human annotation, STEAM estimates the advantage more accurately on this task.
We understand the reviewer's concern about how persuasive the results are. We added Chip Checkout (30 successful and 30 failed episodes) to Table 1, report the results per task, and added the task names to the caption:
On the newly added Chip Checkout, STEAM is again the best on both metrics, and its average over the two tasks is clearly ahead. This shows that STEAM's advantage estimates are more reliable across tasks.