acceptodds
Under review as a conference paper at ICLR 2027

STEAM: Self-Supervised Temporal Ensemble Advantage Modeling for Real-World Robot Learning

Abstract

Real-world robot learning increasingly relies on heterogeneous data, but demonstrations and rollouts often mix useful progress with stalls, corrections, and suboptimal behavior. Effective policy learning therefore requires frame-level advantages that distinguish reliable local progress from failures and regressions. We propose **S**elf-supervised **T**emporal **E**nsemble **A**dvantage **M**odeling (**STEAM**), a label-free method that learns such advantages from expert demonstrations. STEAM trains an ensemble of temporal-offset predictors on frame pairs within expert trajectories, using the normalized temporal offset between two frames as a self-supervised signal. Each predictor maps a frame pair to a distribution over temporal offsets, which is converted into a scalar advantage. STEAM then takes the minimum advantage across the ensemble to score mixed-quality rollout data conservatively. Across real-world bimanual towel folding, chip checkout, table clearing, and cola restocking tasks, STEAM identifies stalls, failures, and recoveries. When combined with CFGRL, STEAM achieves the highest policy success rate on all four tasks, averaging 83.5% against 58.8% for the best baseline.

Primary area:
reinforcement learning
Keywords:
reward modeling, offline reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_msE1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces STEAM, a self-supervised advantage model. Unlike previous robotics value reward model that takes the value as learning objective, STEAM directly learns the advantage for a robotics trajectory and use this label for downstream policy RL.

Critical strengths
  1. Directly learning advantage is a well motivated and well formulized problem

  2. Using model ensample to provide more stable estimation is an interesting design and from the experiment makes great sense.

  3. Paper’s presentation including reward plots, task plots, as well as text is very good and clear.

Critical weaknesses
  1. Directly estimating advantage is not brand new idea, authors should compare with [1]

  2. Table 1 seems only include 1 task, as the main contribution of the paper, this seems less persuasive, please include at least 1 more task or split the table by task

  3. In eq. 4, the baseline of advantage for every frame is the corresponding frame of the longest expert trajectory. I feel little confused here, longest expert traj is bad in average, but does not mean it’s bad at every frame or portion, for example, it can progress very fast at early stage than get stuck in middle and take very long to finish, under this circumstance, the model receive wrong learning signals for the first half portion of every trajectory.

Reference [1] Mao, Yiming, et al. "Arm: Advantage reward modeling for long-horizon manipulation." arXiv preprint arXiv:2604.03037 (2026).

Questions for authors

In table 4, performance drop when ensemble size keep increasing from 3 to 5, this is little anti-intuitive to the results in RL’s critic ensample, where performance usually stable improve with ensemble size to around 10. Can you explain or share some observations here?

3 comments

We thank the reviewer for recognizing the motivation for learning the advantage directly and the ensemble design, and for the specific questions. We answer each point below.

W1. Directly estimating advantage is not brand new idea, authors should compare with [1]. [1] Mao, Yiming, et al. "Arm: Advantage reward modeling for long-horizon manipulation." arXiv preprint arXiv:2604.03037 (2026).

ARM requires human-annotated tri-state labels (progress / stall / regression). Using ARM's official implementation, we faithfully reproduced its full pipeline, including reward model training and policy training, and replaced only the human annotations with labels generated by Claude Opus 5.5; we denote this ARM-VLM. We manually checked 50 intervals stratified by progress / stall / regression (20 / 20 / 10), and 47 of them agree with human judgment (94%). Due to the limited time of the discussion period, we have so far completed the experiment on one task, towel folding:

MethodClust. Acc. / I-AUROCSuccess rate
ARM-VLM63.3 / 58.655.0%
STEAM93.3 / 78.892.3%

ARM-VLM reaches 55.0% on towel folding, below STEAM's 92.3%. Since ARM-VLM keeps ARM's original AW-BC for policy learning whereas STEAM uses CFGRL, this gap also reflects the policy learning method; on clustering accuracy and I-AUROC, the offline metrics unaffected by it, STEAM is again clearly better. This shows that, when neither method relies on human annotation, STEAM estimates the advantage more accurately on this task.

W2. Table 1 seems only include 1 task, as the main contribution of the paper, this seems less persuasive, please include at least 1 more task or split the table by task

We understand the reviewer's concern about how persuasive the results are. We added Chip Checkout (30 successful and 30 failed episodes) to Table 1, report the results per task, and added the task names to the caption:

MethodTowel Clust. Acc.Towel I-AUROCChip Clust. Acc.Chip I-AUROCAvg. Clust. Acc.Avg. I-AUROC
ReWiND63.353.351.762.457.557.9
TimeRewarder88.365.170.071.679.268.4
RoboMeter75.045.530.028.052.536.7
RECAP60.048.380.054.570.051.4
STEAM93.378.881.781.487.580.1

On the newly added Chip Checkout, STEAM is again the best on both metrics, and its average over the two tasks is clearly ahead. This shows that STEAM's advantage estimates are more reliable across tasks.

W3. In eq. 4, the baseline of advantage for every frame is the corresponding frame of the longest expert trajectory. I feel little confused here, longest expert traj is bad in average, but does not mean it’s bad at every frame or portion, for example, it can progress very fast at early stage than get stuck in middle and take very long to finish, under this circumstance, the model receive wrong learning signals for the first half portion of every trajectory.

We need a clarification first: STEAM does not compare a trajectory with the longest one segment by segment. The lengths of the expert demonstrations are used only for speed scaling: Eq. (2) rescales the frame offsets of every demonstration by Lmax⁡/LτkL_{\max}/L_{\tau_k} to a common time scale. The predictor is trained on frame pairs from all expert demonstrations (Eq. 3), and the predictor gives the advantage of each frame from that frame and the frame H steps later.

Uneven pace within a trajectory does add noise to that demonstration's own targets, because Eq. (2) assigns targets by the demonstration's average pace. However, the predictor learns the mapping from visual change to offset, and the same visual change appears in many demonstrations, so this noise averages out (L256–258). In the appendix, the advantage of expert demonstrations drops clearly at pauses and retries and then recovers (Figures 8–11): these segments receive positive targets within their own demonstrations, yet the model scores them low, showing that such outliers are averaged out rather than learned as progress.

Q1. In table 4, performance drop when ensemble size keep increasing from 3 to 5, this is little anti-intuitive to the results in RL’s critic ensample, where performance usually stable improve with ensemble size to around 10. Can you explain or share some observations here?

We thank the reviewer for this observation. We believe the phenomenon is not a performance drop but saturation at M = 3, for the following reasons.

1. M = 5 is not worse; performance saturates. M = 3 and M = 5 reach 92.3% and 90.9% on towel folding (Table 4), with 95% Wilson confidence intervals of 75.9–97.9% and 72.2–97.5% (Fisher's exact test, p = 1.00), so the two are statistically indistinguishable. Table 4 therefore shows that performance saturates after three members rather than drops off.

2. Why it saturates earlier than critic ensembles in RL. In RL, the actor keeps searching during optimization for actions that some critic overestimates, so adding members keeps reducing the errors it can exploit. STEAM's ensemble labels a fixed dataset only once, offline, and the policy is never optimized against it. Moreover, Eq. (6) uses only the rank of each frame within its data pool; once the ensemble minimum pushes overestimated rollout frames below the threshold (Figure 4), additional members rarely change the labels. We measured this: on towel folding, going from M = 3 to M = 5 changes the optimality labels of only 4.87% of the samples. Hence M = 3 already suffices to suppress overestimation on rollouts; adding members barely changes the labels used to train the policy, so policy performance does not improve further.

Thank authors for the clarification, I will maintain my score.

Reviewer_GSQaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes STEAM, a self-supervised method that learns frame-level advantage scores by predicting trajectory-length-normalized temporal offsets between pairs of frames from expert demonstrations. To reduce overestimation on out-of-distribution data, it trains an ensemble of predictors and takes the minimum advantage score. On real robot data, STEAM better distinguishes successful from failed trajectories and detects progress around human interventions than the evaluated baselines. Combined with CFGRL, it improves policy performance across four real-world manipulation tasks.

Critical strengths

The paper is overall well-written and easy to follow. The appendix includes details on the experiment setup and implementation. The experiment shows that STEAM seems to be generally effective in assessing forward and backward progress. The inclusion of negative results and discussions is also helpful to understand the strengths and limitations of the proposed approach.

Critical weaknesses

From the results, it seems like STEAM is quite sensitive to the choice of hyperparameter. For example, the success rate of the towel folding task drops from 92.3% to 54.6% (comparable or lower than other baselines) as bin count N decreases from 32 to 8. Using a smaller lookahead horizon H also decreases the success rate to 80%. This makes it difficult to assess whether the performance gain against baselines is due to different methodology (i.e., to predict advantage rather than value) or the specific modeling choices. An additional ablation study with the same architecture but different supervised targets could answer this question.

Questions for authors
  1. Is the STEAM model trained as single-task or multi-task (language-conditioned)?

  2. Why not directly use normalized temporal difference as the advantage? What’s the reasoning behind introducing the second term in (4)? Why is τmax\tau_{max} used for the second term?

  3. For the baselines, are they using the same hyperparameters, e.g., HH, qexpq^{exp}, qnon−expq^{non-exp}, etc. as STEAM? Do they also use ensembles? Is RECAP using the same number of bins as STEAM?

  4. Is the RoboMeter model also finetuned on the task data or is it evaluated zero-shot?

5 comments

We thank the reviewer for the comments, and in particular for suggesting an ablation that directly tests the source of the gain. We answer each point below.

W1. From the results, it seems like STEAM is quite sensitive to the choice of hyperparameter. For example, the success rate of the towel folding task drops from 92.3% to 54.6% (comparable or lower than other baselines) as bin count N decreases from 32 to 8. Using a smaller lookahead horizon H also decreases the success rate to 80%. This makes it difficult to assess whether the performance gain against baselines is due to different methodology (i.e., to predict advantage rather than value) or the specific modeling choices. An additional ablation study with the same architecture but different supervised targets could answer this question.

We built STEAM-V to ablate the effect of predicting the advantage: it is identical to STEAM, including the ensemble minimum over M = 3 predictors and the quantile thresholds (top 80% of expert data, top 30% of non-expert data), except that the supervision target is a single-frame value (201 bins, as in RECAP), and its advantage is the difference between the values of the two frames. Results on towel folding:

MethodClust. Acc. / I-AUROCSuccess rate
STEAM-V80.0 / 59.260.0%
STEAM93.3 / 78.892.3%

With all modeling choices identical, changing the supervision target from the advantage to the value lowers the success rate from 92.3% to 60.0%, and both offline metrics also drop markedly. This shows that predicting the advantage as the supervision target is one of the key factors behind STEAM's gain.

Regarding the sensitivity to hyperparameters: N and H both determine whether STEAM can read the pace of progress from two frames. N sets the resolution of the offset: with too few bins, progress at different paces falls into the same bin, and N = 2 keeps only the direction (L414–420). H is the time gap between the two frames: if it is too short, the visual change is too small to read progress (Appendix A.5). Our ablations only decrease these two parameters, and the resulting drops show that they are key parameters. All four tasks share the same N and H (Table 6).

Q1. Is the STEAM model trained as single-task or multi-task (language-conditioned)?

STEAM is trained per task: we train a separate ensemble for each task. The language branch of the model encodes the task instruction and provides task-level guidance for the prediction.

Q2. Why not directly use normalized temporal difference as the advantage? What’s the reasoning behind introducing the second term in (4)? Why is τ_max used for the second term?

1. Why not directly use the normalized temporal difference? Computationally, this is exactly what we do. The second term is the same constant for every frame; it changes neither the ranking after the ensemble minimum nor the quantile labels (Eq. 6), so it is omitted in implementation (L269), and the reported values of A_STEAM are also computed without it.

2. Why the second term? The first term alone is not an advantage; an advantage requires a comparison against a baseline. The first term estimates the progress of the segment over H steps, ΔVref\Delta V_{\mathrm{ref}}, measured in steps of the reference expert (L261); the second term is the progress of the baseline over the same H steps. Their difference corresponds to the H-step advantage of Eq. (7), positive if the segment progresses more than the baseline, and negative if it progresses less.

3. Why τ_max? A comparison needs a baseline. We use the slowest expert demonstration, i.e., the longest episode τ_max (L202), as the baseline, and Eq. (2) rescales expert demonstrations of different speeds to a common time scale for comparison. Under Eq. (4), normal expert progress therefore does not receive a negative score in expectation; negative scores go to segments slower than the baseline, such as stalls, regressions, and retries. If the average or the fastest expert were the baseline, a substantial fraction of normal expert progress would also receive negative scores. The baseline is shared by all frames; if each episode subtracted its own offset, the differences in speed between demonstrations would cancel out.

Q3. For the baselines, are they using the same hyperparameters, e.g., H, q^exp, q^non-exp, etc. as STEAM? Do they also use ensembles? Is RECAP using the same number of bins as STEAM?

1. Hyperparameters. The baselines follow RECAP's settings and label all expert demonstrations as optimal. Their threshold on non-expert data (top 30%) and their lookahead (H = 32) are the same as STEAM's, and all policies are trained with the same CFGRL settings (L285–286). The only difference in thresholds is on expert data: filtering expert data is a design unique to STEAM (top 80%), whereas RECAP makes no such distinction.

2. Ensembles. The baselines do not use ensembles; the ensemble minimum is part of the STEAM method. Even so, with a single predictor (M = 1), STEAM still reaches 72.7% on towel folding (Table 4), higher than all baselines (the best is RoboMeter at 60.0%, Table 2).

3. Number of bins. The two methods use different numbers of bins, and they should not be set equal, because they discretize different quantities. STEAM only predicts the local offset between two frames, with range [−Δ~max⁡,Δ~max⁡][-\tilde\Delta_{\max}, \tilde\Delta_{\max}], for which 32 bins are fine enough. RECAP's value must cover all remaining steps from the current frame to the end of the episode, a much larger range, so it uses 201 bins. With the same number of bins, the number of steps per bin would differ greatly between the two methods, and their resolutions would not be comparable. We therefore keep each method's original setting.

Q4. Is the RoboMeter model also finetuned on the task data or is it evaluated zero-shot?

Zero-shot. We directly use the released RoboMeter-4B checkpoint without fine-tuning on our task data, as a representative general-purpose VLM reward model. As with the other baselines, its reward is converted into an advantage by a RECAP-style value model trained on each task (L280–285).

Reviewer_GSQaOpenReview ↗

Thank you for your responses

Thank you for the response and for adding the STEAM-V comparison. It would also be useful to examine how its performance varies with the lookahead horizon H. This would help assess whether the reported gap holds across reasonable settings and better address the original concern about sensitivity to H. The original concern about sensitivity to bin count N remains unresolved.

The comparison with RoboMeter could also be strengthened by fine-tuning its reward model on the same task data, as it is currently evaluated zero-shot.

Thank you also for clarifying that STEAM is trained separately for each task. Although it requires no additional temporal annotations, training and maintaining a separate ensemble for each task adds computational cost and remains a limitation for scaling to a larger set of tasks.

I appreciate the additional experiment and clarifications. However, my main concerns about hyperparameter sensitivity and the baseline comparisons remain, so I will keep my original rating of 2.

Reviewer_YRgaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

STEAM learns frame-level advantages for mixed-quality robot data without rewards or human labels. From expert demonstrations, it builds frame pairs and trains each predictor to classify the length-normalized temporal offset between them into different bins; reversed pairs supply negative offsets. At scoring time, the expected bin is turned into a scalar advantage. The final score is the minimum over an ensemble of 3 predictors. Scores are thresholded separately within the expert pool and the non-expert pool, giving binary optimality labels for CFGRL on a pi0 policy.

Critical strengths
  1. The method is simple and label-free, and it is tested where it matters: real-robot, long-horizon bimanual tasks with real rollouts and human corrections. The gains are large and in the same direction on all four tasks. The ablations line up with the proposed mechanism.
  2. The paper evaluates the advantage signal itself, not only downstream success. The I-AUROC protocol around takeover points and the per-episode curves in A.3 make it possible to see when the signal works and when it does not.
Critical weaknesses
  1. Statistical reporting of the policy results is too thin for the size of the claims. The number of evaluation trials per method and task is never stated, and there are no confidence intervals. The averaged gap over the baselines is probably real, but the paper should show this.
  2. The explanation for the gain is not isolated. Every baseline reward is converted into advantages through a RECAP-style value model. STEAM differs in several ways at once, for example, a distributional bin target; ensemble-min aggregation; per-pool quantile thresholds; no value model. The text attributes the gap to “predicting the advantage directly,” but no experiment separates that factor.
Questions for authors
  1. How many trials were run per method per task in Table 2 and in the ablations? Were initial conditions matched across methods?
  2. Were the baselines labeled with the same per-pool quantiles? How were q_exp and q_nexp chosen? If they were tuned on the evaluation tasks, please say so.
  3. Which task(s) do the 30 successful and 30 failed episodes in Table 1 come from?
  4. Some expert demonstrations contain failed attempts and retries (Fig. 8, 9, 11). During training, those segments receive positive forward offsets. Did you filter them? Does it matter?
1 comment

We thank the reviewer for the detailed comments and answer each point below.

W1. Statistical reporting of the policy results is too thin for the size of the claims. The number of evaluation trials per method and task is never stated, and there are no confidence intervals. The averaged gap over the baselines is probably real, but the paper should show this.

Q1. How many trials were run per method per task in Table 2 and in the ablations? Were initial conditions matched across methods?

For statistically meaningful results, every method and every ablation is evaluated with at least 15 trials on each task, all in the same environment with the same initial settings. The table gives STEAM's average success-rate gap over each baseline across the four tasks (Table 2) with its 95% confidence interval, and the p-value of a task-stratified Cochran–Mantel–Haenszel (CMH) test:

MethodAvg. success-rate gap (pp, 95% CI)CMH p-value
TimeRewarder+24.7 ± 14.21.6 × 10⁻³
RoboMeter+30.5 ± 13.57.2 × 10⁻⁵
RECAP+33.9 ± 13.91.4 × 10⁻⁵

All three intervals exclude 0, and STEAM's improvement over each baseline is statistically significant (p < 0.01).

W2. The explanation for the gain is not isolated. Every baseline reward is converted into advantages through a RECAP-style value model. STEAM differs in several ways at once, for example, a distributional bin target; ensemble-min aggregation; per-pool quantile thresholds; no value model. The text attributes the gap to “predicting the advantage directly,” but no experiment separates that factor.

To isolate this factor, we built STEAM-V, identical to STEAM, including the ensemble minimum over M = 3 predictors and the quantile thresholds (top 80% expert, top 30% non-expert), except that the supervision target is a single-frame value (201 bins, as in RECAP), and its advantage is the difference between the values of the two frames. Results on towel folding:

MethodClust. Acc. / I-AUROCSuccess rate
STEAM-V80.0 / 59.260.0%
STEAM93.3 / 78.892.3%

Changing only the supervision target from the advantage to the value lowers the success rate from 92.3% to 60.0%, and both offline metrics also drop markedly. This shows that predicting the advantage as the supervision target is one of the key factors behind STEAM's gain.

Q2. Were the baselines labeled with the same per-pool quantiles? How were q_exp and q_nexp chosen? If they were tuned on the evaluation tasks, please say so.

1. Same quantiles? On non-expert data, yes: all methods take the top 30%, with human corrections counted as non-expert data. On expert data, the baselines follow RECAP and label all expert demonstrations as optimal, whereas STEAM keeps only the top 80%, a design unique to STEAM.

2. How they were chosen. Expert demonstrations also contain suboptimal segments such as stalls and retries (Figures 8, 9, 11), so q_exp conservatively keeps only the top 80%. Non-expert data contain more suboptimal samples; following RECAP, q_nexp keeps the top 30%.

3. Tuned on the evaluation tasks? No. Both values were set in advance and are shared by all four tasks (Table 6).

Q3. Which task(s) do the 30 successful and 30 failed episodes in Table 1 come from?

They come from towel folding. The scale of the per-episode advantage sum differs across tasks, so Table 1 clusters episodes within a single task (Appendix A.4). We added the task name to the Table 1 caption and added Chip Checkout (30 successful and 30 failed episodes), reported per task:

MethodTowel Clust. Acc.Towel I-AUROCChip Clust. Acc.Chip I-AUROCAvg. Clust. Acc.Avg. I-AUROC
ReWiND63.353.351.762.457.557.9
TimeRewarder88.365.170.071.679.268.4
RoboMeter75.045.530.028.052.536.7
RECAP60.048.380.054.570.051.4
STEAM93.378.881.781.487.580.1

Q4. Some expert demonstrations contain failed attempts and retries (Fig. 8, 9, 11). During training, those segments receive positive forward offsets. Did you filter them? Does it matter?

We did not filter these segments, and their effect is limited. First, the predictor learns offsets from the visual change between two frames. A failed attempt usually returns the scene to an earlier state, and such visual changes are labeled as negative offsets in the many reversed pairs; retry segments make up only a small part of the expert data, so their positive targets are averaged out (L256–258). Second, in expert demonstrations with retries, A_STEAM drops during the failed attempt and recovers after the successful retry (Figures 8, 9, 11; Appendix A.3), showing that the predictor does not learn retries as progress.

Comments on the paper

1 comment

We thank all three reviewers for their constructive feedback. The reviewers found the motivation for learning the advantage directly well grounded (msE1), the method simple and annotation-free and validated on long-horizon real-robot tasks with rollouts and human corrections (YRga), and the direct evaluation of the advantage signal itself helpful for seeing where the method works (YRga, GSQa). The main additions and revisions are as follows:

  1. Same-architecture comparison of advantage vs. value (YRga, GSQa). We added STEAM-V, which changes only the supervision target to a single-frame value and keeps everything else identical to STEAM. It reaches 60.0% on towel folding, below STEAM (92.3%, Table 2), showing that predicting the advantage as the supervision target is one of the key factors behind STEAM's gain.
  2. Statistical reporting (YRga). For statistically meaningful results, every method and every ablation is evaluated with at least 15 trials on each task. We report 95% confidence intervals for STEAM's average success-rate gap over each baseline and pool the four tasks with a task-stratified Cochran–Mantel–Haenszel test: all three intervals exclude 0, and STEAM's improvement over each of the three baselines is statistically significant (p < 0.01).
  3. Table 1 (YRga, msE1). We added Chip Checkout to Table 1 and report the results per task.
  4. Baseline configuration (YRga, GSQa). We clarify the baseline settings: the baselines follow RECAP's hyperparameters, and their threshold on non-expert data and their lookahead are the same as STEAM's; filtering expert data is a design unique to STEAM.
  5. Comparison with ARM (msE1). Section 2 of the paper already cites and discusses ARM. During the discussion period, we reproduced ARM with VLM labels in place of human annotations (ARM-VLM). It reaches 55.0% on towel folding, below STEAM (92.3%, Table 2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.