acceptodds
Under review as a conference paper at ICLR 2027

Event2Reward: Learning Robotic Rewards from Progress Relations

Abstract

Reward models trained with normalized temporal labels from successful demonstrations can mistake setbacks for progress during exploration, turning losses of progress into positive shaping rewards that hinder policy improvement. Real suboptimal and failed executions offer corrective experience, but their absolute progress is difficult to label. We introduce Event2Reward, a reward model that learns from within-trajectory progress relations supported by task events in real executions. These relations directly supervise the progress predictions used for reward shaping. We combine them with temporal labels from successful demonstrations to anchor the numerical progress scale. We curate an event-annotated dataset from approximately 100K raw real-robot execution records across multiple sources and adapt Event2Reward using a small set of target-task executions spanning successful, suboptimal, and failed outcomes. On 40 tasks across four LIBERO suites, Event2Reward improves suite-average policy success over the strongest baseline in each suite by 9.96–13.76 percentage points under matched per-suite interaction budgets. In real-world experiments with a matched 30-minute wall-clock training budget, Event2Reward achieves an average success rate of 60.0% on three SERL insertion tasks, compared with 37.8% for Robometer, and 90.0% on three DSRL manipulation tasks, compared with 81.7% for Robometer. Ablations support the complementary roles of temporal labels and progress relations in downstream policy learning. We will release our model, dataset annotations, and processing tools.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot learning, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3ufqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, authors introduce event based reward model. Reward model learning signal comes from semantically meaningful events inside the trajectory rather than temporal anchors. Experiments show that event based reward model achieve both better reward modeling as well as downstream policy training performance.

Critical strengths
  1. Event based reward model by learning from contrastive signal inside trajectory is novel and makes reward model directly learning from fail/suboptimal trajectories possible.

  2. Compared to SOTA reward models, proposed method achieve better policy learning effect in both LIBERO simulation benchmark as well as several non-trivial manipulation tasks.

Critical weaknesses
  1. Though with LLM assisted, but the human labeling/annotation seems still non-trivial work for 100K rollouts. The authors should provide efficiency related report on data curation and annotation process, including token usage, human labor hours etc.

  2. The data and annotation visualization from page 19-26 are good examples, but they are all pick-and-place tasks, it remains questionable if a model solely trained on these tasks can achieve accurate reward modeling on precision assembly tasks authors used for downstream policy training like (a) Plug Insert (b) RAM Insert (c) Ethernet Insert. Can authors provide visualizations of reward model estimation of these tasks’ executions?

  3. In table 4, authors classify the robot execution events into 4 categories, though a concise high-level abstract but this seems cannot work well on more detailed scenarios reward model should react, e.g. suboptimal execution - robot still progressing, but slower and more easy to make mistakes.

Questions for authors

In Table 4, what’s the difference between Progress stall and Unrecovered regression?

8 comments
AuthorsOpenReview ↗

Overview of our responses

Thank you for your careful reading of our paper and your specific questions about annotation effort, reward prediction for precision assembly, and event definitions.

To accompany our responses, we have organized an anonymous supplementary page for your review with a recording of the annotation process, precision-assembly videos synchronized with progress predictions, and simulation and real-robot examples illustrating stalls and unrecovered regressions. The page follows the same order as the four comments below, with links to each topic.

We encourage you to explore this page alongside the responses below. It provides additional interactive examples with synchronized execution videos and progress predictions, making it easier to examine the model’s behavior during execution and understand the distinction between stalls and unrecovered regressions.

  • [1/4] W1 · Dataset annotation cost. We report GPT token usage and estimated API cost using the publicly released RoboChallenge data from our paper as an example, describe the human annotation work, its throughput, and the person-hours required, and explain how releasing the final annotations will allow others to reuse this investment.
  • [2/4] W2 · Reward prediction on precision assembly. The precision-assembly experiments use reward models adapted with data from each target task. We clarify this setup and provide 18 examples across Plug Insert, RAM Insert, and Ethernet Insert, with execution videos and synchronized progress predictions.
  • [3/4] W3 · Slower progress and likely mistakes. The four event types provide supervision for a continuous progress estimate. Slower execution alone does not require an event label. Our supervision concerns actual progress changes rather than anticipating potential mistakes.
  • [4/4] Q1 · Stall versus unrecovered regression. A stall preserves previously achieved progress, whereas an unrecovered regression involves progress loss that is not restored during the remaining execution. We clarify this distinction through the supervision relations and annotated video examples.
AuthorsOpenReview ↗

[1/4] W1 · How much work does dataset annotation require?

W1. Though with LLM assisted, but the human labeling/annotation seems still non-trivial work for 100K rollouts. The authors should provide efficiency related report on data curation and annotation process, including token usage, human labor hours etc.

Dataset construction combines GPT-assisted annotation with human verification. Human annotators identify event types and mark key time points based on visible state changes in the videos, without assigning precise framewise progress values. The annotation rate is approximately 260–330 trajectories per person-hour, or about 11–14 seconds per trajectory. We manually annotated all 40,857 raw RoboChallenge trajectories used in our paper. Manual annotation took approximately 140 person-hours. Human annotations are compared with GPT proposals. Disagreements are reviewed by a third party, and annotation quality is checked through spot checks. As an example, for the publicly released RoboChallenge data used in our paper, we used GPT-5.6 Sol (medium reasoning) to propose candidate event annotations. Generating candidate event annotations for these data consumed approximately 3.8 billion tokens, with an estimated API cost of US$15,000–19,000.

 

To illustrate the annotation procedure, we provide a continuous screen recording covering all 87 complete episodes from the paper’s Ethernet-insertion training run (Figure R8). One author familiar with the event definitions completed these annotations in 9 minutes 53 seconds of active work, including revisions, averaging approximately 6.8 seconds per trajectory, equivalent to about 530 trajectories per hour. The recording shows the complete process of identifying events, marking key time points, and making revisions.

We will release the Event2Reward dataset, including the final event annotations used for training. Researchers can use the same annotations to train reward models and reproduce the policy-learning experiments without making additional GPT calls or repeating human annotation.

This response and the annotation recording are also available on the corresponding anonymous supplementary page.

AuthorsOpenReview ↗

[2/4] W2 · Reward prediction on precision assembly

W2. The data and annotation visualization from page 19-26 are good examples, but they are all pick-and-place tasks, it remains questionable if a model solely trained on these tasks can achieve accurate reward modeling on precision assembly tasks authors used for downstream policy training like (a) Plug Insert (b) RAM Insert (c) Ethernet Insert. Can authors provide visualizations of reward model estimation of these tasks’ executions?

In the precision-assembly experiments, we adapt the reward model using target-task data. As described in Section 4.4, we use five successful and five suboptimal trajectories for each of Plug Insert, RAM Insert, and Ethernet Insert to adapt the reward model, then freeze it during online policy learning. The pick-and-place annotation examples on pages 20–27 do not represent all the training data used in these experiments.

Following your suggestion, we provide additional prediction visualizations for the three precision-assembly tasks, with six examples per task and 18 in total, showing execution videos synchronized with raw progress predictions (Figure R15). These examples cover insertion, pauses, withdrawal, renewed approach, and incomplete executions, allowing the model’s responses to be examined alongside changes in task state.

We also provide a video of the paper’s online plug-insertion training run using Event2Reward reward shaping, showing the progression from human-guided warm-up to autonomous policy exploration and reaching 10 consecutive successful insertions after approximately 15 minutes (Figure R7).

Reviewer_qD2ZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a reward model for robot learning that learns from within trajectory progress relations, in addition to normalized temporal labels from successful trajectories. This model is able to give better feedback about the progress when the robot stalls or does some unclear behavior (e.g. letting object slip). To do so, they first collect and annotate an event annotated dataset, using GPT and human annotators. Then, the Event2Reward model is trained on this data to predict temporal progress, and progress relations, which is the difference between the prediction and a reference / recovery score. The model is then finetuned on demonstrations of the task of interest, and is used as a reward model for RL. They show their reward model outperforms other reward models on libero and real world manipulation tasks.

Critical strengths
  • the idea of having a reward model that can give feedback about failure / recovery is nice and different than other approaches
  • the experimental results seem generally good, although i have some concerns.
Critical weaknesses
  • although the method gets higher success than baselines, it is really unclear why this method is doing better. there are no ablations or visualizations or analyses in the main text.
  • the authors should compare against a reward function defined by a capable VLM, such as GPT-6 astra. since VLMs are better now, they are a natural point of comparison.
  • why is there only 4 libero tasks? that seems cherry picked to me.
  • little discussion of what data is needed to adapt to the reward model to the task. this itself seems like it requires some thought and exploration
Questions for authors

See weaknesses

6 comments
AuthorsOpenReview ↗

Overview of our responses

Thank you for your careful review and for highlighting the need for clearer explanations and broader evidence. We address your four concerns below. We encourage you to visit our anonymous response page alongside the responses below. It brings together additional video examples, progress-prediction curves synchronized with the videos, and experimental figures, making it easier to compare model predictions with the actual execution and understand the results. Key numerical results and explanations are also included in each response.

  • [1/4] W1 · Why Event2Reward outperforms the baselines. Event2Reward directly corrects the progress predictions that determine shaping rewards, addressing errors that can reward actions causing setbacks. The supervision ablations in §4.1 further show that accurate trajectory comparisons do not necessarily translate into effective rewards for policy learning.
  • [2/4] W2 · Comparison with a strong VLM. Our offline comparison with GPT-6 Astra on trajectories excluded from adaptation shows complementary strengths across the prediction metrics. For online reward shaping, however, Astra’s substantially higher single-query latency makes it difficult to integrate directly.
  • [3/4] W3 · Task coverage. The original main comparison covers all 40 tasks across four LIBERO suites; the four-task setting concerns the supervision ablation. We have extended that ablation to all tasks in LIBERO-10 and LIBERO-Object, with Progress + Relation achieving the highest mean success and normalized evaluation AUC in both suites.
  • [4/4] W4 · Adaptation data. Adaptation uses successful demonstrations and previously collected policy rollouts, with event types and key moments annotated for suboptimal or failed executions rather than manual progress scores for every frame. §3.5 describes the adaptation data, while §4.3 presents the expanded data-efficiency experiments: on LIBERO-Object, an average of 10 adaptation trajectories per task already matches the policy success achieved with 20.
AuthorsOpenReview ↗

[1/4] W1 · Why does it outperform the baselines?

W1. although the method gets higher success than baselines, it is really unclear why this method is doing better. there are no ablations or visualizations or analyses in the main text.

A key reason is that Event2Reward directly corrects the progress predictions used to compute shaping rewards. Shaping rewards depend on changes in progress between adjacent states. Robometer uses a preference head to judge which execution is better and a separate progress head to predict task progress. Even when the preference head correctly ranks a successful trajectory above a failed one, the progress head may still mistake a drop for progress. Although the two heads share a backbone, the preference loss does not directly penalize this local error. These local predictions directly determine the shaping rewards received by the policy. If the model mistakes an actual setback for progress, it may produce positive shaping feedback, encouraging the policy to learn incorrect actions. Event2Reward instead directly constrains post-drop progress to be lower than pre-drop progress and penalizes predictions that violate this relation.

The linked videos illustrate this difference. In Figure R9, the bottle drops and is later picked up again. Event2Reward’s progress prediction decreases after the drop and rises as the robot re-grasps and lifts the bottle, while Robometer’s prediction changes relatively little and does not clearly reflect the regression and recovery. Figure R10 provides another drop-and-recovery example with a ketchup bottle. These examples illustrate that the ability to compare entire executions does not guarantee correct progress feedback during an execution.

 

The supervision ablations in §4.1 and Appendix C.1 further show that learning to compare trajectories does not necessarily yield more effective shaping rewards. Progress + Preference, which uses Robometer-style trajectory-comparison learning, achieves an average accuracy of 95% on unseen success–failure comparisons in LIBERO-10 (Table 10), confirming that it learned the comparison task. However, in the policy-learning experiments on four LIBERO-10 tasks, its normalized evaluation AUC is 19.4%, close to the 19.6% achieved by Progress-only, while Progress + Relation achieves 36.7% (Figure 3 and Table 9). Thus, in these experiments, high trajectory-comparison accuracy did not translate into better policy-learning performance.

Table R1.

SupervisionNormalized evaluation AUC (%)
Progress-only19.6 ± 10.1
Progress + Preference19.4 ± 14.3
Relation-only8.3 ± 7.9
Progress + Relation36.7 ± 7.8

Normalized evaluation AUC averaged across four LIBERO-10 tasks, reported as mean ± SD over five environment seeds. Progress + Preference is a supervision ablation, not the full Robometer system.

These supervision ablations were already presented in Figure 3 of §4.1 in the main text, with the corresponding numerical results in Appendix C.1, Table 9. We have also extended the supervision ablations to all ten LIBERO-10 tasks and all ten LIBERO-Object tasks, with complete results provided in our response to W3 on task coverage.

AuthorsOpenReview ↗

[2/4] W2 · How does Event2Reward compare with a strong VLM?

W2. the authors should compare against a reward function defined by a capable VLM, such as GPT-6 astra. since VLMs are better now, they are a natural point of comparison.

 

1.Offline reward prediction

We agree that a strong general-purpose VLM such as GPT-6 Astra is a natural reward baseline. We conducted an offline comparison with Astra on 200 LIBERO-Object trajectories excluded from reward-model adaptation, covering 1,731 query points. Both models estimate task progress from the same task information and visual observations. We evaluate whether their predictions reflect the annotated execution events. The table below summarizes the results for stalls and intervals where progress has stopped.

During a stall, the progress score should remain stable. Once progress has stopped, it should not rise further. For each model, we use its score at the annotated start of the event as the reference and compare later scores with that value.

For example, if the reference score is 0.40 and a later prediction is 0.45, the upward deviation is 0.05. This measures a change in the model’s prediction.

 

Table R2.

What do we measure?Event2RewardAstra
During stalls, how far do scores move from the reference? ↓ Average deviation, counting both increases and decreases.0.03460.0423
During stalls, what fraction of scores stay within ±0.02 of the reference? ↑ Fraction of predictions within ±0.02 of the reference score.37.27%40.45%
After progress stops, how far do scores rise above the reference? ↓ Average upward deviation. Scores at or below the reference contribute zero.0.01810.0283
After progress stops, how often are scores above the reference? ↓ Fraction above the reference, regardless of the size of the increase.46.94%32.72%

Scores range from 0 to 1. Each metric is computed within a trajectory, then averaged across trajectories of that event type. ↑ Higher is better; ↓ lower is better.

 

During stalls, Event2Reward has a smaller mean absolute deviation from the reference score, whereas Astra performs slightly better in the fraction of predictions within ±0.02 of the reference. After progress stops, Event2Reward has a smaller average upward deviation, but Astra has a lower proportion of scoring points above the reference. Thus, for erroneous increases after progress stops, Event2Reward performs better at limiting their magnitude, whereas Astra produces overly high predictions less often. The videos and progress curves illustrate how both models respond during these intervals.

The complete offline metrics table reports all eight offline metrics, with each model performing better on different metrics. This experiment compares progress predictions only; we did not train a policy using Astra rewards.

 

2.Online reward shaping

For online reward shaping, query latency differs substantially. Astra’s measured median query latency is 20.30 seconds, compared with 0.205 seconds for Event2Reward on an H200 GPU. This much higher latency substantially increases the time cost of obtaining rewards and makes Astra difficult to integrate directly into online reward shaping.

 

For online reward shaping, query latency differs substantially.

ModelMedian latency per scoring call (seconds) ↓
Event2Reward (H200 GPU)0.205
GPT-6 Astra20.30

Astra’s much higher latency substantially increases the time cost of obtaining rewards and makes it difficult to integrate directly into online reward shaping.

 

We also provide a video of the online plug-insertion training run reported in the paper. With Event2Reward reward shaping, the policy achieves 10 consecutive autonomous successes in approximately 15 minutes, including human-guided warm-up and robot resets (Figure R7).

Reviewer_64X3OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Event2Reward, a reward model for progress-based reward shaping that combines normalized temporal labels from successful demonstrations with within-trajectory progress relations derived from event annotations (stalls, regressions, recoveries) on real suboptimal and failed executions. The authors also curate an event-annotated dataset from a large collection of raw real-robot execution records, and evaluate the learned rewards for downstream policy learning in LIBERO simulation suites and on real-robot insertion and manipulation tasks, reporting improvements over learned-reward baselines such as Robometer and Robo-Dopamine.

Critical strengths
  • The paper addresses a well-motivated problem: reward models trained only on normalized temporal labels from successful demonstrations "can mistake setbacks for progress during exploration," and the observation that "suboptimal and failed executions offer corrective experience" but lack valid absolute progress labels is a clean framing of the gap.
  • The literature review is thorough and well-organized, covering visual reward/value models, progress supervision from demonstrations, and reward learning from non-ideal experience, with the positioning of this work relative to each line clearly stated.
  • The event taxonomy in Section 3.2 (progress stall, regression and recovery for successful executions; no further progress and unrecovered regression for failed executions) is a good categorization, and the event-annotated dataset built on it would be a very useful resource for the community if released as promised.
  • The empirical evaluation is broad, spanning controlled supervision ablations, 40 LIBERO tasks across four suites, and six real-world tasks under two RL frameworks (SERL and DSRL).
Critical weaknesses

Method clarity and replicability

  • Several core technical components are insufficiently defined in the main text. The phrase "a soft encoding with mean yty_t" (Sec. 3.3) is not explained - it is unclear what distribution family this soft encoding is, how its spread is set, and how sensitive results are to it.
  • The relation loss notation is confusing. ℓe(t)\ell_e(t) in Eq. (1) is introduced without being explicitly defined as the per-timestep penalty for event ee, and the margin mem_e is described only as "determined by the training rule, not an annotated amount of progress loss," deferring all actual loss and margin definitions to Appendix B.3. The main text should be self-contained enough to replicate the core objective.
  • The sentence "We average the corresponding penalties within each event and then across events to obtain LrelL_{rel}" leaves the aggregation across the four event types and across trajectories ambiguous (e.g., are events weighted equally regardless of interval length or trajectory count?).
  • "Rollout-based joint training" appears for the first time as a section title in Sec. 4.3 without a recap. It is unclear whether this refers to the same joint objective and continued-training stage described in Sec. 3.3 (L270–283) or to something additional; the paragraph should briefly restate what the stage consists of before ablating it.
  • Relatedly, the "w/o joint training" variant is underspecified: it is unclear whether this removes specific rollout data (e.g., event-bearing rollouts), specific loss terms (e.g., LrelL_{rel}), or the entire continued-training stage. Since the authors themselves note that "this comparison evaluates the training stage as a whole, including its data mixture and additional optimization," the ablation does not isolate the contribution of relation supervision from extra data and extra optimization at this stage.
  • The annotation pipeline depends on a proprietary VLM (two-pass annotation with GPT-5.6 Sol) plus human review, which limits reproducibility of the dataset-construction process itself even if the resulting annotations are released.

Novelty relative to prior work

  • The core ingredients are individually close to existing work: Robometer already combines temporal progress labels from demonstrations with supervision from real failures (success prediction and trajectory preferences), and WVM's Suboptimal-Value-Bench already converts annotated hesitation/retry intervals into reference progress structure. The contribution reduces mainly to replacing trajectory-level preferences with event-anchored within-trajectory margin constraints, which is a sensible but incremental delta.

Experimental rigor and support for claims

  • The real-world comparison uses a matched 30-minute wall-clock budget, but it is not reported how the actual number of training steps differs across methods under this setup, nor whether other resources (VRAM, RAM, reward-inference latency) are comparable. A comparison at matched training steps would help disentangle reward quality from per-step computational cost.
  • The efficiency claims do not account for annotation cost. The claim that Event2Reward reaches the rollout-success criterion in 15 minutes vs. 45 for Sparse ignores the time and human/VLM resources required to annotate event rollouts and adaptation trajectories; these should be counted or at least discussed when claiming faster robot learning.
  • The paper reports that Event2Reward outperforms Robometer but provides little analysis of why, given that Robometer also leverages progress and preference signals from real failures.
  • The interesting finding that Relation-only predictions "collapse to nearly identical values close to zero across trajectory stages" is reported but not explained; an analysis of why relation-only supervision degenerates would strengthen the complementarity claim.

Presentation

  • Figure 1: the text in the middle performance panels is too small, and headline results do not need to appear in the overview figure.
  • In Figure 2, it is unclear whether the upper and lower parts of the left-side images are related to each other; if not, they should be separated more clearly, as the current layout confuses the input/output of the model.
Questions for authors
  • What exactly is the "soft encoding with mean yty_t" used for the progress cross-entropy, and how sensitive is performance to its design?
  • Could the authors explicitly define ℓe(t)\ell_e(t) — is it the per-timestep penalty for event ee — and state the margin rules for all four event types in the main text?
  • Why do Relation-only progress predictions collapse to near-zero values? Can the authors provide more analysis or intuition for this degenerate behavior?
  • Is "rollout-based joint training" in Sec. 4.3 identical to the joint objective and continued-training stage of Sec. 3.3? What precisely is removed in the "w/o joint training" variant - data, loss terms, or the entire stage?
  • What do "20 trajectories/task (avg.)" and "4 trajectories/task (avg.)" mean exactly, and how were these numbers chosen?
  • How were the per-suite interaction budgets and training timesteps chosen?
  • Under the matched 30-minute wall-clock budget, how do the actual training steps and resource usage (VRAM/RAM, reward-inference latency) differ across methods? How does the comparison look at matched training steps?
  • Why are the time and resources spent on event annotation (for both the rollout dataset and target-task adaptation data) not counted in the learning-efficiency comparisons?
  • Why does Event2Reward outperform Robometer, which also exploits progress and preference signals from real failures? Can the authors provide a more detailed mechanistic analysis?

if the reviewer's concerns on the novelty of the method and comments on experiments section are addressed, the reviewer is willing to increase rating.

12 comments
AuthorsOpenReview ↗

Overview of our responses

Thank you for your detailed review and your specific comments on the novelty of our method and the experimental evidence. In response, we have clarified the differences between Event2Reward, Robometer, and WVM, added supervision ablations, and expanded the explanations of the training objective, real-robot experiments, and annotation costs.

To make our responses easier to examine, we have organized an anonymous response page around eight topics, bringing together video examples, progress-prediction curves, and experimental results. We encourage you to read the responses below alongside this page. Its synchronized videos and prediction curves show progress, stalls, and regressions during robot execution and how the models respond to these changes. Figures R3–R5 show the differences in the methods’ predictions when an object is dropped and re-grasped. Key results are also stated in the corresponding OpenReview comments.

  • [1/8] W7 · Novelty. We combine supervision of how much of a task is completed in successful demonstrations with supervision of whether events advance or set back the task in real suboptimal and failed executions. These event relations directly constrain the progress predictions used to compute shaping rewards, without requiring complete progress curves for suboptimal and failed executions.
  • [2/8] W10, Q9 · Why Event2Reward outperforms Robometer. Robometer's trajectory-preference supervision does not directly penalize local prediction errors in the progress head, whereas Event2Reward directly corrects these progress relations. In controlled experiments, Progress + Preference, which uses Robometer-style trajectory-comparison learning, achieves 95% comparison accuracy but a normalized policy-evaluation AUC of 19.4%, versus 36.7% for Progress + Relation. We also provide additional results on all ten LIBERO-Object tasks.
  • [3/8] W11, Q3 · Relation-only collapse. With relation supervision alone, a model can reduce the loss by shrinking prediction differences without correcting an incorrect progress relation. Successful demonstrations provide explicit progress targets that calibrate predictions across task stages and constrain this degeneration.
  • [4/8] W1–W3, W13, Q1–Q2 · Training objective. We have reorganized Section 3 to clarify what the model predicts, the progress targets from successful demonstrations, the relation penalties for the four event types, and how the losses are aggregated. The original Figure 2 has been removed and replaced by a new Figure 2 showing real-robot examples of the four event types.
  • [5/8] Q4, W4–W5 · Rollout joint training. Rollout joint training is the continued-training stage in the original §3.3. Its ablation skips this stage while retaining target-task adaptation. The revised §4.3 clarifies the purpose of this comparison. The results show that joint training yields higher success rates at every adaptation-data level and reaches comparable performance with less target-task data.
  • [6/8] Q5, Q6 · Data sizes and budgets. The averages of 20 and 4 trajectories per task correspond to totals of 200 and 40 adaptation trajectories across ten tasks. We have added an intermediate 100-trajectory setting and clarified these data sizes in the revised §4.3. This response also explains how the data sizes and per-suite interaction budgets were chosen.
  • [7/8] W8, Q7 · The 30-minute real-robot comparison. We explain the use of a matched wall-clock training budget and provide the actual environment steps, actor and critic updates for the three tasks, reward-inference latency, and GPU memory usage. Training time includes initial experience collection and resets, with reward inference scheduled between episodes.
  • [8/8] W9, W6, Q8 · Annotation cost and reproducibility. The 15-minute versus 45-minute comparison measures online robot-learning time. Target-task annotation and reward-model adaptation costs are reported separately. For the publicly released RoboChallenge data used in our paper, we report the API costs of generating candidate event annotations and human annotation throughput. We will release the final event annotations so researchers can use the data without repeating annotation.
AuthorsOpenReview ↗

[1/8] W7 · What does event supervision add beyond Robometer and WVM?

W7. The core ingredients are individually close to existing work: Robometer already combines temporal progress labels from demonstrations with supervision from real failures (success prediction and trajectory preferences), and WVM's Suboptimal-Value-Bench already converts annotated hesitation/retry intervals into reference progress structure. The contribution reduces mainly to replacing trajectory-level preferences with event-anchored within-trajectory margin constraints, which is a sensible but incremental delta.

We agree that using successful demonstrations and non-ideal executions to train reward models has precedents. Our core contribution is to directly supervise the progress predictions used to compute shaping rewards through progress relations between states before and after events. For real suboptimal and failed executions, it is difficult to label the task progress of each state. For example, when a robot picks up a cup and then drops it, it is difficult to determine the progress value to assign after the drop, but we can determine that progress is lower after the drop than before it. Event2Reward uses these relations directly as supervision and combines them with progress targets from successful demonstrations to train the model, without assigning specific progress values to these states in suboptimal and failed executions or prescribing complete progress curves for them.

 

Comparison with Robometer

Robometer learns from real failed executions through success prediction and trajectory preferences, but this supervision does not directly identify which transitions within an execution should be penalized. In the cup example above, even if the robot picks up the cup again and completes the task, preferring this successful trajectory over a failed one does not identify the drop as a transition to penalize. Since shaping rewards depend on differences between adjacent progress predictions, Ft=γpt+1−ptF_t=\gamma p_{t+1}-p_t, Event2Reward directly constrains post-drop progress to be lower than pre-drop progress, providing the local supervision needed for negative shaping feedback at the drop.

The video comparison (Figure R3) shows a bottle being dropped and then picked up again: Event2Reward’s progress prediction decreases after the drop and rises after re-grasping, while Robometer’s prediction does not clearly reflect the loss of progress caused by the drop. Robometer’s incorrect progress predictions produce incorrect reward feedback, encouraging the policy to learn incorrect actions. Our experiments also show that Event2Reward performs better in downstream policy learning (Table 9).

 

Comparison with WVM

WVM’s Suboptimal-Value-Bench constructs reference progress curves from annotated hesitation and retry intervals. These curves not only indicate that a regression occurred, but also specify how much progress decreases and how it subsequently recovers. In real executions, however, these changes depend on what the robot accomplishes. For example, after a bottle is dropped, it may remain on the table even as the arm keeps moving, so progress should not recover merely with the passage of time. Event2Reward directly supervises the model to predict lower progress after the drop than before it, and an increase after re-grasping. We do not need to assign a progress value to each of these states or prescribe a curve for the recovery.

This video illustration (Figure R4) makes the distinction concrete. After the bottle drops, the reference curve constructed using WVM’s rule returns to its pre-drop value at 10.3 seconds, but the bottle is still on the table and the robot has not yet picked it up again. Event2Reward’s progress prediction decreases after the drop and rises during re-grasping and lifting.

In addition, the reference curves in WVM’s Suboptimal-Value-Bench end at task completion, whereas our event supervision does not require an execution to ultimately succeed. Even if the robot never re-grasps the dropped cup and ultimately fails to complete the task, we can still determine that progress is lower after the drop than before it and use this relation to train the progress model. The real-robot rollout dataset we constructed contains 46,958 failed executions, accounting for 64.7% of all trajectories (Appendix A.1, Table 2). WVM cannot use these failed executions for training through the reference-curve supervision described above, whereas Event2Reward can use their event relations to train the progress model. Beyond the supervision method itself, we also define four types of task-progress events and construct a corresponding event-annotated dataset, enabling progress relations in real suboptimal and failed executions to be systematically annotated and used for reward-model training.

AuthorsOpenReview ↗

[2/8] W10 · Q9 · Why does Event2Reward outperform Robometer?

W10. The paper reports that Event2Reward outperforms Robometer but provides little analysis of why, given that Robometer also leverages progress and preference signals from real failures.

Q9. Why does Event2Reward outperform Robometer, which also exploits progress and preference signals from real failures? Can the authors provide a more detailed mechanistic analysis?

A key reason is that Event2Reward directly corrects the progress predictions used to compute shaping rewards. Robometer uses a preference head to compare entire executions and a separate progress head to predict task progress. Although the two heads share a backbone, the preference loss does not directly penalize local errors in the progress head. Thus, even when the model correctly ranks a successful trajectory above a failed one, it may still mistake a drop for progress. Event2Reward instead directly constrains post-drop progress to be lower than pre-drop progress and penalizes predictions that violate this relation.

The video example (Figure R5) illustrates this difference. The bottle drops onto the table and is later picked up again. Robometer’s progress predictions do not clearly distinguish the regression after the drop from the subsequent recovery. Event2Reward predicts a clear decrease after the drop and an increase as the robot re-grasps and lifts the bottle. This example illustrates that the ability to compare entire executions does not guarantee correct progress feedback during an execution.

 

The supervision ablations in §4.1 and Appendix C.1 further show that learning to compare trajectories does not necessarily yield more effective shaping rewards. Progress + Preference, which uses Robometer-style trajectory-comparison learning, achieves an average accuracy of 95% on unseen success–failure comparisons in LIBERO-10 (Table 10), confirming that it learned the comparison task. However, the results in §4.1 show a normalized evaluation AUC of 36.7 ± 7.8% for Progress + Relation versus 19.4 ± 14.3% for Progress + Preference (Figure 3 and Table 9). Progress-only achieves 19.6 ± 10.1% (Table 9). These results show that, in this experiment, despite achieving 95% trajectory-comparison accuracy, Progress + Preference performs similarly to Progress-only in policy learning, while Progress + Relation, which directly supervises progress relations, performs better.

In addition to the experiments in §4.1 of the main paper, we provide results on all ten LIBERO-Object tasks to examine the same difference in supervision on another suite. Each condition uses five environment seeds. Within 200k environment steps, Progress + Relation achieves 57.76 ± 6.59% evaluation success, compared with 44.28 ± 9.04% for Progress + Preference, 48.24 ± 7.35% for Progress-only, and 41.36 ± 4.08% for Relation-only (Figure R1).

The comparison between the full Event2Reward and Robometer systems is reported in §4.2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of responses and revisions

We thank all three reviewers for their careful reading and constructive comments. We respond to each reviewer in separate OpenReview comments and summarize the new evidence and manuscript revisions here.

 

1. Anonymous response pages

We have organized a dedicated response page for each reviewer, with video examples, synchronized progress-prediction curves, and experimental figures grouped by the concerns raised in your review. We encourage you to explore these visualizations to see how the model predictions correspond to the actual executions. The overview page provides the Area Chair and all reviewers with a summary of our responses and direct links to the supporting evidence. These pages are fully anonymous and do not track visitors, in accordance with ICLR’s requirements for demonstration links. Key numerical results and explanations are also included in our OpenReview comments.

 

2. Additional evidence

  • Broader supervision ablation. We extended the ablation from four LIBERO-10 tasks to all ten LIBERO-10 tasks and all ten LIBERO-Object tasks. Progress + Relation achieves the highest mean success and normalized evaluation AUC in both suites (AUC 28.36% vs. at most 17.58% on LIBERO-10, and 24.06% vs. at most 22.22% on LIBERO-Object).
  • How the rewards respond to setbacks. Videos with synchronized progress predictions show how Event2Reward and Robometer respond when an object is dropped and re-grasped.
  • Strong VLM baseline. An offline comparison with GPT-6 Astra on 200 LIBERO-Object trajectories excluded from adaptation (1,731 query points) shows complementary strengths across prediction metrics. This comparison evaluates progress predictions and does not include policy training with Astra rewards.
  • Real-robot budget. For the 30-minute comparison, we report the environment steps, actor and critic updates, reward-inference latency and GPU memory of each method.
  • Annotation cost. For the RoboChallenge data used in the paper, generating candidate event annotations consumed about 3.8 billion tokens (estimated API cost US$15,000–19,000), and human annotation runs at about 260–330 trajectories per person-hour. A screen recording shows the annotation of all 87 episodes of one real-robot run, requiring 9 minutes 53 seconds of active annotation time.
  • Precision assembly. We provide multiple examples across Plug Insert, RAM Insert and Ethernet Insert, showing execution videos with synchronized progress predictions. We also provide a video of online plug-insertion training with Event2Reward, in which the policy achieves 10 consecutive autonomous successes in approximately 15 minutes.

 

3. Manuscript revisions

To address the reviewers' concerns that the training objective and training stages were insufficiently explained, we have reorganized Section 3 to clarify the model's prediction target, progress-supervision targets, relation penalties, loss aggregation, and training stages (§3.5). We have also replaced the original Figure 2 with real-robot examples of the four event types to remove the ambiguity it created about model inputs and outputs.

To clarify the purpose of the joint-training ablation and the meaning of the adaptation data sizes, we have rewritten §4.3 and added an intermediate 100-trajectory setting to show performance across adaptation data sizes. To further explain why Relation-only collapses, we have added training diagnostics and controlled experiments in Appendix C.1.2.

Changes relative to the original submission are highlighted in blue in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.