- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, authors introduce event based reward model. Reward model learning signal comes from semantically meaningful events inside the trajectory rather than temporal anchors. Experiments show that event based reward model achieve both better reward modeling as well as downstream policy training performance.
Critical strengths
-
Event based reward model by learning from contrastive signal inside trajectory is novel and makes reward model directly learning from fail/suboptimal trajectories possible.
-
Compared to SOTA reward models, proposed method achieve better policy learning effect in both LIBERO simulation benchmark as well as several non-trivial manipulation tasks.
Critical weaknesses
-
Though with LLM assisted, but the human labeling/annotation seems still non-trivial work for 100K rollouts. The authors should provide efficiency related report on data curation and annotation process, including token usage, human labor hours etc.
-
The data and annotation visualization from page 19-26 are good examples, but they are all pick-and-place tasks, it remains questionable if a model solely trained on these tasks can achieve accurate reward modeling on precision assembly tasks authors used for downstream policy training like (a) Plug Insert (b) RAM Insert (c) Ethernet Insert. Can authors provide visualizations of reward model estimation of these tasks’ executions?
-
In table 4, authors classify the robot execution events into 4 categories, though a concise high-level abstract but this seems cannot work well on more detailed scenarios reward model should react, e.g. suboptimal execution - robot still progressing, but slower and more easy to make mistakes.
Questions for authors
In Table 4, what’s the difference between Progress stall and Unrecovered regression?
8 comments
[1/4] W1 · How much work does dataset annotation require?
W1. Though with LLM assisted, but the human labeling/annotation seems still non-trivial work for 100K rollouts. The authors should provide efficiency related report on data curation and annotation process, including token usage, human labor hours etc.
Dataset construction combines GPT-assisted annotation with human verification. Human annotators identify event types and mark key time points based on visible state changes in the videos, without assigning precise framewise progress values. The annotation rate is approximately 260–330 trajectories per person-hour, or about 11–14 seconds per trajectory. We manually annotated all 40,857 raw RoboChallenge trajectories used in our paper. Manual annotation took approximately 140 person-hours. Human annotations are compared with GPT proposals. Disagreements are reviewed by a third party, and annotation quality is checked through spot checks. As an example, for the publicly released RoboChallenge data used in our paper, we used GPT-5.6 Sol (medium reasoning) to propose candidate event annotations. Generating candidate event annotations for these data consumed approximately 3.8 billion tokens, with an estimated API cost of US$15,000–19,000.
To illustrate the annotation procedure, we provide a continuous screen recording covering all 87 complete episodes from the paper’s Ethernet-insertion training run (Figure R8). One author familiar with the event definitions completed these annotations in 9 minutes 53 seconds of active work, including revisions, averaging approximately 6.8 seconds per trajectory, equivalent to about 530 trajectories per hour. The recording shows the complete process of identifying events, marking key time points, and making revisions.
We will release the Event2Reward dataset, including the final event annotations used for training. Researchers can use the same annotations to train reward models and reproduce the policy-learning experiments without making additional GPT calls or repeating human annotation.
This response and the annotation recording are also available on the corresponding anonymous supplementary page.
[2/4] W2 · Reward prediction on precision assembly
W2. The data and annotation visualization from page 19-26 are good examples, but they are all pick-and-place tasks, it remains questionable if a model solely trained on these tasks can achieve accurate reward modeling on precision assembly tasks authors used for downstream policy training like (a) Plug Insert (b) RAM Insert (c) Ethernet Insert. Can authors provide visualizations of reward model estimation of these tasks’ executions?
In the precision-assembly experiments, we adapt the reward model using target-task data. As described in Section 4.4, we use five successful and five suboptimal trajectories for each of Plug Insert, RAM Insert, and Ethernet Insert to adapt the reward model, then freeze it during online policy learning. The pick-and-place annotation examples on pages 20–27 do not represent all the training data used in these experiments.
Following your suggestion, we provide additional prediction visualizations for the three precision-assembly tasks, with six examples per task and 18 in total, showing execution videos synchronized with raw progress predictions (Figure R15). These examples cover insertion, pauses, withdrawal, renewed approach, and incomplete executions, allowing the model’s responses to be examined alongside changes in task state.
We also provide a video of the paper’s online plug-insertion training run using Event2Reward reward shaping, showing the progression from human-guided warm-up to autonomous policy exploration and reaching 10 consecutive successful insertions after approximately 15 minutes (Figure R7).
Overview of our responses
Thank you for your careful reading of our paper and your specific questions about annotation effort, reward prediction for precision assembly, and event definitions.
To accompany our responses, we have organized an anonymous supplementary page for your review with a recording of the annotation process, precision-assembly videos synchronized with progress predictions, and simulation and real-robot examples illustrating stalls and unrecovered regressions. The page follows the same order as the four comments below, with links to each topic.
We encourage you to explore this page alongside the responses below. It provides additional interactive examples with synchronized execution videos and progress predictions, making it easier to examine the model’s behavior during execution and understand the distinction between stalls and unrecovered regressions.