acceptodds
Under review as a conference paper at ICLR 2027

Not Every Step Is Progress: Event-Grounded Credit Reassignment for Robot Policy Learning

Abstract

Learning from deployment experience can improve robot policies beyond demonstrations. Sparse episode-level outcomes, however, obscure how credit should be assigned within a rollout. Successful rollouts may include locally ineffective behavior, whereas failed rollouts may still exhibit substantial productive execution. Even dense progress or value supervision can conflate effective task progress with elapsed interaction time. To address these issues, we introduce **Event-Grounded Credit Reassignment (ECR)**, which learns local transition effectiveness from episode outcomes, human interventions, and sparse annotations of failure onset. ECR models uncertainty about when ineffective execution begins and marginalizes over plausible onset locations to construct soft supervision for otherwise unlabeled transitions. The resulting estimates can then revise value targets for advantage-conditioned policy learning and reward signals for offline and online reinforcement learning. We instantiate these settings with RECAP, IQL, and RL Token, respectively. Across four simulation tasks and four real-world tasks, ECR improves the mean success rate over RECAP by 31.0 percentage points and achieves 2.78× the mean throughput of RECAP. In offline evaluation on the same task suite, ECR achieves a mean AUROC of 95.3% and a mean AP of 93.3% for distinguishing effective from ineffective transitions. Additional experiments with IQL and RL Token show consistent gains in offline and online reinforcement learning settings. Our results demonstrate that correcting local credit assignment improves policy learning from mixed-quality robot experience.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, Credit Assignment, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wxiuOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper considers the problem of learning a credit assignment model leveraging successful and failed rollouts, as well as human interventions. Given human annotations on episode outcome, human correction, and failure timestep, the proposed approach learns the effectiveness logit ftf_t and onset logit btb_t. These logits are then converted to revise value signals for RL with, e.g., RECAP, IQL and RLToken. The experiment results on Libero-Pro and real robot experiments show that the learned progress estimator outperforms other existing models such as RynnValue-8B, Robo-Dopamine 2.0-8B, Robometer-4B and SVM.

Critical strengths

The paper identifies an important assumption commonly made by progress modeling: progress is often modeled as uniformly across successful demonstrations. This is in general not true for successful but suboptimal trajectories. Moreover, most progress modeling algorithms cannot effectively use failure demonstrations, which this paper aims to address.

Critical weaknesses
  1. The scalability of the proposed approach is questionable. The approach presented in this paper relies heavily on human demonstrations (6 seconds per trajectory per annotator). Moreover, the ECR models are trained per-task, meaning that new annotation needs to be obtained for each new task. This makes it difficult to use the approach for general purpose robotics progress modeling.

  2. The experiment results are limited. Only 1 out of the 10 tasks from each Libero-Pro suite was evaluated, and over only 2 random seeds. The comparison with most baselines is not entirely fair: most models are held fixed and evaluated zero-shot, whereas the approach uses on-task annotated data to train task specific models. A more fair comparison would require fine-tuning the baseline models on the same set of data.

Questions for authors
  • Are ECR models trained per-task?
  • If all baselines are trained with the same set of annotated data, are they still performing worse than ECR?
  • Since PACT (mentioned in the related work section) is closely related to ECR, why is it not selected as another baseline for comparison?
  • Does the observation obtained from a single task from Libero-Pro suites also hold over the entire suite, i.e., all ten tasks?
  • In Table 7, why are detection and policy scores for different models computed differently?
  • How to speed up or reduce human annotation?
9 comments
AuthorsOpenReview ↗

Response to Reviewer wxiu [1/4]

We thank the reviewer for the careful review and constructive feedback. We address the questions in detail below.

1. Annotation efficiency and cost — W1 (annotation cost) and Q6

“The approach presented in this paper relies heavily on human demonstrations (6 seconds per trajectory per annotator). … How to speed up or reduce human annotation?”

Additional temporal annotation is required only for failed trajectories, which account for 11.1% of our training data. Each task contains 80 successful trajectories, including demonstrations, and 10 failed trajectories (§4.1; Appendix B.1). Each failed trajectory requires one onset timestamp; successful trajectories require no manual temporal annotation.

In our rollout collection, operators record candidate onsets via a hotkey (Appendix B.3). A marker denotes the start of a new ineffective segment; repeated attempts within an uninterrupted ineffective segment do not create new onsets. Successful episodes produce no failure-suffix supervision. For failed episodes, we retain the last valid onset in the last autonomous segment, excluding earlier errors from which execution recovered. We clarify this rule in Appendix B.3.

The reported 6.0 seconds per failed trajectory measures offline labeling of existing data, not additional labeling time after online onset recording. In Appendix B.2, three annotators independently labeled the same 80 failed trajectories with collection-time markers hidden; timing included playback, revisiting, and confirmation.

Onset recording can be integrated into rollout collection, eliminating the need for a separate onset-labeling pass even at larger collection scales. Each failed trajectory contributes one retained timestamp, supervising both onset localization and the final ineffective suffix. These sparse markers enable local credit assignment within failed experience without frame-by-frame training annotation.

2. Task-specific policy post-training — W1 (task-specific training) and Q1

“Moreover, the ECR models are trained per-task, meaning that new annotation needs to be obtained for each new task. This makes it difficult to use the approach for general purpose robotics progress modeling. … Are ECR models trained per-task?”

Yes, ECR is trained separately for each task in our experiments. We study robot policy post-training: using target-task demonstrations, autonomous rollouts, and human corrections to efficiently improve a pretrained VLA policy’s execution performance on that task. Target-task policy optimization is also used in representative post-training approaches such as π*₀.₆ / RECAP [1] and RL Token [2], while SARM [3] provides a related example of single-task reward modeling. ECR contributes local credit-assignment signals to this setting.

We evaluated ECR’s generalization to unseen trajectories within each task using five-fold cross-validation. Across four simulation tasks and four real-world tasks, ECR achieved macro-average AUROCs of 97.1 ± 0.4% and 93.5 ± 2.5%, respectively (Appendix Table 16). Test trajectories were excluded from both estimator fitting and normalization statistics, and their onset labels were not used during inference (Appendix C.2). These results support reusing the estimator trained for each task to assess local transition effectiveness in new rollouts, extending the benefit of sparse event supervision beyond the annotated training trajectories.

References

[1] Amin et al. π*₀.₆: a VLA That Learns From Experience. In RSS, 2026.

[2] Xu et al. RL Token: Bootstrapping Online RL with Vision-Language-Action Models. In CoRL, 2026.

[3] Chen et al. SARM: Stage-Aware Reward Modeling for Long Horizon Robot Manipulation. In ICLR, 2026.

AuthorsOpenReview ↗

Response to Reviewer wxiu [2/4]

3. Coverage of all ten Object tasks and independent training runs — W2 (task coverage and seeds) and Q4

“The experiment results are limited. Only 1 out of the 10 tasks from each Libero-Pro suite was evaluated, and over only 2 random seeds. … Does the observation obtained from a single task from Libero-Pro suites also hold over the entire suite, i.e., all ten tasks?”

ECR’s gains in average success rate and throughput persisted over the complete LIBERO-Pro Object suite. We evaluated all ten tasks under the Position perturbation setting, using three independent training seeds per method and 100 evaluation episodes per task for each trained policy.

RECAP, Robo-Dopamine 2.0 (FT) + RECAP, and ECR + RECAP were compared under matched training and evaluation conditions.

Table R1. Policy performance across all ten LIBERO-Pro Object tasks under Position perturbations. Mean ± sample SD across three independent training runs.

MethodSuccess rate (%) ↑Speedup (×) ↑Throughput (tasks/h) ↑
RECAP63.5 ± 1.81.42 ± 0.0362.0 ± 1.9
Robo-Dopamine 2.0 (FT) + RECAP87.8 ± 1.41.67 ± 0.02114.0 ± 2.3
ECR + RECAP97.4 ± 1.11.91 ± 0.02154.4 ± 3.0

ECR improved the average success rate by 9.6 percentage points and throughput by 35.4% over Robo-Dopamine 2.0 (FT) + RECAP. Its mean success rate and throughput were higher on nine of the ten tasks. Results for individual tasks are provided in Appendix F.1, Table 15.

AuthorsOpenReview ↗

Response to Reviewer wxiu [3/4]

4. Baselines trained on target-task data — W2 (baseline adaptation) and Q2

“The comparison with most baselines is not entirely fair: most models are held fixed and evaluated zero-shot, … A more fair comparison would require fine-tuning the baseline models on the same set of data. … If all baselines are trained with the same set of annotated data, are they still performing worse than ECR?”

ECR maintains its advantage in transition effectiveness and policy performance after all three baselines are fine-tuned on target task data. We fine-tuned RynnValue [4], Robo-Dopamine 2.0 [5], and Robometer [6] on the same trajectory splits as ECR across four simulation and four real-world tasks, retaining their native training objectives. All models were evaluated on the same held-out autonomous transitions using the same dense transition labels and metrics. Downstream policy comparisons used RECAP with identical learning settings.

Table R2. Transition effectiveness on held-out trajectories across four simulation and four real-world tasks (Appendix Table 16). Four-task macro-averages: five-fold mean ± sample standard deviation (%).

MethodSim. AUROC ↑Sim. AP ↑Sim. Success AUROC ↑Sim. Failure AUROC ↑Real AUROC ↑Real AP ↑Real Success AUROC ↑Real Failure AUROC ↑
RynnValue (FT)68.2 ± 0.770.7 ± 1.266.1 ± 2.072.7 ± 1.968.0 ± 2.064.2 ± 3.267.3 ± 1.568.5 ± 4.6
Robo-Dopamine 2.0 (FT)81.9 ± 0.984.0 ± 1.779.0 ± 0.985.0 ± 2.575.5 ± 1.673.2 ± 2.373.7 ± 1.675.6 ± 3.1
Robometer (FT)73.4 ± 1.275.4 ± 2.272.2 ± 1.577.5 ± 3.768.5 ± 1.867.0 ± 3.466.9 ± 2.574.3 ± 3.0
ECR97.1 ± 0.496.8 ± 0.597.3 ± 0.496.2 ± 1.193.5 ± 2.589.8 ± 3.894.5 ± 2.191.6 ± 4.4

ECR outperforms all three fine-tuned baselines in both AUROC and AP across the simulation and real-world evaluations. Its macro-average AUROCs reach 97.1 ± 0.4% and 93.5 ± 2.5%, respectively (Table R2).

Table R3a. Simulation policy performance within RECAP (100 trials/task). Trial counts are per method; task columns report success rates (%, ↑).

MethodSpatialObjectGoalLongMean success rate (%)Speedup on successful episodes (×)Throughput (tasks/h)
RynnValue (FT)58.081.048.095.070.51.3880.9
Robo-Dopamine 2.0 (FT)79.090.068.085.080.51.43109.0
Robometer (FT)72.081.061.087.075.31.2685.9
ECR91.099.098.098.096.51.66173.5

Table R3b. Real-world policy performance within RECAP (15 trials/task). Task columns report success rates (%, ↑).

MethodPutStoreInsertFastenMean success rate (%)Speedup on successful episodes (×)Throughput (tasks/h)
RynnValue (FT)86.760.066.726.760.01.2746.1
Robo-Dopamine 2.0 (FT)93.380.060.053.371.71.3357.4
Robometer (FT)80.066.760.040.061.71.2641.1
ECR100.086.780.066.783.31.6982.8

ECR achieves higher policy success than all three fine-tuned baselines on each of the eight tasks. Mean success rates reach 96.5% in simulation and 83.3% on the real robot, with the highest aggregate speedup and throughput in both domains (Tables R3a–R3b). These results show that ECR’s advantage after baseline fine-tuning extends from local transition discrimination to downstream policy performance.

For the onset model itself, Table 4 compares ECR variants under matched supervision. Relative to Direct, Event-CDF improves simulation AP from 88.1% to 96.8% and policy success from 88.8% to 96.5%.

References

[4] Huang et al. RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance. arXiv:2608.09853, 2026.

[5] Xu et al. Robo-Dopamine 2.0: History-Conditioned and OOD-Aware Process Reward Modeling for Robotic Manipulation. arXiv:2608.15680, 2026.

[6] Liang et al. Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons. In RSS, 2026.

Reviewer_U8JjOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed Event-Grounded Credit Reassignment (ECR) approach that learns whether a completed robot transition was locally effective, using demonstrations, human interventions, and one failure-onset annotation per failed rollout. An onset distribution over each event window produces soft effectiveness targets through its cumulative distribution. The learned signal revises value targets or augments rewards for IQL and RL Token. The contribution is the Event-CDF coupling and its use in policy supervision. Across four simulation and four real-world tasks, the proposed approach reports mean success gain over RECAP. Five-fold, trajectory-held-out evaluation tests autonomous transitions, and matched Direct, MAP-target, and stop-gradient ablations support the coupling design.

Critical strengths

ECR clearly identifies a weakness of outcome-based supervision. The notions are: productive behavior can occur inside failed episodes, and mistakes can occur inside successful ones. Its event-onset distribution provides a concrete way to produce soft effectiveness targets for completed transitions.

The downstream pattern is strong. ECR improves success over RECAP on all four simulation and four physical tasks.

The effectiveness evaluation is helpful. Held-out splits are defined by trajectory rather than random frames, copies remain together, and autonomous-only evaluation excludes demonstration and human-action anchors. Reporting discrimination separately within successful and failed episodes tests more than recovery of an episode-level outcome label.

The paper conducts a genuinely matched Direct baseline. With matched features, architecture, annotations and updates, Event-CDF improves AP and policy success.

Critical weaknesses

For each candidate onset, Eq.1 labels every earlier transition effective and every later transition ineffective. Marginalizing these labels gives a non-increasing target within the window. An error followed by recovery and another error cannot be represented faithfully by that target, even though such behavior motivates distinguishing local effectiveness from episode outcome. B.2 defines the final ineffective suffix, and Sec. 5 acknowledges the single-onset limitation. Therefore, an earlier ineffective interval can be given favorable supervision because it precedes that final suffix. This restricts the event-derived training targets, not the global expressiveness of the effectiveness predictor.

The held-out results and successful-trajectory examples show that the learned predictor can detect multiple errors, so it would not be apt to claim it is incapable of recovery-sensitive scoring. My concern is the unmeasured extent and consequence of the target bias. Results separated by windows with and without earlier error-recovery cycles would test whether Event-CDF helps in the very cases where its assumption is least accurate.

The full approach has large gains, but the more specific CDF claim rests on a smaller difference cf., Table 4.

The incremental benefit of Event-CDF needs stronger uncertainty and comparison controls. C.1 makes clear that the two simulation seeds are evaluation seeds for a single trained checkpoint. There is no corresponding real-world component ablation, and each real task has 15 autonomous trials. The AP error bars across trajectory folds address a different source of uncertainty from variation in policy training.

The comparison with pretrained progress models also needs a precise interpretation. B.5 keeps those models fixed, and several receive fewer sampled frames and camera views, without executed actions, compared with ECR. For instance, Robometer was designed for progress and trajectory preferences. Therefore, the large Tab 3 gap supports the practical benefit of a task-trained transition estimator….. it does not isolate the onset model. The matched Direct/MAP/stop-gradient experiment in D.4 substantially addresses that issue, which is a major strength. Independent training repetitions and a real-world version of that comparison would make the attribution more convincing.

Questions for authors

How often do event windows contain an ineffective interval followed by recovery before the final onset? How do Direct, MAP targets, and Event-CDF compare on that subset, including the retained quality of recovery actions?

What variability does the CDF-versus-MAP policy improvement show across independent training runs? Can the same matched-supervision comparison be reported on the real robot with trial counts and uncertainty?

How is agreement among onset and ineffective-interval annotators measured, and how sensitive are the results to plausible onset shifts? Does the learned onset distribution assign calibrated uncertainty to ambiguous boundaries, or is its demonstrated role principally a soft latent assignment?

5 comments
AuthorsOpenReview ↗

Response to Reviewer U8Jj [1/4]

We thank the reviewer for the careful review and constructive feedback. We address the questions in detail below.

1. Single-onset supervision and recovery — W1 and Q1

“Results separated by windows with and without earlier error-recovery cycles would test whether Event-CDF helps in the very cases where its assumption is least accurate. … How often do event windows contain an ineffective interval followed by recovery before the final onset? How do Direct, MAP targets, and Event-CDF compare on that subset, including the retained quality of recovery actions?”

Event-CDF outperforms Direct and MAP-target on all four metrics in windows containing earlier error–recovery cycles. We group event windows according to whether an earlier ineffective interval followed by effective recovery occurs before the final ineffective segment within the same window. Among 493 evaluated event windows, 87 (17.6%) contain such earlier error–recovery cycles, and the remaining 406 do not.

Following the trajectory-held-out protocol in Appendix C.2, we compare matched Direct, MAP-target, and Event-CDF variants in both groups. AP and AUROC measure transition discrimination. For the earlier error–recovery intervals, earlier-error recall measures the fraction of ineffective transitions correctly classified as ineffective, while effective-recovery recall measures the fraction of subsequent effective recovery transitions correctly classified as effective.

Table R1. Transition-effectiveness evaluation across four simulation tasks, grouped by whether earlier error–recovery occurs before the final ineffective segment within the same event window. Four-task macro-averages; five-fold mean ± sample standard deviation (%).

Event-window groupMethodAP ↑AUROC ↑Earlier-error recall ↑Effective-recovery recall ↑
With earlier error–recoveryDirect82.1 ± 1.080.7 ± 1.079.6 ± 0.890.9 ± 0.2
With earlier error–recoveryMAP-target88.0 ± 2.686.8 ± 3.584.2 ± 0.594.2 ± 0.7
With earlier error–recoveryEvent-CDF94.8 ± 0.594.6 ± 0.689.0 ± 0.896.6 ± 0.2
Without earlier error–recoveryDirect90.3 ± 1.092.2 ± 0.9——
Without earlier error–recoveryMAP-target94.8 ± 2.695.9 ± 2.1——
Without earlier error–recoveryEvent-CDF97.7 ± 0.598.2 ± 0.4——

Within the error–recovery subset, Event-CDF exceeds MAP-target by 6.8 percentage points in AP and 7.8 points in AUROC. It also achieves 89.0% earlier-error recall and 96.6% effective-recovery recall, identifying earlier ineffective actions while preserving subsequent effective recovery.

This behavior is consistent with how ECR combines event supervision with transition-level prediction. The single-onset structure constrains the event-derived targets; the effectiveness head predicts from observation–action context without a monotonicity constraint. It also learns from demonstration and human-correction anchors, as well as annotated failure suffixes. These complementary signals allow it to distinguish local ineffective execution from subsequent effective recovery. Event-CDF further marginalizes over candidate onset locations, providing soft supervision instead of committing to the single hard boundary used by MAP-target. The matched comparison supports the benefit of this soft supervision even in windows containing earlier error–recovery cycles.

AuthorsOpenReview ↗

Response to Reviewer U8Jj [2/4]

2. Independent training and matched real-world comparison — W2 and Q2

“The incremental benefit of Event-CDF needs stronger uncertainty and comparison controls. … What variability does the CDF-versus-MAP policy improvement show across independent training runs? Can the same matched-supervision comparison be reported on the real robot with trial counts and uncertainty?”

Event-CDF improves policy success across three independent training runs on each of two real-world tasks, Insert and Store. For every run, we retrained both the effectiveness estimator and the downstream RECAP policy. The comparison follows the matched protocol in Appendix D.4, varying soft Event-CDF versus hard MAP-target supervision.

Each method was evaluated over 120 real-world trials. Table R2 reports per-run success counts and mean ± sample standard deviation across independent runs. SR and TPH follow Appendix C.1.

Table R2. Matched comparison on two real-world tasks across three independent training runs, with 20 trials per trained policy. SR and TPH are reported as mean ± sample standard deviation across runs.

TaskMethodSuccesses/trials by runSR (%) ↑TPH ↑
InsertMAP-target13/20, 13/20, 11/2061.7 ± 5.848.8 ± 6.9
InsertEvent-CDF16/20, 17/20, 16/2081.7 ± 2.987.8 ± 4.5
StoreMAP-target13/20, 14/20, 15/2070.0 ± 5.053.9 ± 6.4
StoreEvent-CDF17/20, 17/20, 18/2086.7 ± 2.988.4 ± 6.8
Macro-averageMAP-target—65.8 ± 1.451.3 ± 1.7
Macro-averageEvent-CDF—84.2 ± 1.488.1 ± 3.2

On each task, every Event-CDF run achieves a higher success rate than every MAP-target run. Mean success increases from 61.7% to 81.7% on Insert and from 70.0% to 86.7% on Store. Across the three runs, success-rate standard deviations are 2.9 percentage points for Event-CDF on both tasks, compared with 5.8 on Insert and 5.0 on Store for MAP-target. Mean throughput increases by approximately 80% and 64%, respectively.

These independent-training results extend the matched component analysis in Table 4 to real-world policy learning: under the matched protocol, Event-CDF yields consistent success-rate improvements across the evaluated training runs and higher mean throughput on both tasks.

AuthorsOpenReview ↗

Response to Reviewer U8Jj [3/4]

3. Interpretation of value/progress-model comparisons — W3

“The comparison with pretrained progress models also needs a precise interpretation. … the large Tab 3 gap supports the practical benefit of a task-trained transition estimator … it does not isolate the onset model.”

We evaluate the value/progress models using inference setups based on their original papers, with task-observation adaptations described in Appendix B.5. Following Table 7, we use decreases in predicted value or progress to detect ineffective transitions. All models are evaluated on the same held-out autonomous transitions using the same human annotations. To make the comparison fairer, we additionally fine-tuned RynnValue [1], Robo-Dopamine 2.0 [2], and Robometer [3] on the same target-task trajectory splits as ECR, using their native training objectives. The following tables report both held-out transition discrimination and downstream RECAP performance across four simulation and four real-world tasks.

Table R3. Transition effectiveness. Four-task macro-averages per domain: five-fold mean ± sample SD (%).

MethodSim. AUROC ↑Sim. AP ↑Sim. Success AUROC ↑Sim. Failure AUROC ↑Real AUROC ↑Real AP ↑Real Success AUROC ↑Real Failure AUROC ↑
RynnValue (FT)68.2 ± 0.770.7 ± 1.266.1 ± 2.072.7 ± 1.968.0 ± 2.064.2 ± 3.267.3 ± 1.568.5 ± 4.6
Robo-Dopamine 2.0 (FT)81.9 ± 0.984.0 ± 1.779.0 ± 0.985.0 ± 2.575.5 ± 1.673.2 ± 2.373.7 ± 1.675.6 ± 3.1
Robometer (FT)73.4 ± 1.275.4 ± 2.272.2 ± 1.577.5 ± 3.768.5 ± 1.867.0 ± 3.466.9 ± 2.574.3 ± 3.0
ECR97.1 ± 0.496.8 ± 0.597.3 ± 0.496.2 ± 1.193.5 ± 2.589.8 ± 3.894.5 ± 2.191.6 ± 4.4

Table R4a. Simulation policy performance within RECAP (100 trials/task). Trial counts are per method; task columns report success rates (%, ↑).

MethodSpatialObjectGoalLongMean success rate (%)Speedup on successful episodes (×)Throughput (tasks/h)
RynnValue (FT)58.081.048.095.070.51.3880.9
Robo-Dopamine 2.0 (FT)79.090.068.085.080.51.43109.0
Robometer (FT)72.081.061.087.075.31.2685.9
ECR91.099.098.098.096.51.66173.5

Table R4b. Real-world policy performance within RECAP (15 trials/task). Task columns report success rates (%, ↑).

MethodPutStoreInsertFastenMean success rate (%)Speedup on successful episodes (×)Throughput (tasks/h)
RynnValue (FT)86.760.066.726.760.01.2746.1
Robo-Dopamine 2.0 (FT)93.380.060.053.371.71.3357.4
Robometer (FT)80.066.760.040.061.71.2641.1
ECR100.086.780.066.783.31.6982.8

These results have been added to Tables 1–3 in the revised main paper.

ECR remains ahead after task-specific fine-tuning of the baselines. Relative to Robo-Dopamine 2.0 (FT), mean policy success increases by 16.0 percentage points in simulation and 11.7 points on the real robot, with throughput gains of 59.1% and 44.2%, respectively.

For the onset model itself, Table 4 compares ECR variants under matched supervision. Relative to Direct, Event-CDF improves simulation AP from 88.1% to 96.8% and policy success from 88.8% to 96.5%. The independent training runs reported in Response 2 further demonstrate its advantage over MAP targets in real-world policy learning.

References

[1] Huang et al. RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance. arXiv:2608.09853, 2026.

[2] Xu et al. Robo-Dopamine 2.0: History-Conditioned and OOD-Aware Process Reward Modeling for Robotic Manipulation. arXiv:2608.15680, 2026.

[3] Liang et al. Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons. In RSS, 2026.

Reviewer_JfecOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Event-Grounded Credit Reassignment (ECR), a framework for improving credit assignment in robot policy learning from mixed-quality execution trajectories. The authors argue that episode-level success signals and conventional progress-based reward models may fail to accurately capture the effectiveness of individual transitions, as successful trajectories can contain ineffective actions and failed trajectories can include meaningful progress.

To address this issue, ECR learns transition-level effectiveness from expert demonstrations, human interventions, and sparse failure-onset annotations. The proposed Event-CDF coupling models uncertainty over the onset of ineffective execution and marginalizes over possible onset locations to generate soft supervision for otherwise unlabeled transitions. The learned effectiveness estimates are subsequently incorporated into policy optimization through value-target revision for RECAP and reward correction for IQL and RL Token.

Critical strengths

Flexible integration with different policy optimization frameworks.

The proposed effectiveness estimator is not restricted to a single downstream learning algorithm. The authors demonstrate its integration with RECAP through value-target revision and with IQL and RL Token through reward correction. This flexibility makes the method potentially useful across advantage-conditioned imitation learning and offline/online reinforcement learning settings.

Critical weaknesses

In the current experiment setting, Robometer and RynnValue are evaluated primarily through progress difference based policy conditioning, rather than their native reward-shaping RL interfaces. Furthermore, ECR utilizes task specific intervention and failure-onset supervision, while the pretrained baselines are evaluated without corresponding adaptation. This might be explaining why ECR exhibit some improvement on the selected tasks. It would be helpful to clarify how the baseline interfaces and input configurations were selected. In particular, extending Table 5 to compare IQL with ECR against IQL with Robometer or RynnValue rewards would provide a more direct assessment of the proposed reward correction mechanism. The current comparison between IQL and IQL+ECR are less convincing than to compare with the other reward estimators directly.

Missing baselines

The experimental evaluation does not include recent methods such as SARM (ICLR 2026) and ReWiND (CoRL 2025), which explicitly aim to improve progress estimation under execution failures and non-monotonic behavior. Both methods provide continuous progress estimates that could potentially be incorporated into the existing progress-based policy-conditioning interface. Although SARM was originally tested in long horizon tasks, some potential adaptations should be able to compare the effectiveness of SARM versus the proposed method. Nevertheless, a downstream policy comparison, particularly with ReWiND, or a clearer explanation for why they are not discussed in the paper would be helpful. Without these comparisons, the authors claim on the practical advantages of the current framework over more advanced progress estimation approaches are not supported.

Concerns about generalizability and scalability of human supervised credit assignment

Another major concern is the generalizability and scalability of the proposed credit reassignment framework. The current experiments rely on task-specific demonstrations, human interventions, and annotated failure onsets to train the effectiveness estimator. Although the method achieves strong performance under current evaluations, all evaluations are conducted on tasks for which task-specific training data and supervision are available. It remains unclear whether the learned effectiveness estimator can generalize across different tasks, policies, or previously unseen failure patterns without additional human supervision.

Furthermore, the reliance on human interventions raises concerns about how well the approach can scale. While the authors report an average failure-onset annotation time of only six seconds, this still seems to be a huge cost if ECR can only be used under a single task setting. The paper also lacks an analysis of how performance scales with the availability and quality of these supervision signals. Additionally, ECR has not been tested on more complex, long-horizon tasks involving repeated errors, recoveries, or previously unseen failure modes.

Questions for authors
  1. How does the proposed method compare to the recent progress estimation models such as SARM and ReWiND? Can the authors provide some insights as well as empirical evidence of how they compare with the proposed framework?
  2. Can ECR learn reusable execution-quality features and does its failure detector remains reliable as the policy changes? It would be nice for the authors to answer this by running some cross-task/cross-policy experiments. It is unclear whether ECR learns a generalizable notion of transition effectiveness or primarily exploits task-specific human feedback to correct credit assignments in a narrow set of previously observed execution scenarios.
  3. Recent agentic frameworks, such as ASPIRE, have achieved nearly 100% across all tasks in LIBERO-PRO. How does the proposed method compare to ASPIRE? Note that this is not a weakness per se as ASPIRE was released after the submission deadline, but it is still highly relevant since the concept of learning reuseable skills is quite similar to the fundamental ideas behind credit assignment.
7 comments
AuthorsOpenReview ↗

Response to Reviewer Jfec [1/4]

We thank the reviewer for the constructive feedback. We address the questions in detail below.

1. Baseline adaptation, inputs, and downstream interfaces — W1

“Robometer and RynnValue are evaluated primarily through progress difference based policy conditioning, rather than their native reward-shaping RL interfaces. … the pretrained baselines are evaluated without corresponding adaptation. … clarify how the baseline interfaces and input configurations were selected. … extending Table 5 to compare IQL with ECR against IQL with Robometer or RynnValue rewards …”

ECR maintains its advantage in transition effectiveness and policy performance after baseline fine-tuning. We fine-tune RynnValue, Robo-Dopamine 2.0, and Robometer on the same target-task trajectory splits across four simulation and four real-world tasks, retaining their original training objectives (Tables R1–R2b).

Table R1. Transition effectiveness. Four-task macro-averages per domain: five-fold mean ± sample SD (%).

MethodSim. AUROC ↑Sim. AP ↑Sim. Success AUROC ↑Sim. Failure AUROC ↑Real AUROC ↑Real AP ↑Real Success AUROC ↑Real Failure AUROC ↑
RynnValue (FT)68.2 ± 0.770.7 ± 1.266.1 ± 2.072.7 ± 1.968.0 ± 2.064.2 ± 3.267.3 ± 1.568.5 ± 4.6
Robo-Dopamine 2.0 (FT)81.9 ± 0.984.0 ± 1.779.0 ± 0.985.0 ± 2.575.5 ± 1.673.2 ± 2.373.7 ± 1.675.6 ± 3.1
Robometer (FT)73.4 ± 1.275.4 ± 2.272.2 ± 1.577.5 ± 3.768.5 ± 1.867.0 ± 3.466.9 ± 2.574.3 ± 3.0
ECR97.1 ± 0.496.8 ± 0.597.3 ± 0.496.2 ± 1.193.5 ± 2.589.8 ± 3.894.5 ± 2.191.6 ± 4.4

Table R2a. Simulation policy performance within RECAP (100 trials/task). Trial counts are per method.

MethodMean success rate (%) ↑Speedup on successful episodes (×) ↑Throughput (tasks/h) ↑
RynnValue (FT)70.51.3880.9
Robo-Dopamine 2.0 (FT)80.51.43109.0
Robometer (FT)75.31.2685.9
ECR96.51.66173.5

Table R2b. Real-world policy performance within RECAP (15 trials/task).

MethodMean success rate (%) ↑Speedup on successful episodes (×) ↑Throughput (tasks/h) ↑
RynnValue (FT)60.01.2746.1
Robo-Dopamine 2.0 (FT)71.71.3357.4
Robometer (FT)61.71.2641.1
ECR83.31.6982.8

In both simulation and the real world, ECR outperforms all three fine-tuned baselines in AUROC and AP, as well as mean policy success and throughput. These results have been added to Tables 1–3 in the revised main paper.

We use the baselines’ published input configurations and RECAP as a common VLA post-training framework under identical downstream learning settings. For RynnValue, Robo-Dopamine 2.0, and Robometer, interval progress increments determine the relative ranking of execution segments for policy conditioning. ECR revises value targets using transition effectiveness and constructs policy conditions from the resulting value estimates. Interval endpoints, the task-wise top-30% conditioning rule, and actor training settings are kept identical across methods (Appendices B.5–B.6; Table 7).

Inputs follow published inference protocols. RynnValue and Robometer use the instruction and four uniformly sampled frames from the causal primary-camera history. Robo-Dopamine 2.0 uses start/goal references from a fixed training-split demonstration, multi-view state pairs, and observed rollout-history panels. We use its forward progress prediction for offline evaluation, RECAP conditioning, and IQL reward shaping.

Direct comparison through native reward-shaping RL interfaces. RynnValue and Robometer follow the potential-based shaping protocol in RynnValue; Robo-Dopamine 2.0 uses its forward progress prediction as the potential. The fine-tuned models add κm(γΦh+1(m)−Φh(m))\kappa_m(\gamma\Phi_{h+1}^{(m)}-\Phi_h^{(m)}) to the same task reward, with hh indexing chunks (Appendix E). Table R3 compares policy success on two real-world tasks, Insert and Store, under the same IQL setup.

Table R3. IQL success rates (%, ↑) on two real-world tasks (15 trials/task). Avg. denotes the equal-task mean.

MethodInsertStoreAvg.
IQL46.760.053.3
IQL + RynnValue (FT)40.060.050.0
IQL + Robo-Dopamine 2.0 (FT)53.360.056.7
IQL + Robometer (FT)40.046.743.3
IQL + ECR73.373.373.3

ECR achieves the highest success rate on both tasks, with mean success of 73.3%, compared with 56.7% for the strongest fine-tuned baseline, Robo-Dopamine 2.0, and 53.3% for IQL alone. ECR retains its advantage when the fine-tuned baselines are evaluated through their native reward-shaping RL interfaces.

AuthorsOpenReview ↗

Response to Reviewer Jfec [2/4]

2. Comparison with SARM and ReWiND — W2 and Q1

“The experimental evaluation does not include recent methods such as SARM (ICLR 2026) and ReWiND (CoRL 2025), … a downstream policy comparison, particularly with ReWiND, or a clearer explanation for why they are not discussed in the paper would be helpful. … How does the proposed method compare to the recent progress estimation models such as SARM and ReWiND?”

A common challenge for value and progress models that rely on temporal supervision is that advancing time does not necessarily indicate effective behavior. SARM improves progress alignment through semantic stage decomposition, while ReWiND introduces non-monotonic progress supervision through video rewinding. Their dense progress targets remain derived from demonstration timing: within-stage linear interpolation in SARM and normalized temporal indices reversed with the video in ReWiND [1, 2]. Consequently, stalls or ineffective repetitions within demonstrations may still receive increasing progress targets, creating a mismatch between progress supervision and local behavioral effectiveness. ECR uses intervention and failure-onset events to supervise transition effectiveness, directly addressing local credit assignment in mixed-quality executions.

We selected RynnValue, Robo-Dopamine 2.0, and Robometer as representative pretrained value and progress models. Following the reviewer’s suggestion, we broaden the comparison to include SARM and ReWiND. The evaluation covers transition effectiveness and downstream RECAP performance across four simulation and four real-world tasks, with five-fold cross-validation for transition effectiveness.

Table R4. Transition effectiveness: SARM, ReWiND, and ECR. Four-task macro-averages per domain; reporting follows Table R1.

MethodSim. AUROC ↑Sim. AP ↑Sim. Success AUROC ↑Sim. Failure AUROC ↑Real AUROC ↑Real AP ↑Real Success AUROC ↑Real Failure AUROC ↑
SARM62.5 ± 1.968.1 ± 3.355.8 ± 3.769.6 ± 7.471.4 ± 2.265.3 ± 1.768.6 ± 2.576.7 ± 2.8
ReWiND56.5 ± 2.265.6 ± 4.152.4 ± 4.557.4 ± 4.358.1 ± 1.556.0 ± 1.757.3 ± 2.452.7 ± 1.5
ECR97.1 ± 0.496.8 ± 0.597.3 ± 0.496.2 ± 1.193.5 ± 2.589.8 ± 3.894.5 ± 2.191.6 ± 4.4

On the same evaluated autonomous transitions and labels, ECR achieves higher AUROC and AP in both domains, including higher AUROC within successful and failed trajectories.

Downstream comparison within RECAP. We convert SARM and ReWiND progress estimates into interval progress increments through the same baseline policy-conditioning interface.

Table R5a. Simulation policy performance within RECAP (100 trials/task). Trial counts are per method; task columns report success rates (%, ↑).

MethodSpatialObjectGoalLongMean success rate (%)Speedup on successful episodes (×)Throughput (tasks/h)
SARM52.078.052.079.065.31.1865.0
ReWiND50.073.049.074.061.51.1258.3
ECR91.099.098.098.096.51.66173.5

Table R5b. Real-world policy performance within RECAP (15 trials/task). Task columns report success rates (%, ↑).

MethodPutStoreInsertFastenMean success rate (%)Speedup on successful episodes (×)Throughput (tasks/h)
SARM80.053.366.726.756.71.1937.4
ReWiND66.746.746.720.045.01.1227.2
ECR100.086.780.066.783.31.6982.8

Under the same RECAP training setup, ECR achieves higher success rates on all eight tasks and higher mean throughput in both domains than SARM and ReWiND. Full transition-discrimination and policy results are reported in Appendix F.5 (Table 24 and Tables 25–26, respectively).

References

[1] Chen et al. SARM: Stage-Aware Reward Modeling for Long Horizon Robot Manipulation. In ICLR, 2026.

[2] Zhang et al. ReWiND: Language-Guided Rewards Teach Robot Policies without New Demonstrations. In CoRL, 2025.

AuthorsOpenReview ↗

Response to Reviewer Jfec [3/4]

3. Generalization across policies and execution conditions — W3 (generalization) and Q2

“… generalize across different tasks, policies, or previously unseen failure patterns without additional human supervision. … does its failure detector remains reliable as the policy changes? … long-horizon tasks involving repeated errors, recoveries, or previously unseen failure modes.”

ECR addresses robot policy post-training, aiming to efficiently improve pretrained VLA performance on target tasks. We train ECR separately for each task to learn local credit assignment from demonstrations, autonomous rollouts, and human corrections. Improving policies on individual target tasks is also central to downstream training in π*₀.₆ / RECAP (RSS 2026) [1] and RL Token (CoRL 2026) [2]; HIL-SERL (Science Robotics 2025) likewise trains policies separately for individual tasks using human-in-the-loop RL [3].

To address your concern about generalization, we further evaluate the trained models under changes in policies and execution conditions (revised manuscript, Appendix F.4, Tables 21–23). ECR models trained on individual tasks can be reused across policies without retraining for the evaluated policy.

Cross-policy evaluation. We collect 20 independent episodes per task from each of the RECAP and SVM policies, totaling 160 across four real-world tasks. These episodes are excluded from ECR training, model selection, and normalization. All five original-fold ECR models remain frozen and evaluate the same episodes without onset labels at inference. Table R6 reports mean ± sample SD across these models, with tasks weighted equally; evaluation follows Appendix C.2.

Table R6. ECR transition effectiveness on new policy rollouts (%).

TaskRECAP AP ↑RECAP AUROC ↑SVM AP ↑SVM AUROC ↑
Put93.5 ± 1.196.2 ± 1.192.1 ± 1.295.3 ± 0.8
Store93.1 ± 1.193.4 ± 2.292.1 ± 0.990.9 ± 2.2
Insert84.3 ± 7.087.6 ± 6.386.4 ± 6.388.7 ± 4.8
Fasten87.8 ± 5.493.5 ± 1.284.7 ± 6.693.5 ± 1.4
Macro-average89.7 ± 2.492.6 ± 2.388.8 ± 1.992.1 ± 0.9

Without additional training, frozen ECR achieves 92.6% and 92.1% macro-average AUROC on RECAP and SVM rollouts, respectively. Both are within 1.4 percentage points of the original held-out AUROC of 93.5% (Table 3), supporting reuse across policies.

Changes in execution conditions. On the Insert task, we evaluate the same policy checkpoint across all conditions without retraining, with 15 trials per condition (Appendix C.1).

Table R7. Policy success under changed execution conditions on the Insert task (%, ↑).

MethodOriginalChanged backgroundChanged objectBoth changed
ECR + RECAP80.080.073.366.7

The same policy completes the task under the tested scene variations without retraining. Success is 80.0% with the original or changed background, 73.3% with a changed object, and 66.7% when both change (Table R7).

Repeated errors and human corrections. We evaluate held-out episodes with at least two human interventions. Evaluation covers autonomous actions before and after interventions, excludes human-controlled actions, and uses each episode's held-out-fold ECR model.

Table R8. Transition effectiveness on episodes with repeated interventions (%); five-fold mean ± sample SD.

TaskSubset / all episodesAP ↑AUROC ↑
Put32 / 4096.1 ± 1.998.3 ± 1.3
Fasten23 / 4084.2 ± 5.094.2 ± 1.6

ECR achieves 98.3% and 94.2% AUROC on these Put and Fasten subsets, supporting effectiveness discrimination across repeated interventions and resumed autonomous execution.

Previously unseen failure modes. ECR is supervised to predict local transition effectiveness, rather than classify a predefined set of failure categories. Interventions and failure-onset events provide effectiveness supervision without separate labels for each failure mode. An unseen failure category therefore does not, by itself, imply that ECR cannot identify the ineffective behavior within it. For new failures that the existing model does not reliably recognize, sparse event annotations from subsequent rollouts can support ECR updates, incorporating these experiences into further credit-assignment learning and policy improvement.

[1] Amin et al. π*₀.₆: a VLA That Learns From Experience. In RSS, 2026.

[2] Xu et al. RL Token: Bootstrapping Online RL with Vision-Language-Action Models. In CoRL, 2026.

[3] Luo et al. Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning. Science Robotics, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.