- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes TPEC (Trajectory Progress Efficiency Consensus), a self-supervised framework for curating robot demonstration data. Compact progress predictors are trained on small random reference subsets from the candidate pool to regress normalized timestamps. Each predictor assigns a whole-trajectory progress-efficiency score based on net predicted progress relative to total predicted variation. Eligible scores from predictors not trained on the evaluated trajectory are combined into a consensus ranking, which guides action-chunk selection under a fixed filter ratio with temporal non-maximum suppression (NMS).
The authors highlight three contributions: (1) curation using uncurated in-dataset references without quality labels or online policy rollouts; (2) whole-trajectory efficiency consensus, with scores invariant to positive affine transformations, for action-chunk selection; and (3) data efficiency and scalability, including reduced-data fine-tuning and lower curation-preparation cost, with claimed applicability to large-scale pretraining.
Experiments cover RoboMimic and RoboTwin with Diffusion Policy students, multitask π0 fine-tuning on 2,500 demonstrations across 50 tasks, three real-robot manipulation tasks, component ablations, and preparation-time measurements.
Critical strengths
-
Low-cost curation without curated references, policy rollouts, or policy gradients. TPEC trains compact progress predictors on small random in-dataset reference sets and reuses the resulting scores across filter ratios. The reported preparation times are substantially lower than those of gradient-based methods such as CUPID, making this a promising lower-cost alternative in the evaluated settings.
-
Broad evaluation across simulation and real-robot settings, including multitask VLA fine-tuning. The paper evaluates TPEC with task-specific Diffusion Policy students and the large VLA model π0, extending beyond a single benchmark setting. This breadth strengthens the practical relevance of the study compared with evaluations limited to one narrow setup.
Critical weaknesses
-
The added value of learned progress scoring over simple heuristics is not convincingly established. The progress predictors regress normalized timestamps rather than task completion or downstream policy utility, and the theory does not establish that higher whole-trajectory efficiency scores imply higher policy success rates. On Square fresh scenes, Shortest trajectory achieves a higher mean success rate than TPEC (80.67% versus 71.33%), although the paired 95% confidence interval includes zero. Comparisons with other simple heuristics, such as end-effector path length, pause ratio, and reversal count, are missing. The fresh-scene paired intervals for TPEC versus One vote also include zero, while adapted SCIZOR-DINOv2 and WARP-RM-DINOv2 implementations differ in configuration and cannot isolate the proposed scoring mechanism. These gaps leave the added value of learned progress scoring and consensus unclear.
-
The robustness of the reported data-efficiency gains and preservation of important behavioral coverage remain insufficiently established. Whole-trajectory ranking may downrank rare recovery behaviors or alternative strategies because every action chunk inherits its source trajectory's consensus score; temporal NMS enforces within-trajectory spacing but does not ensure coverage across behavior modes. The multitask π0 results lack between-seed uncertainty, and the appendix flags two reported means inconsistent with the stated three-seed, 50-episode-per-seed protocol. The main Square and Transport fresh-scene comparisons among TPEC, One vote, and Shortest trajectory use only nominal budget 30%. Some Full-data comparisons also differ in training-update counts, limiting conclusions about the source of observed gains. Finally, the large-scale pretraining claim lacks quantitative results.
Questions for authors
-
Could the authors provide a score-utility analysis on one representative benchmark and compare TPEC with additional simple heuristics, such as end-effector path length, pause ratio, and reversal count? Given that the reported fresh-scene paired confidence intervals include zero, what evidence supports a consistent advantage of consensus over One vote or Shortest trajectory?
-
If rare recovery or alternative-strategy modes can be identified in the existing data, could the authors report their retention rates at high filter ratios and any corresponding effects on downstream policy performance? Otherwise, please acknowledge this as a limitation.
-
Appendix A.2 acknowledges two success-rate means that do not match the stated granularity of three seeds and 50 episodes per seed: Square PH TPEC at the nominal budget 90% (93.00%) and Can MH Random-NMS at the same budget (95.67%). Could the authors explain their calculation or provide per-seed success counts or corrected values? Between-seed uncertainty for the multitask π0 results and, if feasible, an equal-training-update Full-data comparison would also help assess the reported gains.
-
Could the authors provide quantitative results and a clear evaluation protocol for large-scale pretraining, or narrow the corresponding claims in the abstract and contributions?
3 comments
Q2. Retention of recovery and alternative strategies
Thank you for asking which recovery and alternative-strategy behaviors remain in the selected data. At 37,597 windows on Transport MH, TPEC covers 154 demonstrations, Shortest covers 163, and Random-NMS covers all 288. We counted each annotated event once and considered it retained if a selected window contained its key action. The table covers nominal filter ratios of 90% and 70%, corresponding to 10% and 30% retention.
| Nominal retention | Selection | Candidate events | Events in retained trajectories | Events with a retained key action | Retention (%) |
|---|---|---|---|---|---|
| 10% | TPEC | 6 | 0 | 0 | 0.00 |
| 10% | Shortest | 6 | 0 | 0 | 0.00 |
| 10% | Random-NMS | 6 | 6 | 4 | 66.67 |
| 30% | TPEC | 6 | 2 | 2 | 33.33 |
| 30% | Shortest | 6 | 0 | 0 | 0.00 |
| 30% | Random-NMS | 6 | 6 | 6 | 100.00 |
At nominal 30% retention, TPEC selected windows from the source trajectories of two of the six annotated events and retained both key actions. The corresponding policies reached 57.4% overall task success for TPEC, 51.0% for Shortest, and 37.4% for Random-NMS. We will add the event-level analysis to the paper, including whether events were omitted during trajectory ranking or window selection, and discuss how targeted rollouts on recovery scenes could measure the effect of retaining these key actions.
Q3. Matched updates, rollout counts, and π₀ uncertainty
Matched optimizer updates
We appreciate your suggestion to compare at a common optimizer-update budget. In addition to the Transport MH comparison in Q1, the matched-update Can MH comparison gave TPEC a 4.67-point mean gain over Full (99.00% versus 94.33%), using three training seeds and 100 shared scenes per seed.
Supplementary evaluation at nominal 70% retention
This evaluation uses 100 scenes per training seed. The 95% intervals use a crossed bootstrap over training seeds and scenes.
| Task | Seeds | TPEC70 success (%) | Full success (%) | TPEC70 minus Full (points, 95% CI) |
|---|---|---|---|---|
| Transport MH | 29–31 | 48.33 | 37.00 | +11.33 [−0.67, 23.00] |
| Square MH | 29–31 | 90.33 | 83.00 | +7.33 [0.00, 15.00] |
| Square MH | 24–26 | 84.33 | 84.33 | 0.00 |
Nominal 90% rollout counts
Thank you for flagging the two entries in Appendix A.2. The recorded counts under the three-seed, 50-episode-per-seed protocol give the means below. We will correct the table in the revised version.
| Task and method | Source | Submitted mean (%) | Seed 24 | Seed 25 | Seed 26 | From recorded counts (%), mean ± seed SD |
|---|---|---|---|---|---|---|
| Square PH, TPEC90 | Table 5 | 93.00 | 45/50 | 46/50 | 48/50 | 92.67 ± 3.06 |
| Can MH, Random-NMS90 | Table 4 | 95.67 | 46/50 | 47/50 | 50/50 | 95.33 ± 4.16 |
Supplementary evaluation at nominal 90% retention
In a separate evaluation with 100 scenes per seed, we evaluated TPEC90 on Square PH and Random90 on Can MH. We report seeds 27–29 and 24–26 separately for each setting. The 90% selections in this protocol use temporal NMS radius zero.
| Task | Method | Training seeds | Success counts in seed order | Success (%), mean ± seed SD |
|---|---|---|---|---|
| Square PH | TPEC90 | 27–29 | 93/100, 91/100, 91/100 | 91.67 ± 1.15 |
| Can MH | Random90 | 27–29 | 92/100, 94/100, 97/100 | 94.33 ± 2.52 |
| Square PH | TPEC90 | 24–26 | 92/100, 92/100, 88/100 | 90.67 ± 2.31 |
| Can MH | Random90 | 24–26 | 94/100, 97/100, 95/100 | 95.33 ± 1.53 |
Multitask π₀ fine-tuning
At 50% retention, TPEC gains 4.46 percentage points on Clean and 0.93 points on Domain Randomized (DR) over Random in the 50-task aggregate point estimates in Table 19. We will make the aggregation explicit in Table 19 and report variability across training seeds once repeat runs are available.
| Fine-tuning method | Clean success (%) | DR success (%) |
|---|---|---|
| Full | 43.12 | 15.10 |
| Random, 50% retained | 44.10 | 16.22 |
| TPEC, 50% retained | 48.56 | 17.15 |
Q4. Curation at pretraining scale
Thank you for asking for a quantitative pretraining comparison and for the opportunity to clarify its relationship to the multitask fine-tuning results.
On ABC-130K, the full pretraining pool corresponds to approximately 3,540 hours across 129,182 source trajectories and 200 tasks. TPEC retained 30% of the valid starts within each task for π₀ pretraining, yielding 114,698,441 of 382,328,149 valid window starts, or approximately 1,062 hours. We count one timestep per valid start on the original 30 Hz timeline. Each start indexes a contiguous 50-step action sequence.
The TPEC30 pretraining run used a PaliGemma-initialized Gemma 2B backbone and a randomly initialized 300M-parameter action expert. Training used bf16 on eight GPUs on a single node, a global batch size of 512, and seed 42. The configured schedule targets 500,000 optimizer updates, with 1,000 warmup updates followed by a constant learning rate of 5 × 10⁻⁵ and an EMA decay of 0.99.
For the real-robot Pick sausages task, we fine-tuned both ABC-130K checkpoints on the full task training set and evaluated each in 30 trials. The checkpoint pretrained on the TPEC30 subset achieved 70.0% success, 36.7 percentage points above the checkpoint pretrained on all ABC-130K data. For comparison, the Full π₀ baseline in Figure 5 achieved 76.7% success after fine-tuning on the full task training set.
| Checkpoint used for fine-tuning | Approx. ABC-130K hours | ABC-130K window starts | Mean task score | Success (%) |
|---|---|---|---|---|
| ABC-130K, full | 3,540 | 382,328,149 | 1.5667 | 33.3 |
| ABC-130K, TPEC (30% retained per task) | 1,062 | 114,698,441 | 2.5333 | 70.0 |
| Full π₀ | — | — | — | 76.7 |
We are extending the ABC-130K checkpoint comparison to additional real-robot fine-tuning tasks and will report the pretraining setup and downstream results in Section 5.4. Thank you again for your suggestions on the comparisons and evaluation protocols.
Thank you for reading our paper carefully and for your detailed, constructive feedback. We appreciate your recognition of TPEC's low curation cost and your suggestions for testing learned scoring against simpler alternatives.
In our supplementary matched-update experiment on Transport MH, TPEC achieved 57.4% success with 20.33% of the candidate windows. It exceeded Full-raw by 16.8 percentage points and the same-budget Random-NMS baseline by 20.0 points.
Q1. Learned scoring, simple heuristics, and consensus
Utility of a learned score
On Transport MH, we formed 35 pairs of high- and low-score trajectories matched on duration, end-effector path length, pause ratio, and reversal count. Both trajectories in each pair contributed the same number of windows, for 7,140 windows per group. Mean policy success was 21.67% for the high-score group and 15.00% for the low-score group, a difference of +6.67 percentage points (95% CI [−3.33, 16.67]).
At nominal 10% retention on Transport MH, policies trained on One-vote selections reached 46.4% success versus 21.4% for Random-NMS across five training seeds. The progress head regresses normalized timestamps from observations. When a trajectory returns to an earlier task stage, the predicted progress can decrease. For the same positive net progress, this reversal increases total variation and lowers the trajectory score.
Matched Transport MH comparison
Thank you for suggesting the heuristic controls. We trained each policy for 150,000 optimizer updates and evaluated its final EMA checkpoint on 100 shared fresh scenes per seed. Each selection method retained 37,597 windows, the same count as at nominal 30% retention in Table 1. This is 20.33% of the 184,960 candidates in the supplementary pool. Full-raw used all 184,960 windows. The additional ranking comparisons used the same candidate pool and temporal NMS. One-vote uses one eligible out-of-sample head from the TPEC panel per trajectory. The heuristic rankings use end-effector path length, pause ratio, and reversal count.
PathLength30 had the highest mean success among the three heuristics at 53.6%. The 95% intervals below use paired seed-level differences on shared scenes.
Consensus and Shortest
Against One-vote, TPEC has higher mean success in all ten matched task–budget settings in Tables 9 and 10, with an average gain of 5.07 percentage points before rounding the individual means. Against Shortest, TPEC has higher mean success in six of those ten settings and ties in one. On Square MH fresh scenes, Shortest reached 80.67% and TPEC reached 71.33% (Table 12).
At nominal 30% retention, the paired fresh-scene intervals leave the direction of both comparisons uncertain. For TPEC minus One-vote, the intervals were [−10.0, 24.0] on Square MH and [−12.0, 16.67] on Transport MH. For TPEC minus Shortest, they were [−26.67, 8.0] and [−16.0, 14.67], respectively (Appendix A.7). In the matched-update Can MH comparison, TPEC, One-vote, and Shortest reached 99.00%, 97.67%, and 98.00%, respectively.
On Transport MH at nominal 10% retention, Shortest covers 64 demonstrations versus 51 for TPEC, and 4.25% of its selected steps are near-stationary versus 11.10% for TPEC. At nominal 30% retention, the near-stationary fractions are 7.80% for Shortest and 10.12% for TPEC, and both selections contain zero fully idle windows. The PauseRatio30 comparison holds window count and training updates fixed.