acceptodds
Under review as a conference paper at ICLR 2027

Data That Selects Itself: Scalable Self-Curation for Robot Learning

Abstract

Robot imitation learning relies on demonstrations of varying quality, making effective data curation essential for scalable and data-efficient policy learning. However, learning a useful curation signal remains challenging when trusted reference demonstrations and policy-rollout feedback are unavailable. We present Trajectory Progress Efficiency Consensus (TPEC), a self-supervised framework that learns to rank demonstrations from small, uncurated reference sets sampled directly from the candidate dataset. Using lightweight progress predictors, TPEC aggregates whole-trajectory efficiency scores into a consensus ranking for action-chunk selection, enabling data curation without curated references or online policy rollouts. Experiments demonstrate the effectiveness of TPEC across demonstration-quality regimes and policy scales. On RoboTwin 2.0 and in real-robot experiments, TPEC matches or exceeds full-data joint fine-tuning of the billion-parameter VLA model π0 using only 50% of the available action chunks. Furthermore, TPEC scales to large-scale pretraining, highlighting its potential for practical robot learning pipelines. Enabled by its in-dataset selection strategy, TPEC achieves curation-preparation speedups of 2,257× over the gradient-based CUPID and 4.53× over the progress-reward-based WARP-RM in VLA data curation. Code will be public.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Imitation learning, Data Curation, Self-Supervision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_H3fgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes TPEC (Trajectory Progress Efficiency Consensus), a self-supervised framework for curating robot demonstration data. Compact progress predictors are trained on small random reference subsets from the candidate pool to regress normalized timestamps. Each predictor assigns a whole-trajectory progress-efficiency score based on net predicted progress relative to total predicted variation. Eligible scores from predictors not trained on the evaluated trajectory are combined into a consensus ranking, which guides action-chunk selection under a fixed filter ratio with temporal non-maximum suppression (NMS).

The authors highlight three contributions: (1) curation using uncurated in-dataset references without quality labels or online policy rollouts; (2) whole-trajectory efficiency consensus, with scores invariant to positive affine transformations, for action-chunk selection; and (3) data efficiency and scalability, including reduced-data fine-tuning and lower curation-preparation cost, with claimed applicability to large-scale pretraining.

Experiments cover RoboMimic and RoboTwin with Diffusion Policy students, multitask π0 fine-tuning on 2,500 demonstrations across 50 tasks, three real-robot manipulation tasks, component ablations, and preparation-time measurements.

Critical strengths
  1. Low-cost curation without curated references, policy rollouts, or policy gradients. TPEC trains compact progress predictors on small random in-dataset reference sets and reuses the resulting scores across filter ratios. The reported preparation times are substantially lower than those of gradient-based methods such as CUPID, making this a promising lower-cost alternative in the evaluated settings.

  2. Broad evaluation across simulation and real-robot settings, including multitask VLA fine-tuning. The paper evaluates TPEC with task-specific Diffusion Policy students and the large VLA model π0, extending beyond a single benchmark setting. This breadth strengthens the practical relevance of the study compared with evaluations limited to one narrow setup.

Critical weaknesses
  1. The added value of learned progress scoring over simple heuristics is not convincingly established. The progress predictors regress normalized timestamps rather than task completion or downstream policy utility, and the theory does not establish that higher whole-trajectory efficiency scores imply higher policy success rates. On Square fresh scenes, Shortest trajectory achieves a higher mean success rate than TPEC (80.67% versus 71.33%), although the paired 95% confidence interval includes zero. Comparisons with other simple heuristics, such as end-effector path length, pause ratio, and reversal count, are missing. The fresh-scene paired intervals for TPEC versus One vote also include zero, while adapted SCIZOR-DINOv2 and WARP-RM-DINOv2 implementations differ in configuration and cannot isolate the proposed scoring mechanism. These gaps leave the added value of learned progress scoring and consensus unclear.

  2. The robustness of the reported data-efficiency gains and preservation of important behavioral coverage remain insufficiently established. Whole-trajectory ranking may downrank rare recovery behaviors or alternative strategies because every action chunk inherits its source trajectory's consensus score; temporal NMS enforces within-trajectory spacing but does not ensure coverage across behavior modes. The multitask π0 results lack between-seed uncertainty, and the appendix flags two reported means inconsistent with the stated three-seed, 50-episode-per-seed protocol. The main Square and Transport fresh-scene comparisons among TPEC, One vote, and Shortest trajectory use only nominal budget 30%. Some Full-data comparisons also differ in training-update counts, limiting conclusions about the source of observed gains. Finally, the large-scale pretraining claim lacks quantitative results.

Questions for authors
  1. Could the authors provide a score-utility analysis on one representative benchmark and compare TPEC with additional simple heuristics, such as end-effector path length, pause ratio, and reversal count? Given that the reported fresh-scene paired confidence intervals include zero, what evidence supports a consistent advantage of consensus over One vote or Shortest trajectory?

  2. If rare recovery or alternative-strategy modes can be identified in the existing data, could the authors report their retention rates at high filter ratios and any corresponding effects on downstream policy performance? Otherwise, please acknowledge this as a limitation.

  3. Appendix A.2 acknowledges two success-rate means that do not match the stated granularity of three seeds and 50 episodes per seed: Square PH TPEC at the nominal budget 90% (93.00%) and Can MH Random-NMS at the same budget (95.67%). Could the authors explain their calculation or provide per-seed success counts or corrected values? Between-seed uncertainty for the multitask π0 results and, if feasible, an equal-training-update Full-data comparison would also help assess the reported gains.

  4. Could the authors provide quantitative results and a clear evaluation protocol for large-scale pretraining, or narrow the corresponding claims in the abstract and contributions?

3 comments

Thank you for reading our paper carefully and for your detailed, constructive feedback. We appreciate your recognition of TPEC's low curation cost and your suggestions for testing learned scoring against simpler alternatives.

In our supplementary matched-update experiment on Transport MH, TPEC achieved 57.4% success with 20.33% of the candidate windows. It exceeded Full-raw by 16.8 percentage points and the same-budget Random-NMS baseline by 20.0 points.

Q1. Learned scoring, simple heuristics, and consensus

Utility of a learned score

On Transport MH, we formed 35 pairs of high- and low-score trajectories matched on duration, end-effector path length, pause ratio, and reversal count. Both trajectories in each pair contributed the same number of windows, for 7,140 windows per group. Mean policy success was 21.67% for the high-score group and 15.00% for the low-score group, a difference of +6.67 percentage points (95% CI [−3.33, 16.67]).

At nominal 10% retention on Transport MH, policies trained on One-vote selections reached 46.4% success versus 21.4% for Random-NMS across five training seeds. The progress head regresses normalized timestamps from observations. When a trajectory returns to an earlier task stage, the predicted progress can decrease. For the same positive net progress, this reversal increases total variation and lowers the trajectory score.

Matched Transport MH comparison

Thank you for suggesting the heuristic controls. We trained each policy for 150,000 optimizer updates and evaluated its final EMA checkpoint on 100 shared fresh scenes per seed. Each selection method retained 37,597 windows, the same count as at nominal 30% retention in Table 1. This is 20.33% of the 184,960 candidates in the supplementary pool. Full-raw used all 184,960 windows. The additional ranking comparisons used the same candidate pool and temporal NMS. One-vote uses one eligible out-of-sample head from the TPEC panel per trajectory. The heuristic rankings use end-effector path length, pause ratio, and reversal count.

Training conditionSeed 24Seed 25Seed 26Seed 27Seed 28Success (%), mean ± seed SD
TPEC3063/10063/10054/10052/10055/10057.4 ± 5.22
Shortest3051/10045/10055/10047/10057/10051.0 ± 5.10
Random-NMS3033/10028/10046/10040/10040/10037.4 ± 6.99
Full-raw30/10045/10044/10048/10036/10040.6 ± 7.40
One-vote3050/10047/10060/10054/10041/10050.4 ± 7.16
ReversalCount3058/10051/10054/10048/10048/10051.8 ± 4.27
PathLength3057/10046/10055/10057/10053/10053.6 ± 4.56
PauseRatio3031/10038/10038/10032/10032/10034.2 ± 3.49

PathLength30 had the highest mean success among the three heuristics at 53.6%. The 95% intervals below use paired seed-level differences on shared scenes.

ComparisonMean difference (percentage points)95% paired t interval
TPEC30 − Full-raw+16.8[+3.2, +30.4]
TPEC30 − Random-NMS30+20.0[+5.3, +34.7]
TPEC30 − Shortest30+6.4[−4.2, +17.0]
TPEC30 − One-vote30+7.00[−5.66, +19.66]
TPEC30 − ReversalCount30+5.60[+0.15, +11.05]
TPEC30 − PathLength30+3.80[−6.64, +14.24]
TPEC30 − PauseRatio30+23.20[+15.78, +30.62]

Consensus and Shortest

Against One-vote, TPEC has higher mean success in all ten matched task–budget settings in Tables 9 and 10, with an average gain of 5.07 percentage points before rounding the individual means. Against Shortest, TPEC has higher mean success in six of those ten settings and ties in one. On Square MH fresh scenes, Shortest reached 80.67% and TPEC reached 71.33% (Table 12).

At nominal 30% retention, the paired fresh-scene intervals leave the direction of both comparisons uncertain. For TPEC minus One-vote, the intervals were [−10.0, 24.0] on Square MH and [−12.0, 16.67] on Transport MH. For TPEC minus Shortest, they were [−26.67, 8.0] and [−16.0, 14.67], respectively (Appendix A.7). In the matched-update Can MH comparison, TPEC, One-vote, and Shortest reached 99.00%, 97.67%, and 98.00%, respectively.

On Transport MH at nominal 10% retention, Shortest covers 64 demonstrations versus 51 for TPEC, and 4.25% of its selected steps are near-stationary versus 11.10% for TPEC. At nominal 30% retention, the near-stationary fractions are 7.80% for Shortest and 10.12% for TPEC, and both selections contain zero fully idle windows. The PauseRatio30 comparison holds window count and training updates fixed.

Q2. Retention of recovery and alternative strategies

Thank you for asking which recovery and alternative-strategy behaviors remain in the selected data. At 37,597 windows on Transport MH, TPEC covers 154 demonstrations, Shortest covers 163, and Random-NMS covers all 288. We counted each annotated event once and considered it retained if a selected window contained its key action. The table covers nominal filter ratios of 90% and 70%, corresponding to 10% and 30% retention.

Nominal retentionSelectionCandidate eventsEvents in retained trajectoriesEvents with a retained key actionRetention (%)
10%TPEC6000.00
10%Shortest6000.00
10%Random-NMS66466.67
30%TPEC62233.33
30%Shortest6000.00
30%Random-NMS666100.00

At nominal 30% retention, TPEC selected windows from the source trajectories of two of the six annotated events and retained both key actions. The corresponding policies reached 57.4% overall task success for TPEC, 51.0% for Shortest, and 37.4% for Random-NMS. We will add the event-level analysis to the paper, including whether events were omitted during trajectory ranking or window selection, and discuss how targeted rollouts on recovery scenes could measure the effect of retaining these key actions.

Q3. Matched updates, rollout counts, and π₀ uncertainty

Matched optimizer updates

We appreciate your suggestion to compare at a common optimizer-update budget. In addition to the Transport MH comparison in Q1, the matched-update Can MH comparison gave TPEC a 4.67-point mean gain over Full (99.00% versus 94.33%), using three training seeds and 100 shared scenes per seed.

Supplementary evaluation at nominal 70% retention

This evaluation uses 100 scenes per training seed. The 95% intervals use a crossed bootstrap over training seeds and scenes.

TaskSeedsTPEC70 success (%)Full success (%)TPEC70 minus Full (points, 95% CI)
Transport MH29–3148.3337.00+11.33 [−0.67, 23.00]
Square MH29–3190.3383.00+7.33 [0.00, 15.00]
Square MH24–2684.3384.330.00

Nominal 90% rollout counts

Thank you for flagging the two entries in Appendix A.2. The recorded counts under the three-seed, 50-episode-per-seed protocol give the means below. We will correct the table in the revised version.

Task and methodSourceSubmitted mean (%)Seed 24Seed 25Seed 26From recorded counts (%), mean ± seed SD
Square PH, TPEC90Table 593.0045/5046/5048/5092.67 ± 3.06
Can MH, Random-NMS90Table 495.6746/5047/5050/5095.33 ± 4.16

Supplementary evaluation at nominal 90% retention

In a separate evaluation with 100 scenes per seed, we evaluated TPEC90 on Square PH and Random90 on Can MH. We report seeds 27–29 and 24–26 separately for each setting. The 90% selections in this protocol use temporal NMS radius zero.

TaskMethodTraining seedsSuccess counts in seed orderSuccess (%), mean ± seed SD
Square PHTPEC9027–2993/100, 91/100, 91/10091.67 ± 1.15
Can MHRandom9027–2992/100, 94/100, 97/10094.33 ± 2.52
Square PHTPEC9024–2692/100, 92/100, 88/10090.67 ± 2.31
Can MHRandom9024–2694/100, 97/100, 95/10095.33 ± 1.53

Multitask π₀ fine-tuning

At 50% retention, TPEC gains 4.46 percentage points on Clean and 0.93 points on Domain Randomized (DR) over Random in the 50-task aggregate point estimates in Table 19. We will make the aggregation explicit in Table 19 and report variability across training seeds once repeat runs are available.

Fine-tuning methodClean success (%)DR success (%)
Full43.1215.10
Random, 50% retained44.1016.22
TPEC, 50% retained48.5617.15

Q4. Curation at pretraining scale

Thank you for asking for a quantitative pretraining comparison and for the opportunity to clarify its relationship to the multitask fine-tuning results.

On ABC-130K, the full pretraining pool corresponds to approximately 3,540 hours across 129,182 source trajectories and 200 tasks. TPEC retained 30% of the valid starts within each task for π₀ pretraining, yielding 114,698,441 of 382,328,149 valid window starts, or approximately 1,062 hours. We count one timestep per valid start on the original 30 Hz timeline. Each start indexes a contiguous 50-step action sequence.

The TPEC30 pretraining run used a PaliGemma-initialized Gemma 2B backbone and a randomly initialized 300M-parameter action expert. Training used bf16 on eight GPUs on a single node, a global batch size of 512, and seed 42. The configured schedule targets 500,000 optimizer updates, with 1,000 warmup updates followed by a constant learning rate of 5 × 10⁻⁵ and an EMA decay of 0.99.

For the real-robot Pick sausages task, we fine-tuned both ABC-130K checkpoints on the full task training set and evaluated each in 30 trials. The checkpoint pretrained on the TPEC30 subset achieved 70.0% success, 36.7 percentage points above the checkpoint pretrained on all ABC-130K data. For comparison, the Full π₀ baseline in Figure 5 achieved 76.7% success after fine-tuning on the full task training set.

Checkpoint used for fine-tuningApprox. ABC-130K hoursABC-130K window startsMean task scoreSuccess (%)
ABC-130K, full3,540382,328,1491.566733.3
ABC-130K, TPEC (30% retained per task)1,062114,698,4412.533370.0
Full π₀———76.7

We are extending the ABC-130K checkpoint comparison to additional real-robot fine-tuning tasks and will report the pretraining setup and downstream results in Section 5.4. Thank you again for your suggestions on the comparisons and evaluation protocols.

Reviewer_dPWZOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a self-supervised framework for robot demonstration curation using small, randomly sampled uncurated reference sets. By combining lightweight progress predictors with trajectory-level efficiency consensus, this paper enables efficient action-chunk selection without quality annotations or online policy rollouts. Extensive experiments across RoboMimic, RoboTwin, multitask VLA fine-tuning, and real-robot tasks demonstrate its effectiveness and computational efficiency

Critical strengths
  • The paper addresses the important and practical challenge of scalable demonstration curation for robot imitation learning, particularly when curated reference demonstrations and online policy feedback are unavailable.
  • The paper clearly positions TPEC relative to existing data curation methods, highlighting its ability to construct selection signals without quality-based reference preselection or computationally expensive policy-dependent evaluations.
  • TPEC achieves effective and computationally efficient data curation through a simple yet effective approach.
  • The experiments provide extensive empirical validation across RoboMimic, RoboTwin, multitask VLA fine-tuning, and real-robot settings, with rich analyses covering a diverse range of experimental configurations.
Critical weaknesses
  • While TPEC demonstrates some robustness to uncurated reference data, its reliability under more extreme quality distributions remains unclear. For instance, if the randomly sampled reference set is dominated by low-quality demonstrations with substantial backtracking, timestamp-based regression may learn an inaccurate progress representation, potentially compromising the subsequent trajectory ranking. It may be worth exploring whether a very small set of trusted, high-quality demonstrations could serve as additional anchors to mitigate this potential failure mode
  • While the trajectory-level scoring is effective, I wonder whether relying solely on temporal NMS for action-chunk selection, where all chunks from the same trajectory share an identical score, might leave some room for further improvement.
  • It would also be interesting to discuss how the filtering ratio should be chosen in practice, given the performance variations across different budgets.
Questions for authors

see weaknesses.

3 comments

Thank you for your careful reading and for asking how reference composition and chunk selection affect TPEC in practice. We appreciate the opportunity to test an extreme reference composition and examine how adding anchors affects selection and policy performance.

Q1. Reference composition and anchors

Experimental design

On Transport MH, we fixed the selection budget at 37,597 windows and tested four compositions of 48 reference trajectories, ranging from random references to 24/48 and 48/48 high-backtracking references, plus a condition with 44/48 high-backtracking references and four anchors. We used physical-state proxies to identify high-backtracking trajectories and select anchors. We trained the progress heads on the reference set and drew windows from the full candidate pool. Each reference set was divided into eight groups of six trajectories, one group per head. All eight heads passed the eligibility checks in all 12 panels.

R0, R1, and R2 denote three reference-set seeds, distinct from policy training seeds. For each reference seed, Spearman correlation and selected-window overlap compare a stress panel with its corresponding random-reference panel. The random-versus-random row reports the three pairwise comparisons between random panels. Each policy success rate averages three policy seeds. For R0, we evaluated the same checkpoints on both the 50- and 100-scene sets.

Reference compositionSpearman ρWindow overlap (%)R0, 50 scenes success (%)R0, 100 scenes success (%)R1, 100 scenes success (%)R2, 100 scenes success (%)
Random vs. random, across seeds0.926–0.95287.90–91.67————
Random——50.6755.3354.6754.67
24/48 high-backtracking0.929–0.94588.91–90.2452.6755.6756.3355.33
48/48 high-backtracking0.920–0.93488.60–90.2858.0058.3356.6756.33
44/48 high-backtracking + 4 anchors0.924–0.93688.91–90.1562.6755.3357.3356.33

Extreme reference composition

Across R0–R2, the all-high-backtracking condition retains 88.60–90.28% of the windows selected with the corresponding random references. This range lies within the 87.90–91.67% overlap observed between random reference draws. On the 100-scene sets, policy success was 58.33%, 56.67%, and 56.33% for R0–R2, compared with 55.33%, 54.67%, and 54.67% with random references. On the R0 50-scene set, success was 58.00% versus 50.67%. The all-high-backtracking condition yielded higher mean policy success than random references in all four evaluations.

Effect of anchors

Replacing four high-backtracking references with anchors changes the R0 mean success from 58.00% to 62.67% on 50 scenes and from 58.33% to 55.33% on 100 scenes. On the 100-scene evaluations for R1 and R2, success changed from 56.67% to 57.33% and stayed at 56.33%, respectively. The observed effect of adding anchors varies across evaluations.

Q1 (continued). Possible explanations for the observed stability

We appreciate the opportunity to clarify how TPEC can retain similar windows when all 48 reference trajectories exhibit high backtracking. Several properties of the references and TPEC may help explain the overlap observed on Transport MH.

  1. High-backtracking references can still contain shared task stages. Consider a demonstration that approaches an object, fails to grasp it, moves back, then approaches again, grasps it, and places it at the goal. The demonstration visits the approach, grasp, and placement stages even with the retry. If such stages are observable and occur in a broadly shared order across demonstrations, the references can still help distinguish earlier and later stages.

  2. The heads learn a state-to-time mapping across multiple trajectories. Each head regresses normalized timestamps from observations pooled across six reference trajectories. Shared relationships between states and temporal position may remain learnable even when individual references include different retries. For instance, states showing an object already placed at the goal tend to occur later than states showing the robot approaching it. A similar approach state can also occur both before and after a failed grasp, with different timestamp targets. If exactly the same input has targets 0.2 and 0.6, a squared-loss predictor estimates a value between them. Frequent or systematic conflicts can distort the learned coordinate.

  3. Efficiency scoring is invariant to positive affine transformations. Different reference subsets may lead heads to predict temporal coordinates with different scales or offsets. Under a positive affine transformation, net progress and total variation scale by the same factor, leaving their ratio and the efficiency score unchanged. For example, the paths (0, 1, 0.5, 1) and (0.2, 0.8, 0.5, 0.8) differ by a positive affine transformation and both have an efficiency score of 0.75. Changes in the shape or ordering of a predicted progress path can still affect the score.

  4. Multiple heads provide complementary assessments. The eight heads are trained on disjoint reference subsets. TPEC aggregates efficiency scores from eligible heads, excluding any head trained on the trajectory being scored. If a particular head is affected by a reference-specific retry pattern, the other heads may moderate its contribution. For a hypothetical non-reference trajectory with eight eligible votes, seven scores of 0.8 and one score of 0.4 yield a consensus of 0.70, including the two neutral votes. Eight scores of 0.8 yield 0.74. Errors shared across heads can persist in the consensus.

  5. The candidate pool remains mixed. The stress test changes the 48 trajectories used to train the heads. Scoring and selection still operate on the full candidate pool. For example, TPEC can select a smooth candidate trajectory if the learned heads identify it, even when all 48 reference trajectories exhibit high backtracking. This result applies to a mixed candidate pool that contains trajectories of varying quality.

The stress test evaluates TPEC as a whole under this reference composition. These properties offer possible explanations for the observed stability.

Q2. Action-chunk selection within trajectories

Thank you for suggesting that we consider task stages when choosing action chunks. This prompted us to measure which stages temporal NMS already covers and whether stage-aware selection can recover any missing stages.

Experimental purpose and design

We conducted a stage-aware selection experiment on Transport MH, using coverage of physical task stages as a proxy for retaining task-relevant portions of each trajectory. We derive coarse stage labels from the simulator's payload and trash goal indicators and whether the lid has moved more than 5 cm from its initial position. A stage is eligible if it persists for at least three consecutive recorded states. Each chunk covers the eligible stages appearing within its real future-action interval, so one chunk may cover multiple stages.

Starting from the original TPEC selection, we test replacements within the same trajectory to cover missing stages while preserving the total window count, per-trajectory quotas, existing stage coverage, and temporal NMS constraints. This isolates the choice of chunks within the selected trajectories while keeping the trajectory ranking and allocation fixed.

Results and interpretation

Nominal budgetSelectionStages covered / eligibleTransitions retained / eligibleWindows replaced
10%NMS and stage-aware replacement204/204147/1470
30%NMS610/611440/4420
30%Stage-aware replacement611/611441/4421

At the nominal 10% budget, temporal NMS already covers all 204 eligible stages and 147 eligible transitions, and stage-aware replacement makes no changes. At 30%, NMS covers 610 of 611 eligible stages and 440 of 442 eligible transitions. Replacing one window covers one additional stage and one additional transition. Temporal NMS retains nearly all eligible task stages and transitions within the selected trajectories at both budgets.

The stage labels capture physical task states. Whether stage-aware replacement retains more recovery behaviors or alternative strategies, or improves downstream policy success, remains open.

Q3. Choosing a retention budget

Thank you for raising the practical question of how to choose a retention budget. We currently start by retaining 50% of the available action chunks. Based on our experiments, retention ratios between 30% and 70% are reasonable candidates to explore, with performance varying across tasks and budgets. The same TPEC trajectory scores can be reused across these ratios with the existing progress heads.

We agree that automatically choosing an appropriate retention ratio is a valuable direction. Future work could investigate adaptive thresholds that select a data budget based on the dataset and task.

Thank you again for your constructive suggestions and for taking the time to review our work.

Reviewer_vZkwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

TPEC selects robot demonstrations without labels. It has a small time-predicting head that scores each trajectory by how consistently its predicted progress moves forward. The head is trained on random parts of the same dataset, and TPEC keeps the top-scoring data. The paper claims it matches full-data π0 with 50% less data and with lower preparation time than the predecessor method, CUPID.

Critical strengths
  1. It is interesting to see this research in robotics, because works in other domains, such as language models, show that with a fraction of the training data, a model can perform as well or even better.
Critical weaknesses
  1. The paper claims scalability, but I don't see any scaling. The abstract says TPEC "scales to large-scale pretraining," but Section 5.4 supports this with two sentences and no numbers.
  2. I don't see a clear performance advantage over previous methods. In most settings, the differences are tiny and within seed noise. The exception is RoboMimic MH at 10% retained data, but there, simply keeping the shortest trajectories does better (60.7 vs 52.7 on Square and 60.0 vs 43.3 on Transport, Tables 9–10).
  3. Random selection does pretty well. On π0, random 50% also beats full data (44.10/16.22 vs 43.12/15.10, Table 19). On RoboMimic PH at 10% retained data, TPEC averages 1.2 points below Random-NMS.
  4. The TPEC results at 70% differ between the main and ablation tables by exactly 10 points with identical standard deviations: Square is 68.00 ± 6.00 in Table 2 and 78.00 ± 6.00 in Table 9, and Transport is 42.67 ± 5.77 in Table 3 and 52.67 ± 5.77 in Table 10. The higher numbers are the ones that support the ablation claim in Section 5.5. Which numbers are correct?
Questions for authors
  1. It seems that being fast is good here, since shortest-trajectory selection matches or beats TPEC. What is the intuition for why that is true?
  2. If you use a B-spline policy, or remove or merge static frames, does training on all the data perform as well?
  3. Could this be a chunk-size issue? Some demonstrations may have static or near-static segments, and if a static segment is about as long as the chunk, the policy stays there. Is TPEC mainly filtering out such demonstrations? Note that the efficiency score is unchanged by pauses (Section 4.2), while shortest-trajectory selection directly penalizes them.
  4. Please report per-seed success counts for the 70% cells in Tables 2, 3, 9, and 10.
6 comments

Thank you for taking the time to read our paper so carefully and for your detailed, constructive feedback. We appreciate your suggestions to test idle-window removal and B-spline action representations and to examine how pauses affect TPEC.

In our matched-update Transport MH experiment, TPEC achieved 57.4% success with 20.33% of the candidate windows. It exceeded Full-raw by 16.8 % and the same-budget Random-NMS baseline by 20.0 points. The B-spline control also improved full-data performance and brought its mean close to Shortest.

W4 and Q4. Corrected 70% TPEC entries

Original evaluation (50 scenes per seed)

Thank you for catching the reporting error in the nominal 70% TPEC entries in Tables 2 and 3. Under the original 50-scene evaluation protocol, the evaluations are checked and the seeds count are provided as follows, We verified the success counts for training seeds 24–26 in the original evaluation, which used 50 rollouts per seed.

TaskTablesSeed 24Seed 25Seed 26Success (%), mean ± seed SD
Square MH2 and 936/5039/5042/5078.00 ± 6.00
Transport MH3 and 1023/5028/5028/5052.67 ± 5.77

The corresponding ablation entries already report the corrected values.

TaskTableSubmitted TPEC (%)Corrected TPEC (%)
Square MHTable 268.00 ± 6.0078.00 ± 6.00
Square MHTable 978.00 ± 6.0078.00 ± 6.00
Transport MHTable 342.67 ± 5.7752.67 ± 5.77
Transport MHTable 1052.67 ± 5.7752.67 ± 5.77

We will correct Tables 2 and 3. Section 5.5 and Figure 6 summarize the ablations in Tables 9 and 10. Calculated before rounding the individual means, TPEC's average gains across the ten task–budget settings are 5.07 percentage points over One-vote, 10.07 over Net progress, and 24.40 over 16-step local efficiency.

Supplementary evaluation (100 scenes per seed)

In a separate evaluation with 100 scenes per seed, we evaluated TPEC70 and Full on seeds 29–31 for both tasks and on seeds 24–26 for Square MH.

TaskMethodTraining seedsSuccess counts in seed orderSuccess (%), mean ± seed SD
Square MHTPEC7029–3189/100, 88/100, 94/10090.33 ± 3.21
Square MHFull29–3180/100, 84/100, 85/10083.00 ± 2.65
Transport MHTPEC7029–3142/100, 48/100, 55/10048.33 ± 6.51
Transport MHFull29–3138/100, 35/100, 38/10037.00 ± 1.73
Square MHTPEC7024–2689/100, 81/100, 83/10084.33 ± 4.16
Square MHFull24–2684/100, 85/100, 84/10084.33 ± 0.58

For seeds 29–31, the mean paired TPEC70 minus Full difference is 11.33 percentage points on Transport MH (95% crossed-bootstrap interval [-0.67, 23.00]) and 7.33 points on Square MH ([0.00, 15.00]).

The corrected original Square MH TPEC70 mean corresponds to 117 successes in 150 rollouts (78.00%). The supplementary evaluation of seeds 24–26 recorded 253 successes in 300 rollouts (84.33%).

W2, W3, and Q1. Shortest and random-selection baselines

Why can shortest-trajectory selection work well?

Thank you for drawing our attention to the Shortest baseline. At the submitted nominal 10% budget, Shortest reaches 60.7% on Square MH and 60.0% on Transport MH, compared with 52.7% and 43.3% for TPEC. At the same window count on each task, it covers more demonstrations and selects a smaller fraction of near-stationary steps. Both properties are possible contributors to its higher success.

TaskMethodWindowsDemonstrations coveredStrict low-motion steps (%)Near-stationary steps (%)
Transport MHShortest1012,532640.004.25
Transport MHTPEC1012,532510.1411.10
Square MHShortest105,000750.1812.52
Square MHTPEC105,000610.6318.79

We count adjacent-state steps within selected windows, excluding padding and counting each occurrence in overlapping windows. All end effectors and relevant objects must satisfy the motion thresholds, and all grippers must satisfy the gripper-change threshold. The per-step limits for strict low motion are 0.2 mm translation, 0.1° rotation, and 0.05 mm gripper change. The near-stationary limits are 2 mm, 1°, and 0.5 mm, with no open/close gripper command switch.

How does TPEC compare with random selection?

Random-NMS is a strong baseline at nominal 10% retention on RoboMimic PH, averaging 1.2 points above TPEC across the four tasks. At the same nominal budget on MH, TPEC exceeds Random-NMS by 23.4 points on Square and 29.3 points on Transport.

We address Random50 on the multitask π₀ benchmark and report a real-robot Random50 comparison in our W3 fine-tuning response.

TPEC versus simple baselines under matched training

We also compared full TPEC with Shortest, Random-NMS, and Full-raw on Transport MH, using 150,000 optimizer updates for every condition. Each selection method retained 37,597 windows, the same window budget as the nominal 30% setting in Table 1. This is 20.33% of the 184,960 candidates in the supplementary pool. Table 1 reports 185,942 candidates for the submitted comparison. We evaluated five training seeds on 100 shared fresh scenes per seed.

MethodSuccess (%), mean ± seed SDTPEC minus method (percentage points)
TPEC3057.4 ± 5.22—
Shortest3051.0 ± 5.10+6.4
Random-NMS3037.4 ± 6.99+20.0
Full-raw40.6 ± 7.40+16.8

Our response to Q2 reports the per-seed results and additional controls.

Q2. Supplementary comparisons at the nominal 30% budget

We appreciate your suggestions to test B-spline actions and idle-data removal directly. On Transport MH, Full-B-spline achieves 50.2% mean success versus 40.6% for Full-raw. Near-static merging reaches 47.6%, compared with 45.8% for its matched random-deletion control. All comparisons use 150,000 training updates, five training seeds, and 100 evaluation scenes per seed.

B-spline policy

Full-B-spline improves the full-data mean by 9.6 percentage points and brings it close to Shortest30 (51.0%), while TPEC30 retains a higher observed mean (57.4%) in this experiment. At the nominal 30% budget, Random-NMS, Shortest, and TPEC each selected 37,597 of the 184,960 candidate windows. The table reports final EMA checkpoints evaluated on 100 shared fresh scenes per seed.

For each 16-step action sequence, we fit cubic B-splines with eight control points to the two arms' absolute positions and local rotation vectors using endpoint-constrained least squares. The policy predicts the spline coefficients and the original gripper commands at their individual timestamps. At inference, we evaluate the splines at the original 16 timestamps, reconstruct the absolute actions, and execute eight actions per policy call. We use the same observation inputs, observation normalization, backbone configuration, and optimization settings as Full-raw, and adapt the input and output dimensions and action normalization to the spline representation.

Training conditionSeed 24Seed 25Seed 26Seed 27Seed 28Success (%), mean ± seed SD
Full-raw30/10045/10044/10048/10036/10040.6 ± 7.40
Full-idle-drop35/10040/10039/10037/10042/10038.6 ± 2.70
Full-random-drop46/10045/10033/10040/10044/10041.6 ± 5.32
Full-B-spline, 8 control points44/10054/10049/10050/10054/10050.2 ± 4.15
Random-NMS3033/10028/10046/10040/10040/10037.4 ± 6.99
Shortest3051/10045/10055/10047/10057/10051.0 ± 5.10
TPEC3063/10063/10054/10052/10055/10057.4 ± 5.22

Q2. Supplementary comparisons at the nominal 30% budget (continued)

Idle-window removal and static or near-static frame processing

At the window level, Full-idle-drop removes 23 of 184,960 candidate windows (0.0124%) and achieves 38.6% mean success. Matched random window deletion achieves 41.6%. A 16-step window qualifies for removal if both arms and the task objects stay within 2 mm and 1° of their initial poses, gripper widths vary by at most 0.5 mm, gripper commands stay fixed, and absolute action-target errors stay within 2 mm and 1°. The filter preserves contact-transition neighborhoods and trajectory boundaries.

At the frame level, we use thresholds of 2 mm/1° for strict stationarity and 10 mm/3° for near stationarity. Eligibility checks cover both arms and the task objects, gripper widths and commands, and action-target errors. We preserve frames within two steps of contact or gripper-command transitions and the first and last two frames of each trajectory. Deletion removes eligible observation–action pairs and concatenates the retained pairs within each demonstration. Merging bounds displacement relative to the first state of each stationary block and retains the block's first pair and exit frame with their original actions.

For each intervention and training seed, the matched control samples individual frames uniformly without replacement from unprotected positions. It removes the same number of frames within each demonstration, matching both the retained frame counts and the number of training windows. All variants use the original action representation and full-pool normalization. Frame processing changes the offline training trajectories, and the policy runs at the original simulator timestep at deployment.

InterventionFrames removedSuccess (%)Random control (%)
Strict deletion958 (0.512%)44.839.8
Strict merging412 (0.220%)41.042.0
Near-static deletion16,704 (8.934%)46.848.2
Near-static merging11,082 (5.927%)47.645.8

Removal percentages are relative to the original 186,976 training frames. Frame-level processing can improve mean success over Full-raw, and two of the four targeted variants achieve higher means than their matched random-deletion controls. Near-static merging reaches 47.6%, compared with 45.8% for its matched control. TPEC30 achieves the highest mean success among these Transport MH conditions (57.4%).

Q3. Pauses and low-motion data

How TPEC scores pauses

Thank you for asking how the progress score treats pauses. An exact plateau in the evaluated progress path adds zero total variation and leaves the endpoint difference unchanged, so the per-head efficiency score stays the same (Section 4.2). Shortest ranks by total trajectory duration, which includes pauses.

Selected data and policy success

On Transport MH at the nominal 30% budget, Shortest30, TPEC30, and Random-NMS30 each select 37,597 of the 184,960 candidate windows. The table compares their selected data with Full-raw, using the motion thresholds defined in our Q1 response. Fully idle windows span 16 steps (H16), and fully idle execution prefixes span eight steps (H8).

SelectionFully idle windows (H16)Fully idle execution prefixes (H8)Strict low-motion steps (%)Near-stationary steps (%)Mean success (%)
Full-raw23 (0.0124%)56 (0.0303%)0.131310.9340.6
Random-NMS305 (0.0133%)12 (0.0319%)0.132710.8737.4
Shortest30000.0227.8051.0
TPEC3002 (0.0053%)0.08410.1257.4

The H16 and H8 percentages use the window count for each selection. The step percentages use the Q1 thresholds and exclude padding. The strict low-motion counts are 3,872 for Full-raw and 795 for Random-NMS30.

Comparison with Shortest

TPEC30 and Shortest30 both contain zero fully idle windows. TPEC30 retains a higher fraction of strict low-motion steps (0.084% versus 0.022%) and near-stationary steps (10.12% versus 7.80%), as well as two fully idle execution prefixes versus zero for Shortest30. TPEC30 achieves higher mean success (57.4% versus 51.0%) with that greater exposure to low-motion data.

Comparison with Random-NMS

TPEC30 and Random-NMS30 retain similar fractions of near-stationary steps (10.12% and 10.87%, a difference of 0.75 percentage points). TPEC30 achieves 57.4% mean success versus 37.4% for Random-NMS30, a 20.0-point gain. Across these two comparisons, TPEC30's success advantage persists at similar or greater exposure to near-stationary steps.

As reported in Q2, removing the 23 fully idle windows out of 184,960 (0.0124%) yields 38.6% mean success, compared with 40.6% for Full-raw and 41.6% for the random-deletion control.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revised manuscript uploaded

We thank the reviewers and the Area Chair for their help. We have uploaded an revised PDF .The revised manuscript includes the following changes relative to the original submission. Page and table numbers refer to the revised version unless otherwise noted.

  • ABC-130K pretraining (§5.4, Table 1, pp. 8–9; Appendix B.3, p. 23). We replaced the brief pretraining discussion with a quantitative comparison and documented the training setup. After fine-tuning on Pick Sausages, the checkpoint pretrained on the 30% of valid window starts selected by TPEC achieved 70.0% success, compared with 33.3% for full-pool pretraining.
  • Result corrections (§§5.4–5.5 and Appendix A.3). We corrected four RoboMimic entries in the original Tables 2–5, now Tables 3–6, and report their per-seed rollout counts in new Table 10. We also recomputed the aggregate real-robot success rate and the mean gains in the ablation study.
  • Additional RoboMimic analyses (Appendices A.4–A.7, pp. 16–20). We added a reference-composition stress test, supplementary fresh-scene evaluations, matched-update comparisons, motion controls, and an analysis of which recovery and alternative-strategy events are retained.
  • Fine-tuning and curation cost (§5.3 and Appendices B.2–B.3). We clarified how the 50-task results are aggregated, added the real-robot Random baseline, and reported curation costs in GPU-hours in Table 27, using unaccelerated CUPID-style reference.
  • Method clarifications (§4.2 and Appendices C.1 and C.4). We clarified how the efficiency score treats plateaus in predicted progress, how predictor eligibility is checked, and what the conditional training-signal bound establishes.

Summary of Rebuttal

Dear Area Chair,

The revised manuscript adds quantitative evidence at pretraining scale, tests TPEC against simple baselines under matched training conditions, and measures reference robustness and behavior coverage. Here are the main concerns of reviewers and the corresponding responds.

Pretraining scale

Concern (vZkw, H3fg). The submission lacked quantitative pretraining results. vZkw also asked how performance varies with data or model size.

Response. We added a fixed-model π₀ comparison on ABC-130K, which spans 129,182 trajectories across 200 tasks. TPEC retained 30% of valid window starts, corresponding to approximately 1,062 hours out of 3,540 hours. After fine-tuning both checkpoints on the same real-robot Pick Sausages dataset, the TPEC checkpoint achieved 70.0% success vs. 33.3% for full-pool pretraining (§5.4, Table 1).

Learned scoring, simple heuristics, and near-static motion

Concern (vZkw, H3fg). Shortest led in some settings. vZkw asked whether B-splines or near-static frame processing could recover TPEC’s gains, while H3fg asked what learned scoring and consensus add beyond simple heuristics.

Response. In the matched-update Transport MH experiment, TPEC reached 57.4%, compared with 53.6% for path length, the strongest added heuristic, 50.2% for full-data B-splines, and 47.6% for near-static merging. Shortest led at nominal 10% retention on Square and Transport. Shortest’s advantage was concentrated at the lowest retention rate, consistent with its inductive bias toward shorter demonstrations, which lets it cover more demonstrations under a tight window budget. However, across tasks and retention budgets, TPEC achieved higher mean success than Shortest in most matched settings (Tables 11 and 12). On Transport MH, TPEC also retained key recovery or alternative-strategy actions that Shortest omitted. Consensus improved mean success over One-vote in all ten task–budget ablations.

Training budgets and random selection

Concern (vZkw, H3fg). Random selection was competitive, and some full-data comparisons used different numbers of optimizer updates.

Response. At 150,000 updates per run across five Transport MH seeds, TPEC used 20.33% of candidate windows and gained 16.8 percentage points over full data and 20.0 over same-budget Random-NMS. Both paired 95% intervals exclude zero. At 50% retention, TPEC also improved over Random on the 50-task π₀ aggregate and achieved 31/90 vs. 19/90 successes across three real-robot tasks.

Reference robustness and behavior coverage

Concern (dPWZ, H3fg). dPWZ asked whether backtracking-heavy references destabilize ranking, whether trusted anchors help, and whether temporal NMS misses task stages. H3fg asked whether trajectory ranking discards rare recovery behaviors.

Response. Using only high-backtracking references yielded 88.6–90.3% window overlap with random references, within the variation between random draws. Adding four anchors had mixed effects. At nominal 30% retention, temporal NMS covered 610/611 eligible stages within selected trajectories. Across the candidate pool, TPEC retained 2/6 annotated rare events, versus 0/6 for Shortest and 6/6 for Random-NMS.

Result verification

Concern (vZkw, H3fg). The reviewers flagged inconsistent RoboMimic means

Response. We corrected the four flagged means and reported the underlying per-seed rollout counts in Table 10. What's more, we conducted repeated and extended experiments to further consolidate the results. The revised manuscript identifies the 50-task π₀ results as aggregate point estimates.

We believe the revised submission merits acceptance because the new evidence substantiates its central contribution. Robot demonstrations can guide their own curation without quality labels or policy rollouts, improving data efficiency under controlled training budgets and remaining practical at VLA pretraining scale.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.