acceptodds
Under review as a conference paper at ICLR 2027

Retiming Robot Demonstrations with a Learned Speed Prior

Abstract

The pace of a robot demonstration is often determined by the teleoperator rather than by the task itself. As a result, the same motion can appear at widely different speeds across demonstrations, and policies trained on such data may inherit operator-specific timing along with task behavior. We propose to learn a speed prior directly from large-scale demonstration data. Given a task instruction and a motion path, it predicts a distribution over execution speed along the path, from which a chosen quantile defines a consistent clock for retiming each demonstration. Existing methods derive the acceleration signal from the very data they accelerate, so each new task costs another fit, annotation, or pair of predefined rates. A single trained speed prior instead covers every task in a corpus. We fit a speed prior on both robot teleoperation data and handheld demonstrations in which a human arm, rather than a robot arm, moves the gripper, and use it to retime robot demonstrations to the handheld pace before fine-tuning a vision-language-action policy. For execution, we pair the policy with a real-time optimal-control layer that tracks predicted action chunks under robot constraints using the predicted pace. On 120 RoboLab tasks, policies fine-tuned on our learned clocks succeed more often and move with up to half the jerk compared with policies fine-tuned on the original demonstration timing and uniformly accelerated to the same speed. On a real robot, our policy stacks cups in 3.43s, 1.65 faster than baseline's 5.67s while succeeding in all 30 trials. Videos and further material are available at https://anonymous.4open.science/w/anony_repo-882B/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TuTfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains a ~15M-parameter transformer "speed prior" on teleoperated DROID data and handheld FastUMI-100K data. Given a local end-effector path window and the task instruction, it predicts a distribution over execution speed, and a chosen quantile defines a new clock for retiming demonstrations. Retimed DROID episodes are used to fine-tune π0.5, which also predicts "tempo channels" for faster deployment, and the policy is paired with a real-time optimal-control layer (DRAFT/SPLICE/SMOOTH) that respects robot limits and joins action chunks smoothly. On 120 RoboLab tasks the policy beats uniform acceleration at matched speed-up (+3.0 pp success on average, lower jerk and peak speed). On a Franka robot it stacks cups in 3.43 s versus 5.67 s with 30/30 successes.

Critical strengths
  1. The idea of learning a task-agnostic speed prior from faster demonstrators (handheld) and transferring it to slower teleoperated data is novel and avoids per-task fitting or annotation. The filtered-BC control (1.04x speed-up despite 1.60x faster demos) convincingly shows that pace transfer, not merely fast data, is what matters.

  2. The real-robot results are strong and the system is thoroughly engineered, with a 1.65x speedup at 100% success on cups and a well-specified controller. The ablation showing the PD baseline fails (3% towel success) versus the proposed controller shows that executing human pace requires this component.

Critical weaknesses
  1. The real-robot evaluation is narrow and the baselines are not very strong: three tasks, 5 episodes for trash cleaning, and no comparison against the closest prior methods (SAIL, DemoSpeedup, RACE, Realtime-VLA V2) run on the same setup. The SAIL comparison is a cross-paper number on a different robot, so the "4.3x faster" claim is not controlled, and the 30-trial success rates (e.g., 0.90 vs 0.97) are too noisy to distinguish conditions.

  2. The contribution bundles a speed prior, tempo channels, and a new controller, and the ablations do not cleanly separate their effects; the default policy under the proposed controller already gets most of the gains in several cases. In RoboLab the controller doesn’t seem to be used at all, and the success gain is modest, while the "half the jerk" claim holds at one matched point (2.8x) and partly reflects the retimed demonstrations being smoother, as the paper itself notes.

Questions for authors
  1. Can you compare directly against at least one prior speedup method (e.g., SAIL or RACE) on the same robot and tasks, and provide statistics for the differences at 30 trials?
  2. How much of the real-robot gain comes from the controller versus the retimed data? Please run the default and expert policies under the PD baseline and under your controller across all tasks, and run Ours under PD.
  3. Is there any evidence of failure modes where the transferred pace is unsafe, such as on tasks with contact-rich or precision phases?
  4. How sensitive are the results to the choice of target domain (shortest quarter, FastUMI) and to the DCT low-pass filter, given the tracking-jitter confound?
3 comments

Thank you for your constructive feedback. We appreciate your positive remarks on the novelty of learning a task-agnostic speed prior from faster demonstrators, the filtered-BC control, and the engineering of the system. Our responses to your specific questions are provided below. The numbers are from the revised pipeline described in our global response.

Q1. Comparison with prior speed-up methods on the same robot and tasks (W1, Q1)

We agree that the cross-paper comparison with SAIL was not controlled, and we have removed it from the main text. We implemented SAIL [1] and RACE [2] on the same policy backbone and data as the default policy and ran them on our robot, a Franka Research 3 with a Robotiq 2F-85 gripper. The two tasks are placing two bananas into a tray within 30 s and nine cubes into a bin within 60 s, without any fine-tuning on them. The cube task is new. Every condition runs 30 episodes (Appendix F, Tables 8 and 9).

ModelControllerBananas placedBothUPHCubes placedAll 9UPH
DefaultOurs (1x)1.6722231.75.900354.0
DefaultRACE1.7724354.76.906420.7
SAILOurs (3x)1.8024303.87.705467.1
OursOurs (1x)1.9729376.77.432446.7
OursOurs (q99)2.0030492.98.2712511.1

On its 0.99 tempo channel, our policy places more items than both methods on both tasks. We agree that success rates from 30 trials do not separate conditions, so we test the number of items placed per episode with a two-sided Mann-Whitney U test. The difference is significant against SAIL (p = 0.011 on bananas and 0.012 on cubes) and against RACE (p = 0.011 and p < 0.001). SAIL is reported at 3x because at 5x it reaches a lower UPH on bananas (272.9 against 303.8) and could not be run safely on cubes.

We also ran both methods on the three fine-tuning tasks. The table reports the successful episodes out of 30 and their mean completion time on stacking cups and folding towel, and the items placed per hour on cleaning trash (Appendix G, Table 10).

ModelControllerCups, successesCups, timeTowel, successesTowel, timeTrash, UPH
DefaultOurs (1x)296.02 s295.52 s444
DefaultRACE37.35 s–––
SAILOurs (3x)225.08 s194.58 s492
OursOurs (1x)294.40 s295.38 s600
OursOurs (q99)283.25 s303.85 s809

SAIL succeeds less often than the default policy on stacking cups and folding towel. Our policy on its 0.99 tempo channel succeeds as often as the default policy, in less time than SAIL, and places more trash per hour. Counting a failed episode as the slowest, its completion time over all 30 episodes is shorter than that of the default policy and of SAIL on both tasks (p < 0.001 in the same test). On folding towel and cleaning trash, the motion of RACE was unsafe for the robot, and we could not complete the runs.

In simulation (RoboLab, 120 tasks), the comparison is the following, where TPR is the throughput with regret per minute, which rewards a fast success and penalizes a failure (Appendix E, Table 6).

MethodSpeed-upSRTPR
Default (1x / 1.5x / 2x / 2.5x)1.00x / 1.56x / 1.90x / 2.48x0.462 / 0.518 / 0.533 / 0.5301.45 / 2.51 / 3.55 / 3.86
Ours (q50 / q90 / q95 / q99)1.35x / 1.81x / 2.00x / 2.38x0.532 / 0.568 / 0.577 / 0.6082.79 / 4.00 / 4.39 / 5.16
SAIL (2x / 3x / 4x / 5x)1.41x / 1.64x / 1.68x / 1.81x0.525 / 0.502 / 0.519 / 0.5162.53 / 2.86 / 3.14 / 3.25
RACE (on the default policy)4.47x0.3701.94

Raising the setting of SAIL does not take it beyond a speed-up of 1.81x, where it is 0.053 below ours in success rate (95% CI [0.021, 0.083]) and 0.75 below in TPR (95% CI [0.32, 1.15]). RACE falls below the default policy at 1x in success rate.

DemoSpeedup, like SAIL, finds the segments that need precision and plays the others faster by a fixed factor, and it sets this factor by hand for each task. We ran SAIL as the method of this kind. Realtime-VLA V2 trains a speed model on the adjustments that an operator makes by hand to a throttle while the policy runs on the robot. It needs new rollouts with an operator, which do not exist for tasks that the policy has not seen, so we did not run it.

Q2. Controller versus retimed data (W2, Q2)

Thank you for suggesting this decomposition. On the two tasks above, each policy now runs under both controllers. The table reports UPH.

PolicyBananas, PDBananas, our controllerCubes, PDCubes, our controller
Default (1x)200.7231.7234.0354.0
Ours (1x)266.5376.7284.0446.7
Ours (q99)394.0492.9338.0511.1

Each column shows the effect of the retimed data under one controller, and each row the effect of the controller for one policy. Both raise UPH, and neither alone reaches what the two give together.

As you suggested, we also ran every policy under PD on the fine-tuning tasks (successful episodes out of 30).

PolicyCups, PDCups, our controllerTowel, PDTowel, our controller
Default (1x)2829429
Expert (1x)99018
Ours (1x)2329029
Ours (q99)1128030

Our policies predict the positions that the arm should reach, and a faster policy places these positions further apart. The PD baseline cannot follow them at this pace. The arm falls up to 20° behind, against about 1° with our controller, and the task fails. On cleaning trash, we stopped the runs under PD for safety.

RoboLab does not use our controller, so the gain in success rate over uniform acceleration at the same speed-up comes from the retimed data alone (95% CIs, Figure 4).

Oursq50 (1.35x)q90 (1.81x)q95 (2.00x)q99 (2.38x)
Gain in SR+0.035 [0.003, 0.067]+0.039 [0.011, 0.068]+0.044 [0.018, 0.071]+0.078 [0.048, 0.110]

The revised Table 1 compares the motion at two matched speed-ups, 2.0x and 2.4x (jerk 0.83 against 1.23 and 1.16 against 2.41). As you note, this smoothness comes from the retimed demonstrations, and this is intended. Retiming with the median clock raises the SPARC of held-out DROID episodes from −3.91 to −3.09, whereas uniform acceleration to the same duration leaves it at −3.87 (Section 5.5).

Q3. Failure modes where the transferred pace is unsafe (Q3)

We counted the events of the contact and precision phases in all 1,200 episodes of each policy on the 120 RoboLab tasks (Appendix E, Table 7).

Default (1x)Default (2.5x)Ours (q99)
Grasps of a target lost within 1 s54.8%68.7%64.5%
Episodes with a target knocked out of the scene2.3%4.4%2.7%
Gripper-table collisions per episode0.991.981.94
Bumps into objects per episode3.915.054.65

A faster pace has a cost for both policies, but it is not larger for the transferred pace than for uniform acceleration at a similar speed-up, and a target object is knocked out of the scene less often.

On the real robot, our policy did not trigger a safety reflex of the arm in any of its 240 zero-shot episodes. On stacking cups, which requires precise placement, its success rate stays between 0.93 and 0.97 on all clocks, against 0.97 for the default policy (Table 3).

Q4. Sensitivity to the target domain and to the DCT (Q4)

When we retime DROID, the target domain could be DROID itself, a fast subset of DROID, or another dataset from faster demonstrators. With DROID itself, the prior reaches a speed-up of only 1.60x at its 0.99 quantile, against 2.38x for ours, and its policy is 0.032 below ours in success rate at the same speed-up (95% CI [0.002, 0.063], Table 14). A fast subset of DROID would be selected by task as much as by demonstrator, because its episodes differ in task. We therefore use another dataset, FastUMI-100K. When we fine-tune on a single task, all episodes share the task, so we use a fast subset, the shortest quarter. It was fixed before any evaluation, and we did not train policies on other fractions.

The setting of the DCT was fixed from the noise level of the tracked FastUMI-100K poses and was not tuned on results.

PriorSpeed-up (median clock)SPARC (retimed episodes)Speed-up with tracking jitter
With DCT1.28x−3.091.48x
Without DCT1.56x−3.452.04x

Without the DCT, the prior is faster but less smooth and less robust to tracking jitter. With it, the residual of the prior is uncorrelated with the removed energy (Spearman correlation 0.008 on DROID and 0.001 on FastUMI-100K), so the transferred pace mainly reflects how handheld demonstrators move rather than how they are tracked (Appendix H).

Once again, we really appreciate your insightful questions. We hope that our response addresses all of your questions.

References

[1] Nadun Ranawaka Arachchige et al., SAIL: Faster-than-Demonstration Execution of Imitation Learning Policies, CoRL 2025
[2] Sunwoo Kim et al., Time Optimal Execution of Action Chunk Policies Beyond Demonstration Speed, ICLR 2026

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer TuTf,

As the discussion period ends soon, we wanted to kindly check whether our response has addressed your concerns. We would be happy to clarify anything further.

Best regards,
The Authors

Reviewer_VSL4OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a framework for accelerating VLA execution by transferring the execution pace of faster demonstrations to slower ones. The key idea is to separate the motion path from its timing. A speed prior is trained to predict a distribution of execution speeds from local path geometry, task instructions, and a demonstration-domain token. By replacing the source-domain token with a faster target-domain token, the method predicts a new speed profile for the original path and uses it to retime demonstrations for VLA fine-tuning. The fine-tuned policy predicts both actions and their execution pace. The authors also propose a constrained execution controller that respects robot dynamics and smoothly connects consecutive action chunks.

The method is evaluated on 120 RoboLab simulation tasks and real-robot manipulation tasks, including cup stacking, towel folding, and trash cleaning. The results show improvements in execution speed and motion smoothness while maintaining or improving task success.

Critical strengths

1. Interesting and intuitive idea.

Learning execution pace from faster demonstrators and transferring it to other trajectories without requiring trajectory correspondence is an interesting approach. It offers an alternative to predefined or task-specific heuristics for demonstration retiming.

2. Practical consideration of robot execution.

The paper recognizes that faster predicted motion is not necessarily feasible for the robot and addresses this issue through a constrained controller that also handles discontinuities between action chunks.

Critical weaknesses

1. The benefit of learning the speed prior is not sufficiently established against stronger alternatives.

In RoboLab, the improvement in success rate over uniform acceleration at matched speed is relatively modest (3%). The main comparisons are against uniform acceleration and training on faster demonstrations alone. However, existing methods such as ISR and DemoSpeedup also use nonuniform demonstration retiming based on motion-related heuristics. Without comparing against these or other simpler adaptive retiming strategies, it is difficult to assess how much is gained specifically by learning a transferable speed prior.

2. A substantial part of the real-world performance improvement comes from the proposed controller.

The controller itself substantially improves the performance of the default policy. For example, towel-folding success increases from 3% with the default PD controller to 87% with the proposed controller, even without the learned speed prior. The authors do compare policies under the same controller, which helps isolate the effect of learned retiming. Nevertheless, the very large improvement from the controller makes it important to distinguish the contribution of speed-prior learning from that of the execution framework. The novelty and necessity of the controller's individual components could also be established more clearly.

Questions for authors

Generalization of the speed prior.

How well does the speed prior transfer to unseen tasks or motion distributions without task-specific adaptation? In particular, how does the method handle cases where the appropriate execution speed depends on physical properties or contact conditions that are not directly observable from path geometry?

5 comments

Thank you for your constructive feedback. We appreciate your positive remarks on the idea of transferring the pace of faster demonstrators without trajectory correspondence and on the practical consideration of robot execution. Our responses to your specific questions are provided below. The numbers are from the revised pipeline described in our global response.

Q1. Benefit of the learned speed prior against stronger alternatives (W1)

We agree that uniform acceleration and training on faster demonstrations alone do not establish this benefit. We implemented ISR [1], SAIL [2], and RACE [3] on the same policy backbone, data, and evaluation as the default policy (RoboLab, 120 tasks).

MethodSpeed-upSR
Default (1x / 1.5x / 2x / 2.5x)1.00x / 1.56x / 1.90x / 2.48x0.462 / 0.518 / 0.533 / 0.530
Ours (q50 / q90 / q95 / q99)1.35x / 1.81x / 2.00x / 2.38x0.532 / 0.568 / 0.577 / 0.608
SAIL (2x / 3x / 4x / 5x)1.41x / 1.64x / 1.68x / 1.81x0.525 / 0.502 / 0.519 / 0.516
ISR1.96x0.243
RACE (on the default policy)4.47x0.370

ISR, SAIL, and DemoSpeedup were designed and evaluated on a few single tasks, with their settings chosen for each task. In contrast, we train one policy on the 74,546 DROID episodes of many tasks and evaluate it on 120 unseen tasks, where the settings cannot be chosen for each task. In this setting, SAIL does not go beyond a speed-up of 1.81x, where it is 0.053 below ours in success rate (95% CI [0.021, 0.083]), and ISR and RACE fall below the default policy at 1x (Appendix E, Table 6). DemoSpeedup sets its speed-up by hand for each task, so we ran SAIL as the method of its kind. The speed prior, in contrast, is trained once across tasks and needs no setting per task.

We also ran SAIL and RACE on the real robot. On the three fine-tuning tasks, SAIL stacks the cups in 22 of 30 episodes and folds the towel in 19, against 28 and 30 for our policy on its 0.99 tempo channel, and places 492 items of trash per hour against 809 (Appendix G, Table 10). RACE stacks the cups in 3 episodes, and on the other two tasks its motion was unsafe for the robot, so we could not complete the runs. On the two tasks without fine-tuning that Q2 describes, our policy places 492.9 bananas per hour against 303.8 for SAIL and 354.7 for RACE, and 511.1 cubes against 467.1 and 420.7 (Appendix F, Tables 8 and 9).

The gain over uniform acceleration is also larger than the 3% of the submission. At the same speed-up, our policy is above uniform acceleration by 0.035 in success rate at 1.35x (95% CI [0.003, 0.067]) and by 0.078 at 2.38x (95% CI [0.048, 0.110]). The gain grows with speed, because uniform acceleration stops improving beyond 2x (Figure 4).

To see what the transfer itself adds, we also trained the same speed prior on DROID alone, which is adaptive retiming without a faster target. Even at its 0.99 quantile, it reaches a speed-up of only 1.60x, and its policy succeeds as often as uniform acceleration at that speed-up (0.519 against 0.518). It speeds up all parts of the motion alike. With the handheld pace, the speed-up is instead largest near objects, where teleoperators are slow, 2.40x within 3 cm of an object against 1.97x beyond 20 cm on the 0.99 tempo channel (Appendix H, Table 14).

Q2. Contribution of the speed prior versus the controller (W2)

We agree that the two must be distinguished. The comparison of Q1 already does so for the speed prior, because no policy uses our controller on RoboLab. On the two real-robot tasks that run without any fine-tuning, placing two bananas into a tray within 30 s and nine cubes into a bin within 60 s, we ran each policy under both controllers with 30 episodes per condition. The table reports the items placed per hour (UPH, Appendix F, Tables 8 and 9).

PolicyBananas, PDBananas, our controllerCubes, PDCubes, our controller
Default (1x)200.7231.7234.0354.0
Ours (1x)266.5376.7284.0446.7
Ours (q99)394.0492.9338.0511.1

Each column fixes the controller and shows the effect of the retimed data, and each row fixes the policy and shows the effect of the controller. Both raise UPH on both tasks, and neither alone reaches what the two give together.

On the three fine-tuning tasks, all policies run under our controller, so the differences between them come from the data. With the 0.95 tempo channel, our policy shortens stacking cups from 6.02 s to 3.50 s and folding towel from 5.52 s to 4.20 s at the same success rate of 0.97, and raises cleaning trash from 444 to 792 items per hour (Table 3).

Q3. Necessity and novelty of the components of the controller (W2)

To see whether the controller is needed, we ran every policy under the PD baseline on the fine-tuning tasks. The table counts the successful episodes out of 30 (Appendix G, Table 10).

PolicyCups, PDCups, our controllerTowel, PDTowel, our controller
Default (1x)2829429
Expert (1x)99018
Ours (1x)2329029
Ours (q99)1128030

Our policies predict the positions that the arm should reach, and a faster policy places these positions further apart. The PD baseline cannot follow them at this pace. The arm falls up to 20° behind, against about 1° with our controller, and the task fails. On cleaning trash, we stopped the runs under PD for safety.

We also ablated the two functions of our controller (Figure 2), with our policy on its 0.99 tempo channel and 30 episodes per condition. Limit enforcement (Draft and Smooth) keeps the plan within the velocity, acceleration, and torque limits of the robot. Entry selection (Splice) joins a new chunk at the earliest point that the arm reaches on time, rather than at its first waypoint. Without both, a cubic spline through the predicted waypoints at the predicted pace replaces the plan (Appendix G, Table 11).

Stacking cups

ControllerSRTPRATRSPARCReflex stops
PD0.370.205.89−3.380 / 30
Ours w/o limit enforcement and entry selection0.402.713.99−2.8415 / 30
Ours w/o entry selection0.807.505.79−4.040 / 30
Ours0.9316.843.25−3.000 / 30

Folding towel

ControllerSRTPRATRSPARCReflex stops
PD0.00−6.00–−3.340 / 30
Ours w/o limit enforcement and entry selection0.00−6.00–−2.5330 / 30
Ours w/o entry selection0.37−0.346.37−4.460 / 30
Ours1.0015.613.85−3.040 / 30

Without both functions, the reference exceeds the limits of the robot, and the safety reflex stops the arm in 15 episodes on cups and in all 30 on towel. In these episodes, SPARC covers only the motion before the stop. With limit enforcement but without entry selection, the reflex stops no episode, but the arm often brakes hard or turns back when it switches chunks. It moves less smoothly, takes longer, and succeeds less often. Our controller builds on established tools (time-optimal path parameterization, Ruckig, and a quadratic program). What is new is planning ahead from a reserved switch point and selecting the entry, so that a chunk runs at its predicted pace within the limits.

Q4. Generalization of the speed prior (Q)

The prior is trained once on DROID and FastUMI-100K and is not adapted to any evaluation task. The policy trained on its retimed data is evaluated on the 120 RoboLab tasks and on the two real-robot tasks of Q2 without fine-tuning on them. For motions that the faster demonstrators cover poorly, we sorted the DROID windows by their distance to the nearest FastUMI-100K window. From the closest fifth to the farthest fifth, the median error of the predicted speed grows from 5.4% to 8.0% while its bias stays below 1%, and the speed-up that the prior transfers stays between 1.26x and 1.30x (Appendix H, Table 15).

We agree that the path geometry alone does not determine the appropriate speed. The revised prior therefore also reads an image at every node of the path, so that it sees the objects and the contacts that the camera shows. The table reports the share of the variance of held-out log speed that the predicted median explains (Appendix H).

Held-out windowsWith imagesWithout images
DROID0.950.91
FastUMI-100K0.930.86

As a check on contact, we counted the contact events of all episodes on RoboLab. A target object is knocked out of the scene in 2.7% of the episodes of our policy at a speed-up of 2.38x, against 4.4% under uniform acceleration to 2.48x, and the gripper collides with the table as often (1.94 against 1.98 times per episode, Appendix E, Table 7).

Once again, we really appreciate your insightful questions. We hope that our response addresses all of your questions.

References

[1] Licheng Yang et al., Improving Robotic Imitation Learning via Trajectory Standardization, arXiv 2026
[2] Nadun Ranawaka Arachchige et al., SAIL: Faster-than-Demonstration Execution of Imitation Learning Policies, CoRL 2025
[3] Sunwoo Kim et al., Time Optimal Execution of Action Chunk Policies Beyond Demonstration Speed, ICLR 2026

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer VSL4,

As the discussion period ends soon, we wanted to kindly check whether our response has addressed your concerns. We would be happy to clarify anything further.

Best regards,
The Authors

Reviewer_BvrKOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem that imitation-learning policies inherit teleoperation-induced hesitation and operator-specific timing, which makes their motion slow. The paper proposes a method that learns a speed prior from DROID, a teleoperation dataset, and FastUMI-100K, a handheld dataset, and retimes DROID trajectories at a FastUMI-derived pace. Furthermore, it combines this with a real-time optimal-control layer that executes the action chunks predicted by the policy, treating the predicted pace as an upper bound and respecting the robot's velocity, acceleration, and torque limits. The method is evaluated in simulation, where chunks are played back without this controller, and on a real robot, where the controller is used, demonstrating fast and smooth execution.

Critical strengths
  • The problem setting is practical and clear, and the goal of removing unnecessary timing that originates from teleoperation is important.
  • Rather than simple uniform acceleration, it learns a pace conditioned on the local path, the gripper opening, and the task instruction, which results in non-uniform speed-ups, e.g., larger speed-ups around grasps and releases than during transit (Table 4).
  • It is complete as a system, covering not only retiming but also real-time execution that accounts for velocity / acceleration / torque constraints.
Critical weaknesses
  • The effect of cross-domain pace transfer via FastUMI is not sufficiently isolated. Since the policy is trained only on retimed DROID episodes, FastUMI affects the results solely through the speed prior. However, it remains unclear whether the improvement requires FastUMI's conditional pace distribution, or whether a speed prior trained on DROID alone, read at a higher quantile matched to the same average speed-up, would yield comparable results. Such a DROID-only control experiment would clarify the contribution of cross-domain transfer.
  • The set of compared methods is limited. The main comparisons are against uniform acceleration and a fast DROID subset, and superiority over more structured retiming / speed-adaptation methods is not sufficiently demonstrated.
  • The validity of the claim "Their demonstrations need not follow the same paths." An implicit overlap assumption between the two datasets cannot be ruled out. Even if exact correspondence is not required, it seems necessary that sufficiently similar local path geometry exists between DROID and FastUMI. This overlap, and the prediction accuracy in regions where target-speed support is weak, are not sufficiently verified.
Questions for authors
  1. How do the authors expect FastUMI-conditioned retiming to compare with a speed prior trained on DROID only and read at a higher quantile to obtain the same average speed-up? In particular, what aspects of the pace, beyond overall speed, do the authors believe are transferred from FastUMI and could not be obtained from the faster DROID demonstrations alone?
  2. How much do the speed distributions conditioned on similar local paths differ between DROID and FastUMI?
  3. How much overlap is there between the local path distributions of DROID and FastUMI?
5 comments

Thank you for your constructive feedback. We appreciate your positive remarks on the practicality of the problem setting, the non-uniform speed-ups that the learned pace gives, and the completeness of the system. Our responses to your specific questions are provided below. The numbers are from the revised pipeline described in our global response.

Q1. Speed prior trained on DROID alone (W1, Q1)

Thank you for suggesting this control. We trained the same speed prior on DROID alone, read it at the same quantiles as ours, and trained and evaluated the policy in the same way as ours. The table below reports the 0.99 quantile, with the speed-up over the default policy at 1x on RoboLab (120 tasks), overall and by the distance of the end effector to the nearest object, together with the success rate (SR). All four quantiles are in Appendix H (Table 14).

PolicyOverallWithin 3 cm of an objectBeyond 20 cmSR
Default (1.5x)1.56x1.55x1.53x0.518
DROID-only prior (q99)1.60x1.62x1.68x0.519
Ours (q99)2.38x2.40x1.97x0.608

The DROID-only prior does not reproduce the gain. It reaches a lower speed-up than ours at the same quantile, and at the same speed-up, its policy is 0.032 below ours in success rate (95% CI [0.002, 0.063]) and not different from uniform acceleration.

What the handheld demonstrations transfer beyond the overall speed is where the motion is sped up. A higher quantile of DROID is still a teleoperated pace, so it speeds up all parts of the motion alike, as uniform acceleration does. With the handheld pace, the speed-up is instead largest near objects, where teleoperators are slow.

Q2. Comparison with structured retiming and speed-adaptation methods (W2)

We agree that the set of compared methods in the submission was limited. We implemented SAIL [1], ISR [2], and RACE [3] on the same policy backbone, data, and evaluation as the default policy (Appendix E, Table 6).

MethodSpeed-upSR
Default (1x / 1.5x / 2x / 2.5x)1.00x / 1.56x / 1.90x / 2.48x0.462 / 0.518 / 0.533 / 0.530
Ours (q50 / q90 / q95 / q99)1.35x / 1.81x / 2.00x / 2.38x0.532 / 0.568 / 0.577 / 0.608
SAIL (2x / 3x / 4x / 5x)1.41x / 1.64x / 1.68x / 1.81x0.525 / 0.502 / 0.519 / 0.516
ISR1.96x0.243
RACE (on the default policy)4.47x0.370

SAIL is best at 2x, where its success rate is close to ours at the same speed-up (−0.012, 95% CI [−0.038, +0.013]). Raising its setting does not take it beyond a speed-up of 1.81x, where it is 0.053 below ours (95% CI [0.021, 0.083]). ISR and RACE fall below the default policy at 1x in success rate.

We also ran SAIL and RACE on the real robot. On the three fine-tuning tasks, SAIL stacks the cups in 22 of 30 episodes and folds the towel in 19, against 28 and 30 for our policy on its 0.99 tempo channel, and places 492 items of trash per hour against 809 (Appendix G, Table 10). RACE stacks the cups in 3 episodes, and on the other two tasks its motion was unsafe for the robot, so we could not complete the runs. On two further tasks that run without fine-tuning, placing bananas into a tray and cubes into a bin, our policy places 492.9 and 511.1 items per hour, against 303.8 and 467.1 for SAIL and 354.7 and 420.7 for RACE (Appendix F, Tables 8 and 9).

Q3. Overlap between the local paths of DROID and FastUMI-100K (W3, Q3)

We agree with this point. The prior needs no matched demonstrations, but it does need similar local windows in both datasets, and the revised manuscript now states this.

To measure the overlap, we took 100,000 training windows per domain and described each window as the prior reads it. For every DROID window, we compared the distance to its nearest FastUMI-100K window with the distance to its nearest DROID window of another episode. A DROID window counts as covered if a FastUMI-100K window lies within the 90th percentile of the DROID nearest-neighbor distances.

Window descriptorCovered DROID windows
Path shape (distances between nodes)86%
Position81%
Position and rotation72%
Position, rotation, and gripper opening (the prior's input)61%

The shapes of local paths overlap largely, and the two domains differ mainly in wrist rotation and gripper use. This analysis and those of Q4 and Q5 are in Appendix H (Table 15).

Q4. Speed distributions on similar local paths (Q2)

For the held-out DROID windows that have a FastUMI-100K window within the median distance between DROID neighbors (64,892 windows, 18%), we compared the recorded speed of the nearest FastUMI-100K windows with the window's own recorded speed. This ratio uses no model. Next to it we report the speed-up that the prior transfers at the same distance to a gripper event.

Distance to a gripper eventHandheld / teleoperated recorded speedSpeed-up transferred by the prior
2 to 4 cm1.80x1.57x
6 to 10 cm1.42x1.39x
12 to 16 cm1.40x1.32x
Transit1.23x1.28x

On similar paths, the handheld pace is faster not by a constant factor but by one that grows towards gripper events. The pace that the prior transfers follows the same profile, without having seen matched pairs.

Q5. Prediction accuracy where FastUMI-100K support is weak (W3)

The handheld pace of a DROID window has no ground truth, so we cannot verify the prediction there directly. Instead, we sort the DROID windows by their distance to the nearest FastUMI-100K training window, split them into five equal groups, and report two checks.

DROID windowsMedian error of the predicted speed (held-out FastUMI-100K)BiasSpeed-up transferred to DROID
Closest 20% to FastUMI-100K5.4%−0.1%1.26x
Middle 20%7.1%−0.5%1.30x
Farthest 20% from FastUMI-100K8.0%−0.9%1.30x

First, on held-out FastUMI-100K windows that are as far from the handheld training data as each group, the predictions get less sharp with weaker support but remain unbiased. Second, the speed-up that the prior transfers to DROID is stable across the groups, so the prior does not behave differently where the support is weak.

Once again, we really appreciate your insightful questions. We hope that our response addresses all of your questions.

References

[1] Nadun Ranawaka Arachchige et al., SAIL: Faster-than-Demonstration Execution of Imitation Learning Policies, CoRL 2025
[2] Licheng Yang et al., Improving Robotic Imitation Learning via Trajectory Standardization, arXiv 2026
[3] Sunwoo Kim et al., Time Optimal Execution of Action Chunk Policies Beyond Demonstration Speed, ICLR 2026

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer BvrK,

As the discussion period ends soon, we wanted to kindly check whether our response has addressed your concerns. We would be happy to clarify anything further.

Best regards,
The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response

We thank all reviewers for their constructive feedback. We have uploaded a revised manuscript, in which all changes are marked in blue. This response describes what has changed since the submission and lists the experiments that we added. All numbers in our responses are from the revised pipeline. We have also updated the project page linked in the abstract with an illustrated overview of the method and videos of the real-robot experiments.

Changes to the method and the simulator

  • Method. The arc length of a path now also counts gripper motion, and the speed prior now reads images, since we too found that the path alone does not determine the appropriate pace.

  • Simulator. In the submission, the simulated robot was not physically set up. The link inertias of the arm were orders of magnitude too small, its joints had no rotor inertia, and the gripper pressed with less than 1 N and closed at no fixed speed. We now simulate the Franka Research 3 that we use, with its inertias and velocity limits, and Robotiq's gripper model at the closing speed that DROID commands. Under this setting, the success rate of the default policy at 1x is 0.462 (0.388 in the submission), and our margin over uniform acceleration is 0.049 (0.030 in the submission).

New experiments

  • Baselines on RoboLab. We implemented SAIL, ISR, and RACE and compared them with our policy (Appendix E, Table 6).
  • Real-robot tasks without fine-tuning. We added a second task, placing cubes into a bin, alongside placing bananas into a tray. On both tasks, we ran SAIL and RACE, and we ran the default policy and our policy under both the PD baseline and our controller (Section 5.3, Appendix F, Tables 8 and 9).
  • Fine-tuning tasks. On the three tasks of the submission, we ran every policy under the PD baseline and added SAIL and RACE (Appendix G, Table 10).
  • Controller ablation. We ablated limit enforcement and entry selection in our controller on stacking cups and folding towel (Appendix G, Table 11).
  • Analysis. We trained the speed prior on DROID alone, measured the overlap between DROID and FastUMI-100K, and read the prior without its images. We also counted the contact events on RoboLab (Appendix H, Tables 14 and 15, and Appendix E, Table 7).
AuthorsOpenReview ↗

Final Remarks

Dear Area Chair and Reviewers,

We thank the reviewers for the discussion. We summarize the main concerns and how the revision addresses them.

  • Stronger baselines (all reviewers). We implemented SAIL, ISR, and RACE on the same policy backbone, data, and evaluation. On RoboLab, the best success rate and throughput with regret (TPR) of our policy are 0.608 and 5.16, against 0.530 and 3.86 for uniform acceleration and 0.525 and 3.25 for SAIL (Table 6). On the real robot, our policy places significantly more items than SAIL and RACE on both tasks without fine-tuning (Tables 8 and 9). On the fine-tuning tasks, SAIL succeeds less often than our policy, and RACE could not be run safely on two of the three tasks (Table 10).
  • Controller versus retimed data (TuTf, VSL4). The default policy and our policy ran under both the PD baseline and our controller, and we ablated the two functions of the controller. Both the data and the controller contribute (Tables 8 to 11).
  • Contribution of the handheld demonstrations (BvrK). A speed prior trained only on DROID does not reproduce the gain, and the local paths of DROID and FastUMI overlap largely (Tables 14 and 15).

We thank Reviewer BvrK for confirming that most concerns are addressed and Reviewer VSL4 for raising the score.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.