acceptodds
Under review as a conference paper at ICLR 2027

Does Imitation Learning Preserve Expert Performance Across Task Execution Speeds? A Benchmark Study

Abstract

Imitation learning seeks to reproduce expert behavior and is commonly evaluated by task success rate, the fraction of evaluation episodes completed successfully. Matching expert success under nominal conditions does not establish how the expert–learner performance difference changes with task conditions. We study how the expert–learner performance difference varies with execution timing. Timing changes are expressed through a scalar speedup factor; a factor of two halves the scheduled durations. We introduce ParcelStow, a benchmark with three contact-rich manipulation tasks—parcel insertion, upright placement, and keyed peg insertion—that varies these durations while fixing geometry, physical parameters, and success criteria. We evaluate Action Chunking with Transformers (ACT), Diffusion Policy, and DAgger using success rates, task-stage completion, and failure diagnostics. On parcel insertion, ACT matches the expert's 100% at nominal timing but falls to 53% at the maximum demonstrated speedup factor, compared with 84% for the expert. On upright placement and keyed peg insertion, ACT has higher nominal success than the expert, but this ordering reverses beyond the demonstrated speedup factors. These results show that nominal task success does not establish preservation of expert performance under changed task conditions, with execution timing providing a concrete counterexample.

Primary area:
applications to robotics, autonomy, planning
Keywords:
imitation learning, robot learning, policy evaluation, task execution speed, expert–learner comparison, benchmark, contact-rich manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WGRCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether an imitation-based policy that matches expert success at nominal speed still preserves the expert–learner gap when execution timing changes. Timing is set by a scalar speedup factor that shortens scheduled durations while geometry, physical parameters, and success criteria stay fixed. The authors introduce ParcelStow and evaluate Action Chunking with Transformers (ACT), Diffusion Policy, and DAgger by success rate, stage completion, and failure diagnostics.

Critical strengths
  • ParcelStow varies execution timing through a speedup factor while holding geometry, physical parameters, and success criteria fixed. Success rate, stage completion, and failure diagnostics are reported together, so the expert–learner gap is not reduced to a single terminal outcome.

  • Final evaluation outcomes do not select policies, checkpoints, or seeds. Experiments settings are also documented in detail. This limits selection on the reported numbers and makes the comparisons easier to reproduce.

Critical weaknesses
  • The motivation for studying execution-speed changes is not sufficiently established. While shifts in lighting or object position are familiar deployment conditions, a scalar speedup of scheduled durations is less clearly a condition that imitation-based policies face. And the paper does not explain why the expert–learner gap under this change is the right quantity to measure. The reference itself makes the comparison harder to interpret. Appendix A.4.1 uses a scripted expert, and Figure 1 shows that its success often drops sharply once the speedup factor exceeds 2. A learner that stays close to a script already failing under fast execution is not shown to preserve competent expert behavior.

  • The analysis of ACT, Diffusion Policy, and DAgger remains descriptive. The paper reports success, stage completion, and failure modes, and ParcelStow localizes many failures to particular stages. However, it does not explain why the three methods diverge as speedup factor changes, or why the localized stages fail. The study therefore reads as a report of benchmark outcomes and contributes little to imitation learning.

Questions for authors
  • Given a scripted expert specified in Appendix A.4.1, what success is that expert expected to retain as the speedup factor increases? If its success falls sharply above a factor of 2 as shown in Figure 1, what does matching or approaching that expert mean?
  • What accounts for the differences among ACT, Diffusion Policy, and DAgger as the speedup factor increases? For the stages where ParcelStow localizes the failures, which evidence explains the failure, and does that explanation differ across the three methods?
3 comments
AuthorsOpenReview ↗

Reply to Reviewer WGRC

Thank you for the review and the questions about motivation, the expert reference, and failure mechanisms.

W1 and Q1. Motivation for Execution Timing and Interpretation of the Expert Reference: An execution schedule specifies the time available for each task phase. Our formulation makes this requirement explicit through selected phase durations Tk/rT_k/r, with geometry, physical parameters, and success criteria fixed. Varying the schedule changes the demands on the demonstrated strategy and the learned policy. We therefore evaluate both actors under the same timing law and paired initial conditions, measuring GL(r)=p^E(r)−p^L(r)G_L(r)=\widehat p_E(r)-\widehat p_L(r) alongside their success curves. The reference curve records how much success the demonstrator retains as the schedule changes. On PARCEL, both actors succeeded in 100/100 nominal episodes, while at the maximum demonstrated factor r=2r=2 the expert retained 84/100 and ACT achieved 53/100. The resulting 31 percentage point difference measures a loss of expert-relative performance while the reference remains successful in most episodes (Sec. 5.1). Contributions, Sec. 1.1, states the formulation, paired estimator, and diagnostic substitution as our contributions.

Our PARCEL tracking diagnostics relate the expert's declining success to its pre-insertion alignment. Median minimum orientation error increased from 2.2∘2.2^\circ at nominal timing to 7.6∘7.6^\circ at r=2r=2, 8.9∘8.9^\circ at r=2.25r=2.25, and 10.3∘10.3^\circ at r=2.5r=2.5, crossing the 10∘10^\circ tolerance. Median arm velocity utilization remained below 0.14 through r=2.5r=2.5 (Appendix D.4, Table 14). These measurements characterize the reference's tracking behavior as timing tightens.

W2 and Q2. Evidence Explaining Method Differences and Localized Failures: We tested the speed input in our ACT policy. During acquisition, whose scheduled durations are independent of rr, we replaced only the speed input with 1, holding weights, phase feature, and schedule fixed. At r=1.5r=1.5, acquisition rose from 141/200 to 198/200 and success from 3/200 to 131/200. Conversely, setting the acquisition input to 1.5 at nominal timing reduced acquisition from 197/200 to 151/200 and success from 191/200 to 7/200. The value 1.5 lies 5.5 standard deviations above the PEG feature's training mean. The paired substitutions identify the speed input as a cause of acquisition failure (Sec. 3.3, Eq. 7, and Sec. 5.3, Table 2, with terminal outcomes in Appendix E, Table 15).

For, Diffusion Policy, we used the same raw speed observation and acquired 197/200 pegs at r=1.5r=1.5, followed by 12/200 at r=1.75r=1.75 (Sec. 5.2). Its acquisition response therefore differed from ACT's under the common observation variable. DAgger's near-zero success supports failure characterization, while Diffusion Policy supplies additional success curves from its own nominal level (Sec. 5.4). The input-substitution evidence explains the acquisition sensitivity of the evaluated ACT checkpoint.

On UPRIGHT at r=2r=2, both ACT and the expert acquired 200/200 objects. Post-release failures accounted for 173/176 ACT failures and 91/95 expert failures, localizing the loss after release (Sec. 5.2). On PARCEL, we continued both actors after acquisition with a common relative-motion controller retaining actor-specific physical states and anchors. At r=2r=2, this controller produced 95/100 successes following expert acquisition and 64/100 following ACT acquisition. The difference persists under a common downstream controller and depends on the conditions produced before handoff (Sec. 5.5).

Thank authors for the response. These measurements clarify how the expert's declining success relates to its pre-insertion alignment. I also appreciate the input-substitution. However, my concern about the application of execution-speed changes remains, and it is still not clear that varying the scheduled speedup is a condition that imitation-based policies commonly face. I will therefore maintain my rating.

AuthorsOpenReview ↗

Follow-up on W1/Q1: Literature and Applications of Execution-Timing Changes

Thank you for your comment and for recognizing the added analyses.

W1 and Q1. Applications of Execution-Timing Changes: DemoSpeedup provides a direct real-robot example: its Conveyer Fast evaluation reused the same trained checkpoints with a conveyor running twice as fast as during data collection. The robot had to acquire, scan, and place a moving bag in a basket, so the external belt speed changed the time available to execute the task (Sec. 4.2).

Duration and frequency commands also appear directly in imitation learning. Variable-speed teaching-playback [1] trained duration- or frequency-conditioned models for pick-and-place and wiping using physical playback at changed speeds. RACE [2] retimes predicted state chunks under joint torque and velocity constraints. TempoVLA [3] learns an explicitly speed-conditioned policy supporting slower and faster execution through variable-speed trajectory augmentation. SpeedTuning [4] learns a speed policy over imitation-trained action chunks, ESPADA [5] downsamples demonstrations by manipulation phase, AutoSpeed [6] learns stage-dependent speed, and SkiP [7] relabels targets to skip noncritical segments while preserving dense supervision in key segments. These are concrete physical execution-speed objectives.

Related work also studies the feedback consequences of execution timing. REMAC [8] learns delay-conditioned corrections for asynchronous execution. BID [9] selects chunks for temporal consistency and reaction, while SGAC [10] adapts action-queue replacement. FASTER [11] streams completed actions to reduce reaction latency, and AutoHorizon [12] adapts when the policy replans. These methods establish the importance of execution scheduling, delay, and feedback alongside physical retiming.

Our scalar speedup factor makes the execution-time requirement controlled and repeatable: selected phase durations become Tk/rT_k/r, while acquisition and settling durations remain fixed. We apply the same schedule to expert and learner, holding geometry, physical parameters, and success criteria fixed. The paired expert reference measures how much demonstrator success is preserved as this requirement changes.

On PARCEL, ACT and the expert both achieved 100/100 successes at nominal timing. At r=2r=2, within the demonstrated interval, the expert achieved 84/100 and ACT achieved 53/100, a 31 percentage point difference (Sec. 5.1). Nominal agreement thus coexisted with a substantial expert-relative deficit under another demonstrated timing requirement.

We expanded Related Work, Sec. 1.2, with the following 12 references, including six ICLR/NeurIPS papers. Ten are published or accepted, and two are labeled preprints.

  1. Variable-speed teaching-playback, Advanced Robotics 2025.
  2. RACE, ICLR 2026.
  3. TempoVLA, preprint 2026.
  4. SpeedTuning, ICRA 2025.
  5. ESPADA, IEEE RA-L 2026.
  6. AutoSpeed, preprint 2026.
  7. SkiP, NeurIPS 2026, accepted.
  8. REMAC, ICLR 2026.
  9. BID, ICLR 2025.
  10. SGAC, NeurIPS 2025.
  11. FASTER, NeurIPS 2026, accepted.
  12. AutoHorizon, ECCV 2026.
Reviewer_kDtZOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ParcelStow, a benchmark for evaluating imitation learning policies’ robustness across varying execution speeds, to address the limitations of existing benchmarks that only assess success rates under nominal speed and fail to verify policy performance under tighter timing constrain.Through evaluations on three contact-rich simulated tasks and stage-level failure analysis, the work finds ACT, Diffusion Policy and DAgger match expert performance at nominal speed, though ACT degrades noticeably under accelerated execution.

Critical strengths
  1. The paper introduces a new benchmark, ParcelStow, to evaluate imitation learning under varying execution speeds, highlighting that matching expert performance at nominal speed does not guarantee robustness to tighter timing constraints. This is an important and underexplored aspect of evaluating imitation learning policies.
  2. The experimental results provide clear evidence supporting the paper’s central claim, espacially in parcel insertion, where success rates drop from 100% at nominal speed to 84% for the expert and 53% for ACT at twice the nominal speed.
Critical weaknesses
  1. Although the parcel insertion results provide clear evidence for the central claim, the effective baseline coverage remains limited. DAgger achieves only 3%, 0.5%, and 0% nominal success across the three tasks, leaving little room to assess degradation under acceleration. Diffusion Policy also starts substantially below expert performance on two tasks, so the strongest evidence comes primarily from ACT. Adding another method with competitive nominal performance would strengthen this aspect of the paper.
  2. Section 5.2 reports 59/200 acquisition failures on PEG at r=1.5r=1.5, despite unchanged acquisition timing. Since the policy receives the speed factor during acquisition, these failures may reflect sensitivity to the speed input rather than an inability to execute faster movements. The paper acknowledges this possibility but does not test it directly. Comparing policies trained with and without the speed input, while keeping phase information and the timing schedule unchanged, would help separate these effects.
  3. Section 1.1.1 discusses DemoSpeedup, SAIL, and SpeedAug, but the experiments do not include any method specifically designed to handle execution-speed variation.
Questions for authors
  1. Section 5.3 describes DAgger’s results as failure characterization rather than evidence of speed degradation. What additional conclusion does this baseline support about ParcelStow? Given its near-zero success, would the authors consider replacing or supplementing it with a method that achieves competitive nominal performance?
  2. Section 1.1.1 discusses speed-aware methods such as DemoSpeedup, SAIL, and SpeedAug. How would these methods perform on ParcelStow compared with ACT under the same speed conditions?
  3. When a learner outperforms the expert at lower speeds but underperforms at higher speeds, how should its overall performance be assessed in ParcelStow? Is any aggregation or weighting across speed factors defined, or are comparisons intended to remain specific to each speed?
4 comments
AuthorsOpenReview ↗

Response to Reviewer kDtZ

Thank you for the detailed review and the suggestion to test speed-input sensitivity.

W2. PEG Acquisition Failures and Speed-Input Sensitivity: We tested the acquisition input directly in our PEG ACT policy (Sec. 3.3, Eq. 7). During the five acquisition phases, whose scheduled durations are independent of rr, we replaced only the speed input with 1, preserving the weights, phase feature, and timing schedule. At r=1.5r=1.5, acquisition rose from 141/200 to 198/200 and success from 3/200 to 131/200. At r=1.75r=1.75, acquisition rose from 0/200 to 198/200 and success from 0/200 to 176/200. The reverse substitution supplies a complementary comparison: setting the acquisition input to 1.5 at nominal timing reduced acquisition from 197/200 to 151/200 and success from 191/200 to 7/200. These results identify the speed input as a cause of acquisition failure (Sec. 5.3, Table 2, with terminal outcomes in Appendix E, Table 15).

At r=2r=2, nominal-input acquisition recovered to 200/200, followed by 200/200 drops after the input returned to 2 during accelerated transport. The acquisition substitution isolates the input effect at acquisition. Holding trained weights fixed estimates sensitivity of the current policy, while training without the feature estimates the performance of a separately learned policy.

W1 and Q1. Baseline Coverage and the Role of DAgger: ACT matches or exceeds expert nominal success across the tasks and supplies the direct test of preservation from nominal agreement. Diffusion Policy provides an additional speed-response comparison. On UPRIGHT, it succeeded in 181/200 nominal episodes against the expert's 185/200, then 57/200 against 105/200 at r=2r=2, widening an existing deficit. On PEG, its success rose from 95/200 nominally to 120/200 at r=1.25r=1.25, then fell to 25/200 at r=1.5r=1.5. DAgger reached at most 3/100, 4/200, and 1/200 successes on PARCEL, UPRIGHT, and PEG. Its contribution is characterization of a standard baseline's failure under the benchmark protocol (Sec. 5.4).

W3 and Q2. Comparison with Speed-Aware Methods: DemoSpeedup uses entropy-guided demonstration downsampling, SAIL uses consistent inference and controller targets with adaptive speed and scheduling, and SpeedAug uses tempo-rich pretraining and reinforcement learning. Our evaluation adds a paired demonstrator reference under the same imposed timing law, so the measured quantity is the expert-learner success difference at each factor (Related Work, Sec. 1.2, and Contributions, Sec. 1.1).

Q3. Aggregation Across Speed Factors: The primary comparison is GL(r)=p^E(r)−p^L(r)G_L(r)=\widehat p_E(r)-\widehat p_L(r) together with both success curves. Its sign records which actor leads at each factor. Sec. 3.2 also defines the uniform mean over the demonstrated evaluation grid, reported in Appendix C: 13.8 percentage points for PARCEL, −2.0-2.0 for UPRIGHT, and 7.8 for PEG. This aggregate summarizes the specified finite grid. Reporting it alongside the full curves preserves the locations and directions of performance changes.

Reviewer_kDtZOpenReview ↗

Follow-up on Q2

Thank you for the additional experiments. The new PEG intervention convincingly addresses my concern. However, I still believe the paper should better establish why the expert–learner gap under execution-speed variation is a meaningful benchmark target, rather than an observed degradation of current policies. This is also why I previously suggested evaluating speed-aware methods such as DemoSpeedup, SAIL, and SpeedAug on ParcelStow. Such an evaluation would better establish ParcelStow as a benchmark for measuring progress in execution-speed robustness.

Thank you for your suggestion. We completed the PEG comparison of the DemoSpeedup state-ACT adaptation, matched ACT25 control, and expert. At the nominal execution-rate factor r=1r=1, success counts were 92, 196, and 186, respectively.

Task-success counts, with 200 episodes per actor and factor are summarized in the table below:

Execution-Rate Factor rrDemoSpeedup State-ACT AdaptationACT25 ControlExpert
0.5116198
0.7560150197
192196186
1.2512176170
1.50124163
1.75022161
200167
2.500175

We implemented entropy-guided downsampling using 10 prior-latent samples, overlapping-chunk Gaussian KDE with bandwidth 1, and the released HDBSCAN labeling and cluster-selection rule. Precision/casual target strides were 2/4, with chunk length 25. The ACT25 control used stride 1. State observations retained task phase and scheduled factor. Both policies used identical original demonstrations, training splits, normalization, optimizer settings, and checkpoint-selection rules. We froze selected checkpoints before held-out evaluation with paired initial conditions. We retained the original simulator, geometry, physics, controller gains, success predicates, temporal aggregation, and phase-duration law, with uncorrupted observations and zero action noise.

The expert-minus-learner gap for the DemoSpeedup state-ACT adaptation was 47% at r=1r=1 (paired 95% CI [39, 55]) and 81.5% at r=1.5r=1.5 ([76, 86.5]). Expert successes under the tested accelerated schedules provide an achievable reference for evaluating the remaining learning deficit at specified execution speeds in this PEG comparison.

Thank you for the additional experiments. I am now inclined to raise my score to 3. But I still maintain that the paper’s logic needs to be further clarified and strengthened, and that you need to further demonstrate that the issue identified by your benchmark is solvable rather than inevitable.

Reviewer_Ew7YOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the ParcelStow benchmark to evaluate whether imitation learners can preserve expert performance under changes in execution speed. It varies the speedup factor across three contact-rich manipulation tasks and compares the success rates of ACT, Diffusion Policy, and DAgger with those of a scripted expert. In particular, on PARCEL, the expert and ACT both achieve 100% success under nominal conditions, whereas at maximum demonstrated speedup factor r=2r=2 the expert achieves 84% and ACT 53%.

Critical strengths
  • The evaluation setup, which treats execution speed as a controlled perturbation and directly compares the performance curves of the expert and the learner, is clear.
  • It is interesting that the success rates of the expert and the learner diverge even at speeds within the demonstrated range (Fig. 1).
Critical weaknesses
  • Performance degradation of imitation policies under changes in execution speed has itself been addressed in prior work, e.g., DemoSpeedup (Guo et al., 2025) . Since the main novelty of this paper is limited to evaluating the expert–learner gap within demonstrated range, its significance feels somewhat limited.
  • The large reversals in performance ranking on UPRIGHT and PEG occur outside the demonstrated range, which makes them difficult to distinguish from ordinary OOD extrapolation failure.
  • The expert is a scripted IK-based controller with runtime correction, and its controller structure differs substantially from that of the learners. Even within the scope of the claims restricted to state-based policies in simulation (Sec. 5.5), this structural difference could be a source of the performance gap.
Questions for authors
  • What new insight do the UPRIGHT and PEG results provide beyond ordinary OOD generalization failure?
  • Is it possible to disentangle, through additional analysis, the effect of the difference in controller structure between the expert and the learners on the performance gap?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer Ew7Y

Thank you for the careful review and the questions about novelty, extrapolation, and controller structure.

W1. Novelty Relative to DemoSpeedup: Our contribution is an imposed timing-law formulation and a paired estimator of the expert learner success difference, together with diagnostics of its changes (Contributions, Sec. 1.1). DemoSpeedup accelerates execution through entropy-guided demonstration downsampling. SAIL combines consistent inference and controller targets with adaptive speed and scheduling. SpeedAug combines tempo-rich pretraining with reinforcement learning. Our comparison evaluates the learner and demonstrator under a common phase-duration law and paired initial conditions, preserving the demonstrator as a measured reference as timing changes (Related Work, Sec. 1.2). The within-demonstration result also extends beyond PARCEL: on PEG, ACT trailed the expert by 14.5 and 13.5 percentage points at r=0.5r=0.5 and 0.750.75, after leading by 4.5 points at nominal timing (Sec. 5.2).

W2 and Q1. Insights Beyond Extrapolation Failure on UPRIGHT and PEG: We tested the speed input directly in our ACT policy. During the five acquisition phases, whose scheduled durations are independent of rr, we replaced only the speed input with 1, preserving the weights, phase feature, and timing schedule. At r=1.5r=1.5, acquisition rose from 141/200 to 198/200 and success from 3/200 to 131/200. At r=1.75r=1.75, acquisition rose from 0/200 to 198/200 and success from 0/200 to 176/200. Conversely, at nominal timing, setting the acquisition input to 1.5 reduced acquisition from 197/200 to 151/200 and success from 191/200 to 7/200. Thus, changing the input alone induced failures at nominal timing, while restoring the nominal input recovered acquisition under the accelerated schedule. This identifies the speed input as a cause of acquisition failure (Sec. 3.3, Eq. 7, and Sec. 5.3, Table 2, with terminal outcomes in Appendix E, Table 15). The value 1.5 lies 5.5 standard deviations above the PEG feature's training mean.

The UPRIGHT reversal has a distinct stage profile. At r=2r=2, ACT and the expert both acquired 200/200 objects. Post-release failures accounted for 173/176 ACT failures and 91/95 expert failures. ACT acquisition collapsed to 0/200 only at r=2.5r=2.5 (Sec. 5.2). These observations localize the earlier loss to post-release behavior, while the PEG substitution identifies input sensitivity during acquisition.

W3 and Q2. Controller Structure and Handoff Analysis: Our PARCEL relative-motion handoff replaces both actors' downstream policies with a common controller, retaining each actor's physical state and actor-specific anchor. Both retained 300/300 parcels to the pre-insertion endpoint across the evaluated factors. Continuing the common controller through insertion at r=2r=2 produced 95/100 successes after expert acquisition and 64/100 after ACT acquisition (Sec. 5.5). The residual difference therefore depends on the conditions produced before handoff, including the physical state and anchor, even with a common downstream controller. The PEG substitution provides complementary evidence within a single policy architecture: changing only its acquisition input recovered acquisition with the same weights.

Thank you for the detailed rebuttal and additional experiments. I appreciate the experiment in which both the expert and ACT use the same downstream controller after acquisition, as well as the added comparison with DemoSpeedup. The former substantially reduces my concern that the observed expert–learner success-rate difference is primarily caused by the difference in controller structure.

My main remaining concern is about significance. The additional PEG experiment usefully shows that speedup-factor conditioning contributes to the failure, but the large performance reversal still occurs under extrapolation outside the demonstrated range. I therefore view this result primarily as a more detailed characterization of extrapolation failure with respect to a conditioning variable, rather than evidence of a qualitatively distinct generalization problem. I recognize that PARCEL shows an expert–learner separation within the demonstrated range and that directly comparing the learner with its expert under the same execution-timing perturbation is informative. However, I remain unconvinced that the overall findings provide sufficient new insight beyond existing understanding of out-of-distribution generalization. I therefore maintain my Weak Reject rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response Summary and Revised PDF

We thank the reviewers for their careful reviews and constructive questions. Our revised PDF includes a new speed-input experiment, a clearer statement of the scientific contributions, and expanded interpretation of the expert-learner comparisons.

New Evidence on the PEG Failure (Reviewers Ew7Y, kDtZ, and WGRC): We tested the speed input in our PEG ACT policy, holding the weights, timing schedule, physics, and phase feature fixed. During the five fixed-duration acquisition phases, we replaced only the observed speedup factor with its nominal value. At r=1.5r=1.5, acquisition increased from 141/200 to 198/200 and success from 3/200 to 131/200. At r=1.75r=1.75, acquisition increased from 0/200 to 198/200 and success from 0/200 to 176/200, compared with the expert's 163/200. The converse substitution induced failure at nominal timing: setting the acquisition input to 1.5 reduced acquisition from 197/200 to 151/200 and success from 191/200 to 7/200. We thus identify the speed input as a cause of acquisition failure, connecting the localized failure to a specific policy input (Sec. 3.3, Eq. 7, and Sec. 5.3, Table 2).

Distinct Failure Patterns Across Tasks: On UPRIGHT at r=2r=2, ACT and the expert both acquired 200/200 objects. Post-release failures accounted for 173/176 ACT failures and 91/95 expert failures, placing the loss after acquisition (Sec. 5.2). On PARCEL, our shared relative-motion controller yielded 95/100 successes after expert acquisition and 64/100 after ACT acquisition at r=2r=2. The difference persisted after replacing downstream action prediction, linking the insertion gap to the conditions produced before handoff (Sec. 5.5).

Formulation and Contributions: We now state explicitly how our formulation extends timing evaluation beyond the success of an accelerated policy. We impose the same phase-duration law on a learner and its demonstrator, estimate their signed success-rate difference on paired initial conditions, and distinguish demonstrated speeds from extrapolation. Our contributions comprise this expert-relative formulation, the three-task benchmark, empirical findings, and diagnostic speed-input substitution (Sec. 1.1). On PARCEL, equal nominal success separated by 31 percentage points within the demonstrated range. On UPRIGHT and PEG, nominal ACT advantages of 4.5 points reversed beyond their demonstrated ranges. The revised motivation explains why the expert's success curve is needed to interpret these differences.

Changes in the Revised Manuscript:

  • Abstract and Sec. 1: We strengthened the motivation for imposed execution timing and the expert reference, added a Contributions subsection (Sec. 1.1), clarified the relation to acceleration methods (Sec. 1.2), and incorporated the speed-input finding.
  • Secs. 3.2-3.3: We specified a uniformly weighted summary of the expert-learner difference over demonstrated evaluation factors and defined the speed-input substitution in Eq. 7.
  • Sec. 4.3 and new Sec. 5.3, Table 2: We specified the PEG substitution protocol and reported both directions of the experiment on the final evaluation banks.
  • Secs. 5.2, 5.4-5.6, and 6: We connected stage outcomes to the PEG input effect and UPRIGHT post-release failures, clarified the roles of ACT, Diffusion Policy, and DAgger, strengthened the handoff interpretation, and incorporated the diagnostic finding into the conclusion.
  • Appendices C-E: Appendix C includes the baseline summary (Table 8) and scalar expert-learner summaries. Appendix D.4, Table 14, supports the expert-tracking interpretation. New Appendix E, Table 15, provides the complete terminal outcomes for the speed-input substitutions.
AuthorsOpenReview ↗

Completed DemoSpeedup Comparison and Revised Manuscript

We thank the reviewers for the continued discussion. We completed a comparison of a DemoSpeedup state-ACT adaptation, matched chunk-25 ACT control, and the expert on keyed-peg insertion (PEG), with 200 episodes per actor at each of eight speedup factors (0.5, 0.75, 1, 1.25, 1.5, 1.75, 2, and 2.5).

Nominal success rates were 46% for the adaptation, 98% for the control, and 93% for the expert. At a speedup factor of 1.5, the success rates were 0%, 62%, and 81.5%, respectively.

We added a summary in Sec. 5.4 and the complete evaluation protocol and success rates in Appendix F, Table 16 and updated the manuscript PDF, abstract, and supplementary material accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.