- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates whether an imitation-based policy that matches expert success at nominal speed still preserves the expert–learner gap when execution timing changes. Timing is set by a scalar speedup factor that shortens scheduled durations while geometry, physical parameters, and success criteria stay fixed. The authors introduce ParcelStow and evaluate Action Chunking with Transformers (ACT), Diffusion Policy, and DAgger by success rate, stage completion, and failure diagnostics.
Critical strengths
-
ParcelStow varies execution timing through a speedup factor while holding geometry, physical parameters, and success criteria fixed. Success rate, stage completion, and failure diagnostics are reported together, so the expert–learner gap is not reduced to a single terminal outcome.
-
Final evaluation outcomes do not select policies, checkpoints, or seeds. Experiments settings are also documented in detail. This limits selection on the reported numbers and makes the comparisons easier to reproduce.
Critical weaknesses
-
The motivation for studying execution-speed changes is not sufficiently established. While shifts in lighting or object position are familiar deployment conditions, a scalar speedup of scheduled durations is less clearly a condition that imitation-based policies face. And the paper does not explain why the expert–learner gap under this change is the right quantity to measure. The reference itself makes the comparison harder to interpret. Appendix A.4.1 uses a scripted expert, and Figure 1 shows that its success often drops sharply once the speedup factor exceeds 2. A learner that stays close to a script already failing under fast execution is not shown to preserve competent expert behavior.
-
The analysis of ACT, Diffusion Policy, and DAgger remains descriptive. The paper reports success, stage completion, and failure modes, and ParcelStow localizes many failures to particular stages. However, it does not explain why the three methods diverge as speedup factor changes, or why the localized stages fail. The study therefore reads as a report of benchmark outcomes and contributes little to imitation learning.
Questions for authors
- Given a scripted expert specified in Appendix A.4.1, what success is that expert expected to retain as the speedup factor increases? If its success falls sharply above a factor of 2 as shown in Figure 1, what does matching or approaching that expert mean?
- What accounts for the differences among ACT, Diffusion Policy, and DAgger as the speedup factor increases? For the stages where ParcelStow localizes the failures, which evidence explains the failure, and does that explanation differ across the three methods?
3 comments
Thank authors for the response. These measurements clarify how the expert's declining success relates to its pre-insertion alignment. I also appreciate the input-substitution. However, my concern about the application of execution-speed changes remains, and it is still not clear that varying the scheduled speedup is a condition that imitation-based policies commonly face. I will therefore maintain my rating.
Follow-up on W1/Q1: Literature and Applications of Execution-Timing Changes
Thank you for your comment and for recognizing the added analyses.
W1 and Q1. Applications of Execution-Timing Changes: DemoSpeedup provides a direct real-robot example: its Conveyer Fast evaluation reused the same trained checkpoints with a conveyor running twice as fast as during data collection. The robot had to acquire, scan, and place a moving bag in a basket, so the external belt speed changed the time available to execute the task (Sec. 4.2).
Duration and frequency commands also appear directly in imitation learning. Variable-speed teaching-playback [1] trained duration- or frequency-conditioned models for pick-and-place and wiping using physical playback at changed speeds. RACE [2] retimes predicted state chunks under joint torque and velocity constraints. TempoVLA [3] learns an explicitly speed-conditioned policy supporting slower and faster execution through variable-speed trajectory augmentation. SpeedTuning [4] learns a speed policy over imitation-trained action chunks, ESPADA [5] downsamples demonstrations by manipulation phase, AutoSpeed [6] learns stage-dependent speed, and SkiP [7] relabels targets to skip noncritical segments while preserving dense supervision in key segments. These are concrete physical execution-speed objectives.
Related work also studies the feedback consequences of execution timing. REMAC [8] learns delay-conditioned corrections for asynchronous execution. BID [9] selects chunks for temporal consistency and reaction, while SGAC [10] adapts action-queue replacement. FASTER [11] streams completed actions to reduce reaction latency, and AutoHorizon [12] adapts when the policy replans. These methods establish the importance of execution scheduling, delay, and feedback alongside physical retiming.
Our scalar speedup factor makes the execution-time requirement controlled and repeatable: selected phase durations become , while acquisition and settling durations remain fixed. We apply the same schedule to expert and learner, holding geometry, physical parameters, and success criteria fixed. The paired expert reference measures how much demonstrator success is preserved as this requirement changes.
On PARCEL, ACT and the expert both achieved 100/100 successes at nominal timing. At , within the demonstrated interval, the expert achieved 84/100 and ACT achieved 53/100, a 31 percentage point difference (Sec. 5.1). Nominal agreement thus coexisted with a substantial expert-relative deficit under another demonstrated timing requirement.
We expanded Related Work, Sec. 1.2, with the following 12 references, including six ICLR/NeurIPS papers. Ten are published or accepted, and two are labeled preprints.
- Variable-speed teaching-playback, Advanced Robotics 2025.
- RACE, ICLR 2026.
- TempoVLA, preprint 2026.
- SpeedTuning, ICRA 2025.
- ESPADA, IEEE RA-L 2026.
- AutoSpeed, preprint 2026.
- SkiP, NeurIPS 2026, accepted.
- REMAC, ICLR 2026.
- BID, ICLR 2025.
- SGAC, NeurIPS 2025.
- FASTER, NeurIPS 2026, accepted.
- AutoHorizon, ECCV 2026.
Reply to Reviewer WGRC
Thank you for the review and the questions about motivation, the expert reference, and failure mechanisms.
W1 and Q1. Motivation for Execution Timing and Interpretation of the Expert Reference: An execution schedule specifies the time available for each task phase. Our formulation makes this requirement explicit through selected phase durations Tk/r, with geometry, physical parameters, and success criteria fixed. Varying the schedule changes the demands on the demonstrated strategy and the learned policy. We therefore evaluate both actors under the same timing law and paired initial conditions, measuring GL(r)=pE(r)−pL(r) alongside their success curves. The reference curve records how much success the demonstrator retains as the schedule changes. On PARCEL, both actors succeeded in 100/100 nominal episodes, while at the maximum demonstrated factor r=2 the expert retained 84/100 and ACT achieved 53/100. The resulting 31 percentage point difference measures a loss of expert-relative performance while the reference remains successful in most episodes (Sec. 5.1). Contributions, Sec. 1.1, states the formulation, paired estimator, and diagnostic substitution as our contributions.
Our PARCEL tracking diagnostics relate the expert's declining success to its pre-insertion alignment. Median minimum orientation error increased from 2.2∘ at nominal timing to 7.6∘ at r=2, 8.9∘ at r=2.25, and 10.3∘ at r=2.5, crossing the 10∘ tolerance. Median arm velocity utilization remained below 0.14 through r=2.5 (Appendix D.4, Table 14). These measurements characterize the reference's tracking behavior as timing tightens.
W2 and Q2. Evidence Explaining Method Differences and Localized Failures: We tested the speed input in our ACT policy. During acquisition, whose scheduled durations are independent of r, we replaced only the speed input with 1, holding weights, phase feature, and schedule fixed. At r=1.5, acquisition rose from 141/200 to 198/200 and success from 3/200 to 131/200. Conversely, setting the acquisition input to 1.5 at nominal timing reduced acquisition from 197/200 to 151/200 and success from 191/200 to 7/200. The value 1.5 lies 5.5 standard deviations above the PEG feature's training mean. The paired substitutions identify the speed input as a cause of acquisition failure (Sec. 3.3, Eq. 7, and Sec. 5.3, Table 2, with terminal outcomes in Appendix E, Table 15).
For, Diffusion Policy, we used the same raw speed observation and acquired 197/200 pegs at r=1.5, followed by 12/200 at r=1.75 (Sec. 5.2). Its acquisition response therefore differed from ACT's under the common observation variable. DAgger's near-zero success supports failure characterization, while Diffusion Policy supplies additional success curves from its own nominal level (Sec. 5.4). The input-substitution evidence explains the acquisition sensitivity of the evaluated ACT checkpoint.
On UPRIGHT at r=2, both ACT and the expert acquired 200/200 objects. Post-release failures accounted for 173/176 ACT failures and 91/95 expert failures, localizing the loss after release (Sec. 5.2). On PARCEL, we continued both actors after acquisition with a common relative-motion controller retaining actor-specific physical states and anchors. At r=2, this controller produced 95/100 successes following expert acquisition and 64/100 following ACT acquisition. The difference persists under a common downstream controller and depends on the conditions produced before handoff (Sec. 5.5).