acceptodds
Under review as a conference paper at ICLR 2027

Revisiting Open-Loop Execution in Robotics: Toward Reactive, Higher-Performing Policies

Abstract

*Action chunking* — the practice of predicting a sequence of actions and executing a prefix open-loop — has emerged as a key enabler of recent progress in imitation learning for robotic manipulation. However, executing long open-loop prefixes reduces reactivity, limiting policies' ability to correct for errors. Further, the mechanisms underlying these performance benefits remain poorly understood: prior works cite mitigating compounding errors, absorbing inference latency, or smoothing motions, but provide limited controlled evidence or guidance for preserving reactivity. In this work, we argue that long open-loop execution primarily helps short-context policies imitate "non-Markovian demonstrations”. Across four simulation and two real-world tasks, we show that expert non-Markovianity shapes the relationship between task success and open-loop execution horizon. Further, we investigate the impact of compounding errors — the prevailing explanation for long open-loop execution in prior work — and find that while they matter, expert non-Markovianity has a much stronger impact in our experimental setting. Finally, we show that when policies are provided with a sufficiently long context in the high data regime, open-loop execution is no longer beneficial and the most reactive, closed-loop policies perform best. Overall, our findings motivate long-context, reactive policies as a more principled and performant paradigm. Website: https://open-loop-action-chunking.netlify.app/

Primary area:
applications to robotics, autonomy, planning
Keywords:
Action Chunking, Imitation Learning, Diffusion Policy, Reactive Control

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_UHHiOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper revisits why action chunking improves imitation-based manipulation policies. Predicting an action sequence and executing a long prefix open-loop has driven much of the recent progress, but a long open-loop prefix reduces reactivity, and the source of the performance gain is unclear. The authors argue that long open-loop execution mainly helps short-context policies imitate non-Markovian demonstrations. Across four simulation tasks and two real-world tasks, expert non-Markovianity shapes the relationship between success and the open-loop horizon.

Critical strengths
  • The paper studies why long open-loop execution helps imitation learning, a question that bears directly on how manipulation policies are trained and executed. Each experiment is constructed with strong motivation, including non-Markovian demonstrations, compounding error, and context length, making the argument easy to follow.

  • The experiments separate expert non-Markovianity from compounding error, and compare influence of each factor on the execution horizon. Carrying that comparison through four simulation tasks and two real-world tasks gives the claim a solid basis.

  • The empirical results are convincing, and the analysis of these results provides a helpful and practical guidance for the design of observation horizon and execution horizon.

Critical weaknesses
  • Some experimental details are missing. ACT introduced Temporal Ensembling, under which the policy predicts reactively before the full action chunk has been executed. The paper does not appear to state whether this design is used. Without that information, it is difficult to judge how much reactivity the evaluated policies actually give up.

  • The reasoning in Section 4.2 needs further explanations. It assumes that if a long execution horizon could mitigate compounding errors, mitigating compounding errors (by increasing data scale) should disproportionately improve performance at short execution horizons. The step from the first statement to that prediction is not spelled out, so readers may be confused about the expected change in the curve.

Questions for authors
  • Do the ACT and Diffusion Policy experiments use temporal ensembling? Temporal ensembling allows policies to reactively predict actions without waiting until every action in the prediction horizon has been executed. Does that weaken the paper’s motivation concerning reduced reactivity?

  • In the left plot of Figure 4, increasing the data scale does not change the shape of the curve, which agrees with the paper’s analysis. However, reducing the number of demonstrations from 1000 to 400 and then to 200 reduces state coverage and would be expected to increase compounding error. At N_demo = 200 and 400, where compounding error should be larger, lengthening the execution horizon brings performance close to that of the policies trained with 1000 demos. Does this indicate that a long execution horizon could mitigate compounding error, as stated in prior works?

  • In the center plot of Figure 6, why does increasing the context length gradually reduce the success rate? What distinguishes this setting from the other two settings in that figure?

3 comments

We thank the reviewer for their feedback and questions. In our first comment, we will discuss temporal ensembling and Figure 6. In the next one we will address concerns about Figure 4 and compounding errors section:

  1. Temporal Ensembling: Our work does not use temporal ensembling and we do not believe this weakens our motivation regarding reduced reactivity. Temporal ensembling is an inference-time trick from the ACT [1] work which allows action consistency and has been hypothesized to reduce compounding errors. However, all our experiments focus on an iterative generative control policy (particularly Diffusion Policy [2]) and not on ACT. Diffusion, flow, and other similar policies can learn multimodal action distributions and temporal ensembling can lead to mode collapse when applied to such policies, negating some of their benefits. This is probably why Diffusion Policy and other works using similar policy representations, such as LBM [3] and some models from openpi [4], do not report using temporal ensembling. Overall, we did not run experiments on this design choice because the works we are directly building on and aim to propose explanations and solutions for do not use it. That said, we agree that a reference to temporal ensembling should be included in the paper for the reader as appropriate context on how ACT, one of the two works (the other being Diffusion Policy) that introduced chunked open-loop execution, incorporates observations more frequently. We already mention this in Appendix Section A.1, and will move it to the main body’s related works and make it more explicit. Additionally, we will, in fact, add additional motivation for our study and solutions: because iterative generative policies often do not use solutions like temporal ensembling that ACT did, other solutions are needed for incorporating observations more frequently when rolling out such policies. This will directly feed into the motivation and background for our work.
  2. Figure 6 Center: The Figure 6 center plot is with N=200 non-Markovian demos. Long-context learning means a higher-dimensional input space for the policy, which means learning would be harder with limited demonstrations. In the center plot, due to limited demonstrations, the difficulty of learning at longer context lengths dominates over the benefits of reactivity, and thus performance drops. But if you see Figure 6 right, when we run the same analysis with N=1000 demos from the same expert, the benefits of short-horizon reactivity dominate and long-context short-horizon policies outperform short-context long-horizon ones. Commenting on this impact of data scale is exactly why we have two different data scales next to each other. We want to highlight that the optimal performance settings differ at scale and as robotics heads toward training models at scale, we want to promote long-context short execution horizon policies as the more performant regime. We comment on the trade-off between too long a context for the given data scale v/s reactivity in the paragraph under Table 1 right now. Regarding comparison of Figure 6 Left (200 human demos) v/s Figure 6 Center (200 non-Markovian demos): we agree that for the same number of demonstrations, we see different long-context learning regimes for different demonstration sources. Performance of long-context policies is known to be sensitive to a lot of factors, including task type and data. Thus, the amount of demonstrations needed for the pitfalls of long-context learning to not dominate seems to be different for these two experts. However, for a fixed expert distribution (Figure 6 center and right), our analysis stands. Moreover, the analysis about the shape of the curve as context length increases stands consistently, including in Figure 6 center.

[1] Zhao et al. 2023, https://arxiv.org/abs/2304.13705

[2] Chi et al. 2023, https://arxiv.org/abs/2303.04137

[3] TRI LBM Team 2025, https://arxiv.org/abs/2507.05331

[4] Physical Intelligence, https://github.com/Physical-Intelligence/openpi

We now comment on the feedback for figure 4 and compounding errors analysis:

  1. Figure 4 and Compounding Errors Analysis: We thank the reviewer for pointing out the analysis of the N=400 curve with the non-Markovian expert, which is currently missing direct commentary in the text. Overall, our analysis from this section is that compounding errors do help (as reported in prior works), but we do not see the effect as being as strong as what we see from expert Markovianity. For instance, for Figure 4 left (scripted non-Markovian), we see the optimal execution horizon as: 6 at N=100, 6 at N=200, 10 at N=400, and 8 at N=1000. If compounding errors were indeed being significantly mitigated by longer execution horizons and were the driving force behind longer execution horizons, then increasing data (akin to lowering compounding errors) should have lowered the optimal execution horizon (but here it in fact seems to raise it slightly). However, if we switch the nature of data (Figure 4 right) to a Markovian expert, the optimal execution horizons are: 3/4 at N=100, 1 at N=200, 3 at N=400, and 1 at N=1000. Thus, fixing the expert and increasing data (mitigating compounding errors) barely shifts the execution horizon to smaller values, but fixing the data and changing the expert shifts it significantly toward smaller values in all cases. This leads to our analysis that compounding errors do matter, but not as much as other factors. Our HG-DAgger experiment confirms this. That said, we will add the reviewer’s suggested comparison between the N=400 and N=1000 curve of the non-Marlovian expert at longer execution horizons. This will strengthen our claim that in our experimental setting compounding errors do have an effect as shown by limited evidence, but not as strong as that of expert Markovianity.
  2. Clarity of Section 4.2: We appreciate the feedback on this. We believe this is linked to the question about the analysis for Figure 4. Given the additional page for revision, we will add a summary of our discussion about Figure 4. We hope this will help address clarity concerns of Section 4.2. Overall, we are claiming that controlling for the factor that more strongly impacts execution horizon length should shift the shape of the success horizon curve more sharply. While the effect of compounding errors is only seen modestly, the effect of changing expert nature is seen strongly across experiments.

I appreciate the authors' detailed explanation and engagement, which has addressed my concerns regarding Temporal Ensembling and clarity of Section 4.2. I will therefore maintain my current score.

Reviewer_b6xpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits the relationship between observation context length and open-loop action execution horizon in robotic imitation learning. It hypothesizes that non-Markovian expert demonstrations make longer execution horizons beneficial when policies observe only one or two frames. The authors investigate this hypothesis through experiments across four simulated and two real-world manipulation tasks. Their results suggest that longer observation histories can enable shorter execution horizons and improve task performance when sufficient training data are available. They also show that the benefits of longer histories depend on data scale.

Critical strengths
  1. The paper revisits an important and widely adopted design choice. It examines the relationship between observation history and open-loop execution length, encouraging the robotics community to reconsider settings that are often treated as established defaults. Its analysis offers a useful perspective on why longer execution horizons benefit short-context policies and when increasing observation history may reduce this dependence.

  2. The empirical investigation is extensive and thoughtfully designed. The authors examine their hypothesis from several complementary angles, including comparisons between Markovian and non-Markovian experts, dataset-size ablations, corrective training, and observation-history variations. Experiments across four simulated and two real-world tasks strengthen the credibility of the findings. The investigation of data dependence also clarifies the conditions under which longer observation histories are beneficial.

Critical weaknesses
  1. The computational cost and actual responsiveness of longer observation histories are insufficiently evaluated. The paper advocates longer contexts and shorter execution horizons for reactive control, but longer histories and more frequent policy queries can increase computational demands. The simulation experiments abstract away inference latency. Although the real-world implementation uses feature caching and interpolation, the paper does not systematically report how context length affects inference latency, effective replanning frequency, or response time to environmental changes. Measurements on the same hardware are needed to establish whether the proposed configuration delivers a net improvement in responsiveness under practical computational constraints.

  2. The proposed information-deficit explanation would benefit from a more direct controlled test. The extensive experiments support the hypothesis that limited observation history makes non-Markovian demonstrations difficult to imitate, but they do not directly establish whether access to the missing expert state removes the benefit of longer execution horizons. A particularly informative experiment would retain the same non-Markovian demonstrations while providing a short-context policy with the scripted expert’s hidden plan or counter. If this policy no longer benefits from long execution horizons, the result would more directly substantiate the paper’s central explanation that missing information drives this dependence.

Questions for authors

See the weaknesses.

7 comments

We thank the reviewer for their feedback and questions. We have the following comments on the question regarding the computational cost of long-context policies: we agree that long-context policies can increase inference time computational demands. However, we do not believe our work proposes pushing policies toward a regime where these increased computational demands are prohibitive or difficult to address:

  1. We suggest increasing relatively recent context length to longer than just one or 2 recent frames to capture expert non-Markovianity. We are not suggesting extremely long memory through video encoders, VLM query, or language token summary as mitigations to the expert Markovianity problems. Our long-context experiments simply increase history of Diffusion Policy to 8-20 frames, and recent works in long-context learning [1, 2] have been able to successfully run experiments in simulation and hardware using context lengths similar to or longer than ours and do not report inference latency as a major challenge in their settings.
  2. We do provide hardware experiments which show the benefits of long-context, short execution horizon policies as opposed to short-context long execution horizon policies, even with a very basic setting. In our hardware experiments, we did not find it necessary to use novel tricks beyond what is common in robotics to remediate the effects of inference latency; for example, basic feature caching and linear interpolation were sufficient (both are common). We did not implement KV caching or any advanced trajectory smoothing methods to compensate for for the increased computational cost, and yet the long-context, reactive policies out-performed the short-context baselines.
  3. Our focus is on providing the community with takeaways on the interaction between the nature of data commonly seen in robotic behavior cloning and the hyperparameters (like context length and execution horizon) we use when learning our policies. Studying this interaction through a rigorous controlled experiment requires designing experiments which abstract away other possible confounding factors. For instance, we cannot make claims about the nature of data until we abstract away latency. In Section 2’s “current explanation for open-loop experiments”, we do explain this reasoning in the context of prior works. Running the majority of our experiments in simulation helps us make the claim about the nature of data statistically significantly. The hardware results then show that our simulation takeaways continue to hold when we add back the additional complications of hardware experiments, adding further merit to the presented claims and making them practically relevant.

[1] Agarwal et al. 2026, https://arxiv.org/abs/2606.16447

[2] Torne et al. 2025, https://arxiv.org/abs/2505.09561

Thank you for the clarifications. However, quantitative evidence on computational cost and actual responsiveness remains missing, and the proposed control experiment providing hidden expert information to the short-history policy remains unaddressed. These concerns remain unresolved, so I maintain my original rating.

AuthorsOpenReview ↗

More direct test of the information-deficit claim:

We appreciate the suggestion for this experiment! We trained a policy on 1000 demonstrations from our scripted non-Markovian expert in FurnitureSim one-leg, but, per the reviewer’s suggestion, additionally condition the policy on the expert's hidden state hh at every timestep: the expert's current FSM state, how long it has been in that state, and the values of its internal timers and counters. With hh as an input, the demonstrations are Markovian w.r.t. the policy’s observations.

We have included the results in Appendix K, Figure 18 (we uploaded a revised paper draft).

The results show that making the expert’s non-Markovian behavior observable makes short execution horizons optimal again, supporting that expert non-Markovianity w.r.t. the policy’s observations requires long execution horizons.

However, we do want to explain that this experiment comes with some caveats that are difficult to resolve because of the nature of imitation learning. Firstly, hh cannot simply be provided to the policy at the start of an episode. Some subsets of hh are fixed per episode (e.g., the expert's sampled waypoint offsets), but some change during the rollout (e.g. the expert’s internal counters and timers). To provide these values at test time, we run the expert's FSM alongside the policy as an oracle: at each step the oracle observes the current simulator state, updates its FSM, and passes the resulting h to the policy. Its actions are discarded.

Secondly, the oracle's FSM only advances when its transition thresholds are met. Due to the nature of imitation learning, the expert policy and the learned policy will drift at inference time [1], and it is possible that the policy completes a step of the task without satisfying the expert's exact transition condition. There is no easy way to remove expert-learner distribution shifts (or behavior cloning would be solved), so, to minimize this effect, we resort to a “hack” – we loosen the oracle's transition thresholds during evaluation so it advances even when the policy is slightly inaccurate. (Note that for this reason, we did not run the experiment with 200 scripted non-Markovian expert demonstrations – the 200-demo policy diverges from the expert distribution much more quickly).

Even with loosened FSM transition thresholds, a small percentage of rollouts continue to receive OOD hidden-state inputs from the oracle. Furthermore, there is no easy way to quantify this effect; while we can detect episodes that see OOD inputs from the oracle, we cannot easily distinguish episodes where OOD inputs caused the policy to fail from episodes where policy failure (not due to OOD inputs) lead to OOD inputs. As a result, this experiment conflates compounding errors with expert Markovianity – the additional hidden-state inputs provided by the expert make the policy more sensitive to compounding errors than before.

Because we expect lower execution horizons to be more sensitive to compounding errors (Section 4.2), we hypothesize that performance at lower execution horizons might be even higher without the OOD hidden-state inputs confounding factor. But this remains to be verified in future work.

As such, we have placed the results of this experiment in the appendix (Section K, Figure 18). Still, we believe this was an insightful experiment and supports the central claim of the paper, and we thank the reviewer for the idea.

[1] Ross et al 2011, https://arxiv.org/abs/1011.0686

Reviewer_MhKAOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies why and how long open-loop action chunking horizons improve modern imitation-learning policies. It argues the main cause is that short-context policies (1–2 frames) are trained to imitate non-Markovian human experts in data collection: pauses, latent counts aliasing, and unobserved intent create hidden-state unobserved. So a policy that replans often switches between modes or gets stuck in cycles.

Across four simulated tasks (FurnitureSim one-leg, Push-T, GearInsertion, Kitchen), Diffusion Policy trained on human demonstrations show an inverted-U curve of success versus execution horizon, while policies trained on scripted Markovian experts decline almost monotonically as action chunk horizon increases, with the best horizon at 1 step. Injecting non-Markovian behaviors into the scripted expert brings the inverted U shape plot back.

Authors also experimented with the remedies such as data scaling from 100 to 1,000 demos, three rounds of HG-DAgger, and noise injection in the appendix for one of the usual explanation, compounding errors. Overall, they raise success rates but shift the curve only modestly once there are 200 or more demonstrations.

Finally, giving the policy longer context (up to 20 frames) removes the need for long open-loop execution: the best horizon drops to 1–2 steps in simulation. On two challenging real robot manipulation tasks, these reactive policies match or beat the standard short context & long action horizon policy.

Critical strengths
  1. A well designed controlled study of a common but not well understood practice in modern imitation learning: Long open-loop action chunking execution is used in most modern imitation-learning systems, but the usual explanations for it have rarely been tested or proven directly.

The authors isolate one factor by keeping the task, architecture, and training fixed and changing only the expert who collected the data: human, scripted Markovian, and scripted with non-Markovian behaviors added back in. The same empirical result pattern (U shape plot) holds across four simulated tasks. For completeness, authors also test the competing explanation: compounding errors. The failure analysis (hovering, repeated alignment, switching between subtasks) and the appendix discussion of hidden-state aliasing and cycling explain why the curve has the shape it does. My positive recommendation of this work is because it studies a widely used common practice in depth to provide research community a testable claim that others could benefit and build on.

  1. This paper goes beyond diagnosing the problem and propose a working fix: with longer context, the policy can replan every step or two without losing performance. This holds on the simulated tasks and on two dexterous bimanual real-robot tasks requiring reactiveness. On those tasks the long-context, short-horizon policies performed competitively. Table 1 shows a complete comparison between T_o and T_exec and their tradeoffs. This gives practitioners both better understanding and concrete guidance: train with 4–8 frames of context and use shorter execution horizons. That kind of reactivity allows robot to solve tasks that are more dynamic and contact-rich.
Critical weaknesses

None.

I would recommend authors experiment and study pretrained VLA models as well, especially on more real robot tasks. It would be even more interesting to study and compare the scaling properties of longer context policies against markovian VLA policies.

Questions for authors
  1. Would it be possible to conduct the scripted non-Markovian expert experiments on more simulation tasks? Even though the results and findings on FurnitureSim one-leg is insightful, it would be stronger to verify and demonstrate this on more tasks.
2 comments

Results on VLAs: We thank the reviewer for the suggestion! We agree these would be interesting experiments, and could make the results more applicable to practitioners training generalist robot policies. For a few reasons, we believe this warrants an independent study and may be better suited as future work.

For instance, a corresponding study on VLAs should answer the questions – how do different degrees and sources of expert non-Markovianity during pretraining interact and affect the optimal execution horizon? Does non-Markovianity in the pretraining data determine the optimal execution horizon, or can task-specific post-training override it? Can language conditioning minimize the effects of expert non-Markovianity?

We believe these questions are interesting and important, but require another set of thoughtfully designed experiments, and would be better addressed in a more comprehensive study as follow up work.

Scripted Non-Markovian Experts for Other Tasks: We agree such an experiment can strengthen our work. However, given the time and resources required to run these experiments, we were not able to do so with the short turnaround time. Creating a scripted non-Markovian expert for a task requires analyzing the nature of human data, and writing and tuning scripted equivalents to inject features of non-Markovianity. Moreover, our evaluations across tasks are fairly extensive: we determine ideal training length through preliminary experiments, and evaluate 500 trials for multiple checkpoints at each execution horizon.

We did add a new experiment to Appendix K, as suggested by reviewer b6xp, which provides further evidence supporting our expert non-Markovianity claim (we show that adding the non-Markovian expert’s hidden state features to the policy’s context restores Texec=1T_{\text{exec}} = 1 -- please see Appendix K for details). While not the same as adding breadth across tasks, we hope the depth it adds to our FurnitureSim one-leg task will make the evidence supporting our claim stronger.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.