acceptodds
Under review as a conference paper at ICLR 2027

What Data Is Needed for Offline RL in Continuous Control? A Geometric Perspective

Abstract

Learning from high-quality data forms the foundation of most modern AI applications, ranging from large language models to generalist robotic systems. However, collecting sufficient expert data to satisfy the increasing demands of robust performance is infeasible in general. In this paper, we study the question: what properties must offline data exhibit to enable near-optimal behavior in continuous control? We present a rigorous theoretical analysis of the offline RL problem, and outline the inability of common notions of coverage to discriminate between benign and intractable problem instances. To this end, we propose new notions of coverage which capture the inherent geometric nature of continuous control. Via novel analyses leveraging stability-theoretic tools and these geometric notions of coverage, we identify regimes where offline RL is tractable versus difficult, and we demonstrate how popular model-based algorithmic and data practices can achieve polynomial regret bounds under these regimes. We perform experiments on manipulation and locomotion tasks and find strong alignment with our theoretical predictions, as well as empirical evidence that our recommended practices benefit a wider range of model-based and model-free frameworks.

Primary area:
learning theory
Keywords:
reinforcement learning, model based, offline RL, continuous state-action space

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_zPaBOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that concentrability — the standard likelihood-ratio notion of data coverage in offline RL theory — is the wrong primitive for deterministic continuous control, and proves it is orthogonal to problem difficulty: there are benign instances with exponentially bad concentrability and intractable instances with perfect concentrability, and no poly(H)ϵ\mathrm{poly}(H)\epsilon guarantee can hold uniformly even when the learned model is ϵ\epsilon-accurate everywhere reachable under the true dynamics. As a replacement, the authors propose a geometric notion of coverage built on dynamical stability and Wasserstein proximity to the expert distribution rather than density ratios. Using this, they identify two tractable regimes for pessimistic model-based offline RL: when the data comes from an open-loop stable system, restricting to sufficiently long action-chunked policies forces closed-loop stability across the whole candidate model class and yields a polynomial-in-horizon regret bound; when it does not, they introduce a restricted controllability condition and show that expert action-noise injection supplies the excitation needed to identify first-order model error, again giving a polynomial bound. Experiments on robomimic Lift, Reacher-v5 and HalfCheetah-v5 with four offline RL algorithms support both prescriptions, including the predicted negative result that chunking should not help on the open-loop unstable HalfCheetah.

Critical strengths

It is easy to assert that concentrability is a poor fit for deterministic continuous dynamics; this paper does the harder thing of showing it is orthogonal to difficulty in both directions, with fully explicit constructions (Setting C.1’s bump-function system; Setting D.4’s 16tanh⁡16\tanh misranking construction).

Critical weaknesses

With R∈[0,1]\mathrm{R}\in[0,1] and JH=∑h=1HRJ_H=\sum_{h=1}^H \mathrm{R}, the trivial regret bound is HH. Theorem 4.5 gives O(H2)(sup⁡fJmodel(f;μh)+WP⋆)O(H^2)\big(\sup_f \mathrm{J_{model}}(f;\mu_h)+W^\star_{\mathbb{P}}\big), so it is informative only when WP⋆≲1/HW^\star_{\mathbb{P}}\lesssim 1/H and Jmodel≲1/H\mathrm{J_{model}}\lesssim 1/H. But WP⋆W^\star_{\mathbb{P}} in Assumption 3.1 is a fixed horizon-independent constant, and the paper’s own selling point for W2W_2 (Lem. C.13) is precisely that it is horizon-uniformly bounded — i.e. O(1)O(1), not O(1/H)O(1/H). The same applies to the (H−ℓ)2(H-\ell)^2 terms in Thm. 4.12.

Questions for authors

Since JH∈[0,H]J_H\in[0,H], Thm. 4.5’s O(H2)(sup⁡fJmodel+WP⋆)O(H^2)(\sup_f \mathrm{J_{model}}+W^\star_{\mathbb{P}}) improves on the trivial bound only when WP⋆=O(1/H)W^\star_{\mathbb{P}}=O(1/H). Lem. C.13 argues W2W_2 between stable rollouts is horizon-uniform, i.e. Θ(1)\Theta(1). Can you exhibit a concrete instance and dataset for which WP⋆=O(1/H)W^\star_{\mathbb{P}}=O(1/H) holds non-trivially, or state the intended non-vacuous regime explicitly?

3 comments

We thank you for your nuanced critique. Indeed, the upperbounds we present in the main body and the proof structures used were chosen in order to present a clean version of the theory, eliciting all the important qualities without necessarily the optimal scaling. To that end, we focus our theoretical analysis on controlling the trajectory rollouts before the reward is applied, ensuring that they don't deviate exponentially in horizon and produce the exponentially large regret present in the lowerbounds of Appendix D. The trivial bound, while true, is uninformative as it hides this interplay before saturation.

This dependence can be improved through application of Lemma C.17, which uses an interpolating distribution to ensure that the additive Wasserstein penalty scales with JmodelJ_{\text{model}}. We opted not to apply this to W2(P⋆,μ)W_2(\mathbb P^\star, \mu) in the main body, as the long statement obfuscates the main concept of the subexponential scaling.

In our rebuttal revision, will specify this point in the text surrounding where this lemma appears.

A simpler argument can be seen for a fixed interpolant family, under mild assumptions: this can reduce the scaling on the Wasserstein penalties to sub-linear in HH. We sketch the argument below.

  • Construct auxiliary joint state-action distribution νhk\nu_h^k, where νhk=μh\nu_h^k = \mu_h for all h≤kh \leq k, but we roll out the actions of π⋆\pi^\star (or π⋆+σuz\pi^\star + \sigma_u z) for all further timesteps.
  • O(H)W2(⋯ )O(H) W_2(\cdots) can then be obtained whenever Ch(νhk)≲exp⁡(h−k)\mathcal C_h(\nu_h^k) \lesssim \exp(h-k): notably, this regime covers the cases examined in Appendix C. This may be observed by trading off the multiplicative transfer of Ch(νhk)\mathcal C_h(\nu_h^k) with the Wasserstein penalty of W2(P⋆,νhk)≲ρh−kWP⋆W_2(\mathbb P^\star, \nu_h^k) \lesssim \rho^{h-k} W_{\mathbb P}^\star (obtained through the stability of the h−k−1h-k-1 final steps).
  • Optimizing over kk gives that for h−k≈log⁡Hlog⁡(1/ρ)h - k \approx \frac{\log H}{\log(1/\rho)} (using only multiplicative transfer when h≵log⁡Hh \not\gtrsim \log H), we obtain a final poly(H)\mathrm{poly}(H) scaling on JmodelJ_{\text{model}} and O(H)O(H) scaling on WP⋆W_{\mathbb P}^\star. This scaling can further be made sublinear for h−k≈(1+ϵ)log⁡Hlog⁡(1/ρ)h - k \approx \frac{(1 + \epsilon)\log H}{\log(1/\rho)}.

Thank you for your clarification. I will raise my score to 4.

Thank you for your consideration and for the productive discussion!

Reviewer_nHSFOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the conditions under which offline reinforcement learning is tractable in continuous control systems. The authors argue that conventional notions of data coverage, such as concentrability, are insufficient to characterize the difficulty of offline RL in continuous state-action spaces. They introduce a geometric perspective based on dynamical stability and establish negative results demonstrating that small model prediction errors can lead to exponentially large policy evaluation errors. Experiments on robotic manipulation and locomotion tasks are conducted using several model-based and model-free offline RL algorithms.

Critical strengths
  • Understanding what properties offline data must satisfy to support reliable policy learning is an important question, particularly in continuous control, where collecting high-quality demonstrations can be expensive.

  • The connection between dynamical stability, geometric coverage, and model evaluation errors offers a useful perspective on the limitations of conventional concentrability-based analysis.

Critical weaknesses

W1. Limited theoretical novelty compared with closely related prior work.

My primary concern is the limited conceptual novelty of the proposed theoretical framework.

The two main directions of the paper—leveraging action chunking in open-loop stable systems and using exploratory data collection in systems without guaranteed open-loop stability—have already been extensively investigated by Zhang et al. (2025).

In particular, the prior work establishes a closely related stability-theoretic framework based on EISS, Jacobian linearization, and controllability-based exploration. The present paper follows essentially the same conceptual decomposition, extending the analysis from policy estimation errors in imitation learning to dynamics-model estimation errors in offline RL.

While the authors acknowledge and build upon these results, the fundamental technical novelty of this extension is not sufficiently clarified. Although the proposed restricted controllability condition and regret analysis constitute additional contributions, it remains unclear what new theoretical challenges are addressed beyond those considered in the existing framework.

The authors should more explicitly distinguish their novel theoretical contributions from prior work and explain why the extension to pessimistic model-based offline RL represents a substantial advance rather than a relatively direct adaptation of existing stability and excitation arguments.

W2. Limited experimental evaluation.

The experimental evaluation is rather limited, covering only three environments: robomimic Lift, MuJoCo Reacher-v5, and HalfCheetah-v5.

More importantly, the experimental results are largely expected and do not reveal particularly novel empirical insights. For instance, moderate action chunking improving performance in relatively stable manipulation tasks, while longer open-loop execution degrades performance in locomotion tasks, is consistent with the intuitive trade-off between reduced replanning and the loss of feedback. Similarly, incorporating exploratory trajectories can improve data diversity and coverage, making the observed performance gains unsurprising.

Although the authors evaluate several offline RL algorithms, the experiments primarily demonstrate these intuitive trends rather than uncovering new phenomena or providing compelling evidence for the proposed theoretical contributions. Given that the paper does not propose a new algorithm, I would expect more carefully designed case studies that directly validate its theoretical claims. The absence of such experiments makes the empirical evaluation particularly weak.

W3. Insufficient empirical validation of the theoretical contributions.

The connection between the theoretical results and the experimental evaluation is rather weak.

The main theoretical guarantees are established for a specific pessimistic model-based offline RL formulation under explicit stability and controllability assumptions. However, the experiments evaluate existing offline RL algorithms without establishing whether the learned policies and dynamics models satisfy the conditions required by these guarantees.

More importantly, the experiments do not directly examine the key theoretical quantities. In particular, Theorem 4.5 relies on stability properties of the candidate dynamics models, while Theorem 4.12 introduces restricted controllability and dynamics-model Jacobian mismatch as central components of the analysis. None of these properties is directly measured or validated experimentally.

Although evaluating model-free methods may help demonstrate the broader applicability of the proposed practices, the theoretical analysis focuses specifically on pessimistic model-based offline RL. However, only two distinct model-based algorithm families, MOPO and MOBILE, are evaluated. Consequently, the empirical evaluation provides limited direct evidence for the proposed model-based theoretical guarantees.

Reference:

Zhang, Thomas T., et al. "Action chunking and exploratory data collection yield exponential improvements in behavior cloning for continuous control." arXiv preprint arXiv:2507.09061 (2025).

Questions for authors

Q1. What is the fundamental theoretical novelty relative to Zhang et al. (2025)?

Q2. How do the experiments directly support the new theoretical results?

Reference:

Zhang, Thomas T., et al. "Action chunking and exploratory data collection yield exponential improvements in behavior cloning for continuous control." arXiv preprint arXiv:2507.09061 (2025).

4 comments

We thank you for your review; as your critical weaknesses section contained an extra point concerning the intuitiveness of our empirical findings, we format our response below to the sections detailed there.

W1. The distinction between our work and [1] is not simply an extension of stability arguments to offline RL: behavior cloning operates in a favorable regime of offline RL, and learns to reproduce an expert policy given perfect expert action coverage. The general offline RL problem introduces fundamental challenges of concentrability, policy misranking, and the closed-loop control of state-space perturbations that do not arise in the setting of [1]. While our upper bounds build upon related stability tools, our analysis addresses these additional challenges through geometric coverage conditions and establishes provable guarantees for offline RL.

Beyond these positive guarantees, our main contributions are:

Appendix C, referenced in Section 2.2 and Theorem 2.4, shows that exponential restricted concentrability as used in works such as [2, 3] need not imply difficult policy evaluation. We rigorously establish a continuous control setting in which restricted concentrability on model error and Bellman error grows exponentially in horizon despite admitting benign policy evaluation. This demonstrates that prior offline RL upperbounds varying with concentrability are vacuous: a finding orthogonal to the imitation learning setting of [1]. Appendix C.1 additionally establishes a novel connection between Wasserstein distance and stability.

Theorem 2.5, studied in Appendix D, demonstrates settings where no dataset interventions can prevent the exponential policy evaluation problem. This is a novel contribution unique to the offline RL setting, contradictory to the positive noise-injection results for behavior cloning in [1]. We further establish settings where this leads to policy misranking, resulting in pessimistic and certainty-equivalent offline RL algorithms failing and selecting suboptimal policies despite arbitrarily small model error. Our analysis also establishes that global EISS need not transfer between dynamics models even under exact agreement of their values and first derivatives along the expert trajectory.

These are fundamentally different questions from those studied in [1]: rather than only characterizing error propagation in imitation learning, we identify limitations of existing offline RL analyses, establish failure modes for policy selection, and provide geometric coverage and stability conditions under which provable offline RL is nevertheless possible.

W2. We note that the fact that these empirical trends are intuitive is the motivation behind our theoretical analysis: standard offline RL guarantees fail to explain why action chunking and exploratory data collection mitigate error propagation in continuous-control settings. Our contribution is not the discovery of these interventions, but a theoretical framework that properly matches the empirical findings. This is contrast to current understanding arising from standard RL frameworks, which we demonstrate do not provide meaningful understanding or guidance for these interventions.

W3. To address these concerns, we are adding experiments measuring closed-loop Jacobian error across dataset compositions, together with controlled numerical experiments following our theoretical setting and assumptions. These experiments allow us to examine the predicted relationship between geometric excitation, model accuracy, and policy performance, and to evaluate the theoretical mechanisms in settings with known stability and controllability properties. Given these controlled experiments, we observe the trends we predict in theory, verifying that the core quantities in our analysis are correct.

Our assumptions characterize sufficient conditions for favorable learning guarantees, rather than conditions that must hold exactly for the predicted behavior to arise. HalfCheetah-v5 illustrates this distinction: despite the expert policy being able to correct and recover from perturbations to its motion, its periodic locomotion dynamics need not exhibit the trajectory-wise EISS contraction assumed in our analysis. The persistence of our predicted trends in this more general setting suggests that the underlying geometric mechanisms extend beyond the formal assumptions of our theorems, justifying that our problem characterization meaningfully balances tractability and general applicability.

In addition, we note that our experimental datasets include direct expert-policy coverage through the mg-dense dataset for Lift and our constructed datasets, as well as action-noise-injected coverage in the excitation experiments. These provide controlled settings for studying the relationship between data coverage, model error, and downstream performance.

Finally, we noticed that several of the concerns raised in your review overlap with those of another reviewer, although the overall recommendations differ. We would appreciate any clarification on which aspects of our work most strongly influenced your assessment, and which concerns you would consider most important for us to address in reconsidering your recommendation.

[1] Zhang et al. "Action chunking and exploratory data collection yield exponential improvements in behavior cloning for continuous control." ICLR, 2025.

[2] Uehara and Sun. "Pessimistic model-based offline reinforcement learning under partial coverage." ICLR, 2021.

[3] Xie et al. "Bellman-consistent pessimism for offline reinforcement learning." NeurIPS, 2021.

Thank you for the detailed response and the additional evidence, which helped address my concerns. I encourage the authors to incorporate the key points from our discussion and the new results into the final manuscript to improve clarity and strengthen the contributions. Overall, I’ll raise my rating to 3.

Thank you for the productive discussion: we will incorporate this descriptive framework relating the model class with the data coverage into our final revision, including our discussion of real-world analogs, and we will work towards incorporating our specific three-class extension to Setting D.1 in a camera ready version.

Reviewer_QKcxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the data coverage conditions required for effective offline reinforcement learning in continuous control. The authors argue that traditional concentrability-based coverage notions are insufficient to characterize problem difficulty in continuous state-action spaces. To address this, they introduce geometric coverage conditions based on Wasserstein distance, dynamical stability, and restricted controllability. Theoretical results characterize regimes where offline RL can suffer exponentially growing policy-evaluation errors and establish polynomial regret guarantees under suitable conditions. In particular, the paper studies how action chunking can improve learning in open-loop stable systems and how exploratory action perturbations can provide useful geometric coverage in unstable systems. Experiments on robotic manipulation and MuJoCo locomotion tasks support these insights across model-based and model-free offline RL algorithms.

Critical strengths
  • The paper addresses an important and fundamental question in offline RL. Understanding what properties of offline datasets enable near-optimal policies is highly relevant, particularly in continuous-control settings where collecting expert demonstrations can be expensive.

  • The geometric perspective on data coverage is interesting and well motivated. The paper identifies limitations of traditional concentrability assumptions and introduces geometric notions of coverage based on Wasserstein distance and dynamical stability. This provides a useful alternative perspective on why certain offline RL problems are tractable while others remain difficult.

  • The theoretical contributions are substantial. The paper establishes negative results showing that conventional coverage conditions may fail to distinguish benign from difficult continuous-control problems. It also develops positive results showing how action chunking and exploratory data collection can yield polynomial regret guarantees under appropriate stability and controllability assumptions.

  • The experiments provide useful empirical support for the theoretical insights. The authors evaluate several model-based and model-free offline RL algorithms, including TD3+BC, IQL, MOPO, and MOBILE. The results suggest that action chunking benefits stable manipulation tasks, while exploratory data collection improves performance in more challenging continuous-control environments.

Critical weaknesses
  • The novelty relative to closely related work should be clarified. Prior work by Zhang et al. [1] has already studied how action chunking and exploratory data collection mitigate exponential error compounding in continuous-control behavior cloning using stability-theoretic analysis. While extending these insights to offline RL is meaningful, the paper should more clearly highlight the new theoretical challenges and contributions beyond adapting these existing results to pessimistic model-based RL.

  • The theoretical guarantees rely on strong assumptions that may limit their practical applicability. These include deterministic dynamics, known rewards, realizable dynamics models, stability conditions, and restricted controllability. In particular, the geometric coverage and controllability conditions depend on the expert policy and true dynamics, which are generally unknown in offline RL. The paper does not sufficiently discuss how these conditions could be estimated or verified from practical offline datasets.

  • The empirical evaluation does not directly verify the proposed geometric coverage conditions. Although the results align qualitatively with the theoretical predictions, the experiments do not explicitly measure the Wasserstein coverage terms, controllability Gramians, or stability constants. Furthermore, the HalfCheetah experiment is discussed as a setting where the expert may not satisfy the closed-loop stability assumptions required by the analysis. This makes it difficult to determine how directly the theoretical guarantees explain the observed improvements.

  • The empirical evaluation is relatively limited compared with the scope of the theoretical claims. The main experiments focus on RoboMimic Lift, MuJoCo Reacher, and HalfCheetah. A broader evaluation covering different manipulation tasks, dynamics characteristics, and naturally collected suboptimal datasets would provide stronger evidence that the proposed geometric principles generalize across continuous-control problems.

  • The theoretical insights are not fully translated into actionable data collection guidelines. Although the paper demonstrates the benefits of injecting exploratory noise into expert behavior, it remains unclear how to determine the required noise level, proportion of exploratory data, or action-chunk length from an available offline dataset. More explicit guidance on these choices, ideally connected to the theoretical quantities, would strengthen the practical significance of the work.

[1]: Zhang, Thomas T., et al. "Action chunking and data augmentation yield exponential improvements in behavior cloning for continuous spaces." ICLR 2026.

Questions for authors
  1. Can the authors clarify the main theoretical contributions beyond Zhang et al., which already studies action chunking, exploratory data collection, and stability-based error compounding in continuous-control behavior cloning?

  2. How can the geometric coverage conditions, particularly the Wasserstein distance and restricted controllability assumptions, be estimated or verified in practice when the expert policy and true dynamics are unknown?

  3. Can the authors provide empirical diagnostics of the proposed geometric coverage and stability quantities? This would help establish a more direct connection between the theoretical analysis and the observed performance improvements.

  4. Can the authors extend the experiments to more continuous-control environments and naturally collected suboptimal datasets, rather than primarily relying on expert trajectories with synthetic action-noise injection?

  5. How should practitioners determine suitable action-chunk lengths and exploratory noise levels using the proposed framework? Can the theoretical analysis provide more concrete guidance for these choices?

5 comments

We thank you for your review: we address your questions sequentially below.

Q1. The distinction between our work and [1] is not simply an extension of stability arguments to offline RL: behavior cloning operates in a favorable regime of offline RL, and learns to reproduce an expert policy given perfect expert action coverage. The general offline RL problem introduces fundamental challenges of concentrability, policy misranking, and the closed-loop control of state-space perturbations that do not arise in the setting of [1]. While our upper bounds build upon related stability tools, our analysis addresses these additional challenges through geometric coverage conditions and establishes provable guarantees for offline RL.

Beyond these positive guarantees, our main contributions are:

Appendix C, referenced in Section 2.2 and Theorem 2.4, shows that exponential restricted concentrability as used in works such as [2, 3] need not imply difficult policy evaluation. We rigorously establish a continuous control setting in which restricted concentrability on model error and Bellman error grows exponentially in horizon despite admitting benign policy evaluation. This demonstrates that prior offline RL upperbounds varying with concentrability are vacuous: a finding orthogonal to the imitation learning setting of [1]. Appendix C.1 additionally establishes a novel connection between Wasserstein distance and stability.

Theorem 2.5, studied in Appendix D, demonstrates settings where no dataset interventions can prevent the exponential policy evaluation problem. This is a novel contribution unique to the offline RL setting, contradictory to the positive noise-injection results for behavior cloning in [1]. We further establish settings where this leads to policy misranking, resulting in pessimistic and certainty-equivalent offline RL algorithms failing and selecting suboptimal policies despite arbitrarily small model error. Our analysis also establishes that global EISS need not transfer between dynamics models even under exact agreement of their values and first derivatives along the expert trajectory.

These are fundamentally different questions from those studied in [1]: rather than only characterizing error propagation in imitation learning, we identify limitations of existing offline RL analyses, establish failure modes for policy selection, and provide geometric coverage and stability conditions under which provable offline RL is nevertheless possible.

Q2. Standard offline RL theory guarantees rely on concentrability assumptions, which depend on unknown policies and environment dynamics and are not generally verifiable from offline data. Our geometric coverage and restricted controllability conditions serve an analogous theoretical role: they characterize the geometric properties required for learning, rather than quantities assumed to be known to the algorithm. Our contribution is to establish geometric conditions that better characterize continuous-control problems.

Regarding the strength of the assumptions: this paper is a response to how ill-fit the conditions and assumptions of previous work are to the continuous control setting. To this end, we isolate the simplest, cleanest theoretically tractable setting that still captures the central difficulties of continuous-control offline RL, and we demonstrate that even in this favorable setting, prior analyses still fail to meaningfully characterize the difficulty of problem instances. Furthermore, we provide lowerbounds demonstrating that exponential failures can still persist even with these assumptions. This elicits the multifaceted difficulty of the setting, and demonstrates why our novel analysis is necessary. The assumptions are therefore deliberate abstractions that allow us to isolate these phenomena, rather than conditions that must be estimated in practice. Finally, we demonstrate that these abstractions lead to meaningful insights into method design.

Crucially, our geometric conditions provide an alternative to standard concentrability, exploiting the geometry of reachable distributions and dynamical stability to obtain non-vacuous guarantees where conventional analyses may fail. Our contribution is to establish and analyze these conditions, not to claim that they can be certified from arbitrary offline data.

Q3. To address these concerns, we are adding experiments measuring closed-loop Jacobian error across dataset compositions, together with controlled numerical experiments following our theoretical setting and assumptions. These experiments allow us to examine the predicted relationship between geometric excitation, model accuracy, and policy performance, and to evaluate the theoretical mechanisms in settings with known stability and controllability properties. Given these controlled experiments, we observe the trends we predict in theory, verifying that the core quantities in our analysis are correct.

Our assumptions characterize sufficient conditions for favorable learning guarantees, rather than conditions that must hold exactly for the predicted behavior to arise. HalfCheetah-v5 illustrates this distinction: despite the expert policy being able to correct and recover from perturbations to its motion, its periodic locomotion dynamics need not exhibit the trajectory-wise EISS contraction assumed in our analysis. The persistence of our predicted trends in this more general setting suggests that the underlying geometric mechanisms extend beyond the formal assumptions of our theorems, justifying that our problem characterization meaningfully balances tractability and general applicability.

Finally, our experimental datasets include direct expert-policy coverage through the mg-dense dataset for Lift and our constructed datasets, as well as action-noise-injected coverage in the excitation experiments. These provide controlled settings for studying the relationship between data coverage, model error, and downstream performance.

Q4. Our experiments are designed to test the specific geometric mechanisms identified by our theoretical analysis, and are intended to represent a carefully curated set of examples rather than a broad suite. Importantly though, our evaluation already includes independently collected suboptimal trajectories. Appendix G, Figure 5 (right) (to-be Figure 6 upon revision) varies the fraction of the HalfCheetah-v5 dataset drawn from the medium-v0 dataset provided by [3]. This dataset consists of trajectories generated by a partially trained TQC policy (6 million training steps) [4], rather than synthetic perturbations of expert actions. The periodic nature of the policy's locomotion, together with the exclusion of absolute horizontal position from the observed state, allows these suboptimal trajectories to overlap geometrically with expert trajectories.

For RoboMimic Lift, we use the mg-dense dataset, which [5] specifically designed to be representative of dataset compositions used in common offline RL benchmarks.

Q5. While these prescriptions were not made explicit in the main text, our theoretical results do provide concrete guidance for all three choices, which we will clarify in the revision.

For chunk length ℓ\ell, we turn to Proposition E.3 (Proposition 3.1 in [1]), which provides a sufficient lowerbound on chunk length in order to guarantee stability of the candidate induced system. This threshold is horizon-independent and scales with the loglog of system theoretic constants. We would therefore recommend choosing the shortest chunk length satisfying this stability requirement, so as to avoid unnecessary open-loop execution. This is explored empirically in Figure 1: notably, these optimal chunk lengths are small in practice.

With respect to σu\sigma_u and the fraction of action-noise injected data, we first note that in our theoretical results, the noise injected distribution is a theorem object used to relate the excitation properties of the dataset distribution, rather than an algorithmic knob.

That being said, Prop 4.11 establishes the tradeoff ∥Jacobian error∥2≲1σu2∥Model error∥2+σu2\| \text{Jacobian error} \|^2 \lesssim \frac{1}{\sigma_u^2} \| \text{Model error} \|^2 + \sigma_u^2. Optimizing over σu\sigma_u suggests σu≈∥Model error∥\sigma_u \approx \sqrt{\| \text{Model error} \|}, resulting in ∥Jacobian error∥2≲∥Model error∥\| \text{Jacobian error} \|^2 \lesssim \| \text{Model error} \|. Similarly, the bound present in Theorem 4.12 indicates an inverse relationship between σu\sigma_u and regret within an admissible range. These bounds reveal a fundamental tradeoff between expert proximity and sufficient excitation: increasing exploratory noise improves excitation but eventually incurs greater distributional mismatch and approximation error. This heuristic is supported by empirical evidence in Figure 4 (right).

For the dataset composition, our theory shows that exclusively expert data need not be optimal for offline RL. The coverage terms instead favor a mixture of expert data and exploratory data, going against the previous standards of single- or all-policy concentrability. Our experiments support this conclusion, with a 50/50 mixture providing a strong empirical heuristic.

[1] Zhang et al. "Action chunking and exploratory data collection yield exponential improvements in behavior cloning for continuous control." ICLR, 2025.

[2] Uehara and Sun. "Pessimistic model-based offline reinforcement learning under partial coverage." ICLR, 2021.

[3] Xie et al. "Bellman-consistent pessimism for offline reinforcement learning." NeurIPS, 2021.

[4] Younis et al. "Minari."

[5] Raffin et al. "Stable-baselines3: Reliable reinforcement learning implementations." JMLR, 2021.

[6] Mandlekar et al. "What matters in learning from offline human demonstrations for robotic manipulation." CoRL, 2021.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.