- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that concentrability — the standard likelihood-ratio notion of data coverage in offline RL theory — is the wrong primitive for deterministic continuous control, and proves it is orthogonal to problem difficulty: there are benign instances with exponentially bad concentrability and intractable instances with perfect concentrability, and no guarantee can hold uniformly even when the learned model is -accurate everywhere reachable under the true dynamics. As a replacement, the authors propose a geometric notion of coverage built on dynamical stability and Wasserstein proximity to the expert distribution rather than density ratios. Using this, they identify two tractable regimes for pessimistic model-based offline RL: when the data comes from an open-loop stable system, restricting to sufficiently long action-chunked policies forces closed-loop stability across the whole candidate model class and yields a polynomial-in-horizon regret bound; when it does not, they introduce a restricted controllability condition and show that expert action-noise injection supplies the excitation needed to identify first-order model error, again giving a polynomial bound. Experiments on robomimic Lift, Reacher-v5 and HalfCheetah-v5 with four offline RL algorithms support both prescriptions, including the predicted negative result that chunking should not help on the open-loop unstable HalfCheetah.
Critical strengths
It is easy to assert that concentrability is a poor fit for deterministic continuous dynamics; this paper does the harder thing of showing it is orthogonal to difficulty in both directions, with fully explicit constructions (Setting C.1’s bump-function system; Setting D.4’s misranking construction).
Critical weaknesses
With and , the trivial regret bound is . Theorem 4.5 gives , so it is informative only when and . But in Assumption 3.1 is a fixed horizon-independent constant, and the paper’s own selling point for (Lem. C.13) is precisely that it is horizon-uniformly bounded — i.e. , not . The same applies to the terms in Thm. 4.12.
Questions for authors
Since , Thm. 4.5’s improves on the trivial bound only when . Lem. C.13 argues between stable rollouts is horizon-uniform, i.e. . Can you exhibit a concrete instance and dataset for which holds non-trivially, or state the intended non-vacuous regime explicitly?
3 comments
Thank you for your clarification. I will raise my score to 4.
Thank you for your consideration and for the productive discussion!
We thank you for your nuanced critique. Indeed, the upperbounds we present in the main body and the proof structures used were chosen in order to present a clean version of the theory, eliciting all the important qualities without necessarily the optimal scaling. To that end, we focus our theoretical analysis on controlling the trajectory rollouts before the reward is applied, ensuring that they don't deviate exponentially in horizon and produce the exponentially large regret present in the lowerbounds of Appendix D. The trivial bound, while true, is uninformative as it hides this interplay before saturation.
This dependence can be improved through application of Lemma C.17, which uses an interpolating distribution to ensure that the additive Wasserstein penalty scales with Jmodel. We opted not to apply this to W2(P⋆,μ) in the main body, as the long statement obfuscates the main concept of the subexponential scaling.
In our rebuttal revision, will specify this point in the text surrounding where this lemma appears.
A simpler argument can be seen for a fixed interpolant family, under mild assumptions: this can reduce the scaling on the Wasserstein penalties to sub-linear in H. We sketch the argument below.