- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes HeFLO, a method that incorporates a flow-regularised term into offline policy learning to constrain the policy near pre-collected expert behaviour, thereby ensuring training stability at every step. The method is evaluated on seven MuJoCo environments.
Critical strengths
The paper provides thorough theoretical analysis in the LQR setting, including the stability and convergence of HeFLO, which offers good interpretability.
Critical weaknesses
Although Section 4.5 candidly summarises the limitations, these points are significant:
- Assumption 1 is not proved, yet it serves as the theoretical foundation for training stability throughout the entire process.
- The experimental results are insufficient. In Table 1, HeFLO outperforms baselines in only half of the environments, and recent relevant methods such as FQL are not compared. Moreover, no ablation study is conducted against a simple MSE behaviour cloning baseline.
Questions for authors
- Given that the authors admit in Section 4.5 that Assumption 1 remains unproven and may fail, how should the reliability of Theorem 1 be assessed in real‑world applications?
- The threshold depends on unknown system parameters. How should be selected in practice? Are there actionable guidelines?
- The paper acknowledges that a simple MSE regulariser could play a similar role. Can the authors supplement an ablation study to justify the necessity of the flow regulariser?
- As mentioned in the Related Work section, can the authors include comparisons with recent relevant methods such as FQL?
7 comments
Response to the comment 1/4 from reviewer m59u
Question 1: Given that the authors admit in Section 4.5 that Assumption 1 remains unproven and may fail, how should the reliability of Theorem 1 be assessed in real-world applications?
Response: We thank the reviewer for this insightful comment. We have reworded Appendix D.3 accordingly. The logic we intended was: TD-learning error bounds for LQR (Yang et al., 2019) hold under a near-on-policy condition. In other words, the data distribution must be close to the stationary distribution of the policy being evaluated. Figure 3 shows that this condition is approximately met along training. That makes a bounded plausible; it does not verify it, let alone uniformly over . We should have presented it as a necessary-condition check, not as validation, and the revised appendix now says exactly this.
Where the assumption sits in the literature. Assumption 1 is an inexact policy evaluation assumption of the kind that is standard in convergence analyses of policy-gradient and policy-mirror-descent methods. Much of that literature assumes exact access to or to the true gradient, e.g. the softmax policy-gradient analysis of Mei et al. (2020), the exact policy-mirror-descent rates of Xiao (2022) and Lan (2023), and the optimal rate of Johnson et al. (2023), and the inexact variants (Xiao, 2022, Assumption 1), (Cen et al., 2022, Eq. 26), (Zhan et al., 2023, Assumption 2), (Alfano et al., 2023, Assumption A1) assume (in sup-norm or a weighted norm) and show that the exact-case rate is preserved up to a neighbourhood proportional to (or for the variant). Theorem 1 has the same structure, with the exact-critic case recovered at . What is specific to our setting is that the bound must hold uniformly over and for the aggregate critic of a mixture of configurations.
Regarding assessment in practice, Assumption 1 can be monitored from data alone, without knowledge of the dynamics. Holding out a subset of (stratified by configuration), one can compute at each actor iterate the held-out Bellman residual
which requires only the current actor, critic and transitions. It vanishes iff the critic satisfies the Bellman equation on the data support, and the value error is bounded by the Bellman residual up to a factor depending on the closed-loop mixing rate, so a small, non-increasing residual along training is direct evidence for bounded on the iterates; evaluating it for actors perturbed within a ball around addresses uniformity over .
Where it is in the revised manuscript (blue text): Appendix D.3 retitled and rewritten: opening paragraph with the literature positioning above and the necessary-condition reading of Figure 3. Section 4.5(ii) states what a self-contained proof would need, how enters the bound, and points to the appendix. The eight cited papers were added to the bibliography.
Response to the comment 2/4 from reviewer m59u
Question 2: The threshold depends on unknown system parameters. How should be selected in practice? Are there actionable guidelines?
Response: We thank the reviewer for this insightful question. We agree that the thresholds depend on the unknown system family.Theorem 1 provides the shape of the dependency on , and this shape makes the selection actionable. (a) First of all, should be above the threshold. Below the threshold no guarantee holds and training can diverge; above it, stability and convergence hold for every . (b)Second, over-regularising is cheap. The deployment price is bounded uniformly in by a quantity depending only on expert suboptimality and flow fidelity, and the contraction factor at step size , is which improves with and saturates at . Under uncertainty, should therefore be chosen on the large side. This gives the following guidelines.
-
Normalise. Express in units of . This is similar to the normalisation of the BC weight by the mean in TD3+BC (Fujimoto & Gu, 2021).
-
Sweep and detect the threshold from data. Train on a coarse logarithmic grid of and monitor, on held-out transitions, the regulariser , which is the observable surrogate of the confinement guaranteed by Theorem 1(i), together with the held-out Bellman residual of our response to the first question. An that drifts upward during training, or a growing Bellman residual, indicates that is below the threshold. Select the smallest in the stable regime or, any larger value.
We thank the reviewer for the thoughtful and constructive feedback. We have carefully addressed the concerns raised, and your comments have helped us make the manuscript clearer and stronger. We hope the revised manuscript better conveys our primary contribution: stability and convergence guarantees for policy-gradient offline RL under fixed, un-resampleable domain randomisation, which, to our knowledge, are the first of their kind. The empirical section complements this analysis by examining confinement and variability across seeds and configurations in nonlinear systems, helping assess the practical relevance of the theoretical insights.