acceptodds
Under review as a conference paper at ICLR 2027

HeFLO: Flow-Regularized Offline Policy Learning under System Heterogeneity and Its Convergence Analysis

Abstract

Offline reinforcement learning (RL) learns control policies from pre-collected data without environment interaction, which is essential where online exploration is costly or unsafe. In practice such data are gathered from heterogeneous systems that share a dynamical structure but may differ in physical parameters. Additionally, the experts may themselves be specialized to particular configurations. We propose HeFLO: Heterogeneous FLow-regularized Offline policy learning, which minimizes an infinite-horizon average cost over a bounded range of parametric uncertainty with flow-based behaviour-cloning regularization. The objective is a domain randomization objective, and HeFLO is its offline, model-free, demonstration-regularized counterpart. For heterogeneous linear quadratic regulators we prove that a regularizer above an explicit and initialization-dependent threshold confines every iterate to the stability radius of the demonstrated behaviour, so the closed loop is stable for every configuration throughout training. Additionally, the iterates converge geometrically to a neighbourhood whose radius depends on the demonstration set. Empirically, HeFLO reaches near-expert performance on average over the configuration family across heterogeneous MuJoCo environments.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Offline Reinforcement Learning, Convergence Analysis, LQR Analysis, Flow matching, Domain Randomization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_m59uOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HeFLO, a method that incorporates a flow-regularised term into offline policy learning to constrain the policy near pre-collected expert behaviour, thereby ensuring training stability at every step. The method is evaluated on seven MuJoCo environments.

Critical strengths

The paper provides thorough theoretical analysis in the LQR setting, including the stability and convergence of HeFLO, which offers good interpretability.

Critical weaknesses

Although Section 4.5 candidly summarises the limitations, these points are significant:

  1. Assumption 1 is not proved, yet it serves as the theoretical foundation for training stability throughout the entire process.
  2. The experimental results are insufficient. In Table 1, HeFLO outperforms baselines in only half of the environments, and recent relevant methods such as FQL are not compared. Moreover, no ablation study is conducted against a simple MSE behaviour cloning baseline.
Questions for authors
  1. Given that the authors admit in Section 4.5 that Assumption 1 remains unproven and may fail, how should the reliability of Theorem 1 be assessed in real‑world applications?
  2. The α\alpha threshold depends on unknown system parameters. How should α\alpha be selected in practice? Are there actionable guidelines?
  3. The paper acknowledges that a simple MSE regulariser could play a similar role. Can the authors supplement an ablation study to justify the necessity of the flow regulariser?
  4. As mentioned in the Related Work section, can the authors include comparisons with recent relevant methods such as FQL?
7 comments

We thank the reviewer for the thoughtful and constructive feedback. We have carefully addressed the concerns raised, and your comments have helped us make the manuscript clearer and stronger. We hope the revised manuscript better conveys our primary contribution: stability and convergence guarantees for policy-gradient offline RL under fixed, un-resampleable domain randomisation, which, to our knowledge, are the first of their kind. The empirical section complements this analysis by examining confinement and variability across seeds and configurations in nonlinear systems, helping assess the practical relevance of the theoretical insights.

AuthorsOpenReview ↗

Response to the comment 1/4 from reviewer m59u

Question 1: Given that the authors admit in Section 4.5 that Assumption 1 remains unproven and may fail, how should the reliability of Theorem 1 be assessed in real-world applications?

Response: We thank the reviewer for this insightful comment. We have reworded Appendix D.3 accordingly. The logic we intended was: TD-learning error bounds for LQR (Yang et al., 2019) hold under a near-on-policy condition. In other words, the data distribution must be close to the stationary distribution of the policy being evaluated. Figure 3 shows that this condition is approximately met along training. That makes a bounded ϵQ\epsilon_Q plausible; it does not verify it, let alone uniformly over B\mathcal B. We should have presented it as a necessary-condition check, not as validation, and the revised appendix now says exactly this.

Where the assumption sits in the literature. Assumption 1 is an inexact policy evaluation assumption of the kind that is standard in convergence analyses of policy-gradient and policy-mirror-descent methods. Much of that literature assumes exact access to QπQ^\pi or to the true gradient, e.g. the softmax policy-gradient analysis of Mei et al. (2020), the exact policy-mirror-descent rates of Xiao (2022) and Lan (2023), and the optimal rate of Johnson et al. (2023), and the inexact variants (Xiao, 2022, Assumption 1), (Cen et al., 2022, Eq. 26), (Zhan et al., 2023, Assumption 2), (Alfano et al., 2023, Assumption A1) assume ∥Q^−Qπ∥≤ϵ\lVert\hat Q-Q^\pi\rVert\le\epsilon (in sup-norm or a weighted L2L^2 norm) and show that the exact-case rate is preserved up to a neighbourhood proportional to ϵ\epsilon (or ϵ\sqrt\epsilon for the L2L^2 variant). Theorem 1 has the same structure, with the exact-critic case recovered at ϵQ=0\epsilon_Q=0. What is specific to our setting is that the bound must hold uniformly over B\mathcal B and for the aggregate critic of a mixture of configurations.

Regarding assessment in practice, Assumption 1 can be monitored from data alone, without knowledge of the dynamics. Holding out a subset of D\mathcal{D} (stratified by configuration), one can compute at each actor iterate KkK_k the held-out Bellman residual

ϵ^BR(Kk)=(1Nho∑[Q^Kk(x,u)−c(x,u)+J^Kk−Q^Kk(x′,πKk(x′,η))]2)1/2,\hat\epsilon_{\mathrm{BR}}(K_k)=\Bigl(\tfrac{1}{N_{\mathrm{ho}}}\textstyle\sum\bigl[\hat Q_{K_k}(x,u)-c(x,u)+\hat J_{K_k}-\hat Q_{K_k}(x',\pi_{K_k}(x',\eta))\bigr]^2\Bigr)^{1/2},

which requires only the current actor, critic and transitions. It vanishes iff the critic satisfies the Bellman equation on the data support, and the value error is bounded by the Bellman residual up to a factor depending on the closed-loop mixing rate, so a small, non-increasing residual along training is direct evidence for bounded ϵQ\epsilon_Q on the iterates; evaluating it for actors perturbed within a ball around KkK_k addresses uniformity over B\mathcal{B}.

Where it is in the revised manuscript (blue text): Appendix D.3 retitled and rewritten: opening paragraph with the literature positioning above and the necessary-condition reading of Figure 3. Section 4.5(ii) states what a self-contained proof would need, how ϵQ\epsilon_Q enters the bound, and points to the appendix. The eight cited papers were added to the bibliography.

AuthorsOpenReview ↗

Response to the comment 2/4 from reviewer m59u

Question 2: The threshold depends on unknown system parameters. How should be selected in practice? Are there actionable guidelines?

Response: We thank the reviewer for this insightful question. We agree that the α\alpha thresholds depend on the unknown system family.Theorem 1 provides the shape of the dependency on α\alpha, and this shape makes the selection actionable. (a) First of all, α\alpha should be above the threshold. Below the threshold no guarantee holds and training can diverge; above it, stability and convergence hold for every α\alpha. (b)Second, over-regularising is cheap. The deployment price J(K⋆)−J(KJ⋆)J(K^\star)-J(K^\star_J) is bounded uniformly in α\alpha by a quantity depending only on expert suboptimality and flow fidelity, and the contraction factor at step size γ=1/(2L)\gamma=1/(2L), is 1−ασmin⁡−LJ4(LJ+ασmax⁡)1-\tfrac{\alpha\sigma_{\min}-L_J}{4(L_J+\alpha\sigma_{\max})} which improves with α\alpha and saturates at 1−σmin⁡/(4σmax⁡)1-\sigma_{\min}/(4\sigma_{\max}). Under uncertainty, α\alpha should therefore be chosen on the large side. This gives the following guidelines.

  1. Normalise. Express α\alpha in units of J^K0/σmin⁡\hat J_{K_0}/\sigma_{\min}. This is similar to the normalisation of the BC weight by the mean ∣Q∣|Q| in TD3+BC (Fujimoto & Gu, 2021).

  2. Sweep and detect the threshold from data. Train on a coarse logarithmic grid of α\alpha and monitor, on held-out transitions, the regulariser R(Kk)=E∥πKk(x,η)−μθb(x,η)∥22\mathcal{R}(K_k)=\mathbb{E}\Vert\pi_{K_k}(x,\eta)-\mu_{\theta_b}(x,\eta)\Vert_2^2, which is the observable surrogate of the confinement ∥Kk−Kb∥F≤r(α)\Vert K_k-K_b\Vert_F\le r(\alpha) guaranteed by Theorem 1(i), together with the held-out Bellman residual of our response to the first question. An R(Kk)\mathcal{R}(K_k) that drifts upward during training, or a growing Bellman residual, indicates that α\alpha is below the threshold. Select the smallest α\alpha in the stable regime or, any larger value.

Reviewer_NSmNOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HeFLO, an offline actor-critic method for learning a single policy from demonstrations collected across heterogeneous system configurations. A flow model is first trained to represent the demonstrated behavior and is then used as a behavioral regularizer during policy optimization.

The paper studies an infinite-horizon average-cost objective and provides theoretical analysis for heterogeneous LQR systems. Under its assumptions, sufficiently strong behavioral regularization keeps policy iterates within a uniformly stabilizing region and yields geometric convergence to an error neighborhood whose size reflects distribution shift, heterogeneity, critic error, and finite configuration sampling.

Empirically, HeFLO is evaluated on heterogeneous MuJoCo environments and achieves performance competitive with IQL, CQL, and TD3+BC, while the authors also investigate the effects of behavioral regularization and reward relabeling.

Critical strengths

The paper studies a relevant and practically motivated setting: offline RL from demonstrations collected across heterogeneous system configurations, where further environment interaction is unavailable. The distinction between standard domain randomization and its fixed-data offline counterpart is clearly motivated.

The theoretical analysis is substantial. For heterogeneous LQR, the paper provides explicit stability and convergence guarantees and characterizes how distribution shift, heterogeneity, critic error, and finite configuration sampling contribute to the convergence error floor.

The stability interpretation of behavioral regularization is particularly interesting. Rather than merely viewing BC regularization as a way of mitigating offline distribution shift, the analysis shows how sufficiently strong regularization around demonstrated behavior can keep all policy iterates within a uniformly stabilizing region.

The paper is transparent about its assumptions and limitations, including the restriction of the theory to LQR, the unproven critic-accuracy assumption, and the fact that flow matching itself is not essential to the theoretical results.

Critical weaknesses

The role of the flow model is insufficiently motivated. The theoretical guarantees do not rely on flow matching specifically, but only on an accurate behavioral anchor. Moreover, the deployed stochastic actor is itself noise-conditioned. It is therefore unclear what flow matching contributes beyond simpler behavioral regularization. This is particularly relevant because the pathwise MSE regularizer may penalize an actor for selecting one valid mode when the flow maps the same noise realization to another. A simple BC-vs-flow ablation seems necessary.

The empirical comparisons are limited. HeFLO is evaluated only against generic offline RL baselines (IQL, CQL, TD3+BC), while relevant alternatives include expressive generative policies such as Diffusion-QL, robust offline RL, and offline meta-RL/context-adaptation methods designed for heterogeneous or latent dynamics.

The experiments therefore do not convincingly validate the central motivation. HeFLO is competitive but does not consistently outperform the simple baselines. Consequently, the results demonstrate that the method works, but provide limited evidence that flow regularization itself is beneficial or that HeFLO is particularly effective at handling heterogeneous dynamics.

Questions for authors

see weaknesses

4 comments
AuthorsOpenReview ↗

Response to the comment 1/3 from reviewer NSmN

Question 1: The role of the flow model is insufficiently motivated. The theoretical guarantees do not rely on flow matching specifically, but only on an accurate behavioral anchor. Moreover, the deployed stochastic actor is itself noise-conditioned. It is therefore unclear what flow matching contributes beyond simpler behavioral regularization. This is particularly relevant because the pathwise MSE regularizer may penalize an actor for selecting one valid mode when the flow maps the same noise realization to another. A simple BC-vs-flow ablation seems necessary.

Response: It is a very interesting comment. Section 4.5(vi) flagged this, and the revised manuscript adds the ablation Table (Table 6 in the Appendix D.5 of the revised manuscript). The guarantees of Section 4 hold for any behavioural anchor of comparable fidelity (Section 4.5(vi)), so the empirical question is whether the flow anchor matters on heterogeneous data. We keep the data, critic, actor network, α\alpha and seeds of HeFLO fixed and exchange only the regulariser R\mathcal{R} in (3.4): (a) the pathwise flow loss of HeFLO; (b) a mean-squared-error anchor towards a deterministic MSE behaviour-cloning anchor, E∥πζ(x,η)−ab∥2\mathbb{E}\lVert\pi_\zeta(x,\eta)-a^b\rVert^2 with aba^b is the action taken from D\mathcal D at the corresponding state inside HeFLO's pipeline.

Table : Anchor ablation (reproduction of Table 6): normalised score, mean ±\pm std; only the regulariser differs.

EnvironmentHeFLO (flow anchor)MSE-BC anchor
InvertedPendulum-v5100 ±\pm 0.0100 ±\pm 0.0
InvertedDoublePendulum-v5102.63 ±\pm 0.097.61 ±\pm 5.05
Reacher-v5100.17 ±\pm 4.7398.11 ±\pm 4.90
Pusher-v5104.71 ±\pm 3.81100.22 ±\pm 3.68
Swimmer-v5103.89 ±\pm 0.23104.15 ±\pm 0.71
Hopper-v5119.05 ±\pm 0.07117.57 ±\pm 0.04
Walker2d-v5109.54 ±\pm 2.66108.41 ±\pm 1.91

Result. Table 6 shows modest improvements with the flow anchor over MSE-BC, while both variants achieve comparable performance. These results align with the theory's emphasis on the presence and strength of a behavioural anchor, and the revised manuscript says so (Section 5, Section 4.5(vi) and the Conclusion).

What, then, does the flow model contribute? (i) Theoretically, our analysis does not rely on anything specific to flow matching. Theorem 1 needs an anchor of fidelity ϵb\epsilon_b and a regulariser that is quadratic in the actor parameters, and both anchors provide this. Section 4.5(vi) said so already, and the ablation is consistent with it. (ii) Empirically, on the two environments tested, no measurable gain: the stability and low variance of HeFLO in Table 1 are attributable to the presence and strength of a behavioural anchor, not to the anchor's parametrisation. (iii) Where it could matter. Experts trained on different δj\delta_j take different actions at a shared state xx, so the pooled behaviour distribution pD(u∣x)p_{\mathcal D}(u\mid x) is a mixture over configurations. Therefore, a regression anchor clones its conditional mean, which need not be a good action for any single configuration, whereas the flow represents the mixture and the pathwise regulariser pulls the actor toward a sample from it (now explained in Section 5). The agreement of the two anchors indicates that, for the ranges Sδ\mathcal S_\delta and the m≤5m\le5 configurations used here, the conditional mean remains a good action. The flow anchor is thus a more expressive default as it performs on par here and keeps the method applicable when the demonstrations are genuinely multimodal, rather than a necessity, and the revised text no longer suggests otherwise.

Where it is in the revised manuscript (blue text): Table 6 and the paragraph “Anchor ablation” in Appendix D.5; paragraph “Anchor ablation and additional baselines” with the mixture argument and the outcome in Section 5; Section 4.5(vi) ; the paragraph “Why pathwise coupling, and its cost” in Section 3.2 and the Conclusion were reworded accordingly.

AuthorsOpenReview ↗

Response to the comment 2/3 from reviewer NSmN

Question 2: The empirical comparisons are limited. HeFLO is evaluated only against generic offline RL baselines (IQL, CQL, TD3+BC), while relevant alternatives include expressive generative policies such as Diffusion-QL, robust offline RL, and offline meta-RL/context-adaptation methods designed for heterogeneous or latent dynamics.

Response: We appreciate the reviewer's suggestion to compare our method with existing offline RL methods that include expressive generative policies . Keeping that suggestion in mind, we compare our method against FQL (Park et al., 2025) and Diffusion-QL (Wang et al., 2023), which are expressive generative-policy methods that operate under exactly our assumptions, see the following table. We also report a robust offline baseline, Robust Fitted Q-Iteration (RFQI) (Panaganti et al., 2022), with the caveat of Appendix B. RFQI optimises a worst-case objective over an uncertainty set whereas our objective is average-case over ρδρ_δ, so the comparison measures the price of the worst-case objective on our data rather than ranking two methods for the same task. Finally we present the comparison results in the following table. Given the limited time, we can only perform simulation for some of the algorithms provided by the reviewers.

Table: Additional comparison results (reproduction of Table 7): normalised score, mean ±\pm std;

EnvironmentFQLRFQIDiffusion-QLHeFLO
InvertedPendulum-v5100.00 ± 0.00100.00 ± 0.00100.00 ± 0.00100.00 ± 0.00
Pusher-v5101.27 ± 5.67101.07 ± 5.64103.78 ± 5.23104.70 ± 3.81
InvertedDoublePendulum-v5102.61 ± 0.39100.64 ± 13.84102.61 ± 0.40102.63 ± 0.00
Reacher-v5105.06 ± 4.62108.64 ± 4.65106.58 ± 6.07100.17 ± 4.73
Swimmer-v5101.91 ± 10.90105.17 ± 10.86102.51 ± 9.50103.89 ± 0.23
Hopper-v5107.99 ± 19.43115.80 ± 1.28116.82 ± 0.71119.05 ± 0.07

Where it is in the revised manuscript (blue text): Table 7 and the paragraph “Additional baselines.” in Appendix D.5

AuthorsOpenReview ↗

Response to the comment 3/3 from reviewer NSmN

Question 3: The experiments therefore do not convincingly validate the central motivation. HeFLO is competitive but does not consistently outperform the simple baselines. Consequently, the results demonstrate that the method works, but provide limited evidence that flow regularization itself is beneficial or that HeFLO is particularly effective at handling heterogeneous dynamics.

Response: We thank the reviewer for distinguishing competitive performance from evidence for the specific benefits of flow regularization and handling heterogeneity. We agree these are separate claims and that competitive results alone do not establish either advantage. We address both below using our theory and results.

New comparisons: The revised manuscript adds an ablation table (Table 6, Appendix D.5). Section 4 guarantees hold for any behavioural anchor of comparable fidelity (Section 4.5(vi)), so the empirical question is whether the flow anchor matters on heterogeneous data.

Table 6 shows modest improvements with the flow anchor over MSE-BC, with comparable performance overall. This aligns with our theory's emphasis on the presence and strength of a behavioural anchor (Section 5, Section 4.5(vi), Conclusion).

What, then, does the flow model contribute? (i) Theoretically, our analysis does not rely on flow matching specifically. Theorem 1 needs an anchor of fidelity ϵb\epsilon_b and a regulariser quadratic in the actor parameters, which both anchors provide (Section 4.5(vi)). (ii) Empirically, on the two environments tested, there is no measurable gain: HeFLO's stability and low variance in Table 1 stem from the presence and strength of a behavioural anchor, not its parametrisation. (iii) Where it could matter. Experts trained on different δj\delta_j take different actions at a shared state xx, making the pooled behaviour distribution pD(u∣x)p_{\mathcal{D}}(u | x) a mixture over configurations. A regression anchor clones its conditional mean, which need not be a good action for any single configuration, whereas the flow represents the mixture and the pathwise regulariser pulls the actor toward a sample from it (Section 5). Their agreement indicates that for the ranges Sδ\mathcal{S}_\delta and m≤5m \le 5 configurations here, the conditional mean remains a good action. The flow anchor is thus a more expressive default that performs on par here and keeps the method applicable when demonstrations are genuinely multimodal, rather than a necessity, and the revised text no longer suggests otherwise.

Does HeFLO handle heterogeneous dynamics? “Beats every baseline on every row” is the wrong test, because the theory does not predict higher scores. It predicts that for α\alpha above the threshold (4.9), the learned policy stabilises every configuration in Sδ\mathcal{S}_\delta, giving almost no failures and small seed-to-seed spread (Remark 2), and that the objective settles at a floor (4.11) set by the heterogeneity itself (Appendix F.8). Existing results support each part:

  • Test configurations are drawn independently of training ones, so every score in Table 1 evaluates unseen configurations (Section 5).
  • HeFLO is the only method scoring ≥100\ge 100 on every family, with the lowest seed-to-seed std in 6 of 7 (mean 1.6 vs. 3.1 / 9.8 / 9.8 for TD3+BC / CQL / IQL; Table 1). Five of seven rows are at the expert ceiling, so they cannot rank methods; they only show baseline failures (IQL on InvertedPendulum, 57.1±14.657.1 \pm 14.6; CQL on InvertedDoublePendulum, 78.7±34.178.7 \pm 34.1). On Hopper and Walker2d, HeFLO achieves the best score with 5–9×5\text{--}9\times smaller std.
  • The α\alpha-sweeps (Appendix D.4) show the predicted threshold: convergence above an environment-dependent value, failure below it.
  • Learning curves (Appendix D.2) reach and hold a plateau. In the LQR case (Appendix D.1), where theory applies exactly, HeFLO reaches 99.41±1.7499.41 \pm 1.74 of the expert.

Why TD3+BC is close. TD3+BC is also an anchored method (fixed-weight MSE anchor), so our theory predicts it should be stable, and it is. CQL and IQL fail because they lack an explicit actor-side anchor. HeFLO adds the stability certificate (threshold and floor), a policy gradient without differentiating through the critic, and lower variance.

What we do not claim. Not that HeFLO is the only method able to handle heterogeneity, or that the flow anchor is strictly needed. The revised abstract and Conclusion claim a stability-certified method matching or exceeding strong baselines on average with markedly lower variance. Directly testing the floor (e.g., widening Sδ\mathcal{S}_\delta or varying mm) requires new datasets and is left as future work (Appendix D.5).

Where it is in the revised manuscript (blue text): Table 6 and “Anchor ablation” in Appendix D.5; Section 4.5(vi); “Why pathwise coupling, and its cost” in Section 3.2; and the Conclusion.

We thank the reviewer for the thoughtful and constructive feedback. We have carefully addressed the concerns raised, and your comments have helped us make the manuscript clearer and stronger. We hope the revised manuscript better conveys our primary contribution: stability and convergence guarantees for policy-gradient offline RL under fixed, un-resampleable domain randomisation, which, to our knowledge, are the first of their kind. The empirical section complements this analysis by examining confinement and variability across seeds and configurations in nonlinear systems, helping assess the practical relevance of the theoretical insights.

Reviewer_ujcVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a flow-regularized offline policy-learning method for heterogeneous dynamical systems. A flow-based behavior model is used as a regularizer in offline actor-critic learning. For heterogeneous LQR systems, the authors provide stability and convergence analysis.

Critical strengths
  1. The paper clearly motivates the problem that offline data may be collected from multiple systems with different physical parameters, and that it is useful to learn a single policy that can operate across such heterogeneous configurations.

  2. The paper provides a theoretical analysis for the heterogeneous LQR setting.

  3. The paper provides empirical evaluations on several heterogeneous MuJoCo environments.

Critical weaknesses
  1. The theoretical scope is substantially narrower than the general problem formulation. The paper starts from a general heterogeneous MDP formulation, while the theoretical guarantees are developed only for LQR systems. This creates some disconnect between the general presentation of the method and the scope of the analysis. It would be helpful either to frame the problem more explicitly around the LQR setting from the beginning, or to discuss how the main arguments could potentially extend to more general dynamical systems. In particular, the paper could clarify what properties of the dynamics and policy class are essential for establishing stability and convergence beyond LQR.

  2. The practical justification of Assumption 1 is unclear. Assumption 1 requires the critic approximation error to remain uniformly bounded over the relevant policy region. However, the empirical observation that the learned-policy state distribution approaches the expert state distribution does not directly imply that the critic approximation error is small, especially uniformly over the policy region. Since this assumption plays an important role in the convergence analysis, it would be helpful to clarify under what conditions it can be verified or justified in practice.

  3. The additional benefit of flow matching is not sufficiently demonstrated. The paper states that the theoretical guarantees do not rely on properties specific to flow matching and would also hold for an MSE-based behavioral regularizer with comparable fidelity. However, there is no experimental comparison between flow regularization and standard behavior cloning. Therefore, it is difficult to determine whether the observed performance improvement comes from the flow model itself or simply from regularizing the learned policy toward the demonstrations. Such an ablation seems particularly important given that flow regularization is emphasized as a main component of the proposed method.

Questions for authors
  1. The paper begins with a general heterogeneous MDP formulation, while the theoretical results are limited to LQR systems. Are there identifiable conditions on more general nonlinear dynamics, costs, or policy classes under which similar stability or convergence arguments could be established?

  2. How can Assumption 1 be verified or justified in practice? In particular, why does convergence of the learned-policy state distribution toward the expert distribution provide evidence for a uniformly bounded critic approximation error? Could the authors directly evaluate the critic error in the experiments?

  3. What additional benefit does the flow model provide compared with a standard behavior-cloning or MSE regularizer? Since the theoretical guarantees do not depend specifically on flow matching, could the authors provide an ablation comparing flow regularization with a simpler behavioral regularizer?

4 comments
AuthorsOpenReview ↗

Response to the comment 1/3 from reviewer ujcV

Question 1: The paper begins with a general heterogeneous MDP formulation, while the theoretical results are limited to LQR systems. Are there identifiable conditions on more general nonlinear dynamics, costs, or policy classes under which similar stability or convergence arguments could be established?

Response: We appreciate the reviewer's observation and agree that the distinction between the general problem formulation and the specific LQR analysis should be made explicit. In the revised manuscript, we clarify that the general heterogeneous-MDP formulation (Section 3) is the setting and algorithm, while the LQR family is the analysed instance. We have modified the abstract and introduction to state that the guarantees are for heterogeneous LQR with linear state-feedback policies, and that the MuJoCo results are empirical.

On the substantive question, here is what the proof of Theorem 1 actually uses, separated into properties that are generic and ones that are LQR-specific. We have added a new paragraph of Section 4.4, with this discussion explicitly highlighting which aspects of the argument could plausibly extend beyond LQR and which are inherently tied to the linear-quadratic structure.

Generic (used, and plausibly available beyond LQR):

  1. A stability margin around the demonstrated behaviour. The anchor KbK_b lies in the uniformly stabilising set K\mathcal K with radius rb>0r_b > 0 (Eq. (4.4)). Part (i) of Theorem 1 is pure confinement: a large enough α\alpha keeps R(Kk)\mathcal R(K_k) below D′D', hence KkK_k within r′<rbr' < r_b of KbK_b. The nonlinear analogue is a robustness margin of the expert closed loop (e.g. an input-to-state-stability or contraction margin: if the demonstrated policy renders every configuration exponentially stable/contracting with margin β\beta, then every policy within a β\beta-dependent neighbourhood does too, for Lipschitz ff). Nothing about linearity is needed for this step beyond the existence of such a margin.

  2. Smoothness of JJ on the stabilising set. We use LJ=sup⁡B∣∇2J∣L_J=\sup_{\mathcal B}\\|\nabla^2 J\\|. For smooth dynamics and costs, JJ is smooth on the interior of the stabilising set (stationary distributions depend smoothly on the policy under uniform ergodicity).

  3. A regulariser that is strongly convex in the actor parameters. With a linear one-step actor, R\mathcal R is an exact quadratic with curvature σmin⁡\sigma_{\min} (Assumption 2), so L=J+α2R\mathcal L=J+\tfrac{\alpha}{2}\mathcal R is μ\mu-strongly convex on B\mathcal B with μ=ασmin⁡−LJ\mu=\alpha\sigma_{\min}-L_J as soon as α>LJ/σmin⁡\alpha > L_J/\sigma_{\min}. This is the mechanism behind part (ii): the regulariser dominates the non-convexity of JJ locally. It carries over verbatim to any actor that is linear in its parameters (e.g. a fixed feature map with a trained last layer); for fully nonlinear actors one would need gradient dominance of R\mathcal R on B\mathcal B, which is where our argument stops.

  4. A uniform-in-B\mathcal B critic bound and coverage (Assumptions 1–2), which are standard offline-RL requirements and not LQR-specific.

LQR-specific:

  1. The closed-form quadratic critic (Proposition 1) is what gives Lemma 2 the true gradient and the computed direction share the same policy-gradient matrix EKE_K and differ only through the covariance weighting. This is what lets us decompose the error floor into the interpretable terms ϵcov\epsilon_{\mathrm{cov}} and ϵhet\epsilon_{\mathrm{het}}. In a general MDP the mismatch ∇J−g\nabla J-g exists but has no such structure; one would bound it directly by a concentrability-type constant times the state-distribution shift, losing the explicit heterogeneity-coupling term.

  2. Real-analyticity of K↦PK(δ)K\mapsto P_K(\delta), which gives finite uniform constants on B\mathcal B for free.

Where it is in the revised manuscript (blue text): Scope sentences in the abstract and at the end of the first paragraph of Section 1; new paragraph “What the proof uses, and what is specific to LQR” in Section 4.4; added sentence in Section 4.5(i).

AuthorsOpenReview ↗

Response to the comment 2/3 from reviewer ujcV

Question 2: How can Assumption 1 be verified or justified in practice? In particular, why does convergence of the learned-policy state distribution toward the expert distribution provide evidence for a uniformly bounded critic approximation error? Could the authors directly evaluate the critic error in the experiments?

Response: We thank the reviewer for this insightful comment. We have reworded Appendix D.3 accordingly. The logic we intended was: TD-learning error bounds for LQR (Yang et al., 2019) hold under a near-on-policy condition. In other words, the data distribution must be close to the stationary distribution of the policy being evaluated. Figure 3 shows that this condition is approximately met along training. That makes a bounded ϵQ\epsilon_Q plausible; it does not verify it, let alone uniformly over B\mathcal B. We should have presented it as a necessary-condition check, not as validation, and the revised appendix now says exactly this.

Where the assumption sits in the literature. Assumption 1 is an inexact policy evaluation assumption of the kind that is standard in convergence analyses of policy-gradient and policy-mirror-descent methods. Much of that literature assumes exact access to QπQ^\pi or to the true gradient, e.g. the softmax policy-gradient analysis of Mei et al. (2020), the exact policy-mirror-descent rates of Xiao (2022) and Lan (2023), and the optimal rate of Johnson et al. (2023), and the inexact variants (Xiao, 2022, Assumption 1), (Cen et al., 2022, Eq. 26), (Zhan et al., 2023, Assumption 2), (Alfano et al., 2023, Assumption A1) assume ∥Q^−Qπ∥≤ϵ\lVert\hat Q-Q^\pi\rVert\le\epsilon (in sup-norm or a weighted L2L^2 norm) and show that the exact-case rate is preserved up to a neighbourhood proportional to ϵ\epsilon (or ϵ\sqrt\epsilon for the L2L^2 variant). Theorem 1 has the same structure, with the exact-critic case recovered at ϵQ=0\epsilon_Q=0. What is specific to our setting is that the bound must hold uniformly over B\mathcal B and for the aggregate critic of a mixture of configurations.

Regarding assessment in practice, Assumption 1 can be monitored from data alone, without knowledge of the dynamics. Holding out a subset of D\mathcal{D} (stratified by configuration), one can compute at each actor iterate KkK_k the held-out Bellman residual

ϵ^BR(Kk)=(1Nho∑[Q^Kk(x,u)−c(x,u)+J^Kk−Q^Kk(x′,πKk(x′,η))]2)1/2,\hat\epsilon_{\mathrm{BR}}(K_k)=\Bigl(\tfrac{1}{N_{\mathrm{ho}}}\textstyle\sum\bigl[\hat Q_{K_k}(x,u)-c(x,u)+\hat J_{K_k}-\hat Q_{K_k}(x',\pi_{K_k}(x',\eta))\bigr]^2\Bigr)^{1/2},

which requires only the current actor, critic and transitions. It vanishes iff the critic satisfies the Bellman equation on the data support, and the value error is bounded by the Bellman residual up to a factor depending on the closed-loop mixing rate, so a small, non-increasing residual along training is direct evidence for bounded ϵQ\epsilon_Q on the iterates; evaluating it for actors perturbed within a ball around KkK_k addresses uniformity over B\mathcal{B}.

Where it is in the revised manuscript (blue text): Appendix D.3 retitled and rewritten: opening paragraph with the literature positioning above and the necessary-condition reading of Figure 3. Section 4.5(ii) states what a self-contained proof would need, how ϵQ\epsilon_Q enters the bound, and points to the appendix. The eight cited papers were added to the bibliography.

AuthorsOpenReview ↗

Response to the comment 3/3 from reviewer ujcV

Question 3: What additional benefit does the flow model provide compared with a standard behavior- cloning or MSE regularizer? Since the theoretical guarantees do not depend specifically on flow matching, could the authors provide an ablation comparing flow regularization with a simpler behavioral regularizer?

Response: It is a very interesting comment. Section 4.5(vi) flagged this, and the revised manuscript adds the ablation Table (Table 6 in the Appendix D.5 of the revised manuscript). The guarantees of Section 4 hold for any behavioural anchor of comparable fidelity (Section 4.5(vi)), so the empirical question is whether the flow anchor matters on heterogeneous data. We keep the data, critic, actor network, α\alpha and seeds of HeFLO fixed and exchange only the regulariser R\mathcal{R} in (3.4): (a) the pathwise flow loss of HeFLO; (b) a mean-squared-error anchor towards a deterministic MSE behaviour-cloning anchor, E∥πζ(x,η)−ab∥2\mathbb{E}\lVert\pi_\zeta(x,\eta)-a^b\rVert^2 with aba^b is the action taken from D\mathcal D at the corresponding state inside HeFLO's pipeline.

Table: Anchor ablation (reproduction of Table 6): normalised score, mean ±\pm std; only the regulariser differs.

EnvironmentHeFLO (flow anchor)MSE-BC anchor
InvertedPendulum-v5100 ±\pm 0.0100 ±\pm 0.0
InvertedDoublePendulum-v5102.63 ±\pm 0.097.61 ±\pm 5.05
Reacher-v5100.17 ±\pm 4.7398.11 ±\pm 4.90
Pusher-v5104.71 ±\pm 3.81100.22 ±\pm 3.68
Swimmer-v5103.89 ±\pm 0.23104.15 ±\pm 0.71
Hopper-v5119.05 ±\pm 0.07117.57 ±\pm 0.04
Walker2d-v5109.54 ±\pm 2.66108.41 ±\pm 1.91

Result. Table 6 shows modest improvements with the flow anchor over MSE-BC, while both variants achieve comparable performance. These results align with the theory's emphasis on the presence and strength of a behavioural anchor, and the revised manuscript says so (Section 5, Section 4.5(vi) and the Conclusion).

What, then, does the flow model contribute? (i) Theoretically, our analysis does not rely on anything specific to flow matching. Theorem 1 needs an anchor of fidelity ϵb\epsilon_b and a regulariser that is quadratic in the actor parameters, and both anchors provide this. Section 4.5(vi) said so already, and the ablation is consistent with it. (ii) Empirically, on the two environments tested, no measurable gain: the stability and low variance of HeFLO in Table 1 are attributable to the presence and strength of a behavioural anchor, not to the anchor's parametrisation. (iii) Where it could matter. Experts trained on different δj\delta_j take different actions at a shared state xx, so the pooled behaviour distribution pD(u∣x)p_{\mathcal D}(u\mid x) is a mixture over configurations. Therefore, a regression anchor clones its conditional mean, which need not be a good action for any single configuration, whereas the flow represents the mixture and the pathwise regulariser pulls the actor toward a sample from it (now explained in Section 5). The agreement of the two anchors indicates that, for the ranges Sδ\mathcal S_\delta and the m≤5m\le5 configurations used here, the conditional mean remains a good action. The flow anchor is thus a more expressive default as it performs on par here and keeps the method applicable when the demonstrations are genuinely multimodal, rather than a necessity, and the revised text no longer suggests otherwise.

Where it is in the revised manuscript (blue text): Table 6 and the paragraph “Anchor ablation” in Appendix D.5; paragraph “Anchor ablation and additional baselines” with the mixture argument and the outcome in Section 5; Section 4.5(vi) ; the paragraph “Why pathwise coupling, and its cost” in Section 3.2 and the Conclusion were reworded accordingly.

We thank the reviewer for the thoughtful and constructive feedback. We have carefully addressed the concerns raised, and your comments have helped us make the manuscript clearer and stronger. We hope the revised manuscript better conveys our primary contribution: stability and convergence guarantees for policy-gradient offline RL under fixed, un-resampleable domain randomisation, which, to our knowledge, are the first of their kind. The empirical section complements this analysis by examining confinement and variability across seeds and configurations in nonlinear systems, helping assess the practical relevance of the theoretical insights.

Comments on the paper

1 comment

We thank the reviewers for the thoughtful and constructive feedbacks. We have carefully addressed the concerns raised, and your comments have helped us make the manuscript clearer and stronger. We hope the revised manuscript better conveys our primary contribution: stability and convergence guarantees for policy-gradient offline RL under fixed, un-resampleable domain randomisation, which, to our knowledge, are the first of their kind. The empirical section complements this analysis by examining confinement and variability across seeds and configurations in nonlinear systems, helping assess the practical relevance of the theoretical insights.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.