acceptodds
Under review as a conference paper at ICLR 2027

PACE: Path-Aware Constraints on Exploration in Frozen Flows for Offline-to-Online Reinforcement Learning

Abstract

Flow policies capture complex, multimodal offline behaviors, yet direct value optimization through iterative generation can destabilize training. Existing methods instead train a flow teacher to imitate offline data and guide the actor that interacts with the environment. During the online phase, however, updating this teacher on newly collected experience can undermine its learned behavioral structure and destabilize policy improvement. Preserving a stable behavioral reference while continually improving its guidance remains an open problem. To address this, we introduce Path-Aware Constraints on Exploration (PACE) in frozen flows for offline-to-online RL, which freezes the pretrained flow and uses an evolving critic to guide exploration in its latent noise space. The explored and original outputs jointly supervise the actor, enabling a frozen teacher to offer both stable behavioral guidance and continually updated knowledge. Using the flow's fixed geometry, PACE further constrains deformation along complete noise-to-action paths to reduce out-of-distribution risk. Experiments on 35 OGBench tasks demonstrate improved sample efficiency and final performance under limited online interaction.

Primary area:
reinforcement learning
Keywords:
offline-to-online reinforcement learning, flow matching, expressive policies

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sEqEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PACE, an O2O RL method that leverages a frozen behavior-cloning flow policy (flow teacher). The authors argue that continually updating a flow teacher with online experience can distort its learned behavioral distribution and destabilize one-step flow policy (student) learning. To address this issue, PACE freezes the pretrained flow teacher and performs learned Q-value guided exploration in its latent noise space to generate improved action targets for the flow student. In particular, the proposed method introduces a path-aware constraint that accounts for the deformation along the entire flow generation trajectory, aiming to restrict latent exploration in directions that induce large path deformations.

Critical strengths

S1 (Interesting motivation). The paper shows that continually updating the flow teacher during online learning can spread its learned behavioral distribution into low-density regions, potentially degrading the performance of the one-step actor. This observation provides a clear motivation for maintaining a frozen flow teacher.

S2 (Path-aware constraints for latent exploration). PACE introduces a path-aware constraint that considers deformation along the entire flow generation trajectory, rather than only at the latent inputs or final actions. The paper derives a quadratic approximation of path deformation, enabling constrained Q-value-guided latent exploration without repeatedly evaluating perturbed flow trajectories.

S3 (Strong empirical results). PACE demonstrates strong empirical performance across 35 OGBench tasks, outperforming existing baselines in average final success rate.

Critical weaknesses

W1 (Unclear contribution and interaction of the two distillation objectives). In Eq. (4), the actor is trained with both the original distillation loss toward the frozen flow teacher (α∥aπ−aB∥2\alpha\|a_{\pi}-a_{B}\|^2) and the additional loss toward the explored action target (αβ∥aπ−aimp∥2\alpha\beta\|a_{\pi}-a_{imp}\|^2). However, β\beta is set to 10 or higher across all task families, assigning a substantially larger coefficient to the exploration-target loss. Since the two objectives may encourage the actor to move toward different actions (aB≠aimpa_{B}\neq a_{imp}), it is unclear how they interact during optimization and whether the original distillation term remains necessary. An ablation removing the original distillation term, along with an analysis of the relative loss or gradient magnitudes, would help clarify the contribution of each objective and the motivation for choosing relatively large values of β\beta.

W2 (Insufficient ablation studies and implementation details for adaptive latent exploration). The paper lacks sensitivity analyses of important components in adaptive latent exploration, such as the metric trust radius ρ\rho in Eq. (2). Furthermore, the clipping operation in Eq. (3) is not clearly specified, including its clipping bounds and implementation. Additional ablation studies and implementation details would help assess the robustness and reporducibility of the proposed method.

W3 (Increased hyperparameter complexit). Compared with FQL and FINO, PACE introduces several additional hyperparameters, including the latent step size η\eta, metric trust radius ρ\rho, metric damping λ\lambda, and exploration-target weight β\beta (Tables 3 and 4). In particular, β\beta is selected separately for each task family. However, the paper does not provide sufficient sensitivity analyses or explain how these additional hyperparameters were selected. This raises concerns about the robustness of the reported performance and the additional tuning effort required by PACE.

W4 (Limited justification for reducing OOD exploration). The paper argues that path-aware constraints reduce OOD exploration and mitigate the risk of Q-value overestimation. However, the primary goal in O2O RL should be to avoid exploiting overestimated Q-values for OOD actions, rather than reducing OOD exploration itself. In fact, exploring OOD actions can be essential for discovering better behaviors beyond the offline dataset, particularly when the dataset is suboptimal. Therefore, reducing the OOD rate does not necessarily indicate more effective online policy improvement. Further analysis is needed to justify whether the proposed constraints selectively suppress unreliable OOD actions without unnecessarily restricting beneficial exploration.

W5 (Task-dependent benefits of freezing the flow teacher). The motivation for freezing the flow teacher is supported by substantial improvements on Cube-Double-Task4 and Puzzle-4×4-Task4 (Table 5 and Figure 6). However, the frozen teacher variant performs worse than standard FQL on both AntMaze tasks. These results suggest that freezing the flow teacher is not consistently beneficial across different environments. Moreover, maintaining a fixed behavioral reference may restrict the actor from learning new high-value actions that deviate substantially from the offline behavior distribution. Further analysis would help clarify when freezing the flow teacher is beneficial and why continually updating it can destabilize actor learning.

W6 (Insufficient analysis of baseline performance and the contribution of entropy-guided sampling). The continually updated-flow variant of PACE exhibits unstable training on Cube-Double-Task4 and Puzzle-4×4-Task4, resulting in failed runs (Table 2), whereas standard FQL achieves nonzero final performance on Puzzle-4×4-Task4 (Table 5). Further analysis of these training failures would help clarify the source of instability. Moreover, PACE adopts the entropy-guided action selection of FINO, which is not used by several other baselines, including FQL. An additional ablation without this action-selection mechanism would help isolate the contribution of the proposed method and provide a more controlled comparison with these baselines.

W7 (Missing computational cost analysis). PACE requires additional computations, including backward flow integration, flow Jacobian evaluation, path metric construction. Although the authors reduce the computational burden by generating exploration targets for only a subset of each training batch, the paper does not provide quantitative comparisons of training time or GPU memory consumption against FQL and FINO. Such comparisons would help assess the practical efficiency of the proposed method.

Questions for authors

Could you provide empirical results and explanation regarding the Weaknesses above, especially W1, 2, 4, 6, and 7?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3 (Part 1/3)

We thank Reviewer 3 for the constructive review and insightful suggestions. We have conducted the requested ablations, sensitivity analyses, and computational benchmarks, which we detail below and will incorporate into the revised manuscript. Unless specified otherwise, results in the additional experiments below are four-seed mean final success rates on Task 4 after 500k online interactions.

W1 — Contribution and interaction of the two distillation objectives. Table 2 already evaluates removing the explored target aimpa_{\text{imp}} (dropping mean success by 12.112.1\\% across tasks). To directly address the reviewer’s question, we ran the complementary ablation removing only the original distillation loss toward the frozen teacher output aBa_B while retaining the direct QQ objective and the explored target aimpa_{\text{imp}}. Removing the original distillation term nearly eliminated online success on both Cube Double and Puzzle 4×44\times4, confirming that aBa_B remains indispensable as a global behavioral anchor.

Gradient diagnostics during training explain why a relatively large β\beta (β≥10\beta \ge 10) coexists harmoniously with this anchor rather than overwhelming it:

  1. Comparable Weighted Gradient Norms: Because aimpa_{\text{imp}} is constructed via a small local trust-region step (ρ=0.05\rho=0.05) starting from zπ=Φ1−1(s,aπ)z_\pi = \Phi_1^{-1}(s, a_\pi), its unweighted residual ∣aπ−aimp∣2\\|a_\pi - a_{\text{imp}}\\|_2 is substantially smaller than ∣aπ−aB∣2\\|a_\pi - a_B\\|_2. After multiplying by β\beta, the two distillation losses produce comparable actor-gradient norms.
  2. Directional Complementary Role: The two gradients exhibit mildly negative median cosine similarities, meaning the original teacher target aBa_B actively counterbalances aimpa_{\text{imp}} precisely when value-guided exploration attempts to pull the actor away from the offline behavioral manifold.

W2 & W3 — Implementation details, hyperparameter selection, and sensitivity analyses.

  1. Clipping Specification in Eq. (3): In Eq. (3), we clip both the decoded base action Φ1(s,zπ)\Phi_1(s, z_\pi) and the explored action Φ1(s,zπ+δ)\Phi_1(s, z_\pi + \delta) coordinatewise to the valid action bounds [−1,1][-1, 1], add their residual difference to the current actor output aπa_\pi, and clip the final target aimpa_{\text{imp}} to [−1,1][-1, 1] (Appendix B.1).
  2. How β\beta Was Selected (No Fine-Grained Per-Task Tuning): To select β\beta, we originally evaluated only three coarse values (β∈10,30,100\beta \in \\{10, 30, 100\\}) on a single task (Task 4) and a single seed per family, and then reused the selected value across all five tasks in that family. As shown in Table 8, PACE exhibits strong performance across both β=10\beta=10 and β=30\beta=30 without requiring delicate tuning (whereas β=100\beta=100 is overly aggressive on manipulation and humanoid tasks).
  3. Sensitivity to ρ\rho, η\eta, and λ\lambda: As shown in Table 9, halving or doubling the trust radius ρ∈0.025,0.05,0.10\rho \in \\{0.025, 0.05, 0.10\\}, proposed step scale η∈5,10,20\eta \in \\{5, 10, 20\\}, or metric damping λ∈0.005,0.01,0.02\lambda \in \\{0.005, 0.01, 0.02\\} yields consistently high final success and AUC, demonstrating robustness around the shared defaults.

Table 8: Sensitivity to exploration-target weight β\beta (Final online success %, 4 seeds on Task 4).

Task Familyβ=10\beta = 10β=30\beta = 30β=100\beta = 100
AntMaze Large98.098.098.098.098.098.0
AntMaze Giant90.590.595.595.596.596.5
AntSoccer Arena89.589.584.584.585.585.5
HumanoidMaze Medium79.579.594.594.550.050.0
HumanoidMaze Large37.037.035.535.50.00.0
Cube Double76.576.571.571.50.00.0
Puzzle 4×44\times499.599.599.599.550.550.5

Table 9: Sensitivity to path-exploration hyperparameters ρ\rho, η\eta, and λ\lambda (4 seeds on Task 4, reported as AUC / Final %).

Task FamilyParameterLow / Default / HighLow (AUC / Final)Default (AUC / Final)High (AUC / Final)
Cube DoubleTrust radius ρ\rho0.0250.025 / 0.050.05 / 0.100.1046.646.6 / 76.076.052.552.5 / 76.576.555.555.5 / 82.582.5
Puzzle 4×44\times4Trust radius ρ\rho0.0250.025 / 0.050.05 / 0.100.1062.462.4 / 96.596.562.762.7 / 99.599.574.274.2 / 99.099.0
Cube DoubleStep scale η\eta55 / 1010 / 202051.851.8 / 78.078.052.552.5 / 76.576.552.752.7 / 78.578.5
Cube DoubleDamping λ\lambda0.0050.005 / 0.010.01 / 0.020.0257.357.3 / 82.582.552.552.5 / 76.576.556.556.5 / 85.585.5

(Continued in Part 2/3 below.)

AuthorsOpenReview ↗

Response to Reviewer 3 (Part 2/3)

W4 — Selective OOD suppression vs. beneficial exploration. We fully agree that blindly suppressing all deviations from the offline dataset would hinder policy improvement. Crucially, Figure 3 and Appendix C.4 demonstrate that PACE's full-path constraint does not reduce OOD incidence by conservatively pulling actions closer to the teacher. On Cube Double (across 256 states and 1,024 samples/state), switching from start/end to full-path constraints:

  • Increases the actor's mean action deviation from the teacher from 0.02860.0286 to 0.03820.0382 (permitting larger exploratory shifts toward higher QQ values), while
  • Simultaneously reducing the empirical OOD rate (measured by an independent offline dataset kk-NN distance doff(s,a)d_{\text{off}}(s,a)) from 3.463.46\\% to 3.123.12\\%.

Combined with the superior performance demonstrated in Tables 1 and 2, this confirms that full-path geometry selectively penalizes high-shear directions that cross into unsupported regions while granting greater freedom along well-supported transport directions.

W5 — Task-dependent effects of freezing the teacher and limited-budget scope. In Table 5, freezing the flow teacher improves FQL on five of seven Task 4 environments and raises overall mean success from 45.045.0\\% to 56.556.5\\%, notably rescuing Cube Double from catastrophic collapse (0.00.0\\% \to 52.5\\%). Even on the two AntMaze tasks where frozen-teacher FQL is slightly lower, it still maintains strong performance (80–8780\text{--}87\\%), and full PACE recovers and surpasses standard FQL (9898\\% on AntMaze Large and 9696\\% on AntMaze Giant, Table 2) via critic-guided latent exploration. Thus, freezing provides a simple yet high-leverage stability foundation across the benchmark.

Regarding whether a fixed reference could limit asymptotic improvement under much larger online budgets: our primary research question addresses policy improvement under a limited online interaction budget (operationalized here as 500k500\text{k} steps, matching prior O2O benchmarks such as FINO). Whether methods eventually converge or change rank as the budget scales toward multi-million steps is an interesting open question, though without a natural upper cutoff. In the revision, we will explicitly scope our claims to the limited-budget O2O regime and note longer-horizon scaling in Section E (Limitations and Future Work).

(Continued in Part 3/3 below.)

AuthorsOpenReview ↗

Response to Reviewer 3 (Part 3/3)

W6 — Training failures in continually updated flow and contribution of candidate selection.

  1. Cause of Non-Finite Failures in Continually Updated Flow (Table 2): Unlike standard FQL—which only evaluates forward teacher outputs for distillation—the continually updated-flow variant also performs backward ODE integration (zπ=Φ1−1(s,aπ)z_\pi = \Phi_1^{-1}(s, a_\pi)) and solves linear systems involving the path Jacobian metric GG (Eq. 2). When online replay shifts destabilize the unfrozen velocity field on complex manipulation tasks (Cube Double and Puzzle 4×44\times4), backward integration and Jacobian inversion encounter ill-conditioned trajectories, leading to numerical overflow (NaN/non-finite values). Freezing the pretrained flow eliminates this instability. This further corroborates the internal consistency of our design—namely, that a fixed flow reference is essential for reliable path-metric computation.
  2. Isolating Entropy-Guided Candidate Selection: To isolate the contribution of PACE's core learning mechanism from FINO-style candidate sampling, Table 10 compares PACE with single-sample collection and single-sample evaluation (K=1K=1 across all tasks, matching standard FQL) against full PACE. Even without candidate selection, single-sample PACE achieves strong results across tasks (e.g., 95.595.5\\% on AntMaze Large, 91.591.5\\% on AntMaze Giant, 90.590.5\\% on AntSoccer, 76.576.5\\% on Cube Double, and 99.599.5\\% on Puzzle 4×44\times4). Candidate selection primarily benefits high-dimensional humanoid locomotion (HumanoidMaze Medium/Large), where action dimensionality is substantially higher. We will include Table 10 in the revision to make this distinction explicit.

Table 10: Ablation of candidate selection mechanism (Task 4, 4 seeds, reported as AUC / Final %).

Task FamilyPACE (Single Collection + Single Eval, K=1K=1)Paper PACE (QQ-Softmax Collection + QQ-Argmax Eval)
AntMaze Large94.594.5 / 95.595.595.495.4 / 98.098.0
AntMaze Giant79.679.6 / 91.591.587.687.6 / 95.595.5
AntSoccer Arena69.869.8 / 90.590.572.372.3 / 87.587.5
HumanoidMaze Medium46.746.7 / 54.054.084.484.4 / 96.096.0
HumanoidMaze Large9.49.4 / 6.06.014.914.9 / 37.037.0
Cube Double52.552.5 / 76.576.552.552.5 / 76.576.5
Puzzle 4×44\times462.762.7 / 99.599.562.762.7 / 99.599.5

W7 — Computational cost analysis. In our target deployment setting, the critical computational cost is inference-time latency rather than training-time compute: distilling into a one-step flow student yields significantly lower inference latency than multi-step iterative flow or diffusion policies, which is essential for real-time, safety-critical scenarios such as high-frequency robotic control. Because backward integration and path-metric computation in PACE are strictly confined to training updates, execution/inference uses the exact same one-step actor architecture as FQL and FINO—fully preserving the inference-speed advantage of FQL-style policies with zero added latency. Even during training, benchmarking on the same GPU under identical batch settings shows that computing exploration targets on a sub-batch of 32 examples adds only 2.39 ms2.39\text{ ms} per update over FQL (7.14 ms7.14\text{ ms} vs. 4.75 ms4.75\text{ ms}) and 2.10 ms2.10\text{ ms} over FINO (7.14 ms7.14\text{ ms} vs. 5.04 ms5.04\text{ ms}), with a modest 86 MiB86\text{ MiB} increase in peak GPU memory (518 MiB518\text{ MiB} vs. 432 MiB432\text{ MiB}).

Reviewer_o3FxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PACE, a flow-matching-based RL method built on FQL. It uses critic gradients to adjust its input noise, producing locally improved action targets. These updates are constrained by changes along the full noise-to-action path. A one-step student learns from both the original teacher outputs and the improved targets. Experiments on 35 OGBench tasks show better average online performance and sample efficiency.

Critical strengths
  1. I find the idea of using locally improved actions as distillation targets reasonable. It allows the student to receive updated guidance while keeping the teacher fixed. The framework is complete and clearly explained.
  2. The experiments cover 35 OGBench tasks and show good improvements in average online performance and sample efficiency. These results support the practical value of the method.
Critical weaknesses
  1. The novelty seems limited. It would also be helpful to discuss how PACE differs from ReFORM and QGF, as well as other methods built on FQL, especially what additional benefits from noise changing.

  2. I still have concerns about long-term stability and dependence on the teacher. The analysis only covers local target construction, and Appendix E acknowledges some late-training declines. It is also unclear how well PACE works when the offline teacher misses important high-value behaviors, since the fixed behavioral reference may continue to limit improvement.

[1] ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulation. [2] Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning.

Questions for authors

Regarding online learning part, how well can PACE improve when the offline teacher poorly represents important high-value behaviors? I think this is could be a more important issue.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 2 (Part 1/2)

We thank Reviewer 2 for the encouraging assessment, noting the clarity of our presentation, the soundness of using locally improved targets with a fixed teacher, and the strength of our 35-task empirical validation. Below, we address your comments regarding novelty and PACE's behavior when the offline teacher poorly covers high-value behaviors.

W1 — Novelty and Comparison with ReFORM, QGF, and FQL-based Methods. We respectfully disagree with the assessment of limited novelty. Rather than being a minor variant of existing latent-steering methods, PACE introduces a distinct paradigm for expressive-policy O2O RL: it uses a frozen flow teacher simultaneously as a stable behavioral anchor and as a full-path Riemannian transport geometry (G=1c∫01Jt⊤Jtdt+λIG = \frac{1}{c}\int_0^1 J_t^\top J_t dt + \lambda I) to regularize critic-guided latent exploration into dual-target supervision for a one-step actor.

We will expand Section 2 to explicitly discuss ReFORM [1], QGF [2], and FQL-based methods along this distinction:

  1. Vs. ReFORM (Offline Noise Reflection): ReFORM is designed for purely offline RL, reflecting latent particles within hard noise boundaries to stay strictly inside the offline support. It does not address online policy improvement, dual-target distillation, or trajectory-level flow geometry.
  2. Vs. QGF (Test-Time Gradient Guidance): QGF applies multi-step critic-gradient guidance to ODE trajectories at test/inference time, incurring substantial inference latency and directly altering the execution sampler. In contrast, PACE amortizes critic-guided latent exploration into training-time distillation targets (aimpa_{\text{imp}}), preserving a single-step actor with zero added inference latency while constraining updates via full-path sensitivity rather than unregularized or endpoint-only steps.
  3. Vs. FQL / FINO: Standard FQL and FINO continually fine-tune the flow teacher on online replay, which can distort learned modes and destabilize student learning (e.g., FQL collapsing to 0.00.0\\% on Cube Double Task 4, Table 5). Freezing the teacher preserves a stable geometric manifold, while latent noise optimization unlocks adaptive guidance from a fixed generator.

Empirical Isolation of Our Contributions (Ablations in Table 2):

  • Benefit of noise-space exploration: Removing aimpa_{\text{imp}} while keeping the frozen teacher and direct QQ objective drops mean Task 4 success by 12.112.1 percentage points (8484\\% \to 72\\%). Conversely, adding aimpa_{\text{imp}} drives substantial gains on complex tasks, raising success from 6464\\% to 9595\\% on HumanoidMaze Medium and from 5858\\% to 7777\\% on Cube Double.
  • Benefit of full-path geometry: Replacing our full-path metric GG with start-point (7979\\%), endpoint (7272\\%), or start-and-end (7070\\%) constraints consistently underperforms full PACE (8484\\%), confirming the unique value of full-trajectory flow geometry.

(Continued in Part 2/2 below.)

AuthorsOpenReview ↗

Response to Reviewer 2 (Part 2/2)

W2 & Q1 — Can PACE Improve When the Offline Teacher Poorly Covers High-Value Behaviors? This is a central question for frozen-prior methods. Empirically, Tables 1 and 6 show that PACE achieves dramatic online improvements even when the offline teacher starts near 00\\% success (indicating poor coverage of complete high-value trajectories):

  • Puzzle 4×44\times4 Tasks 2 & 5: Offline success starts at 2020\\% and 1010\\%, yet PACE reaches 9999\\% and 100100\\% online (whereas FQL and FINO both fail at 00\\%).
  • Cube Double Task 4: Offline success is only 44\\%, yet PACE reaches 7979\\% online (vs. 44\\% for FQL and 1313\\% for FINO).
  • AntMaze Giant Tasks 2 & 4: Starting from 11\\% and 1313\\%, PACE reaches 9595\\% and 9393\\% online.
  • HumanoidMaze Medium Tasks 1 & 4: Starting from 2424\\% and 1313\\%, PACE reaches 9797\\% and 9595\\% online.

Why does a frozen teacher still enable strong improvement here?

  1. Skill Stitching & Mode Reweighting: In OGBench play and navigation datasets, poor initial success typically reflects a lack of complete goal-reaching trajectories rather than a total absence of valid local transitions. The frozen flow captures these valid behavioral primitives, while the evolving online critic identifies high-value transitions and uses path-constrained latent exploration to reweight and stitch local primitives into successful long-horizon policies.
  2. Direct Value Optimization: Alongside distillation, Eq. (4) retains the actor's direct critic objective −Qactor(s,aπ)-Q^{\text{actor}}(s, a_\pi), allowing the one-step student to combine learned primitives as online feedback refines the critic.

Regarding Long-Term Stability & Late-Training Declines: As noted in Appendix E, isolated late-training dips (e.g., AntMaze Giant Task 1) reflect the broader coupled learning dynamics between value estimation and fixed regularization weights, rather than instability in the frozen flow itself—which motivates our future work on adaptive diagnostics and scheduling in Section E. Across all 35 tasks, PACE achieves the highest overall normalized AUC (7272 vs. FINO's 5959, measuring average performance throughout online learning, Section 4.1) and final success (82.182.1\\% vs. 65.065.0\\%), showing consistent stability across the 500k500\text{k} budget. Regarding longer-term asymptotic behavior, our primary research question addresses policy improvement under a limited online interaction budget (operationalized here as 500k500\text{k} steps, matching prior O2O benchmarks such as FINO); how performance evolves as the budget scales toward multi-million steps is an interesting open question without a natural upper cutoff, which we leave for future work in Section E.

Thank you for the detailed response. I appreciate the clarification of how PACE differs from ReFORM, QGF, and other FQL-based methods, as well as the ablations supporting the value of the full-path constraint. I still view the contribution as incremental, and my concerns about dependence on the teacher and late-training stability remain. I therefore maintain my original decision.

Reviewer_ZsLLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper considers offline-to-online RL. The paper identifies that, for previous teacher-student style updates, the teacher is not fixed during online training and may expand its generated distributions to unsupported actions with a limited training budget. To address this, the paper proposes PACE, which fixes the teacher during online training and penalizes the full-path deformation between the student flow and the teacher flow. Empirical results show that PACE outperforms baselines.

Critical strengths
  1. The reported empirical results are strong.
Critical weaknesses
  1. The connection between path-aware constraints and OOD/low density actions is not established rigorously. The proposed metric measures "path deformation" of flow trajectories, but does not directly control action density under the offline behavior distribution. In addition, the reference trajectory is obtained by inverting the current actor’s action, which itself may not lie in a well-supported region. Therefore, small path deformation does not ensure that the resulting action is well supported. While avoiding unsupported actions is central to the full-path constraints, this claim is not rigorously justified.

  2. The distinct benefit of constraining path deformation is not isolated. The ablation studies compare PACE with constraining the start, end, and start & end points, but this does not ensure they have comparable constraint strength on the actions. PACE's better performance may just come from a stronger/weaker constraint on the actual action. I would suggest that the authors isolate the benefit of path deformation by varying the constraint weights for each kind of constraint, and then plot how their performance changes w.r.t. different weights and whether PACE can Pareto-dominate them. Or maybe show that PACE has the best max performance (with the best weights).

Questions for authors
  1. Can you reply to the two points in Critical Weaknesses?

  2. The paper claims that the spread of probability mass into sparser regions yields poorer student performance. How do you justify this claim? There is insufficient evidence that spreading probability mass into sparser regions yields poorer student performance.

  3. Figure 1 shows that distributions of FQL's flow teacher expand a lot after online learning, but how do you justify that this is indeed a bad thing? Is there a direct causal relationship between this expansion and the decrease in performance, and is this relationship general?

  4. What is the clipping in Equation (3) used for?

  5. The paper does not compare with related recent latent-space RL works. Can you provide comparisons and discussions with them, for example, DSRL (Steering Your Diffusion Policy with Latent Space Reinforcement Learning) and ReFORM (Reflected Flows for On-support Offline RL via Noise Manipulation)? Both methods do not constrain path deformation while still achieving better results than FQL, and DSRL has reported its offline-to-online performance.

  6. For the ablation comparing PACE and continually updated flow, why are the offline results different? Why don't they start with the same converged teacher?

  7. The paper claims that the specific challenge of offline-to-online RL comes from the online training budget. Can you provide an ablation to show that with an increasing training budget, FQL shows increasingly better results? And how does PACE compare with FQL with an increasing training budget?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1 (Part 1/3)

Thank you for the careful review. Before addressing each point, we offer one brief clarification regarding the summary: both trajectories in the path cost belong to the same frozen flow teacher. The one-step actor's action determines the reference latent through inversion; the cost measures how a latent perturbation changes the ensuing generation path.

W1 — Connection between path-aware constraints and offline action support.

We agree that small path deformation does not directly guarantee high offline action density. This is consistent with our objective, because PACE's path constraint is intended not to enforce similarity to offline actions, but to regulate critic-guided policy improvement while allowing adaptation beyond the offline behavior distribution. Its role is to provide a behavior-informed geometry for exploration, whose rationale and empirical benefits we clarify below.

1. Direction-aware policy improvement. PACE uses the teacher's transport geometry to impose a direction-aware constraint. Here, directions with greater path sensitivity incur higher costs, while less sensitive directions allows more exploration. This is intended to preserve useful behavioral structure while allowing new actions, reducing the risk of disruptive updates without making exploration uniformly conservative.

Appendix A shows that PACE updates achieve a positive first-order critic gain (g⊤δ>0g^\top\delta>0 for g≠0g\ne0) under the path-metric constraint, realizing the direction-aware exploration. Thus, the constraint regulates critic-guided improvement from the current actor rather than merely enforcing similarity to offline actions.

2. Additional connection to modeled density. Let pθ(a∣s)p_\theta(a\mid s) denote the frozen teacher's conditional density. Under the local smoothness assumptions detailed in our supplementary mathematical response, the exact continuous flow before clipping satisfies

∣log⁡pθ(Φ1(s,zπ+δ)∣s)pθ(aπ∣s)∣≤∣zπ∣2∣δ∣2+12∣δ∣22+LDpath(s,zπ,δ).\left|\log\frac{p_\theta(\Phi_1(s,z_\pi+\delta)\mid s)}{p_\theta(a_\pi\mid s)}\right| \leq \\|z_\pi\\|_2\\|\delta\\|_2+\frac12\\|\delta\\|_2^2+L\sqrt{D_{\mathrm{path}}(s,z_\pi,\delta)}.

Here LL is the local regularity constant specified in the supplement. Intuitively, the change in model log-density relative to the current action is bounded by the latent step and accumulated path deformation. PACE's metric and damping regulate these quantities locally. This establishes a conditional connection to modeled density, without certifying that either action has high offline density. The supplementary response provides the assumptions and derivation.

3. Why Inverting aπa_\pi Does Not Cause Unbounded Drift: Three built-in mechanisms prevent aπa_\pi (and thus zπ=Φ1−1(s,aπ)z_\pi = \Phi_1^{-1}(s, a_\pi)) from drifting outside the well-supported region:

  • Persistent Anchoring to the Frozen Teacher (aBa_B): In Eq. (4), the actor is simultaneously distilled toward the frozen teacher's unperturbed output aB=Φ1(s,ϵ)a_B = \Phi_1(s, \epsilon) with a strong distillation weight α\alpha (e.g., α∈[10,1000]\alpha \in [10, 1000]) throughout training. Moreover, the exploration target aimpa_{\text{imp}} is introduced only after a 100k-step warmup and linear ramp (Appendix B.1), ensuring aπa_\pi is already anchored to the offline support before latent exploration begins.
  • Strict Local Trust Region & Residual Clipping: Each exploration step is bounded within a small trust radius (ρ=0.05\rho = 0.05) and passed through action-difference clipping (Eq. 3), preventing single-step jumps into unsupported regions.

4. Independent Empirical Support: As shown in Figure 3 and Appendix C.4, we evaluated generated actions using an independent offline dataset kk-NN distance metric doff(s,a)d_{\text{off}}(s, a) that does not rely on the flow's path metric. Across 256 states on Cube Double, full-path constraints reduce the empirical OOD rate from 3.463.46\\% (start/end constraints) to 3.123.12\\% while simultaneously permitting larger beneficial action deviations (0.03820.0382 vs. 0.02860.0286).

We will revise Sections 1 and 3.2 to emphasize PACE's intended balance between discovering behaviors beyond the offline data and limiting disruptive updates driven by imperfect critic estimates. We will explain how the critic guides improvement and the path geometry regulates its direction and extent, and interpret the OOD diagnostic jointly with learning performance. We will also add the conditional density analysis to Appendix A, making the rationale and scope of this regularization explicit.

(Continued in Part 2/3 below.)

AuthorsOpenReview ↗

Response to Reviewer 1 (Part 2/3)

W2 — Does the common radius favor Full Path? Under defaults (η=10,ρ=0.05\eta=10, \rho=0.05), the ablation changes only the metric GG. We measured the resulting action displacements on 128 shared state–noise probes for Task 4 of each family. Relative to Full Path, the median displacement ranged from 1.02–1.14×1.02\text{--}1.14\times for start-only, 2.32–4.18×2.32\text{--}4.18\times for endpoint-only, and 0.72–0.96×0.72\text{--}0.96\times for start-and-end. In every family, Full Path lay between endpoint-only and start-and-end, with a displacement very close to start-only. Thus, PACE gained no systematic advantage from imposing a uniformly tighter or looser constraint on action displacement.

We also tested the reviewer’s suggestion of matching action displacement by adjusting the constraint radius ρ\rho. Taking start-and-end as an example (which produced the smallest median step), the default setting already matched Full Path within 4.34.3\\% on Cube Double and Puzzle 4×44\times4. On the five navigation tasks, increasing ρ\rho from 0.050.05 to 0.200.20 still left the median displacement 17–2817\text{--}28\\% below Full Path. Because at least 127 of 128 proposals per task were already accepted in full (κ=1\kappa = 1), further relaxing ρ\rho had virtually no room to enlarge displacement. Increasing displacement further would require changing the unconstrained step scale itself (e.g., increasing η\eta specifically for start-and-end). Introducing a second, method-specific hyperparameter adjustment would confound the ablation, so we consider such joint tuning outside the scope of this comparison. Under the shared defaults (η,ρ)(\eta, \rho), the update rule remains consistent across variants and isolates the geometric effect of replacing the metric GG.

Q2–Q3 — Teacher drift and student performance. Table 5 already provides a controlled performance test of the motivation in Figure 1. Keeping ordinary FQL distillation and using a pretrained, frozen flow teacher outperformed standard FQL on five of seven Task 4 environments, raising mean final success from 45.045.0\\% to 56.556.5\\%. The clearest case is Cube Double, where standard FQL collapsed to 0.00.0\\% while the frozen-teacher variant reached 52.552.5\\%. On that same task, Figure 1 shows that continued FQL updates spread probability mass into sparser regions around the pretrained modes, expanding the projected 99%-mass region to 2.16×2.16\times its offline area. Together, Table 5 establishes the performance degradation under limited online feedback, while Figure 1 illustrates a consistent structural mechanism behind it. We will clarify in Section 3.1 that Figure 1 serves as diagnostic motivation rather than a claim of universal causality across all task regimes.

Q4 — What is the clipping in Equation (3) used for? In Eq. (3), we clip the decoded base and explored actions coordinatewise to the environment's valid action bounds [−1,1][-1, 1], add their residual difference to the current actor action aπa_\pi, and clip the resulting target aimpa_{\text{imp}} to [−1,1][-1, 1] (Appendix B.1). This ensures that both the residual target calculation and the final distillation target strictly respect valid action boundaries.

Q5 — Comparison and discussion with DSRL and ReFORM. Our main comparison focused on the standard offline-pretraining and online-fine-tuning protocol. ReFORM was developed for purely offline RL via reflected flows, whereas DSRL was developed to steer a pretrained generative policy and also reports O2O results. To address the reviewer’s question directly, we evaluated an OGBench adaptation of DSRL-NA on Task 4 of all seven families under a common RL budget (1M1\text{M} offline RL updates, 500k500\text{k} online interactions, 4 seeds): mean final success was 66.066.0\\% for DSRL-NA compared to 84.484.4\\% for PACE. We will include this comparison and discuss both DSRL and ReFORM in Section 2 and Appendix D.

Table 11: Final online success (%, 4 seeds on Task 4).

Task familyDSRL-NAPACE
AntMaze Large95.595.598.098.0
AntMaze Giant46.046.095.595.5
AntSoccer Arena61.061.089.589.5
HumanoidMaze Medium40.040.094.594.5
HumanoidMaze Large32.532.537.037.0
Cube Double89.089.076.576.5
Puzzle 4×44\times498.098.099.599.5
Mean66.066.084.484.4

(Continued in Part 3/3 below.)

AuthorsOpenReview ↗

Response to Reviewer 1 (Part 3/3)

Q6 — Why do the offline scores differ? The continually updated-flow variant follows FQL’s original training protocol: the flow teacher, actor, and critic are initialized together and trained jointly throughout offline RL, with the teacher updated via behavior cloning. In contrast, PACE separately pretrains and freezes the teacher before offline actor–critic training begins. The offline checkpoints are therefore natural outcomes of each pipeline, allowing us to evaluate the end-to-end offline-to-online effect of freezing vs. continually updating the generative reference.

Q7 — Larger online interaction budgets. Our primary research question addresses policy improvement under a limited online interaction budget (operationalized here as 500k500\text{k} steps, matching prior O2O benchmarks such as FINO). Whether methods eventually converge or change rank as the budget scales toward multi-million steps is an interesting open question, though without a natural upper cutoff. In the revision, we will explicitly scope our claims to the limited-budget O2O regime and note longer-horizon scaling in Section E (Limitations and Future Work).

Supplementary mathematical response to W1

This note adds the conditional density bound used in W1 to the path-metric and critic-update analysis in Appendix A.

Notation and assumptions. We follow the notation of the main paper and Appendix A, fixing a state ss and reference latent zz. Let p0p_0 denote the Gaussian source density and pθ(a∣s)p_\theta(a\mid s) the frozen teacher's conditional action density. The latter is a modeled density, distinct from the actor's distribution and the true offline behavior distribution.

Assume vθv_\theta and its spatial Jacobian are continuous, and its ODE induces a differentiable invertible flow over t∈[0,1]t\in[0,1]. For this fixed ss, define

ht(x)=∇x⋅vθ(s,x,t).h_t(x)=\nabla_x\cdot v_\theta(s,x,t).

Assume that, on a neighborhood UU containing both generation paths,

∣ht(x)−ht(y)∣≤L∣x−y∣2for all x,y∈U,t∈[0,1].|h_t(x)-h_t(y)|\leq L\\|x-y\\|_2 \quad\text{for all }x,y\in U,\\ t\in[0,1].

This condition concerns spatial variation of the velocity field's divergence. It is stronger than Lipschitz continuity of the velocity field itself. A sufficient condition on a convex UU is

sup⁡t∈[0,1],,x∈U∣∇x(∇x⋅vθ(s,x,t))∣2≤L.\sup_{t\in[0,1],\\,x\in U} \\|\nabla_x(\nabla_x\cdot v_\theta(s,x,t))\\|_2\leq L.

Bounds of this type also appear in density-error analyses, for example Su et al. (2025), Assumption 3.3, which allows a time-dependent bound. Here LL may depend on the fixed state and neighborhood; no uniform bound over all states is assumed.

Conditional density bound. Under these assumptions, the exact continuous flow before clipping satisfies

∣log⁡pθ(Φ1(s,z+δ)∣s)pθ(Φ1(s,z)∣s)∣≤∣z∣2∣δ∣2+12∣δ∣22+LDpath(s,z,δ).\left|\log\frac{p_\theta(\Phi_1(s,z+\delta)\mid s)} {p_\theta(\Phi_1(s,z)\mid s)}\right| \leq \\|z\\|_2\\|\delta\\|_2+\frac12\\|\delta\\|_2^2 +L\sqrt{D_{\mathrm{path}}(s,z,\delta)}.

Setting z=zπz=z_\pi gives the inequality in our main W1 response. The reference zπz_\pi need not itself be sampled from p0p_0; the statement evaluates the teacher density at the two specified actions.

Derivation. The continuous change-of-variables formula gives

log⁡pθ(Φ1(s,z)∣s)=log⁡p0(z)−∫01ht(Φt(s,z)),dt.\log p_\theta(\Phi_1(s,z)\mid s) =\log p_0(z)-\int_0^1h_t(\Phi_t(s,z))\\,\mathrm dt.

This is the standard density identity for continuous normalizing flows; see Chen et al. (2018), Section 4 and Appendix A.

Apply the identity at z+δz+\delta and zz, subtract, and use the triangle inequality:

∣log⁡pθ(Φ1(s,z+δ)∣s)pθ(Φ1(s,z)∣s)∣≤∣log⁡p0(z+δ)p0(z)∣+∫01∣ht(Φt(s,z+δ))−ht(Φt(s,z))∣,dt.\begin{aligned} \left|\log\frac{p_\theta(\Phi_1(s,z+\delta)\mid s)} {p_\theta(\Phi_1(s,z)\mid s)}\right| &\leq \left|\log\frac{p_0(z+\delta)}{p_0(z)}\right|\\\\ &\quad+\int_0^1 |h_t(\Phi_t(s,z+\delta))-h_t(\Phi_t(s,z))|\\,\mathrm dt. \end{aligned}

The Lipschitz condition and Cauchy-Schwarz over the unit interval give

∫01∣ht(Φt(s,z+δ))−ht(Φt(s,z))∣,dt≤L∫01∣Φt(s,z+δ)−Φt(s,z)∣2,dt≤LDpath(s,z,δ).\begin{aligned} \int_0^1|h_t(\Phi_t(s,z+\delta))-h_t(\Phi_t(s,z))|\\,\mathrm dt &\leq L\int_0^1\\|\Phi_t(s,z+\delta)-\Phi_t(s,z)\\|_2\\,\mathrm dt\\\\ &\leq L\sqrt{D_{\mathrm{path}}(s,z,\delta)}. \end{aligned}

For the standard Gaussian source,

log⁡p0(z+δ)p0(z)=−z⊤δ−12∣δ∣22,\log\frac{p_0(z+\delta)}{p_0(z)} =-z^\top\delta-\frac12\\|\delta\\|_2^2,

hence its absolute value is at most ∣z∣2∣δ∣2+12∣δ∣22\\|z\\|_2\\|\delta\\|_2+\frac12\\|\delta\\|_2^2. Combining these inequalities proves the claim.

Thank you for the clarifications and additional experiments. I appreciate the density-change bound, the explanations of the details, and the new experiments comparing with DSRL-NA. My remaining two concerns are:

W2: Using the same η\eta and ρ\rho across metrics does not ensure comparable update magnitudes. I still want to suggest that the authors vary the constraint weights for each kind of constraint, and then plot how their performance changes w.r.t. different weights and whether PACE can Pareto-dominate them. This can help to study whether PACE's advantage indeed comes from path constraints instead of a better step size.

Q6: The authors' reply confirms that the comparison changes both the offline training and the online updates. So the benefit of freezing the teacher during online updates is not fully isolated. I would suggest a comparison starting from the same teacher, actor, and critic checkpoint.

The empirical results are promising, but these concerns about the mechanism behind why PACE performs better are not yet resolved. I therefore maintain my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.