- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PACE, an O2O RL method that leverages a frozen behavior-cloning flow policy (flow teacher). The authors argue that continually updating a flow teacher with online experience can distort its learned behavioral distribution and destabilize one-step flow policy (student) learning. To address this issue, PACE freezes the pretrained flow teacher and performs learned Q-value guided exploration in its latent noise space to generate improved action targets for the flow student. In particular, the proposed method introduces a path-aware constraint that accounts for the deformation along the entire flow generation trajectory, aiming to restrict latent exploration in directions that induce large path deformations.
Critical strengths
S1 (Interesting motivation). The paper shows that continually updating the flow teacher during online learning can spread its learned behavioral distribution into low-density regions, potentially degrading the performance of the one-step actor. This observation provides a clear motivation for maintaining a frozen flow teacher.
S2 (Path-aware constraints for latent exploration). PACE introduces a path-aware constraint that considers deformation along the entire flow generation trajectory, rather than only at the latent inputs or final actions. The paper derives a quadratic approximation of path deformation, enabling constrained Q-value-guided latent exploration without repeatedly evaluating perturbed flow trajectories.
S3 (Strong empirical results). PACE demonstrates strong empirical performance across 35 OGBench tasks, outperforming existing baselines in average final success rate.
Critical weaknesses
W1 (Unclear contribution and interaction of the two distillation objectives). In Eq. (4), the actor is trained with both the original distillation loss toward the frozen flow teacher () and the additional loss toward the explored action target (). However, is set to 10 or higher across all task families, assigning a substantially larger coefficient to the exploration-target loss. Since the two objectives may encourage the actor to move toward different actions (), it is unclear how they interact during optimization and whether the original distillation term remains necessary. An ablation removing the original distillation term, along with an analysis of the relative loss or gradient magnitudes, would help clarify the contribution of each objective and the motivation for choosing relatively large values of .
W2 (Insufficient ablation studies and implementation details for adaptive latent exploration). The paper lacks sensitivity analyses of important components in adaptive latent exploration, such as the metric trust radius in Eq. (2). Furthermore, the clipping operation in Eq. (3) is not clearly specified, including its clipping bounds and implementation. Additional ablation studies and implementation details would help assess the robustness and reporducibility of the proposed method.
W3 (Increased hyperparameter complexit). Compared with FQL and FINO, PACE introduces several additional hyperparameters, including the latent step size , metric trust radius , metric damping , and exploration-target weight (Tables 3 and 4). In particular, is selected separately for each task family. However, the paper does not provide sufficient sensitivity analyses or explain how these additional hyperparameters were selected. This raises concerns about the robustness of the reported performance and the additional tuning effort required by PACE.
W4 (Limited justification for reducing OOD exploration). The paper argues that path-aware constraints reduce OOD exploration and mitigate the risk of Q-value overestimation. However, the primary goal in O2O RL should be to avoid exploiting overestimated Q-values for OOD actions, rather than reducing OOD exploration itself. In fact, exploring OOD actions can be essential for discovering better behaviors beyond the offline dataset, particularly when the dataset is suboptimal. Therefore, reducing the OOD rate does not necessarily indicate more effective online policy improvement. Further analysis is needed to justify whether the proposed constraints selectively suppress unreliable OOD actions without unnecessarily restricting beneficial exploration.
W5 (Task-dependent benefits of freezing the flow teacher). The motivation for freezing the flow teacher is supported by substantial improvements on Cube-Double-Task4 and Puzzle-4×4-Task4 (Table 5 and Figure 6). However, the frozen teacher variant performs worse than standard FQL on both AntMaze tasks. These results suggest that freezing the flow teacher is not consistently beneficial across different environments. Moreover, maintaining a fixed behavioral reference may restrict the actor from learning new high-value actions that deviate substantially from the offline behavior distribution. Further analysis would help clarify when freezing the flow teacher is beneficial and why continually updating it can destabilize actor learning.
W6 (Insufficient analysis of baseline performance and the contribution of entropy-guided sampling). The continually updated-flow variant of PACE exhibits unstable training on Cube-Double-Task4 and Puzzle-4×4-Task4, resulting in failed runs (Table 2), whereas standard FQL achieves nonzero final performance on Puzzle-4×4-Task4 (Table 5). Further analysis of these training failures would help clarify the source of instability. Moreover, PACE adopts the entropy-guided action selection of FINO, which is not used by several other baselines, including FQL. An additional ablation without this action-selection mechanism would help isolate the contribution of the proposed method and provide a more controlled comparison with these baselines.
W7 (Missing computational cost analysis). PACE requires additional computations, including backward flow integration, flow Jacobian evaluation, path metric construction. Although the authors reduce the computational burden by generating exploration targets for only a subset of each training batch, the paper does not provide quantitative comparisons of training time or GPU memory consumption against FQL and FINO. Such comparisons would help assess the practical efficiency of the proposed method.
Questions for authors
Could you provide empirical results and explanation regarding the Weaknesses above, especially W1, 2, 4, 6, and 7?
5 comments
Response to Reviewer 3 (Part 2/3)
W4 — Selective OOD suppression vs. beneficial exploration. We fully agree that blindly suppressing all deviations from the offline dataset would hinder policy improvement. Crucially, Figure 3 and Appendix C.4 demonstrate that PACE's full-path constraint does not reduce OOD incidence by conservatively pulling actions closer to the teacher. On Cube Double (across 256 states and 1,024 samples/state), switching from start/end to full-path constraints:
- Increases the actor's mean action deviation from the teacher from to (permitting larger exploratory shifts toward higher values), while
- Simultaneously reducing the empirical OOD rate (measured by an independent offline dataset -NN distance ) from to .
Combined with the superior performance demonstrated in Tables 1 and 2, this confirms that full-path geometry selectively penalizes high-shear directions that cross into unsupported regions while granting greater freedom along well-supported transport directions.
W5 — Task-dependent effects of freezing the teacher and limited-budget scope. In Table 5, freezing the flow teacher improves FQL on five of seven Task 4 environments and raises overall mean success from to , notably rescuing Cube Double from catastrophic collapse (). Even on the two AntMaze tasks where frozen-teacher FQL is slightly lower, it still maintains strong performance (), and full PACE recovers and surpasses standard FQL ( on AntMaze Large and on AntMaze Giant, Table 2) via critic-guided latent exploration. Thus, freezing provides a simple yet high-leverage stability foundation across the benchmark.
Regarding whether a fixed reference could limit asymptotic improvement under much larger online budgets: our primary research question addresses policy improvement under a limited online interaction budget (operationalized here as steps, matching prior O2O benchmarks such as FINO). Whether methods eventually converge or change rank as the budget scales toward multi-million steps is an interesting open question, though without a natural upper cutoff. In the revision, we will explicitly scope our claims to the limited-budget O2O regime and note longer-horizon scaling in Section E (Limitations and Future Work).
(Continued in Part 3/3 below.)
Response to Reviewer 3 (Part 3/3)
W6 — Training failures in continually updated flow and contribution of candidate selection.
- Cause of Non-Finite Failures in Continually Updated Flow (Table 2): Unlike standard FQL—which only evaluates forward teacher outputs for distillation—the continually updated-flow variant also performs backward ODE integration () and solves linear systems involving the path Jacobian metric (Eq. 2). When online replay shifts destabilize the unfrozen velocity field on complex manipulation tasks (Cube Double and Puzzle ), backward integration and Jacobian inversion encounter ill-conditioned trajectories, leading to numerical overflow (
NaN/non-finite values). Freezing the pretrained flow eliminates this instability. This further corroborates the internal consistency of our design—namely, that a fixed flow reference is essential for reliable path-metric computation. - Isolating Entropy-Guided Candidate Selection: To isolate the contribution of PACE's core learning mechanism from FINO-style candidate sampling, Table 10 compares PACE with single-sample collection and single-sample evaluation ( across all tasks, matching standard FQL) against full PACE. Even without candidate selection, single-sample PACE achieves strong results across tasks (e.g., on AntMaze Large, on AntMaze Giant, on AntSoccer, on Cube Double, and on Puzzle ). Candidate selection primarily benefits high-dimensional humanoid locomotion (HumanoidMaze Medium/Large), where action dimensionality is substantially higher. We will include Table 10 in the revision to make this distinction explicit.
Table 10: Ablation of candidate selection mechanism (Task 4, 4 seeds, reported as AUC / Final %).
| Task Family | PACE (Single Collection + Single Eval, ) | Paper PACE (-Softmax Collection + -Argmax Eval) |
|---|---|---|
| AntMaze Large | / | / |
| AntMaze Giant | / | / |
| AntSoccer Arena | / | / |
| HumanoidMaze Medium | / | / |
| HumanoidMaze Large | / | / |
| Cube Double | / | / |
| Puzzle | / | / |
W7 — Computational cost analysis. In our target deployment setting, the critical computational cost is inference-time latency rather than training-time compute: distilling into a one-step flow student yields significantly lower inference latency than multi-step iterative flow or diffusion policies, which is essential for real-time, safety-critical scenarios such as high-frequency robotic control. Because backward integration and path-metric computation in PACE are strictly confined to training updates, execution/inference uses the exact same one-step actor architecture as FQL and FINO—fully preserving the inference-speed advantage of FQL-style policies with zero added latency. Even during training, benchmarking on the same GPU under identical batch settings shows that computing exploration targets on a sub-batch of 32 examples adds only per update over FQL ( vs. ) and over FINO ( vs. ), with a modest increase in peak GPU memory ( vs. ).
Response to Reviewer 3 (Part 1/3)
We thank Reviewer 3 for the constructive review and insightful suggestions. We have conducted the requested ablations, sensitivity analyses, and computational benchmarks, which we detail below and will incorporate into the revised manuscript. Unless specified otherwise, results in the additional experiments below are four-seed mean final success rates on Task 4 after 500k online interactions.
W1 — Contribution and interaction of the two distillation objectives. Table 2 already evaluates removing the explored target aimp (dropping mean success by 12.1 across tasks). To directly address the reviewer’s question, we ran the complementary ablation removing only the original distillation loss toward the frozen teacher output aB while retaining the direct Q objective and the explored target aimp. Removing the original distillation term nearly eliminated online success on both Cube Double and Puzzle 4×4, confirming that aB remains indispensable as a global behavioral anchor.
Gradient diagnostics during training explain why a relatively large β (β≥10) coexists harmoniously with this anchor rather than overwhelming it:
W2 & W3 — Implementation details, hyperparameter selection, and sensitivity analyses.
Table 8: Sensitivity to exploration-target weight β (Final online success %, 4 seeds on Task 4).
Table 9: Sensitivity to path-exploration hyperparameters ρ, η, and λ (4 seeds on Task 4, reported as AUC / Final %).
(Continued in Part 2/3 below.)