acceptodds
Under review as a conference paper at ICLR 2027

BiFlow: First-Order Bilevel Coarse-to-Fine Flow Matching Policy for Continuous Control

Abstract

Classical actor-critic methods for continuous control use Gaussian or deterministic MLP actors that produce actions through a single forward pass. While efficient, this one-step state-to-action mapping amortizes critic-based action optimization and can be restrictive in high-dimensional tasks that require iterative action optimization or fine-grained adjustment along the action-value landscape. Diffusion and flow matching policies offer a more expressive alternative by casting action generation as an iterative transport process, but existing methods either employ a monolithic generative actor with shared parameters across the trajectory or decouple coarse proposal and fine refinement with separate objectives. We propose BiFlow, a first-order bilevel coarse-to-fine flow matching policy. At the population level, the lower problem selects a coarse actor under the exact value function of the composed policy, while the upper problem optimizes the refiner; hence the lower solution depends on the refiner through that value function. Our practical solver approximates it with a learned critic, a short -step lookahead, and updates the refiner without unrolling this inner loop. This tractable first-order approximation preserves cross-cycle coupling through the critic trained under the deployed composed policy and the lookahead-induced outer data. We derive a critic comparison for the deployed numerical action, separating weighted flow-matching error, training–deployment source mismatch, and numerical execution error. Empirically, BiFlow achieves strong performance against recent diffusion- and flow-based continuous control baselines.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, diffusion policy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vLPXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BiFlow in the control problem, which combines a coarse actor with a flow-matching refine. During training, a temporary actor is updated through several Q-guided steps to provide starting actions. The flow then learns to move these actions toward better targets found through local Q optimization. The authors formulate this as a bilevel problem and use a first-order approximation. Experiments on eight MuJoCo and DeepMind Control Suite tasks report improvements over the compared baselines.

Critical strengths

I think the coarse-to-fine idea is simple and reasonable. The actor provides an initial action, and the flow learns to refine it toward higher-value actions. This is a sensible design for making fine-grained action adjustments, especially when small changes in the action matter.

Critical weaknesses
  1. The two-level training introduces additional computational cost, which makes its practical benefit less clear.

  2. The relationship to similar approaches that shape the prior distribution of flow matching, such as DSRL, is not discussed systematically enough.

  3. The method description takes up too much space for an idea that could be explained more directly. The experimental section is consequently too compressed, with insufficient analysis in the main paper.

Questions for authors

I feel that the selected tasks do not directly support the authors’ main insight. Could you provide more examples showing the benefits of action refinement?

4 comments
AuthorsOpenReview ↗

Responses to Reviewer vLPX 1/2

We appreciate the reviewer's careful evaluation and constructive feedback. Our responses are provided below.

W1: The two-level training introduces additional computational cost, which makes its practical benefit less clear.

We agree that conventional bilevel optimization can introduce substantial computational overhead, particularly when computing the response gradient. However, a core contribution of BiFlow is making bilevel optimization computationally practical for diffusion/flow-based RL.

This is achieved through two design choices. First, our first-order approximation avoids higher-order differentiation through the inner optimization steps. Second, the refiner is trained via flow matching without backpropagating through flow integration when updating the refiner. At inference, BiFlow only requires one coarse actor forward pass and 8 Euler steps, without inner-loop optimization.

As shown in Table 3, BiFlow has a training cost comparable to SDAC:

MethodTime per Cycle (ms)
SDAC30.5
BiFlow31.0

Despite the comparable training cost, BiFlow achieves higher mean returns across all eight tasks, with substantial improvements over QFlex:

TaskBiFlowQFlexImprovement
Ant-v45989.6 ± 125.74880.8 ± 294.5+22.7%
Dog-run590.4 ± 42.7326.1 ± 25.1+81.0%

These results demonstrate that our first-order bilevel approximation delivers substantial performance gains with manageable computational overhead.


W2: The relationship to similar approaches that shape the prior distribution of flow matching, such as DSRL, is not discussed systematically enough.

Thank you for your suggestion. DSRL learns a latent-noise steering policy for a frozen pretrained diffusion policy, while BiFlow jointly learns a coarse actor and flow refiner from scratch through online RL. We have added a systematic discussion of DSRL and its differences from BiFlow to Section 2.3.

AuthorsOpenReview ↗

Responses to Reviewer vLPX 2/2

Q1: I feel that the selected tasks do not directly support the authors' main insight. Could you provide more examples showing the benefits of action refinement?

Thank you for this constructive comment. We have added a synthetic experiment to Appendix E to further illustrate our main insights. First, action refinement reduces the burden of transporting actions from the source to the targets so that the refiner can focus on local action improvements and limit the effect of fitting error on final action accuracy. Second, bilevel optimization couples the coarse actor and refiner so that source selection and subsequent refinement are jointly optimized using the value of the composed policy.

The experiment is a state-conditioned bandit with state s∈[−1,1]s \in [-1, 1] and two-dimensional action a∈R2a \in \mathbb{R}^2. At each state, the closed-form QQ-function has a broad, low peak (σa=0.5\sigma_a = 0.5, value Q≈1.0Q \approx 1.0) along a smooth arc and a narrow, high peak (σb=0.04\sigma_b = 0.04, value Q≈2.7Q \approx 2.7) along a neighboring arc with high-frequency local perturbations. The two peak centers are separated by approximately 0.40.4. Q-guided FM, QFlex, and BiFlow use the exact QQ-function and its action gradients. Q-guided FM trains a velocity field from a fixed state-conditioned coarse source using Q-guided improvement targets. The table below reports results over five random seeds, where dist. is the mean Euclidean distance from the deployed action to the narrow-peak center and hit @0.04 is the fraction of states with a distance below 0.040.04. The visualization is provided in Figure 3, and the task definition, training configurations, and supplementary results are provided in Appendix E.

Methoddist. ↓hit @0.04 ↑
Q-guided FM0.0261 ± 0.005586.1%
QFlex0.4012 ± 0.00220.0%
BiFlow0.0020 ± 0.0008100.0%

Q-guided FM reaches the narrow-peak region but retains a larger localization error than BiFlow. BiFlow's coarse actions lie near the narrow peak, and the refiner moves them closer to its center. Refinement reduces the mean distance from 0.02100.0210 to 0.00200.0020 by approximately 90.7%90.7\% and increases the mean QQ-value from 2.44872.4487 to 2.71432.7143. These results demonstrate the benefit of local action refinement.

QFlex also uses a coarse policy and a refiner. Its coarse policy optimizes its own action values, so subsequent refinement gains do not directly guide source selection. BiFlow updates its coarse actor using refined-action values to produce coarse actions suited to refinement. These coarse actions help the refiner learn local adjustments near the narrow peak. The two modules reinforce each other to localize deployed actions at the narrow peak and illustrate the benefit of their coupled learning.

Furthermore, on Ant-v4, using the refiner only at evaluation reduces the return by 25.3%25.3\% (Table 2), demonstrating the benefit of online action refinement on a standard continuous-control benchmark.


W3: The method description takes up too much space for an idea that could be explained more directly. The experimental section is consequently too compressed, with insufficient analysis in the main paper.

Thank you for your suggestion. We have simplified the method section by consolidating equations and focusing on the core algorithmic steps. We have added statistical significance tests (Table 4) to the experimental section.

AuthorsOpenReview ↗

Follow-up to Reviewer vLPX

Dear Reviewer vLPX,

Thank you again for your constructive feedback. As the discussion period is approaching its end, we would be happy to provide any further clarifications that may be helpful in addressing your remaining questions or concerns.

Best regards,

The Authors

Thank you for the detailed response and the additional experiment. I appreciate your efforts. The synthetic bandit illustrates how BiFlow can refine actions in a controlled setting. I still find the evidence limited on when fine-grained action refinement explains the gains observed on standard control benchmarks, so I maintain my original decision.

Reviewer_PZnUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BiFlow, a coarse-to-fine actor for off-policy online RL, in which a deterministic coarse actor proposes an action and a flow-matching refiner transports it to the executed action. The two stages are framed as a bilevel problem coupled through the critic of the composed policy, and solved with a first-order scheme that combines a temporary lookahead of the coarse actor, Q-ascent targets for the refiner, and a FOMAML-style update of the retained coarse actor. The authors also provide a Gronwall-type bound relating the deployed action's value to flow-matching and source-mismatch errors. On five MuJoCo and three DMC tasks, BiFlow reports the best final return on all eight tasks.

Critical strengths
  1. BiFlow shows clear gains over strong flow- and diffusion-based baselines, especially on harder locomotion tasks such as Ant and Dog-run.
  2. The additional training cost is modest compared with diffusion-based baselines, and the theoretical analysis is clear about its scope.
Critical weaknesses

Main

  1. The method is very close to QFlex [1]. QFlex already learns a source policy, builds targets by Q-gradient ascent from it, and trains the velocity field with conditional flow matching. The improvements over QFlex need further ablation, e.g., removing the lookahead or updating the coarse actor with its own Q-value, to better clarify the contribution.
  2. Several strong and directly relevant baselines are missing, including FlowRL [2] and FPMD [3], which the paper itself discusses, and modern Gaussian baselines such as CrossQ [4] and BRO [5]. Without them, the claim of state-of-the-art performance is insufficiently supported.
  3. The fairness of the comparison is unclear. BiFlow uses its own update-to-data ratio and tuned settings, while baselines keep their defaults, and some rerun baselines, such as DPMD, fall well below their published results [6]. The authors should report UTD ratios, network sizes, and tuning budgets for all methods.

Minor

The Reproducibility Statement is empty, and no code is provided.

[1] Wei, Yunyue, Chenhui Zuo, and Yanan Sui. "Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flow." ICLR 2026. [2] Lv, Lei, et al. "Flow-Based Policy for Online Reinforcement Learning." NeurIPS 2025. [3] Chen, Tianyi, et al. "One-Step Flow Policy Mirror Descent." arXiv:2507.23675 (2025). [4] Bhatt, Aditya, et al. "CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity." ICLR 2024. [5] Nauman, Michal, et al. "Bigger, Regularized, Optimistic: Scaling for Compute and Sample-Efficient Continuous Control." NeurIPS 2024. [6] Ma, Haitong, et al. "Efficient Online Reinforcement Learning for Diffusion Policy." ICML 2025.

Questions for authors
  1. Could the authors provide confidence intervals for the main results, and explain how they are computed? On several tasks, such as Hopper and Quadruped-walk, the margins are small relative to the standard deviation in Table 1, and it is unclear whether they exceed seed noise.
4 comments
AuthorsOpenReview ↗

Responses to Reviewer PZnU 1/2

We thank the reviewer for the thoughtful comments and helpful suggestions. We address each point in the following.

W1: Further evaluations are needed to distinguish BiFlow from QFlex.

We would like to clarify that our core contribution, the bilevel optimization formulation, distinguishes BiFlow from QFlex. QFlex optimizes the source policy and flow refiner with separate objectives, while BiFlow formulates their learning as a bilevel problem (Sec. 3.1), where the lower-level coarse policy depends on the upper-level refiner through the value function of the composed policy.

Under this bilevel formulation, our practical first-order solver introduces two key components that distinguish BiFlow from QFlex. i) Refiner-aware coarse update. The coarse actor is updated with the value of the refined action. ii) Lookahead source. The refiner is trained on a lookahead source that approximates the lower-level solution.

To validate these two components, we conducted two additional ablations on Ant-v4 (5 seeds).

MethodFinal Return
BiFlow5989.6 ± 125.7
Coarse update using its own Q-value5219.1 ± 147.4
Without lookahead (Kin=0K_{\mathrm{in}}=0)4876.5 ± 215.2

Removing either component leads to a substantial performance drop, by 12.9% for the refiner-aware coarse update and 18.6% for the lookahead source. These results confirm that the two components derived from our bilevel formulation are essential to the performance of BiFlow and set it apart from QFlex.

We have also added a synthetic experiment to Appendix E to illustrate how the differences between BiFlow and QFlex affect optimization behavior. The task has a closed-form Q-function containing a broad, low peak and a narrow, high peak at each state. The table below reports results over five random seeds. The metric dist. is the mean distance from the deployed action to the narrow-peak center for the corresponding state, and hit @0.04 is the fraction of states with a distance below 0.040.04. The visualization is provided in Figure 3 of the revised manuscript.

Methoddist. ↓hit @0.04 ↑
QFlex0.4012 ± 0.00220.0%
BiFlow0.0020 ± 0.0008100.0%

QFlex's coarse and deployed actions both remain around the broad peak. BiFlow's coarse actions lie near the narrow peak, and the refiner moves them closer to its center. These results further illustrate the differences between BiFlow and QFlex.

The comparison with QFlex is reported in Appendix D.3 and the synthetic experiment details are provided in Appendix E.


W2: Several strong and directly relevant baselines are missing, including FlowRL and FPMD, which the paper itself discusses, and modern Gaussian baselines such as CrossQ and BRO.

We have conducted additional experiments with FlowRL [1], a directly relevant flow-based RL baseline. As shown below, BiFlow achieves higher mean returns on all eight tasks.

TaskBiFlowFlowRL
Hopper-v43468.5 ± 115.92714.5 ± 204.6
Walker2d-v44529.0 ± 487.44127.0 ± 457.6
Swimmer-v4136.4 ± 14.649.5 ± 12.3
Ant-v45989.6 ± 125.74708.9 ± 318.1
Humanoid-v46545.0 ± 327.25448.9 ± 478.1
Quadruped-walk961.2 ± 11.1889.8 ± 12.3
Dog-walk896.5 ± 15.0829.5 ± 22.5
Dog-run590.4 ± 42.7426.7 ± 39.5

We have added the FlowRL results to Appendix D.5.

FPMD [2] focuses on efficient one-step flow inference. Its original paper reports that DPMD outperforms FPMD on several MuJoCo tasks. We have already included DPMD and SDAC as representative baselines.

CrossQ [3] has been directly compared with QFlex, which reports superior performance on its high-dimensional control benchmarks. We have already included QFlex as a directly relevant baseline.

BRO [4] primarily improves performance through critic scaling, regularization, and optimistic exploration, using substantially different computational configurations. These improvements are complementary to our focus on flow-based policy optimization, so we prioritize comparisons with methods under more closely aligned training settings.

AuthorsOpenReview ↗

Responses to Reviewer PZnU 2/2

W3: The fairness of the comparison is unclear. BiFlow uses its own update-to-data ratio and tuned settings, while baselines keep their defaults, and some rerun baselines, such as DPMD, fall well below their published results. The authors should report UTD ratios, network sizes, and tuning budgets for all methods.

We thank the reviewer for raising this concern. All methods use the same update-to-data ratio (UTD = 0.2). BiFlow uses a single hyperparameter configuration across all tasks (Appendix Tables 5–6), without task-specific tuning. For baselines, we retain the method-specific default configurations from their public implementations without additional tuning. The network architectures are summarized below.

MethodUTDActor / Generative NetworkCritic
BiFlow0.2Coarse: 2 × 256; FM: 2 × 256Twin Q: 3 × 256 each
TD30.22 × 256Twin Q: 3 × 256 each
QVPO0.2Noise: 2 × 256; Encoder: 2 × 128Twin Q: 3 × 256 each
QSM0.2Noise: 3 × 256; Encoder: 2 × 128Twin Q: 3 × 256 each
DACER0.2Noise: 3 × 256; Encoder: 2 × 128Twin Gaussian Q: 3 × 256 each
SDAC0.2Noise: 3 × 256; Encoder: 2 × 128Twin Q: 3 × 256 each
DPMD0.2Noise: 3 × 256; Encoder: 2 × 128Twin Q: 3 × 256 each
QFlex0.2Coarse: 2 × 256; FM: 2 × 256Twin Q: 3 × 256 each
FlowRL0.2Flow: 2 × 512Twin Q, buffer twin Q, V: 4 × 512 each

Regarding the discrepancy between our reproduced DPMD results and the original paper, two factors should be considered. First, the original paper reports the best mean returns during training, whereas we report final returns under a unified evaluation protocol. Second, our experiments use a public unified implementation (Appendix D), whose configurations may differ from the original implementation. The original paper also uses 5M environment steps for Humanoid, compared with our unified 1M-step budget.

Importantly, even when compared with DPMD's originally reported best mean returns, BiFlow achieves higher mean returns on all four shared tasks below:

TaskDPMD (Reported Best)BiFlow (Final)
Hopper-v43275 ± 553468.5 ± 115.9
Walker2d-v44365 ± 2664529.0 ± 487.4
Swimmer-v479.3 ± 52.5136.4 ± 14.6
Ant-v45683 ± 1385989.6 ± 125.7

While these cross-paper results use different reporting protocols, they provide additional evidence of BiFlow's competitiveness beyond our unified reproductions.

We have expanded Appendices C and D.1 to include the configurations and tuning protocol for both BiFlow and the baselines.


Q1: Confidence intervals and statistical tests are needed to distinguish performance gains from variability across random seeds.

We have added the 95% confidence intervals of the main results in the table below. The intervals are computed over 5 random seeds using the Student-t distribution, i.e., mean±t0.975,4⋅s/5\text{mean} \pm t_{0.975,4} \cdot s/\sqrt{5}. Since our hypothesis is that BiFlow improves over the baselines, we conducted a Welch's t-test between BiFlow and the baseline with the highest mean final return in Table 1 of the revised manuscript on each task.

TaskBiFlow 95% CIStrongest baseline 95% CIp-value
Hopper3468.5 [3324.6, 3612.4]DPMD 3244.9 [3008.5, 3481.3]0.031
Walker2d4529.0 [3923.8, 5134.2]QFlex 3916.5 [3550.6, 4282.4]0.025
Swimmer136.4 [118.3, 154.5]QFlex 120.5 [117.0, 124.0]0.035
Ant5989.6 [5833.5, 6145.7]QFlex 4880.8 [4515.1, 5246.5]< 0.001
Humanoid6545.0 [6138.7, 6951.3]QFlex 5966.5 [5660.7, 6272.3]0.007
Quadruped-walk961.2 [947.4, 975.0]QFlex 941.2 [918.7, 963.7]0.038
Dog-walk896.5 [877.9, 915.1]QFlex 836.6 [798.2, 875.0]0.004
Dog-run590.4 [537.4, 643.4]QFlex 326.1 [294.9, 357.3]< 0.001

BiFlow obtains the highest mean final return on all eight tasks. The task-wise tests yield p<0.05p<0.05 on all eight tasks. We have added a statistical analysis in Appendix D.4, with the complete confidence intervals and test results reported in Table 4.


The Reproducibility Statement is empty.

Thank you for checking this carefully. We have added a Reproducibility Statement to the revised manuscript.


References

[1] Lei Lv et al. Flow-Based Policy for Online Reinforcement Learning. NeurIPS, 2025.

[2] Tianyi Chen et al. One-Step Flow Policy Mirror Descent. arXiv:2507.23675, 2025.

[3] Aditya Bhatt et al. CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity. ICLR, 2024.

[4] Michal Nauman et al. Bigger, Regularized, Optimistic: Scaling for Compute and Sample-Efficient Continuous Control. NeurIPS, 2024.

Reviewer_PZnUOpenReview ↗

Response to authors

Thanks to the authors for the additional experiments and clarifications. The new ablations, baseline comparisons, and explanations of UTD, network architectures, and confidence intervals address most of my concerns and strengthen the credibility of the results.

However, the evaluated action spaces are relatively low-dimensional and do not cover the high-dimensional continuous control settings emphasized in QFlex, such as musculoskeletal control tasks with 80- and 700-dimensional action spaces. The current evidence is therefore insufficient to establish whether the advantages over QFlex extend to these settings. I am inclined to maintain my original score of 2.

AuthorsOpenReview ↗

Responses to Reviewer PZnU

We sincerely thank you for your thoughtful follow-up and constructive feedback. We are pleased that our previous responses have addressed most of your concerns.


Q: The evaluated action spaces are relatively low-dimensional and do not cover high-dimensional continuous control settings, such as musculoskeletal control tasks with 80- and 700-dimensional action spaces.

To further address this concern, following your suggestion, we have added a direct comparison between BiFlow and QFLEX on MyoLeg-Walk, an 80-dimensional musculoskeletal control task evaluated in the QFLEX paper. With fewer than 24 hours remaining before the rebuttal deadline, we prioritized completing this direct comparison and conducted five independent runs for each method.

Since the QFLEX paper reports learning curves but does not provide exact numerical returns for this task, we reran QFLEX using the same training budget and evaluation protocol as BiFlow to enable a direct quantitative comparison.

TaskAction DimensionQFLEXBiFlow
MyoLeg-Walk8013,235.10 ± 907.1514,954.57 ± 941.04

Our reproduced QFLEX result (13,235.10) is consistent in magnitude with the ~13,000 return indicated by the MyoLeg-Walk learning curve in the original QFLEX paper (Fig. 2(b)). BiFlow achieves approximately 13.0% higher mean return than QFLEX.

For a consistent comparison, we apply the same checkpoint-selection rule to both methods. For each method, we take the highest evaluation return observed during training for each seed (as in SDAC and DPMD, Ma et al., 2025) and report the mean and sample standard deviation of these five returns.

For a fair comparison, we matched the hidden-layer sizes of the coarse/source network, flow velocity network, and critic (three hidden layers of 256 units each), as well as the learning rates. We also follow QFLEX's sampling and update schedule: both methods use 80 parallel environments and perform one training update after each parallel sampling step (i.e., after collecting 80 transitions), corresponding to a UTD ratio of (1/80=0.0125). This differs from the UTD ratio in our MuJoCo experiments due to the different parallel sampling configurations. Both methods are trained for 20M total environment steps, counted across all parallel environments. Full configuration details are provided in Appendix D.6.

We hope this additional experiment provides direct evidence of BiFlow's effectiveness on the 80-dimensional musculoskeletal control task and helps address your remaining concern.

In light of this new evidence, we would greatly appreciate it if you could reconsider your original score. We would also be happy to clarify any remaining questions.

Thank you again for your time and constructive feedback!

Reviewer_PnrVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes BiFlow, a flow policy algorithm that uses coarse-to-fine action generation formulated under the bilevel optimization framework. The lower-level actor outputs a coarse action candidate and the upper-level actor refines it. For policy learning, a few inner gradient ascent lookahead steps are first performed. The refiner is then trained using a weighted regression loss, with the coarse actions detached. The coarse parameters are then updated with gradient ascent through the composed policy. Empirical results on MuJoCo and DMC tasks demonstrate superior performance to existing diffusion and flow policy baselines.

Critical strengths
  1. The experimental setup is relatively complete. The included baselines cover a wide range of strong diffusion and flow policy baselines and the evaluation includes both MuJoCo and DMC tasks.
  2. The paper is well-organized and clearly written.
Critical weaknesses
  1. The proposed method does not fully align with the bilevel optimization framework. One core problem in bilevel optimization lies in estimating the response gradient ∂fupper∂ϕ∗∂ϕ∗∂ψ\frac{\partial f_{\mathrm{upper}}}{\partial \phi^{\ast}}\frac{\partial \phi^{\ast}}{\partial \psi}, but the method omits this term in the outer step without justifying why this approximation is accurate.
  2. The paper also did not discuss why the bilevel optimization formulation is necessary and why it is better than a simpler single level optimization formulation max⁡π1,π2Es∼D[Q(s,π2(s,π1(s)))]\max_{\pi_1, \pi_2}\mathbb{E}_{s \sim \mathcal{D}}\left[Q\left(s, \pi_2(s, \pi_1(s))\right)\right].
Questions for authors
  1. The experiments only use 5 MuJoCo tasks while the compared baselines (e.g. DACER and DPMD) are usually evaluated on 8-10 MuJoCo tasks in their original papers. How are the tasks selected in this work?
  2. More related work should be discussed. Currently the paper only contains related work on diffusion and flow policies in Section 2. Related work on bilevel optimization should also be included given its close relevance to the methodology in this paper.
  3. The coarse parameter update in Equations 11 and 12 requires backpropagation through the flow sampling process, which can be unstable and inefficient. Are there ways to avoid this?
  4. Minor issue. Page 14 is blank and should be removed.
4 comments
AuthorsOpenReview ↗

Responses to Reviewer PnrV 1/2

We appreciate the reviewer's detailed review and valuable feedback. We provide our responses to the specific questions below.

W1: Omitting the response gradient in the outer step needs justification.

Omitting the response gradient in the outer step (Eq. (10)) is a practical first-order approximation to the bilevel formulation. Approximate approaches have been widely explored in related optimization settings, including Unrolled GAN [1], DARTS [2], and SCORER [3].

In BiFlow, computing the exact response gradient is challenging. The lower-level solution depends on the refiner through the value function of the composed policy, which is itself approximated by a learned critic. Accurately capturing this dependency would therefore additionally require differentiating through the critic learning process. For this reason, we adopt a first-order approximation.


W2: Why is the bilevel optimization formulation necessary instead of a simpler single-level formulation?

Thank you for this thoughtful question. We address it from the perspectives of optimization and coordination between the coarse actor and refiner.

Optimization perspective.

First, directly optimizing the proposed single-level objective requires backpropagating Q-gradients through the entire flow integration process to update the refiner. This can be costly and unstable [4, 5, 6]. Instead, BiFlow constructs Q-guided targets and trains the refiner via weighted conditional flow matching, avoiding such backpropagation for refiner updates.

Second, source selection and refinement serve different roles, and the effectiveness of refinement depends on the starting points provided by the coarse actor. A coarse actor that provides starting points near high-value regions reduces the transport burden from the source to the targets, allowing the refiner to focus on local adjustments and limit the effect of fitting error on final action accuracy. Since starting-point selection affects the difficulty of refinement, the two stages need coordinated learning.

Coordination perspective.

The bilevel formulation explicitly captures the coupling between coarse source selection and refinement. The lower level selects high-value sources under the composed-policy value function, making source selection refiner-dependent. The upper level then trains the refiner on these sources.

The synthetic experiment in Appendix E illustrates this coordination on a state-conditioned bandit with a broad, low peak and a narrow, high peak. Local variations in the narrow-peak center require precise action localization. The table reports results for three methods with different optimization strategies. The metric dist. is the mean distance from the deployed action to the corresponding narrow-peak center, and hit @0.04 is the fraction of states for which this distance is below 0.040.04.

MethodCoarse/source updatedist. ↓hit @0.04 ↑
Single-stage Q-guided FMFixed source0.0261 ± 0.005586.1%
QFlexCoarse-action values0.4012 ± 0.00220.0%
BiFlowRefined-action values0.0020 ± 0.0008100.0%

Single-stage Q-guided Flow Matching reaches the narrow-peak region but retains a larger localization error than BiFlow. QFlex is a representative method that trains its coarse policy and refiner with separate objectives without bilevel optimization. Its coarse policy optimizes its own action values without feedback from refinement gains and remains attracted to the broad peak. BiFlow uses refined-action values to guide source selection near the narrow peak. These results support coupling source selection and refinement through bilevel optimization.

AuthorsOpenReview ↗

Responses to Reviewer PnrV 2/2

Q1: How are the tasks selected in this work?

We replaced some MuJoCo tasks on which all methods perform near saturation with higher-dimensional and more challenging tasks (e.g., Dog-walk and Dog-run from the DeepMind Control Suite).

To address this concern, we conducted new experiments on two additional MuJoCo tasks, InvertedDoublePendulum-v4 and Pusher-v4 (table below). The methods obtain similar returns on InvertedDoublePendulum-v4, and BiFlow performs comparably to the strongest baseline on Pusher-v4. We have added these results to the Appendix D.2.

MethodInvertedDoublePendulum-v4Pusher-v4
BiFlow9347.92 ± 5.94−21.66 ± 1.38
TD39341.56 ± 3.33−21.95 ± 1.25
QVPO9355.01 ± 1.80−23.53 ± 1.70
QSM9356.85 ± 0.51−24.66 ± 2.80
DACER9328.70 ± 5.51−21.06 ± 0.66
SDAC9348.02 ± 6.07−29.03 ± 1.21
DPMD9349.60 ± 7.25−27.36 ± 1.90

Q2: Related work on bilevel optimization should also be included.

Thank you for this constructive suggestion. We have expanded Section 2 with a discussion of related work on bilevel optimization, including Unrolled GAN [1], DARTS [2], and SCORER [3].


Q3: The coarse parameter update in Equations 11 and 12 requires backpropagation through the flow sampling process, which can be unstable and inefficient. Are there ways to avoid this?

One alternative to avoid backpropagation through flow integration is to update the coarse actor using Q(s,cϕ(s))Q(s,c_\phi(s)). We evaluate this alternative on Ant-v4:

MethodFinal Return (Ant-v4)
BiFlow (Ours)5989.6 ± 125.7
Coarse actor updated with Q(s,cϕ(s))Q(s,c_\phi(s))5219.1 ± 147.4 (−12.9%)

In our experiments, we did not observe notable training instability from retaining backpropagation through flow integration. We also compare the computational cost per training cycle:

MethodTime per Cycle (ms)
BiFlow (Ours)31.0
SDAC30.5

These results suggest that retaining this gradient path provides substantial performance benefits while remaining stable and computationally practical.


References

[1] Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled Generative Adversarial Networks. ICLR, 2017.

[2] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable Architecture Search. ICLR, 2019.

[3] Fernando Martinez, Yingdong Lu, Tao Li, and Juntao Chen. Stackelberg Coupling of Online Representation Learning and Reinforcement Learning. ICLR, 2026.

[4] Seohong Park, Qiyang Li, and Sergey Levine. Flow Q-Learning. ICML, 2025.

[5] Perry Dong, Qiyang Li, Dorsa Sadigh, and Chelsea Finn. EXPO: Stable Reinforcement Learning with Expressive Policies. ICLR, 2026.

[6] Jaehyeok Doo, Byeongguk Jeon, Seonghyeon Ye, Kimin Lee, and Minjoon Seo. Q-Flow: Stable and Expressive Reinforcement Learning with Flow-based Policy. ICML, 2026.

AuthorsOpenReview ↗

Follow-up to Reviewer PnrV

Dear Reviewer PnrV,

Thank you again for your thoughtful feedback. With less than 24 hours remaining in the discussion period, we would be grateful for any further feedback you may have on whether our responses have adequately addressed your concerns.

Best regards,

The Authors

Thank you for the detailed responses and the additional experiments. My concerns have been partially addressed. However, I still find the motivation for the bilevel formulation and its methodological contribution limited. I will therefore keep my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers

Dear Reviewers,

We thank all reviewers for the constructive feedback and valuable suggestions. We have conducted additional experiments and revised the manuscript to address the concerns raised in the reviews. The main updates are summarized below.

  1. Bilevel formulation and first-order approximation: We clarify the motivation for bilevel optimization, justify the first-order approximation, and expand the related work, as discussed in our responses to Reviewer PnrV (W1, W2, Q2).

  2. Additional MuJoCo evaluations: We conduct experiments on InvertedDoublePendulum-v4 and Pusher-v4 to broaden the benchmark coverage, as discussed in our response to Reviewer PnrV (Q1).

  3. Comparison with QFlex: We conduct additional ablations on refiner-aware coarse updates and critic-guided lookahead to clarify the key differences between BiFlow and QFlex, as discussed in our response to Reviewer PZnU (W1).

  4. Additional baselines and evaluation fairness: We add FlowRL comparisons, clarify training configurations and evaluation protocols, and provide confidence intervals and statistical tests, as discussed in our responses to Reviewer PZnU (W2, W3, Q1).

  5. Action refinement analysis: We introduce a synthetic experiment to illustrate the benefits of flow refinement and bilevel coupling, and highlight the online refiner ablation on Ant-v4, as discussed in our response to Reviewer vLPX (Q1).

We hope these revisions address your concerns. Further comments and suggestions are always welcome.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.