BiFlow: First-Order Bilevel Coarse-to-Fine Flow Matching Policy for Continuous Control

Classical actor-critic methods for continuous control use Gaussian or deterministic MLP actors that produce actions through a single forward pass. While efficient, this one-step state-to-action mapping amortizes critic-based action optimization and can be restrictive in high-dimensional tasks that require iterative action optimization or fine-grained adjustment along the action-value landscape. Diffusion and flow matching policies offer a more expressive alternative by casting action generation as an iterative transport process, but existing methods either employ a monolithic generative actor with shared parameters across the trajectory or decouple coarse proposal and fine refinement with separate objectives. We propose BiFlow, a first-order bilevel coarse-to-fine flow matching policy. At the population level, the lower problem selects a coarse actor under the exact value function of the composed policy, while the upper problem optimizes the refiner; hence the lower solution depends on the refiner through that value function. Our practical solver approximates it with a learned critic, a short $K$-step lookahead, and updates the refiner without unrolling this inner loop. This tractable first-order approximation preserves cross-cycle coupling through the critic trained under the deployed composed policy and the lookahead-induced outer data. We derive a critic comparison for the deployed numerical action, separating weighted flow-matching error, training–deployment source mismatch, and numerical execution error. Empirically, BiFlow achieves strong performance against recent diffusion- and flow-based continuous control baselines.