- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes BiFlow in the control problem, which combines a coarse actor with a flow-matching refine. During training, a temporary actor is updated through several Q-guided steps to provide starting actions. The flow then learns to move these actions toward better targets found through local Q optimization. The authors formulate this as a bilevel problem and use a first-order approximation. Experiments on eight MuJoCo and DeepMind Control Suite tasks report improvements over the compared baselines.
Critical strengths
I think the coarse-to-fine idea is simple and reasonable. The actor provides an initial action, and the flow learns to refine it toward higher-value actions. This is a sensible design for making fine-grained action adjustments, especially when small changes in the action matter.
Critical weaknesses
-
The two-level training introduces additional computational cost, which makes its practical benefit less clear.
-
The relationship to similar approaches that shape the prior distribution of flow matching, such as DSRL, is not discussed systematically enough.
-
The method description takes up too much space for an idea that could be explained more directly. The experimental section is consequently too compressed, with insufficient analysis in the main paper.
Questions for authors
I feel that the selected tasks do not directly support the authors’ main insight. Could you provide more examples showing the benefits of action refinement?
4 comments
Responses to Reviewer vLPX 2/2
Q1: I feel that the selected tasks do not directly support the authors' main insight. Could you provide more examples showing the benefits of action refinement?
Thank you for this constructive comment. We have added a synthetic experiment to Appendix E to further illustrate our main insights. First, action refinement reduces the burden of transporting actions from the source to the targets so that the refiner can focus on local action improvements and limit the effect of fitting error on final action accuracy. Second, bilevel optimization couples the coarse actor and refiner so that source selection and subsequent refinement are jointly optimized using the value of the composed policy.
The experiment is a state-conditioned bandit with state and two-dimensional action . At each state, the closed-form -function has a broad, low peak (, value ) along a smooth arc and a narrow, high peak (, value ) along a neighboring arc with high-frequency local perturbations. The two peak centers are separated by approximately . Q-guided FM, QFlex, and BiFlow use the exact -function and its action gradients. Q-guided FM trains a velocity field from a fixed state-conditioned coarse source using Q-guided improvement targets. The table below reports results over five random seeds, where dist. is the mean Euclidean distance from the deployed action to the narrow-peak center and hit @0.04 is the fraction of states with a distance below . The visualization is provided in Figure 3, and the task definition, training configurations, and supplementary results are provided in Appendix E.
| Method | dist. ↓ | hit @0.04 ↑ |
|---|---|---|
| Q-guided FM | 0.0261 ± 0.0055 | 86.1% |
| QFlex | 0.4012 ± 0.0022 | 0.0% |
| BiFlow | 0.0020 ± 0.0008 | 100.0% |
Q-guided FM reaches the narrow-peak region but retains a larger localization error than BiFlow. BiFlow's coarse actions lie near the narrow peak, and the refiner moves them closer to its center. Refinement reduces the mean distance from to by approximately and increases the mean -value from to . These results demonstrate the benefit of local action refinement.
QFlex also uses a coarse policy and a refiner. Its coarse policy optimizes its own action values, so subsequent refinement gains do not directly guide source selection. BiFlow updates its coarse actor using refined-action values to produce coarse actions suited to refinement. These coarse actions help the refiner learn local adjustments near the narrow peak. The two modules reinforce each other to localize deployed actions at the narrow peak and illustrate the benefit of their coupled learning.
Furthermore, on Ant-v4, using the refiner only at evaluation reduces the return by (Table 2), demonstrating the benefit of online action refinement on a standard continuous-control benchmark.
W3: The method description takes up too much space for an idea that could be explained more directly. The experimental section is consequently too compressed, with insufficient analysis in the main paper.
Thank you for your suggestion. We have simplified the method section by consolidating equations and focusing on the core algorithmic steps. We have added statistical significance tests (Table 4) to the experimental section.
Follow-up to Reviewer vLPX
Dear Reviewer vLPX,
Thank you again for your constructive feedback. As the discussion period is approaching its end, we would be happy to provide any further clarifications that may be helpful in addressing your remaining questions or concerns.
Best regards,
The Authors
Thank you for the detailed response and the additional experiment. I appreciate your efforts. The synthetic bandit illustrates how BiFlow can refine actions in a controlled setting. I still find the evidence limited on when fine-grained action refinement explains the gains observed on standard control benchmarks, so I maintain my original decision.
Responses to Reviewer vLPX 1/2
We appreciate the reviewer's careful evaluation and constructive feedback. Our responses are provided below.
We agree that conventional bilevel optimization can introduce substantial computational overhead, particularly when computing the response gradient. However, a core contribution of BiFlow is making bilevel optimization computationally practical for diffusion/flow-based RL.
This is achieved through two design choices. First, our first-order approximation avoids higher-order differentiation through the inner optimization steps. Second, the refiner is trained via flow matching without backpropagating through flow integration when updating the refiner. At inference, BiFlow only requires one coarse actor forward pass and 8 Euler steps, without inner-loop optimization.
As shown in Table 3, BiFlow has a training cost comparable to SDAC:
Despite the comparable training cost, BiFlow achieves higher mean returns across all eight tasks, with substantial improvements over QFlex:
These results demonstrate that our first-order bilevel approximation delivers substantial performance gains with manageable computational overhead.
Thank you for your suggestion. DSRL learns a latent-noise steering policy for a frozen pretrained diffusion policy, while BiFlow jointly learns a coarse actor and flow refiner from scratch through online RL. We have added a systematic discussion of DSRL and its differences from BiFlow to Section 2.3.