- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces FlowOpt Policy, a learned sampling-based optimizer for model predictive control. Unlike methods such as MPPI or CEM—which condense candidate action sequences into a weighted mean before resampling—this approach leverages cost-conditioned attention and flow matching to directly learn how to iteratively transform an entire population of candidate action sequences. The training objective is generated online by a more robust sampling-based "teacher," guiding the model to evolve the current population of candidates toward one with lower costs. Experimental results demonstrate that, given the same rollout budget, this method outperforms various classical optimizers across a range of physical control tasks and TD-MPC2 world-model benchmarks. Furthermore, it exhibits superior out-of-distribution adaptability compared to "Flow Policy," a model that generates actions directly.
Critical strengths
-
Formulating online action optimization as "cost-constrained candidate-ensemble transport" is both conceptually elegant and insightful. This framing naturally preserves the structure among multi-modal candidate actions, making it a promising direction for further research.
-
The paper compares MPPI, CEM, CMA-ES, and DIAL-MPC across tasks including CarNav, HalfCheetah, Walker, Push-T, and the Cheetah-WM and Humanoid-WM environments (based on TD-MPC2). It also includes comparisons with a "Flow Policy" architecture. Notably, in the Push-T and world-model-based tasks, the results demonstrate that leveraging rollout costs during the online optimization process significantly outperforms one-shot, direct action generation.
Critical weaknesses
-
In the -th iteration of the outer loop, the paper performs only a single rollout for the initial population , while keeping fixed across all Euler steps. As moves away from the original candidates, the network continues to rely on the ranks and standardized costs computed at the previous positions—information that becomes invalid once the candidates have shifted.
-
The current ablation studies primarily focus on increasing the number of outer-loop optimization iterations in the CarNav and Push-T environments. This fails to clarify whether the observed performance gains stem from population attention, cost ranking, cost standardization, flow integration, teacher quality, or simply the use of a larger network.
Questions for authors
See Weaknesses
2 comments
Follow-up on Our Revisions and Responses
As we have entered the final day of the discussion period, we wanted to briefly follow up on our response.
Both of your comments were very helpful in improving the paper. The question on the fixed context during flow integration led us to ground this design choice in a principled theoretical foundation based on the Wasserstein gradient-flow perspective, while the request for ablations resulted in a comprehensive component study that identifies which elements drive performance.
We are grateful for the suggestions, as the paper is now much clearer in its exposition and considerably stronger in its experimental evidence. Given these significant improvements, we would greatly appreciate it if you would consider raising your evaluation.
Response to Reviewer aB5c
We are thankful to the reviewer for highlighting the strengths of our work and for their valuable suggestions for demonstrating more clearly several aspects of our method.
W1 (Fixed rollout costs during flow integration)
We thank the reviewer for their point, as it highlights an important distinction between optimization iterations and flow integration.
The context is fixed by design, exactly as in MPPI and CEM. Every sampling-based planner computes its entire update from the costs evaluated at the pre-update samples, and re-evaluates only afterwards. FlowOpt Policy has the same information structure: the population Ak is evaluated once per update. The Nτ Euler steps numerically integrate this single learned update; they are not Nτ separate planning iterations. Indeed, Corollary 1 shows that MPPI and CEM correspond to one Euler step under the same fixed context.
The context Ck is not intended to represent the costs of the transported candidates at their intermediate positions. Instead, the original population Ak, together with its evaluations Ck, forms a fixed scored memory, in which the evaluated candidates remain at the positions where they were evaluated. Each transported candidate queries this memory using its current position and flow time, and carries the score of its own evaluated copy as a label of its origin, not as an estimate of its current cost.
More fundamentally, our revised paper now establishes a principled theoretical foundation for this design, deriving the planning formulation from a heat–proximal splitting of the Wasserstein gradient flow and providing a mathematical justification for the structure of the learned update. The training target of each update (Propositions 1–2) is determined entirely by quantities fixed before the flow starts: the perturbed population and the evaluated candidates with their costs. The optimal velocity field is therefore the conditional expectation of the source-to-target displacement given the current position, the flow time and the fixed context (Proposition 3), which Theorem 1(i) expresses in closed form as a softmax-weighted aggregation of the evaluated candidates. Costs at intermediate positions do not enter. Ck thus plays the role of the conditioning variable of a conditional flow, held fixed during integration.
W2 (Ablation of individual components)
We thank the reviewer for this valuable suggestion. We have added component ablations on CarNav and Push-T (Table 1), isolating the effects of population attention, cost ranking and standardization, flow integration, teacher quality and network size. Each component is varied independently while keeping the deployment rollout budget and evaluation seeds fixed.
The results identify population attention and the number of flow integration steps as the most influential design choices. Removing attention reduces Push-T success by 30.2 percentage points, while using a single Euler step reduces it by 50 points, with substantial cost increases on CarNav as well. Cost conditioning also contributes, particularly ranking on Push-T, while reducing teacher quality progressively degrades performance. Varying network width has more modest effects.
These experiments complement our Flow Policy and newly introduced Learned Gaussian Update (LGU) (Figure 2 and Table 2) baselines , which respectively assess the benefits of optimization using rollout feedback and learning population transport over Gaussian proposal updates.
We sincerely thank the reviewer for their constructive feedback and valuable suggestions, which motivated us to substantially strengthen both the theoretical foundations and experimental evaluation of our work. The resulting revisions have fully addressed the reviewer's points and significantly improved the paper.
In light of these improvements, we hope the reviewer will consider increasing their evaluation.