acceptodds
Under review as a conference paper at ICLR 2027

ArcFlow: Training Flow Policies for Multiple Solver Budgets

Abstract

Flow-matching policies adjust inference cost through the number of solver steps, but reinforcement learning at a single fixed solver budget can hide degradation at cheaper deployment budgets and even erase useful one-step behavior present in a pretrained policy. We introduce Action–Ratio–Clip Flow (ArcFlow), a deployment-aware algorithm for training one flow policy across a specified range of solver budgets. ARCFLOW combines action-aligned supervision, budget exposure, and worst-budget checkpoint selection. For reconstruction objectives, its bridge aggregates temporal probes into one surrogate ratio and one PPO-style clip per action, with a boundary residual exactly equal to deployed one-step reconstruction error. Cycling exposes the shared field to the deployment range, and selection optimizes its performance floor. For native velocity objectives, multi-budget endpoint supervision applies the same principle without replacing the native update. Both realizations preserve the architecture and deployment-time cost. Across DM Control, manipulation, and locomotion, ARCFLOW attains the highest mean one-step return among five flow-policy baselines on 25 of 30 tasks under each method's reported training settings. Component studies identify complementary improvements from boundary supervision, the fixed-grid bridge, budget exposure, and deployment selection. These results establish deployment-budget-aware training as an effective route to strong low-step control with one policy serving multiple inference costs.

Primary area:
reinforcement learning
Keywords:
flow matching, reinforcement learning, flow-based policies

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kkB9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Some modern robot policies are flow-matching policies. They don’t output an action directly. They start from random noise and gradually push it toward a good action. The number of pushes is the solver budget K. Researchers fine-tune these policies with reinforcement learning (RL), usually at one fixed budget, often 4 steps. The motivation for the paper comes from the fact that the RL trained on a given budget can quietly ruin performance at other budgets, especially 1 step. The paper identifies that FPO’s loss multiplies the error by (1 − t)², where t runs from 1 (pure noise) to 0 (final action). At t = 1, that factor is zero. But t = 1 is the only point a one-step solver ever uses. To fix this, ArcFlow compares the model’s velocity to the target at four fixed time points (1, 3/4, 1/2, 1/4). Thus, at t = 1, the error is exactly the one-step action error, so one-step quality is always trained, whatever budget was used to collect data. These four checks are averaged into one score per action, which feeds one PPO-style clipped update. Further, during training, rollouts rotate through K = 1, 2, 3, 4, 6, 8, 12, 16, so the model practices across the full range. Also, each saved model is tested at K = 1, 4, and 16, and the one whose weakest score is highest is kept.

Critical strengths

The positive points of the paper are: a) it addresses the real problem of robot latency, b) a clear diagnosis of why low training loss doesn’t guarantee good one-step actions, c) easier deployment possibility of the developed methodology, and d) complete learning curves are evaluated.

Critical weaknesses

The negative points of the paper are: a) FPO++, DPPO, ReinFlow, and PolicyFlow were adapted onto FPO’s backbone and settings rather than their original settings; b) the main table and the ablation report point estimates without confidence intervals or significance tests; c) the image-policy version uses a different update (max–min gradient blend, backtracking); and d) no real-robot experiments.

Questions for authors

The detailed comments are given below.

  1. FPO++, DPPO, ReinFlow and PolicyFlow were adapted to FPO’s backbone and settings rather than their original settings. DPPO was even moved from diffusion to flow and trained from scratch. The flat, low curves for DPPO and PolicyFlow could partly reflect adaptation issues rather than the methods themselves.
  2. The main table and the ablation report point estimates without confidence intervals or significance tests, and seed counts for the main benchmark aren’t documented.
  3. The update has no exact trust-region or policy-gradient meaning, and the error bounds cover action reconstruction, not return or improvement guarantees.
  4. The four-point grid, the budget sets, the clip constants, and the 0.5 collapse threshold are empirical. Only the plain Euler solver is studied, and selection protects only the budgets it checks.
  5. No real-robot experiments, so the latency motivation is untested in practice.
Reviewer_cHPoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

When deploying flow policies, it is possible to choose the step count: This trades off numerical precision with network evaluations and thus latency. Typically, policies are trained for a fixed solver budget. This paper shows that with lower step counts than the trained ones, these policies heavily degrade in performance. The paper presents ARCFLOW, which mitigates this with a new PPO-Style training paradigm, which has several improvements to make the policy perform well across different cost budgets to mitigate the degradation. This is achieved through sampling across different solver counts, and several more adjustments that are each ablated properly. The paper furthermore gives solid mathematical reasoning and a comparison to SOTA in numerous different benchmark tasks.

Critical strengths
  • Ablation of each component of the proposed method. The method has several “tricks”, and the paper motivates clearly why each of them is present.

  • Mathematical reasoning. The paper gives a clear motivation why there are limitations to the SOTA.

  • No deployment modifications: the method can deploy policies without changing the deployment architecture, making it very straightforward to implement.

  • Benchmarking on a wide set of tasks against SOTA. It shows that the benefits are not just for a hand-selected special case.

Critical weaknesses
  • Albeit the method and its components are properly validated, it does look like a combination of tricks, and does, in my view, consist more of an engineering than a methodical contribution. One part being the heuristic choices of the intervals and grids, with no clear motivation and ablation.

  • No real-world validation. In a time of agentic coding, custom simulation-only validations become less and less convincing. This is a general problem, but if possible, a real-world experiment would be beneficial.

  • The paper’s writing style is mechanistic and feels like thrown together without being concisely structured.

Questions for authors

Clarifications:

  • Can the selection of intervals and grids be more properly motivated or ablated?

Suggestions for future work:

  • It would be enlightening to see the method properly discussed (and ablated) in the context of different integration methods.
Reviewer_J3dyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem that although flow policies can adjust the inference cost through the number of solver steps, reinforcement learning (RL) fine-tuning at a single fixed solver budget can degrade performance on lower budgets and especially one-step performance. To address this problem, the authors introduce Action-Ratio-Clip Flow (ARCFLOW), an RL algorithm that trains a flow policy across a prespecified fixed set of solver budgets. The key ingredients are action-aligned supervision based on an exact one-step reconstruction identity, exposure to the full budget range and worst-budget checkpoint selection. The experiments show that ARCFLOW achieves the highest mean one-step return in 25 out of the 30 demonstrated tasks compared to five flow-policy RL baselines while remaining competitive at higher solver budgets.

Critical strengths

The major strength of this paper is that it identifies cross-budget degradation as critical failure mode in RL fine-tuning and provides a possible cause for it. In short, a flow policy fine-tuned via RL at a specific solver budget might achieve high return at that budget, but its return might collapse when tested under fewer steps and especially one step. The provided explanation is that noise-reconstruction objectives such as the FPO reference implementation have a (1−t)2(1-t)^2 weight so the loss and its gradient vanish at t=1t=1, which is the only point a one-step Euler solver queries and therefore the velocity field receives no supervision at the node used for one-step deployment. The paper also demonstrates the failure mode empirically on the released ReinFlow pipeline showing that RL can erase one-step capability present in the pretrained policy.

Critical weaknesses
  1. The methodological contribution is largely heuristic and more limited than the paper framing suggests. ARCFLOW is a combination of four ingredients: 1) velocity supervision instead of FPO noise reconstruction target, 2) a fixed four-node bridge with h=1/4h=1/4, 3) cycling the rollout budget over the fixed set of budgets S\mathcal{S} and 4) min-over-budgets checkpoint selection. The grid size, the 1/41/4 scale, the cycling order and the selection rule are all design choices without any ablation or analysis. Furthermore, Table 2 shows that the largest single gain (0.350 to 0.534 one-step return) comes from the velocity supervision. Nevertheless, to my best understanding the same modification paired with the one-step degradation of FPO and on the same tasks appears in ReFPO https://arxiv.org/html/2606.21086v1 which is not cited.

  2. The baseline comparison is not matched meaning that ARCFLOW selects checkpoints by worst-KK validation return while the baselines use native selection. Yet Table 2 shows this rule on its own is worth 0.668 to 0.711 at one step, so it is likely that part of the Table 1 margin is due to this selection rather than training. It should be also noted that training compute is also unequal because cycling over S\mathcal{S} uses on average 6.6 network evaluations per collected action, in contrast to 4 for the fixed-budget arms.

  3. The paper restricts itself to a problem setting that does not take KK as input, but without really justifying this restriction and in addition misses comparing against fundamental baselines. These would be KK-conditioned or shortcut-style policy (Frans et al., 2025) and a simple distillation baseline. In addition, ReFPO (mentioned above), which reports FPO one-step degradation on the same MuJoCo stack and proposes a different cause and fix is not compared against.

Questions for authors
  1. If the baselines are given the same worst-KK checkpoint return and number of network evaluations during training, how much of the Table 1 margin on one-step performance would remain?
  2. How does ARCFLOW compare to a KK-conditioned policy trained with the same RL pipeline and to ReFPO's velocity regularizer?
  3. The boundary suppression explanation predicts that one-step degradation should track the (1−t2)(1-t^2) weigthing. Does one-step return in the FPO arm recover if the noise-reconstruction target is kept but the timestep weight is flattened (with no other change)? And given that ReinFlow has no such factor but still collapses at one step (Table 3), how much of the one-step failure does the Section 3 diagnosis explain?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.