- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Some modern robot policies are flow-matching policies. They don’t output an action directly. They start from random noise and gradually push it toward a good action. The number of pushes is the solver budget K. Researchers fine-tune these policies with reinforcement learning (RL), usually at one fixed budget, often 4 steps. The motivation for the paper comes from the fact that the RL trained on a given budget can quietly ruin performance at other budgets, especially 1 step. The paper identifies that FPO’s loss multiplies the error by (1 − t)², where t runs from 1 (pure noise) to 0 (final action). At t = 1, that factor is zero. But t = 1 is the only point a one-step solver ever uses. To fix this, ArcFlow compares the model’s velocity to the target at four fixed time points (1, 3/4, 1/2, 1/4). Thus, at t = 1, the error is exactly the one-step action error, so one-step quality is always trained, whatever budget was used to collect data. These four checks are averaged into one score per action, which feeds one PPO-style clipped update. Further, during training, rollouts rotate through K = 1, 2, 3, 4, 6, 8, 12, 16, so the model practices across the full range. Also, each saved model is tested at K = 1, 4, and 16, and the one whose weakest score is highest is kept.
Critical strengths
The positive points of the paper are: a) it addresses the real problem of robot latency, b) a clear diagnosis of why low training loss doesn’t guarantee good one-step actions, c) easier deployment possibility of the developed methodology, and d) complete learning curves are evaluated.
Critical weaknesses
The negative points of the paper are: a) FPO++, DPPO, ReinFlow, and PolicyFlow were adapted onto FPO’s backbone and settings rather than their original settings; b) the main table and the ablation report point estimates without confidence intervals or significance tests; c) the image-policy version uses a different update (max–min gradient blend, backtracking); and d) no real-robot experiments.
Questions for authors
The detailed comments are given below.
- FPO++, DPPO, ReinFlow and PolicyFlow were adapted to FPO’s backbone and settings rather than their original settings. DPPO was even moved from diffusion to flow and trained from scratch. The flat, low curves for DPPO and PolicyFlow could partly reflect adaptation issues rather than the methods themselves.
- The main table and the ablation report point estimates without confidence intervals or significance tests, and seed counts for the main benchmark aren’t documented.
- The update has no exact trust-region or policy-gradient meaning, and the error bounds cover action reconstruction, not return or improvement guarantees.
- The four-point grid, the budget sets, the clip constants, and the 0.5 collapse threshold are empirical. Only the plain Euler solver is studied, and selection protects only the budgets it checks.
- No real-robot experiments, so the latency motivation is untested in practice.