- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies direct Q-gradient fine-tuning of pretrained flow policies without an actor loss that anchors the policy to demonstrations or its pretrained parameters. It observes that large updates can destroy pretrained performance and proposes Simple Flow Gradient (SFG), which limits the change in generated actions between successive committed policies. The constraint is measured on a fixed set of state–noise pairs and enforced by rescaling the accumulated parameter update at each episode boundary. The method combines this procedure with demonstration replay and an ensemble critic. Experiments primarily evaluate one-step MeanFlow policies, with additional results on multi-step policies and a point-cloud manipulation task.
Critical strengths
The paper examines a practically relevant problem: RL fine-tuning can initially destroy the capabilities acquired through imitation learning. The frozen-critic experiment and the measurements of action changes provide useful evidence that update magnitude deserves attention, even when the policy uses a one-step generator.
The proposed modification is relatively simple and produces promising results in the main comparison. I also appreciate that the paper reports settings where the constraint slows early learning and where the one-step method fails. These results help identify the limits of the approach rather than presenting only favorable outcomes.
Critical weaknesses
(a) The experimental coverage is too limited for me to assess how broadly the method is useful and how large its benefits are likely to be. The main comparison averages over five task settings, three of which are variants of Franka-Kitchen, while the component ablations cover only two tasks. The additional transport and point-cloud experiments are useful, but they do not establish consistent gains across a sufficiently diverse set of conditions. In particular, the one-step method fails on transport, and the multi-step results involve different pretrained policies and additional implementation considerations. Broader evaluation on further Robomimic tasks and suitable OGBench tasks would help determine whether the findings extend beyond the current settings. This matters because the reported benefit appears to depend on the competence of the initial policy and the task: preserving an already useful policy and learning from a weak initialization may require different update behavior.
(b) The paper motivates controlling policy changes, but does not convincingly establish why the proposed functional-step projection is the appropriate way to do so, or why it should be preferred over other constraints. The learning-rate ablations are helpful, but they do not resolve this question. Comparisons with an action-distance penalty relative to the previous policy, another trust-region formulation, or adaptive step-size control would be more informative about the specific contribution. These comparisons should use the same pretrained actor, critic, replay scheme, and moving reference policy. As it stands, the strongest controlled result is that the constraint avoids an early collapse on tool_hang, while the final performance is unchanged; on kitchen-partial, removing it improves early learning and leads to a similar endpoint. I therefore find the central argument incomplete: the paper identifies a failure mode and proposes a plausible remedy, but does not sufficiently explain when this remedy is preferable or whether the advantage comes from its specific design rather than more conservative updates in general. The stronger results of the complete SFG configuration do not by themselves resolve this attribution issue.
Questions for authors
(a) Under what conditions should the functional-step constraint help, and when should it be expected to slow learning? Can the authors connect the contrasting results on tool_hang, kitchen-partial, and transport to properties of the task or the pretrained policy, and provide evidence that this explanation extends beyond the two ablation tasks?
(b) What advantage does episode-level projection provide over a penalty on action changes relative to the previous policy or another adaptive update constraint? A matched comparison examining early performance loss, learning efficiency, and final performance would materially affect my assessment.
(c) Should the main contribution be understood primarily as preserving pretrained competence during the initial phase, or as improving the broader effectiveness of flow-policy fine-tuning? If the latter, what evidence isolates that benefit from the other components of the SFG configuration?
3 comments
W(b). The paper motivates controlling policy changes, but does not convincingly establish why the proposed functional-step projection is the appropriate way to do so, or why it should be preferred over other constraints. The learning-rate ablations are helpful, but they do not resolve this question. Comparisons with an action-distance penalty relative to the previous policy, another trust-region formulation, or adaptive step-size control would be more informative about the specific contribution. These comparisons should use the same pretrained actor, critic, replay scheme, and moving reference policy. As it stands, the strongest controlled result is that the constraint avoids an early collapse on tool_hang, while the final performance is unchanged; on kitchen-partial, removing it improves early learning and leads to a similar endpoint. I therefore find the central argument incomplete: the paper identifies a failure mode and proposes a plausible remedy, but does not sufficiently explain when this remedy is preferable or whether the advantage comes from its specific design rather than more conservative updates in general. The stronger results of the complete SFG configuration do not by themselves resolve this attribution issue.
Q(b). What advantage does episode-level projection provide over a penalty on action changes relative to the previous policy or another adaptive update constraint? A matched comparison examining early performance loss, learning efficiency, and final performance would materially affect my assessment.
The revision adds this matched comparison (Appendix I.1): a penalty on the action change relative to the previous policy, and a parameter-space trust region, each with the same pretrained actor, critic, replay, learning rate, and moving reference (the actor at the start of the episode) as SFG. Their strengths are calibrated so that the committed step at the start of training matches = 0.01. Means over the three ablation tasks (Table 9):
| Constraint | First 0.3M (mean / lowest) | Steps to 0.8 (M) | AUC 0–0.5M | Final |
|---|---|---|---|---|
| SFG projection | 0.71 / 0.33 | 0.24 | 0.75 | 0.96 |
| Penalty on action change | 0.68 / 0.31 | 0.35 | 0.75 | 0.95 |
| Parameter-space trust region | 0.61 / 0.19 | 0.38 | 0.67 | 0.96 |
| No constraint | 0.45 / 0.01 | 0.47 | 0.53 | 0.96 |
- What the projection adds is precise control of the step. It holds the committed functional step at throughout training (0.0100 to 0.0102); the penalty's step drifts (from 0.0086 to 0.0043 on
tool_hang), and the trust region, which bounds parameters rather than actions, needs a different on each task and still drifts up to 2.9 . Adaptive control of itself is left to future work. - The penalty is a viable alternative. It is close to SFG, faster on
kitchen-partialandsquareand slower ontool_hang; on average the projection is slightly higher early and faster, with the same AUC and final score. - Not just more conservative updates. The smaller learning rates of Table 2, without the bound, stay below SFG early on
tool_hangandkitchen-partial(at best 0.56 and 0.53 against 0.70) and do not end higher on any task.
Q(c). Should the main contribution be understood primarily as preserving pretrained competence during the initial phase, or as improving the broader effectiveness of flow-policy fine-tuning? If the latter, what evidence isolates that benefit from the other components of the SFG configuration?
Both, from different components. The bound's role is the first: it preserves the pretrained competence early and leaves the final score unchanged (Table 2; our answer to (a)). The broader effectiveness comes from the recipe as a whole, a direct deterministic -gradient on a one-step generator with the bound, and the revised Table 1 now isolates it from the starting policy and the policy architecture. Seven-task means:
| Method | Initial score | Completion score | Steps to 0.8 (M) |
|---|---|---|---|
| OGPO+CA | 0.43 | 0.91 | 0.56 |
| SFG (10-step flow) | 0.43 | 0.92 | 0.68 |
| SFG (1-step flow) | 0.41 | 0.94 | 0.47 |
| SFG (MeanFlow) | 0.43 | 0.98 | 0.21 |
- Not the starting policy. All four start at 0.41 to 0.43.
- The update rule. SFG (1-step flow) fine-tunes a one-step policy distilled from the same ten-step flow policy that OGPO+CA starts from, and still ends higher and reaches 0.8 sooner (0.94 and 0.47M against 0.91 and 0.56M).
- The one-step generator. Applied directly to the ten-step flow policy, the same update ends at the level of OGPO+CA but is slower (0.68M), since the critic's gradient then passes through the sampler.
We thank Reviewer VoEh for the detailed assessment and for valuing our report of the limits. Changes are in blue in the revised PDF.
W(a). The experimental coverage is too limited for me to assess how broadly the method is useful and how large its benefits are likely to be. The main comparison averages over five task settings, three of which are variants of Franka-Kitchen, while the component ablations cover only two tasks. The additional transport and point-cloud experiments are useful, but they do not establish consistent gains across a sufficiently diverse set of conditions. In particular, the one-step method fails on transport, and the multi-step results involve different pretrained policies and additional implementation considerations. Broader evaluation on further Robomimic tasks and suitable OGBench tasks would help determine whether the findings extend beyond the current settings. This matters because the reported benefit appears to depend on the competence of the initial policy and the task: preserving an already useful policy and learning from a weak initialization may require different update behavior.
Q(a). Under what conditions should the functional-step constraint help, and when should it be expected to slow learning? Can the authors connect the contrasting results on tool_hang, kitchen-partial, and transport to properties of the task or the pretrained policy, and provide evidence that this explanation extends beyond the two ablation tasks?
Coverage. The revision extends every part of the evaluation:
canandlift(Figure 3, Table 1)square(Table 2)door,pen,relocate(Figure 6)transportQ(a). The bound costs little when the pretrained policy is weak and protects it when it is competent but fragile (Table 2, now on three tasks; first 0.3M with and without the bound):
square(θ0 = 0.19) andkitchen-partial(0.32) the update without the bound does not collapse, and the bound does not slow learning (0.73 against 0.69, 0.70 against 0.65; it ends at 0.98 and 1.00).tool_hang(0.50) and ontransportwith a three-step policy (0.54), the update without the bound is at zero for the first 0.3M or throughout; with it,tool_hangkeeps 0.70 early andtransportends at 0.81.squareandkitchen-partialbut falls below θ0 ontool_hang(0.37 against 0.50); δ = 0.01 is safe on all three.