- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes Steerable Flow Editing (SFEdit), an online reinforcement learning framework that jointly optimizes flow-based action generation and subsequent action refinement. For a fixed critic, a KL- and Wasserstein-regularized objective yields an exact tilt–transport factorization: the generator favors actions according to their attainable value after correction, minus movement cost, while a proximal map performs the correction. The same operator supplies actor-learning signals, executed actions, and critic-bootstrap actions. SFEdit learns the resulting distribution through adjoint matching and introduces iQAM, an integrating-factor discretization that improves few-step learning without additional vector–Jacobian products. Experiments on 50 OGBench tasks achieve 87.6% aggregate success, compared with 80.7% for the strongest baseline under the shared evaluation protocol. Additional real-robot experiments demonstrate improved success on screw sorting and plate stacking after 45 minutes of online training.
Critical strengths
- The mathematical formulation of the problem (Eq.5) is interesting. The two-stage pipeline of action sampling, i.e., first sampling actions and then separately editing them (decoupled), has been shown to perform well. A formulation that couples these two stages intuitively makes sense.
- Experiments include diverse baselines, and the proposed algorithm is also verified on real robots.
Critical weaknesses
- The paper's scope is too narrow in the sense that it largely focuses on two specific methods from prior work: Adjoint Matching and Editing. However, tilting and editing are not the only two solution categories for online or offline-to-online reinforcement learning. In fact, the essence of Editing is its ability to use the pretrained policy, rather than training the main policy from scratch. Therefore, the mathematical formulation (Eq.5) itself is not a fundamental approach in solving the online RL problem.
- The presentation is poor. For example, in Section 4, the authors did not explicitly state which part of the algorithm is iQAM. Moreover, most theorems and propositions in Section 4 disturb the readability of the proposed method.
- Offline-to-online RL also improves the policy beyond data, and QAM-E (Tilting+Editing) is originally proposed for solving this. However, the authors solely focus on online RL.
- Iteratively solving Eq.12 increases non-negligible computational burden since each step requires gradient computation over the critic. However, there is no training runtime comparison.
Questions for authors
- It seems that the main motivation of Eq.5 comes from QAM-E. However, I am not convinced why we should use two different distances between distributions, i.e., reverse KL divergence and the Wasserstein-2 distance. Why did the authors choose the Wasserstein-2 distance? Isn’t it more natural to choose the KL divergence?
- How does SFEdit perform in offline and offline-to-online RL setups? QAM-E is originally evaluated on offline and offline-to-online RL setups. Since SFEdit is directly replaceable with QAM-E, I would like to know the performance of SFEdit compared with the original reported results of QAM-E.
- How efficient is SFEdit in terms of training/inference runtime? According to Figure 1, SFEdit’s per-step performance improvement is high, but with more computation per step. Please compare with QAM-E.
12 comments
Response to Reviewer TLQ5 (2/4): the offline-to-online pipeline and the online setting (Q2, W3)
2. Performance in the offline-to-online setup (Q2)
We fully agree: QAM-E was proposed and evaluated in this setting, so the comparison is only complete there. Following your suggestion, we ran QAM, QAM-E, and SFEdit through the full pipeline (1M offline updates, then 500k online steps), so QAM-E also trains its edit network offline. Table R3 lists the released results next to ours.
SFEdit ends the offline phase with the lowest average success of the three methods and the online phase with the highest, above the released QAM-E as well (Table R3).
Table R3. Full offline-to-online pipeline (paper Table 1 and Appendix E.4). Success (%) after the offline → online phase, five tasks per domain. Not run: antmaze-large, puzzle-3×3, and scene, where both released methods finish at 96% or more (cube-double, also near the ceiling, is included). Ours: three seeds, . Bold: best final success per row.
| QAM (ours) | QAM-E (ours) | SFEdit | QAM (released) | QAM-E (released) | |
|---|---|---|---|---|---|
| antmaze-giant | 7.2 → 57.6 | 2.0 → 94.8 | 0.0 → 96.3 | 18.1 → 53.0 | 1.2 → 94.8 |
| cube-quadruple | 14.0 → 59.6 | 0.0 → 78.8 | 4.1 → 80.0 | 2.7 → 71.0 | 6.5 → 79.2 |
| puzzle-4×4 | 11.2 → 73.2 | 34.5 → 100.0 | 32.7 → 100.0 | 0.2 → 3.5 | 39.1 → 98.9 |
| cube-triple | 5.0 → 77.2 | 5.9 → 68.4 | 5.3 → 73.9 | 3.4 → 70.3 | 4.5 → 79.2 |
| humanoidmaze-large | 2.8 → 51.8 | 1.5 → 29.1 | 0.9 → 64.9 | 11.3 → 20.5 | 2.1 → 16.4 |
| humanoidmaze-medium | 66.2 → 84.0 | 58.4 → 88.2 | 66.8 → 89.6 | 66.8 → 83.0 | 58.6 → 82.7 |
| cube-double | 64.4 → 99.8 | 65.0 → 100.0 | 41.0 → 99.8 | 64.2 → 99.7 | 65.0 → 99.8 |
- Averaged over the seven domains, SFEdit goes from 21.5 to 86.4, QAM-E from 23.9 to 79.9, QAM from 24.4 to 71.9, and the released QAM-E from 25.3 to 78.7. The advantage of SFEdit therefore comes from online interaction, the setting it is built for.
- SFEdit and the released QAM-E are close on antmaze-giant (96.3 vs. 94.8), cube-quadruple (80.0 vs. 79.2), and puzzle-4×4 (100.0 vs. 98.9), while SFEdit requires no extra network to learn the edit. There SFEdit is 20–39 points above QAM in our runs.
3. Focus on online RL (W3)
Your point is well taken: offline-to-online RL also improves the policy beyond its data, and the submission did not say that our experiments belong to this setting. They study its online phase, and the revision adds the full pipeline (Table R3). Section 5.1 now opens with the protocol: all methods share an offline-pretrained start with at most 10% success, so the comparison measures what each method gains through interaction.
We focus on this phase for two reasons. An edit can move an action off the data: offline, such an action is scored only by the critic's extrapolation (BCQ [Fujimoto et al., ICML 2019] and BEAR [Kumar et al., NeurIPS 2019]), whereas online every executed edit enters the replay buffer and corrects the critic. And online fine-tuning is a common way to improve large robot policies (ConRFT [Chen et al., RSS 2025], [Physical Intelligence, 2025], and RLT), and in Section 5.3, 45 minutes of it nearly triples the success of the frozen teacher. Theorem 1 holds for any fixed critic, so the update also applies offline, which the revised Discussion names as future work.
Response to Reviewer TLQ5 (3/4): why two different distances, and the scope of Eq. 5 (Q1, W1)
4. Why two different distances, and why rather than KL? (Q1)
This is a very natural question, and your reading is correct: Eq. 5 is motivated by flow and robot policy edit and steering techniques (QAM-E, EXPO, ResiP [Ankile et al., ICRA 2025], PLD [Xiao et al., ICLR 2026], ZPRL [Yu et al., T-RL 2026], RFS [Su et al., ICLR 2026], and others). The submission did not explain why the two terms use different distances, and the revised Section 4.1 now does. KL prices reweighting and prices how far actions move, and QAM derives its edit variants from the same pair.
QAM (its Section 4) maximizes under a KL constraint on an auxiliary policy and a Wasserstein constraint between it and the executed policy ( for QAM-FQL, for QAM-E). Noting that this objective is intractable to optimize directly, QAM approximates the auxiliary policy by its own solution, the critic tilt, and fits the executed policy afterwards. Eq. 5 is the penalty form of the case, and Theorem 1 gives its exact joint solution. Proposition 2 gives the gap that the sequential solution leaves (Appendices A.2 and B.3, Figure 2).
KL is the natural first alternative for the second term, and we checked it: KL does not see distances between actions, so a shift off the reference support costs infinitely much however short it is, and with KL on both terms the optimal "edit" is only a second reweighting, whereas moves every sample by the proximal map (Theorem 1, Table R4).
Table R4. What each pairing of the two terms yields.
| generator term / edit term | what the optimum is |
|---|---|
| KL / (ours) | the generator reweights by the value after the edit, and the editor moves each sample (Theorem 1) |
| KL / KL | two reweightings by functions of the current value, so stays on the support of |
| / | one movement of the reference, no reweighting |
On the critic of Figure 2, a new figure (Figure 8, Appendix B.3) shows that reaching the executed value of the solution of Eq. 5 takes 2.1 times its regularization budget with the two terms solved in sequence, 2.4 times with alone, and 2.9 times with KL alone. The benchmark agrees on the sequential versus joint comparison: QAM's target followed by our edit (QAM+transport) averages 82.3, and the joint solution 85.2 (both without iQAM, Figure 4).
5. Scope of the formulation (W1)
We agree with you on both counts: tilting and editing are not the only solution categories for online or offline-to-online RL, and the essence of editing is indeed that it can reuse a pretrained policy. We do not claim that Eq. 5 is the fundamental route to online RL, and we state the claim more precisely: Eq. 5 generalizes the KL-regularized policy improvement step, of which QAM keeps only the reweighting and FQL only the edit.
- As , the optimum is the reference reweighted by , the target of REPS, MPO, AWAC, and QAM. Theorem 1 replaces by , so any learner of this target can learn and execute . In QAM this takes three lines (rows 1–3 of Table R1).
- QAM is the case . FQL fixes the generator at the behavior-cloning flow (): with enough capacity, its one-step policy applies our edit to that flow's samples, with and . Editing a fixed pretrained policy, the strength you rightly point out, is this case, and Eq. 32 (Appendix B.2) gives what the fixed generator gives up, . On the robot we keep frozen and train a student with about 1/50 of the parameters of its action expert.
- The other solution categories you mention are among our ten baselines: latent-noise steering (DSRL), implicit policy extraction (IFQL), score matching (QSM), learned edits (EXPO, QAM-E), and a Gaussian actor (RLPD). SFEdit finishes above all of them (paper Table 6): QAM reaches 80.7, FQL 72.7, and SFEdit 87.6 (85.2 without iQAM).
Response to Reviewer TLQ5 (1/4): comparison with QAM-E (W3, Q2)
Thank you very much for the detailed and expert review. We are glad that you find Eq. 5 interesting, that coupling the sampling and editing stages "intuitively makes sense" to you, and that you value the baselines and the real-robot results. You identified real gaps: the submission lacked a comparison with QAM-E, the full offline-to-online pipeline, and a runtime analysis, and Section 4 was hard to read.
1. Comparison with QAM-E (W3, Q2)
You are right that QAM-E is the most direct point of comparison: it also tilts and then edits, and SFEdit can replace it directly. It should have been a baseline, and we thank you for asking for it. We ported it to our code base and ran it on all 50 tasks with five seeds (now in Figure 3 and paper Table 6).
Under the shared protocol, SFEdit reaches 87.6, QAM-E 83.8, and QAM 80.7 (Table R2), and SFEdit needs no edit network.
Table R2. QAM-E under the shared protocol, and QAM's released results. Final success (%) after 500k online steps, five tasks per domain. Ours: the same starting checkpoints, β=3, five seeds. Released: their own 1M offline updates, per-domain temperatures, 12 seeds. Bold: best under the shared protocol.
Our port uses the released edit scales (0.1, or 0.9 on puzzle-4×4) and 0.1 on cube-double and scene, where the released 0 would make it plain QAM.