acceptodds
Under review as a conference paper at ICLR 2027

Flow Policy Optimization with Steerable Editing

Abstract

Flow policies offer an expressive and scalable foundation for continuous control, but online reinforcement learning becomes subtle when generated actions are refined before execution. An action with low current value may still be a valuable sample if a small edit can move it into a high-value region, so the generator should account for what its actions can become after correction. We introduce Steerable Flow Editing (SFEdit), a proximal framework that derives distribution learning and action refinement from a single policy-improvement objective. A Wasserstein-smoothed KL regularizer yields an exact tilt–transport factorization: the flow favors actions by the value attainable after correction, net of the movement cost, and a proximal map performs the correction. The scaled proximal displacement provides the actor learning signal, and the same map determines the actions used for interaction and critic backups. We realize the resulting tilt through adjoint matching and introduce iQAM, an integrating-factor discretization that reduces the degradation of few-step learning without additional vector–Jacobian products. On 50 OGBench tasks across ten domains, SFEdit achieves 87.6% final aggregate success, compared with 80.7% for the strongest baseline under the shared evaluation protocol. On two fine-grained real-robot manipulation tasks, it nearly triples the success of a frozen generalist within 45 minutes of online training.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, Reinforcement Learning, Generative Policy

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TLQ5OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Steerable Flow Editing (SFEdit), an online reinforcement learning framework that jointly optimizes flow-based action generation and subsequent action refinement. For a fixed critic, a KL- and Wasserstein-regularized objective yields an exact tilt–transport factorization: the generator favors actions according to their attainable value after correction, minus movement cost, while a proximal map performs the correction. The same operator supplies actor-learning signals, executed actions, and critic-bootstrap actions. SFEdit learns the resulting distribution through adjoint matching and introduces iQAM, an integrating-factor discretization that improves few-step learning without additional vector–Jacobian products. Experiments on 50 OGBench tasks achieve 87.6% aggregate success, compared with 80.7% for the strongest baseline under the shared evaluation protocol. Additional real-robot experiments demonstrate improved success on screw sorting and plate stacking after 45 minutes of online training.

Critical strengths
  • The mathematical formulation of the problem (Eq.5) is interesting. The two-stage pipeline of action sampling, i.e., first sampling actions and then separately editing them (decoupled), has been shown to perform well. A formulation that couples these two stages intuitively makes sense.
  • Experiments include diverse baselines, and the proposed algorithm is also verified on real robots.
Critical weaknesses
  • The paper's scope is too narrow in the sense that it largely focuses on two specific methods from prior work: Adjoint Matching and Editing. However, tilting and editing are not the only two solution categories for online or offline-to-online reinforcement learning. In fact, the essence of Editing is its ability to use the pretrained policy, rather than training the main policy from scratch. Therefore, the mathematical formulation (Eq.5) itself is not a fundamental approach in solving the online RL problem.
  • The presentation is poor. For example, in Section 4, the authors did not explicitly state which part of the algorithm is iQAM. Moreover, most theorems and propositions in Section 4 disturb the readability of the proposed method.
  • Offline-to-online RL also improves the policy beyond data, and QAM-E (Tilting+Editing) is originally proposed for solving this. However, the authors solely focus on online RL.
  • Iteratively solving Eq.12 increases non-negligible computational burden since each step requires gradient computation over the critic. However, there is no training runtime comparison.
Questions for authors
  • It seems that the main motivation of Eq.5 comes from QAM-E. However, I am not convinced why we should use two different distances between distributions, i.e., reverse KL divergence and the Wasserstein-2 distance. Why did the authors choose the Wasserstein-2 distance? Isn’t it more natural to choose the KL divergence?
  • How does SFEdit perform in offline and offline-to-online RL setups? QAM-E is originally evaluated on offline and offline-to-online RL setups. Since SFEdit is directly replaceable with QAM-E, I would like to know the performance of SFEdit compared with the original reported results of QAM-E.
  • How efficient is SFEdit in terms of training/inference runtime? According to Figure 1, SFEdit’s per-step performance improvement is high, but with more computation per step. Please compare with QAM-E.
12 comments
AuthorsOpenReview ↗

Response to Reviewer TLQ5 (1/4): comparison with QAM-E (W3, Q2)

Thank you very much for the detailed and expert review. We are glad that you find Eq. 5 interesting, that coupling the sampling and editing stages "intuitively makes sense" to you, and that you value the baselines and the real-robot results. You identified real gaps: the submission lacked a comparison with QAM-E, the full offline-to-online pipeline, and a runtime analysis, and Section 4 was hard to read.


1. Comparison with QAM-E (W3, Q2)

You are right that QAM-E is the most direct point of comparison: it also tilts and then edits, and SFEdit can replace it directly. It should have been a baseline, and we thank you for asking for it. We ported it to our code base and ran it on all 50 tasks with five seeds (now in Figure 3 and paper Table 6).

Under the shared protocol, SFEdit reaches 87.6, QAM-E 83.8, and QAM 80.7 (Table R2), and SFEdit needs no edit network.

Table R2. QAM-E under the shared protocol, and QAM's released results. Final success (%) after 500k online steps, five tasks per domain. Ours: the same starting checkpoints, β=3\beta = 3, five seeds. Released: their own 1M offline updates, per-domain temperatures, 12 seeds. Bold: best under the shared protocol.

am-giantam-largehm-med.hm-largecube-2cube-3cube-4puz-3puz-4scenemean
QAM (released)53.097.683.020.599.770.371.0100.03.599.869.8
QAM-E (released)94.896.082.716.499.879.279.2100.098.999.884.7
QAM (ours)53.997.487.174.199.876.159.8100.059.998.680.7
QAM-E (ours)90.998.664.651.0100.075.279.5100.080.098.183.8
SFEdit95.994.785.374.6100.067.479.8100.080.098.587.6

Our port uses the released edit scales (0.1, or 0.9 on puzzle-4×4) and 0.1 on cube-double and scene, where the released 0 would make it plain QAM.

  • The lead over QAM-E comes from the two humanoid mazes, where SFEdit is within 2 points of QAM and QAM-E is 21–24 points below both. On the other eight domains SFEdit and QAM-E are within one point on average (SFEdit 0.8 lower).
  • On humanoidmaze-large QAM-E is low under every protocol (51.0 here, 29.1 after our full pipeline, 16.4 released), likely because of the form of its edit: where an edit rarely changes the best sample (Table R7), SFEdit's edit shrinks with the critic's slope and stays small (mean size 0.08), whereas QAM-E samples its edit from a policy with an entropy target, which perturbs every action by up to 0.1 per coordinate. On humanoidmaze-medium the gap appears only when the edit network is trained online from scratch (64.6 here, 88.2 after the full pipeline).
  • Our port is within 1.1 points of the released QAM-E on four of the seven pipeline domains (Table R3), and β=3\beta = 3 is QAM-E's stronger setting: on the eight domains with a nonzero released edit scale, it averages 80.0 at β=3\beta = 3 and 77.1 at its released temperatures.
AuthorsOpenReview ↗

Response to Reviewer TLQ5 (2/4): the offline-to-online pipeline and the online setting (Q2, W3)


2. Performance in the offline-to-online setup (Q2)

We fully agree: QAM-E was proposed and evaluated in this setting, so the comparison is only complete there. Following your suggestion, we ran QAM, QAM-E, and SFEdit through the full pipeline (1M offline updates, then 500k online steps), so QAM-E also trains its edit network offline. Table R3 lists the released results next to ours.

SFEdit ends the offline phase with the lowest average success of the three methods and the online phase with the highest, above the released QAM-E as well (Table R3).

Table R3. Full offline-to-online pipeline (paper Table 1 and Appendix E.4). Success (%) after the offline → online phase, five tasks per domain. Not run: antmaze-large, puzzle-3×3, and scene, where both released methods finish at 96% or more (cube-double, also near the ceiling, is included). Ours: three seeds, β=3\beta = 3. Bold: best final success per row.

QAM (ours)QAM-E (ours)SFEditQAM (released)QAM-E (released)
antmaze-giant7.2 → 57.62.0 → 94.80.0 → 96.318.1 → 53.01.2 → 94.8
cube-quadruple14.0 → 59.60.0 → 78.84.1 → 80.02.7 → 71.06.5 → 79.2
puzzle-4×411.2 → 73.234.5 → 100.032.7 → 100.00.2 → 3.539.1 → 98.9
cube-triple5.0 → 77.25.9 → 68.45.3 → 73.93.4 → 70.34.5 → 79.2
humanoidmaze-large2.8 → 51.81.5 → 29.10.9 → 64.911.3 → 20.52.1 → 16.4
humanoidmaze-medium66.2 → 84.058.4 → 88.266.8 → 89.666.8 → 83.058.6 → 82.7
cube-double64.4 → 99.865.0 → 100.041.0 → 99.864.2 → 99.765.0 → 99.8
  • Averaged over the seven domains, SFEdit goes from 21.5 to 86.4, QAM-E from 23.9 to 79.9, QAM from 24.4 to 71.9, and the released QAM-E from 25.3 to 78.7. The advantage of SFEdit therefore comes from online interaction, the setting it is built for.
  • SFEdit and the released QAM-E are close on antmaze-giant (96.3 vs. 94.8), cube-quadruple (80.0 vs. 79.2), and puzzle-4×4 (100.0 vs. 98.9), while SFEdit requires no extra network to learn the edit. There SFEdit is 20–39 points above QAM in our runs.

3. Focus on online RL (W3)

Your point is well taken: offline-to-online RL also improves the policy beyond its data, and the submission did not say that our experiments belong to this setting. They study its online phase, and the revision adds the full pipeline (Table R3). Section 5.1 now opens with the protocol: all methods share an offline-pretrained start with at most 10% success, so the comparison measures what each method gains through interaction.

We focus on this phase for two reasons. An edit can move an action off the data: offline, such an action is scored only by the critic's extrapolation (BCQ [Fujimoto et al., ICML 2019] and BEAR [Kumar et al., NeurIPS 2019]), whereas online every executed edit enters the replay buffer and corrects the critic. And online fine-tuning is a common way to improve large robot policies (ConRFT [Chen et al., RSS 2025], π0.6∗\pi_{0.6}^{\ast} [Physical Intelligence, 2025], and RLT), and in Section 5.3, 45 minutes of it nearly triples the success of the frozen π0.5\pi_{0.5} teacher. Theorem 1 holds for any fixed critic, so the update also applies offline, which the revised Discussion names as future work.

AuthorsOpenReview ↗

Response to Reviewer TLQ5 (3/4): why two different distances, and the scope of Eq. 5 (Q1, W1)


4. Why two different distances, and why W2W_2 rather than KL? (Q1)

This is a very natural question, and your reading is correct: Eq. 5 is motivated by flow and robot policy edit and steering techniques (QAM-E, EXPO, ResiP [Ankile et al., ICRA 2025], PLD [Xiao et al., ICLR 2026], ZPRL [Yu et al., T-RL 2026], RFS [Su et al., ICLR 2026], and others). The submission did not explain why the two terms use different distances, and the revised Section 4.1 now does. KL prices reweighting and W2W_2 prices how far actions move, and QAM derives its edit variants from the same pair.

QAM (its Section 4) maximizes EπQ\mathbb E_\pi Q under a KL constraint on an auxiliary policy and a Wasserstein constraint between it and the executed policy (ℓ2\ell_2 for QAM-FQL, ℓ∞\ell_\infty for QAM-E). Noting that this objective is intractable to optimize directly, QAM approximates the auxiliary policy by its own solution, the critic tilt, and fits the executed policy afterwards. Eq. 5 is the penalty form of the W2W_2 case, and Theorem 1 gives its exact joint solution. Proposition 2 gives the gap that the sequential solution leaves (Appendices A.2 and B.3, Figure 2).

KL is the natural first alternative for the second term, and we checked it: KL does not see distances between actions, so a shift off the reference support costs infinitely much however short it is, and with KL on both terms the optimal "edit" is only a second reweighting, whereas W22W_2^2 moves every sample by the proximal map TκT_\kappa (Theorem 1, Table R4).

Table R4. What each pairing of the two terms yields.

generator term / edit termwhat the optimum is
KL / W2W_2 (ours)the generator reweights by the value after the edit, and the editor moves each sample (Theorem 1)
KL / KLtwo reweightings by functions of the current value, so π\pi stays on the support of πref\pi_{\mathrm{ref}}
W2W_2 / W2W_2one movement of the reference, no reweighting

On the critic of Figure 2, a new figure (Figure 8, Appendix B.3) shows that reaching the executed value of the β=3\beta = 3 solution of Eq. 5 takes 2.1 times its regularization budget with the two terms solved in sequence, 2.4 times with W2W_2 alone, and 2.9 times with KL alone. The benchmark agrees on the sequential versus joint comparison: QAM's target followed by our edit (QAM+transport) averages 82.3, and the joint solution 85.2 (both without iQAM, Figure 4).


5. Scope of the formulation (W1)

We agree with you on both counts: tilting and editing are not the only solution categories for online or offline-to-online RL, and the essence of editing is indeed that it can reuse a pretrained policy. We do not claim that Eq. 5 is the fundamental route to online RL, and we state the claim more precisely: Eq. 5 generalizes the KL-regularized policy improvement step, of which QAM keeps only the reweighting and FQL only the edit.

  • As κ→0\kappa \to 0, the optimum is the reference reweighted by eβQe^{\beta Q}, the target of REPS, MPO, AWAC, and QAM. Theorem 1 replaces βQ\beta Q by UκU_\kappa, so any learner of this target can learn μ+∝πrefeUκ\mu^+ \propto \pi_{\mathrm{ref}} e^{U_\kappa} and execute TκT_\kappa. In QAM this takes three lines (rows 1–3 of Table R1).
  • QAM is the case κ→0\kappa \to 0. FQL fixes the generator at the behavior-cloning flow (μ=πref\mu = \pi_{\mathrm{ref}}): with enough capacity, its one-step policy applies our edit TκT_\kappa to that flow's samples, with β=1\beta = 1 and κ=1/(2α)\kappa = 1/(2\alpha). Editing a fixed pretrained policy, the strength you rightly point out, is this case, and Eq. 32 (Appendix B.2) gives what the fixed generator gives up, KL(πref∥μ+)≥0\mathrm{KL}(\pi_{\mathrm{ref}} \Vert \mu^+) \ge 0. On the robot we keep π0.5\pi_{0.5} frozen and train a student with about 1/50 of the parameters of its action expert.
  • The other solution categories you mention are among our ten baselines: latent-noise steering (DSRL), implicit policy extraction (IFQL), score matching (QSM), learned edits (EXPO, QAM-E), and a Gaussian actor (RLPD). SFEdit finishes above all of them (paper Table 6): QAM reaches 80.7, FQL 72.7, and SFEdit 87.6 (85.2 without iQAM).
Reviewer_EngjOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Steerable Flow Editing (SFEdit), a framework that combines critic-guided flow policy optimization with action editing.

The key idea is simple and intuitive: when actions will be edited before execution, the flow policy should optimize the value of the edited actions rather than the original generated actions.

The authors formulate a joint policy improvement objective with KL regularization on the generated action distribution and a Wasserstein penalty on the movement between generated and executed actions. They derive an exact tilt–transport factorization, showing that the optimal flow distribution is tilted by the Moreau envelope of the critic, while the final actions are obtained through a proximal editing operator.

Algorithmically, SFEdit extends QAM by replacing the terminal critic gradient with a proximal displacement:

\frac{T_\kappa(a)-a}{\kappa}.$$ This can be interpreted as an implicit, curvature-aware gradient update that accounts for the subsequent editing procedure. The paper additionally introduces iQAM, an integrating-factor discretization of adjoint matching.
Critical strengths
  • Good mathematical formulation and clear intuition. The motivation is natural: optimizing the value of generated actions can be suboptimal if those actions are subsequently edited. The joint optimization objective provides a principled way to connect critic tilting and action editing, rather than combining them heuristically. The resulting tilt–transport factorization is mathematically elegant, and the connection between the proximal displacement and the envelope gradient is particularly interesting.

  • The figures are well designed and effectively communicate the key intuition. In particular, Figures 1 and 2 clearly demonstrate why optimizing the value of actions before editing can be suboptimal.

  • Promising experimental results. The evaluation is reasonably comprehensive, covering 50 OGBench tasks across ten domains and two real-world robotic manipulation tasks. SFEdit achieves 87.6% aggregate success compared with 80.7% for QAM. The ablation comparing QAM with action editing against editing-aware QAM (82.3% vs. 85.2%) provides direct evidence that the proposed target modification can be useful. The real-robot results are also encouraging, demonstrating substantial improvement over the frozen pretrained policy within a limited online training budget.

Critical weaknesses
  • Limited algorithmic novelty. Despite the elegant formulation, the method is essentially QAM with a proximal-gradient replacement for the terminal critic gradient and an additional action-editing procedure.
  • Limited practical gains in some settings. The improvements are concentrated in a few domains, and the editing-aware target provides relatively modest gains over QAM with action editing. The method also introduces additional hyperparameters and computation.
  • Presentation could be improved. Although the figures are clear, the mathematical exposition and wording are sometimes difficult to follow. A simpler explanation of the main algorithmic difference from QAM would improve accessibility.
Questions for authors

no critical questions

2 comments
AuthorsOpenReview ↗

Response to Reviewer Engj: novelty, practical gains, and presentation (W1–W3)

Thank you very much for the positive and thoughtful assessment. We are glad that you find the formulation principled, the tilt–transport factorization elegant, and the figures helpful. We liked your one-line summary, β∇Q(a)→(Tκ(a)−a)/κ\beta \nabla Q(a) \to (T_\kappa(a) - a)/\kappa, so much that we borrowed it: it is now the first row of Table R1 in the general response. We address the three weaknesses in turn.


1. Limited algorithmic novelty (W1)

Your description of the algorithm is accurate: in code, SFEdit is QAM with the terminal critic gradient replaced by a proximal displacement, plus the edit of the executed and bootstrap actions. We agree that the change to QAM's code is small, and the contribution lies in what this change optimizes. SFEdit changes what the generator learns, and the ablations show that each part matters. Prior editors train the generator on the unedited critic tilt (QAM-E), on imitation of data or edited actions (EXPO, PA-RL), or keep it fixed (value-gradient guidance). Proposition 2 shows that a generator trained on the unedited value leaves a gap whichever editor follows, and Theorem 1 gives the target and the edit without this gap, for any learner of an exponential tilt. We build on QAM, the current state of the art for flow policies, because the tilt is the unique critical point of the adjoint matching loss. The change takes three lines of its code, and iQAM a fourth.

  • With the edit in place, replacing QAM's target by the edit-aware target raises the aggregate from 82.3 to 85.2.
  • The target needs the edit: on antmaze-giant (all without iQAM), the edit-aware target alone scores 38.2, below QAM's 53.9, and 95.4 with the edit (Appendix E.1).
  • An edit of the same size in a random direction gives an aggregate of 0.2, so the direction of the edit matters.
  • Both edited actions are needed (Table R10): on antmaze-giant, editing only the executed action gives 39.7 and only the bootstrap action 61.5, against 95.9 with both. On antmaze-large, near the top of the scale, the variants lie between 93.2 and 94.7.

Table R10. Which action is edited (paper Table 2(b)). Final success (%), five tasks per domain, middle columns three seeds, outer columns from paper Table 6 (five seeds). Bold: best per row.

QAM (no edit)executed action onlybootstrap action onlyboth (SFEdit)
antmaze-giant53.939.761.595.9
antmaze-large97.493.294.594.7

2. Limited practical gains in some settings (W2)

We agree with this assessment: the gain of the edit-aware target over QAM with the same edit (82.3 → 85.2) is modest on average, and the improvements of the full method are concentrated in a few domains. Proposition 2 predicts where the target matters: the two targets differ only where the edit changes which sampled action is best. Before online training this happens in 9–41% of states on the cube and puzzle domains but in at most 1.4% on antmaze-large and the humanoid mazes (Table R7, reply to 98Yj), and three domains are near the ceiling for both methods, so a ten-domain average dilutes the effect. The full method gains most where QAM is far from the ceiling: +42.0, +20.0, and +20.1 on the three domains where QAM is below 60%.

On the added hyperparameters and computation: κ\kappa is the one hyperparameter we tune (one value in nine of the ten domains), and the solver settings (M=4M = 4, ρ=1/2\rho = 1/2) are fixed rather than tuned in every task, simulated and real. SFEdit adds no parameters and costs 1–8% more per update than QAM (Table R5), and with the same 45-minute robot budget it leads QAM by 17 points on Screw Sorting (average over three runs).


3. Presentation (W3)

We agree, and your suggestion of a simpler explanation of the difference from QAM was very helpful. Algorithm 1 now shows QAM's version next to every line that SFEdit changes (Table R1 lists the same four changes), the derivation of iQAM moved to the appendix, Lemma 3 became an inline identity (Eq. 9, Lemma 4 in Appendix A.3), and each formal statement of Section 4 is introduced by a sentence that says what it is for.

Thank you again for your support and for these suggestions, which made the paper clearer and more accessible. We are happy to answer any further questions.

Dear reviewer, thank you again for your positive and thoughtful review. As the discussion closes soon, we would like to kindly check whether our responses on novelty, practical gains, and presentation address your remaining points.

  • Table R1 lists the lines in which SFEdit differs from QAM and what each changes, the ablations in Figure 4 and Table R10 show that each part matters, and Algorithm 1 now shows QAM's version next to every changed line.
  • The revision also adds the comparison with QAM-E under the shared protocol (87.6 against 83.8), the full offline-to-online pipeline, and further analysis on robustness and effectiveness of SFEdit.

If you have any further questions, we would be glad to answer them before the deadline, and we would be grateful if you could confirm your assessment in light of the revision.

Reviewer_98YjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes SFEdit, a framework that enables distribution learning and action refinement. SFEdit trains the flow policy to account for the value of actions after editing.

Critical strengths

This paper discusses online reinforcement learning for flow-matching policies and proposes SFEdit to address this problem. The ablation studies show that iQAM, editing and envelope contribute to performance improvements.

Critical weaknesses
  1. The performance improvements are not significant on some tasks, and computational efficiency is not analyzed. SFEdit requires multiple critic gradient evaluations. How much latency does this introduce? Does increasing the number of critic gradient evaluations lead to better performance?

  2. "An action with low current value may still be a valuable sample if a small edit can move it into a high-value region." This makes sense conceptually. However, is this a significant problem in real robotic tasks?

  3. In the antmaze-large task, why is SFEdit's performance not as good as QAM's? Is there any reason for this?

  4. In Figures 1 and 2, the paper shows that SFEdit can achieve a better score in the two-peak critic example, illustrating that the Moreau envelope helps conceptually. Are there similar cases in robotic tasks?

  5. How sensitive is SFEdit to its hyperparameters?

  6. When critic is not accurate, what is the performance of SFEdit

Questions for authors

See the weakness.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 98Yj (1/3): significance, latency, and solver steps (W1)

Thank you very much for the careful review and the concrete questions. We also appreciate your noting that the ablations show the contribution of iQAM, editing, and the envelope. We are also sorry that the method and related presentation were difficult to follow. Besides reorganizing Section 4, here is the method in plain terms (see also Table R1 in the general response). SFEdit edits every sampled action with M=4M = 4 critic-gradient steps (a larger movement scale κ\kappa allows larger edits). It trains the policy on Uκ(a)U_\kappa(a), the value an action reaches after its edit minus the cost of the move, where QAM uses Q(a)Q(a), and it changes one term of QAM's recursion (iQAM). The paper calls the edit the transport and UκU_\kappa the envelope.


1. Significance of the improvements (W1)

You are right that the improvement is not significant on every task, and we agree that the paper should say where the gains are and where they are not, and the revision now does. Over the 50 tasks SFEdit leads QAM by 6.9 points (paired t-test, p=0.037p = 0.037), and the gain comes from the three domains where QAM is below 60%.

  • With tasks as the unit, a Wilcoxon signed-rank test gives p=0.065p = 0.065. Conditioning on the 50 tasks and permuting the method labels among the 10 runs of each task gives p=1.5×10−4p = 1.5 \times 10^{-4} (20,000 permutations, Appendix D.2).
  • The gains are +42.0 on antmaze-giant, +20.0 on cube-quadruple, and +20.1 on puzzle-4×4. On cube-quadruple, QAM's released configuration uses a swept temperature of 1 instead of the shared β=3\beta = 3, and the released QAM reaches 71.0 there, 8.8 points below SFEdit.
  • Three domains are near the ceiling for both methods (cube-double, puzzle-3×3, and scene). On the two humanoid mazes the two are within 2 points, and there an edit changes which sampled action ranks best in fewer than 1.5% of states (Table R7).
  • On cube-triple SFEdit is 8.7 points below QAM, and one of its five tasks, task 2, swings between 26 and 96 across seeds.
  • Three further seeds on all 50 tasks (150 new runs) give an aggregate of 87.3 (per domain, in the order of Table R2: 89.6, 99.1, 84.7, 70.7, 99.9, 70.9, 79.9, 100.0, 80.0, 98.4).

2. How much latency do the extra critic gradients add? (W1)

This is an important practical question, and you are right that the submission left it open, so we measured it for this rebuttal. Per gradient update SFEdit costs 1–8% more than QAM and 2–5% less than QAM-E, and it adds no parameters (Table R5, reply to TLQ5). An action chunk takes 0.46–0.51 ms with SFEdit and 0.34–0.38 ms with QAM, and on the robot every method had the same 45-minute budget, so the comparison includes this cost.


3. Do more critic-gradient evaluations (a larger MM) help? (W1)

This is a very good suggestion, since the submission only used M=4M = 4. We ran M=1M = 1, 2, and 8 on four domains. Yes, where the solver still moves actions after four steps: eight steps add 13 points on puzzle-4×4, and one step costs about 20 points on antmaze-giant and puzzle-4×4 (Table R6).

  • With M=1M = 1 the solve stops after one step, so the policy's target no longer carries the value that the edit reaches (Appendix B.1).
  • M=2M = 2 halves the critic gradients. It is within 3 points of the default on antmaze-giant and cube-quadruple, above it on cube-triple, and 13.3 points below on puzzle-4×4.
  • On puzzle-4×4 the fourth solver step still moves the action by 0.32 (0.014 on antmaze-giant), and M=8M = 8 raises success from 80.0 to 93.3. Elsewhere M=8M = 8 is within 3 points of the default.

Table R6. Solver steps MM (paper Table 2(a) and Appendix E.2). Final success (%), five tasks per domain, three seeds (M=4M = 4: five seeds, paper Table 6). Last-step size ∥yM−yM−1∥\lVert y_M - y_{M-1} \rVert averaged over the second half of training on tasks 1 and 3. Bold: best success per column.

MMantmaze-giantcube-triplecube-quadruplepuzzle-4×4last step, antmaze-giantlast step, puzzle-4×4critic gradients per update
174.174.979.760.00.1220.542
292.974.179.766.70.0530.434
4 (default)95.967.479.880.00.0140.328
894.369.279.993.30.0030.2916
AuthorsOpenReview ↗

Response to Reviewer 98Yj (2/3): real robotic tasks, two-peak cases, and antmaze-large (W2, W4, W3)


4. Does "low value now, high value after a small edit" matter in real robotic tasks, and are there two-peak cases? (W2, W4)

We agree that an argument that "makes sense conceptually" still needs evidence from tasks, and we thank you for pressing on this point. The submission offered only the two-peak illustration, so we added a direct measurement. In simulation it is common in the manipulation domains, in 6–41% of states (Table R7). On the robot SFEdit leads QAM on both tasks, which tests the full method rather than the mechanism alone.

At the start of online training we draw 32 actions per state from the policy and check whether the action with the highest current value Q(a)Q(a) is still ranked first by the edit-aware target Uκ(a)U_\kappa(a).

  • The best sample changes in 41% of states on puzzle-4×4, in 13–18% on antmaze-giant, cube-double, cube-quadruple, and puzzle-3×3, and in 9% on cube-triple. In these states the edit-aware target trains the policy toward a sample that QAM's target ranks lower.
  • It changes in at most 1.4% of states on antmaze-large and both humanoid mazes. There the two targets nearly coincide (Proposition 2), and SFEdit is within 3 points of QAM.
  • A change in ranking does not guarantee a gain: on cube-triple SFEdit is 8.7 points below QAM.

Table R7. How often the edit changes the best of 32 policy samples (Appendix E.2). Measured at the start of online training with each domain's default κ\kappa. Last column: final success of SFEdit minus QAM (paper Table 6). ‡^{\ddagger}: near-saturated final success for both methods. Bold: gains of 20 points or more.

domainbest sample changes after the edit (% of states)mean edit sizeSFEdit − QAM (points)
antmaze-large0.10.00−2.7
humanoidmaze-large1.30.08+0.5
humanoidmaze-medium1.40.05−1.8
scene6.20.12−0.1$^{\ddagger}$
cube-triple8.80.21−8.7
cube-double12.60.23+0.2$^{\ddagger}$
antmaze-giant12.70.23+42.0
cube-quadruple16.90.21+20.0
puzzle-3×318.00.340.0$^{\ddagger}$
puzzle-4×440.90.52+20.1

On the robot, QAM shares the student policy, critic, replay buffer, and time budget with SFEdit but has none of the edit-aware target, the edit, and iQAM. After 45 minutes SFEdit reaches 84.4% on Screw Sorting against 67.8%, and 51.1% against 44.4% on Plate Stacking (three runs of 30 evaluation episodes each).

As for two-peak cases: in Figures 1–2 the critic has a narrow high-value band next to low values. QAM's target suppresses a sample just outside the band, and the edit-aware target favors it because the edit carries it into the band. Grasp success has such bands: in Dex-Net 2.0, predicted grasp robustness stays near zero along the gripper position and spikes where the gripper fits a small opening (Dex-Net 2.0, Mahler et al., RSS 2017, App. B of arXiv 1703.09312). Plate Stacking needs accurate approach angles to grasp the shallow plates, and in our trials SFEdit's gain there comes mainly from such grasps (Section 5.3).


5. Why is SFEdit below QAM on antmaze-large (94.7 vs 97.4)? (W3)

Thank you for the careful look at the per-domain results. This gap deserved an explanation in the paper. The 2.7-point gap is within the seed variation of SFEdit near the top of the scale: with the three further seeds of point 1, SFEdit scores 99.1 on antmaze-large. There the edit barely moves actions (mean size 0.00) and almost never changes which sample ranks first (0.1% of states, Table R7), so SFEdit behaves almost like iQAM, and the other QAM variants are also at 97–99% (QAM 97.4, QAM-E 98.6).

AuthorsOpenReview ↗

Response to Reviewer 98Yj (3/3): hyperparameters and inaccurate critics (W5, W6)


6. How sensitive is SFEdit to its hyperparameters? (W5)

This is a fair question, since SFEdit introduces a new quantity. SFEdit adds one tuned hyperparameter, the movement scale κ\kappa, and performance changes little when it is decreased and falls when it is increased.

  • κ\kappa is 0.02 in nine of the ten domains and 0.2 on antmaze-giant. A quarter of the default scores 86.5 (default 87.6), while twice the default falls to QAM's level (80.7) and four times to 71.5, plausibly because large edits leave the region where the critic was trained (Figure 4, Appendix E.1).
  • Everything else is fixed: M=4M = 4 solver steps with step size ρ=1/2\rho = 1/2 in all 50 simulation tasks and both robot tasks, and β=3\beta = 3 shared with QAM in all ten domains. On the robot, κ=0.05\kappa = 0.05 and β=0.3\beta = 0.3 were fixed after preliminary tests, before the reported runs, and QAM uses the same β\beta (Appendix F).

7. How does SFEdit perform when the critic is inaccurate? (W6)

We are grateful for this question: it is a central concern for a method that follows the critic's gradient at every edit, and it needed a direct test, which we ran for this rebuttal. With a random error as large as the gradient added to every critic gradient, SFEdit stays within 2 points of its clean score and QAM drops by 6–7 points (Table R8).

  • The error (σ=1\sigma = 1) is added to every critic gradient used by the target and the edit, for both methods, and reused at all MM steps of an edit, so it does not average out within an edit.
  • The reported runs also start from critics without online data: an offline checkpoint whose policy succeeds at most 10% of the time in simulation, and a newly initialized critic on the robot (Appendix F). On Screw Sorting SFEdit leads QAM already after 10 minutes of online training (54.4% vs. 44.4%, Table R9).
  • Errors stay bounded: an edit is at most κβG\kappa \beta G long when the critic's slope is bounded by GG (Proposition 3), a critic off by at most ε\varepsilon changes UκU_\kappa by at most βε\beta \varepsilon (Appendix B.2), and, as in QAM, edited next actions are scored by the ensemble mean minus a multiple of its spread, following the common setting (Appendix D.1). What does hurt is a movement scale too large for the critic (point 6), which is why κ\kappa is the one hyperparameter we tune on antmaze-giant.

Table R8. Critic-gradient error (paper Table 2(c) and Appendix E.3). Final success (%), five tasks × three seeds per domain, error of relative size σ=1\sigma = 1, clean scores from paper Table 6 (five seeds). Bold: higher value per column.

antmaze-giant, cleanantmaze-giant, σ=1\sigma = 1cube-quadruple, cleancube-quadruple, σ=1\sigma = 1
QAM53.947.159.852.9
SFEdit95.994.079.879.9

Table R9. Real-robot success early in training. Screw Sorting after 10 and 45 minutes of online training, counted from the end of the 5,000 warm-up steps of every method (45 minutes: Section 5.3). Three runs per method, 30 evaluation episodes each (successes out of 90). Bold: best per column.

after 10 minutesafter 45 minutes
SFEdit54.4% (49/90)84.4% (76/90)
QAM44.4% (40/90)67.8% (61/90)
RLT35.6% (32/90)58.9% (53/90)
DSRL27.8% (25/90)40.0% (36/90)

Thank you again for these questions: each led to an analysis that is now in the paper, which is clearer and more complete because of your review. If our answers address your concerns, we would be very grateful if you would consider updating your score, and we are happy to provide any further detail.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the three reviewers for their careful and constructive reviews, and the AC for handling the discussion. The reviews pointed to real gaps: the comparison with QAM-E, the full offline-to-online pipeline, the cost of the solver, and the readability of Section 4. We agree, and we revised the paper accordingly (main changes in blue in the uploaded PDF). Our central idea is unchanged: if an action will be edited before it is executed, the policy should be trained on the value of the edited action.


(A) QAM-E as a baseline (TLQ5)

Under the shared protocol, SFEdit reaches 87.6% on the 50 OGBench tasks, QAM-E 83.8%, and QAM 80.7% (Table R2), without training an edit network.


(B) Full offline-to-online pipeline (TLQ5)

SFEdit ends the offline phase with the lowest average success and the online phase with the highest: 86.4, against 79.9 for QAM-E, 71.9 for QAM, and 78.7 for the released QAM-E (Table R3).


(C) Cost (all reviewers)

Per gradient update SFEdit costs 1–8% more than QAM and 2–5% less than QAM-E, and it adds no parameters (Table R5).


(D) Why KL is paired with W2W_2 (TLQ5)

Following the statement in QAM, where direct optimization is called intractable and the problem is approximated in two steps, Eq. 5 is its penalty form, and Theorem 1 gives its exact joint solution.


(E) New analyses (98Yj, Engj)

Significance tests, solver steps (Table R6), how often the edit changes the best sample (Table R7), noisy critic gradients (Table R8), early robot success (Table R9), and which action is edited (Table R10).


(F) Clearer Section 4 (all reviewers)

Algorithm 1 shows QAM's version next to every line that SFEdit changes (summarized in Table R1), each formal statement is introduced by what it is for, and the iQAM derivation moved to Appendix C.2. All reported results are unchanged.

Table R1. What SFEdit changes relative to QAM.

QAMSFEdit
signal that trains the policy (terminal adjoint)−β∇Q(xK)-\beta \nabla Q(x_K)−(T^κ(xK)−xK)/κ-(\hat T_\kappa(x_K) - x_K)/\kappa
executed actionaaT^κ(a)\hat T_\kappa(a)
bootstrap action (in the critic's update)a′a^{\prime}T^κ(a′)\hat T_\kappa(a^{\prime})
adjoint recursion (iQAM)λi=diλi+1+2higi\lambda_i = d_i \lambda_{i+1} + 2 h_i g_iλi=di(λi+1+2higi)\lambda_i = d_i (\lambda_{i+1} + 2 h_i g_i)

The edit T^κ\hat T_\kappa (the paper's transport) moves a sampled action aa uphill with M=4M = 4 critic-gradient steps at a cost ∥T^κ(a)−a∥2/(2κ)\lVert \hat T_\kappa(a) - a \rVert^2 / (2\kappa). Row 1 trains the policy on the gradient of the envelope UκU_\kappa (the value after the edit minus this cost), which equals the edit's displacement divided by κ\kappa, so one solve gives both. Row 4 (iQAM) rescales one term of QAM's adjoint recursion at no extra cost.


Numbering and seeds

Tables R1–R10 are in our responses below, and all other numbers refer to the revised PDF (submission Table 4 is now Table 6, Proposition 4 is Proposition 3). New runs in Tables R3, R6, R8, and R10 use three seeds, and rows repeated from the paper's Table 6 use five.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.