acceptodds
Under review as a conference paper at ICLR 2027

RetraQ: Manifold-Constrained Critic Guidance for Flow Policies in RL

Abstract

A learned critic can improve a pretrained flow-matching policy by steering sampled actions toward higher predicted value. However, even a critic that fits the behavior data well can provide misleading action gradients. Gradient components pointing away from the data-supported region are weakly constrained by behavior data and can lead guidance to exploit critic extrapolation errors. We introduce RetraQ (Retraction-based Q-guidance), which uses geometry encoded by the pretrained flow to constrain both the direction and the destination of critic guidance. A single velocity evaluation of the frozen flow defines a denoising map. RetraQ filters the critic gradient with the map's Jacobian and applies the map itself to correct the updated action. Our analysis characterizes when these operations approximate tangent filtering and retraction onto the behavior manifold. The same update guides actions at test time and provides targets for policy distillation and online reinforcement learning. In controlled studies, RetraQ improves true return while keeping refined actions close to the data-supported region, and its distillation targets remain more stable than those of raw critic ascent. In offline-to-online training on LIBERO and OGBench, RetraQ achieves higher mean final success than the evaluated critic-guidance baselines. On a real-world plug-insertion task, test-time RetraQ improves the success rate of a behavior-cloned policy from % to %. Online training further increases success to % in only 30 minutes and % within one hour. These results show that a pretrained flow policy provides not only an expressive action distribution, but also local geometry for reliable policy improvement.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, policy improvement, flow matching, generative policies, robot learning, critic guidance

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_jBVuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RetraQ, a critic-guidance method for pretrained flow-matching policies. Its key motivation is that critic values on a behavior manifold determine tangential derivatives but leave normal derivatives unconstrained. RetraQ therefore uses a frozen flow policy’s denoising map to approximate two geometric operations: its Jacobian filters the critic gradient, and the map itself corrects the updated action. The resulting actions can be executed directly or used as targets for policy distillation and offline-to-online learning.

The paper provides local geometric analysis, controlled experiments with known manifolds, simulation results, and a real-robot plug-insertion experiment. The approach is well motivated and practically appealing. However, I have concerns about whether the empirical gains establish the proposed geometric mechanism, particularly the contribution of Jacobian filtering beyond endpoint denoising.

Critical strengths
  1. A clear and relevant motivation.
  2. A simple implementation that reuses the pretrained policy.
  3. Promising online results. RetraQ achieves higher mean success than the evaluated baselines on the three LIBERO suites. The real-robot results also demonstrate improvements, both at test time and during online learning.
  4. Transparent discussion of limitations. The paper reports high-dimensional filtering failures, repeated offline-distillation degradation, and uncertainty in several comparisons.
Critical weaknesses
  1. The real-robot results do not sufficiently separate critic guidance from the denoising map’s own effect.

Appendix C.1 reports a median map-induced displacement of (|R(a)-a|_2=1.62), approximately 65 times the deployed critic-induced displacement target of (0.025). The calibration measures movement relative to (R(a)), rather than relative to the original action. Thus, the map itself contributes a substantial action modification independently of the critic.

This does not imply that the smaller critic-induced correction is unimportant. However, it makes map-only (R(a)) and Raw+R (R(a+\eta g)) essential controls on the robot. The map-only results on the toy problem and OGBench do not establish its effect on plug insertion. Without these controls, it remains unclear how much of the robot improvement requires critic guidance and how much specifically requires Jacobian filtering.

  1. The incremental benefit of explicit gradient filtering is not yet firmly established.

Endpoint denoising accounts for most of the improvement in the component ablations. On LIBERO, filtering adds approximately 1.9–4.3 percentage points over Raw+R, but the three-seed results do not establish a statistically reliable advantage. On OGBench, Raw+R slightly exceeds RetraQ, while the RoboMimic confidence interval for their difference includes zero.

This distinction also matters theoretically. For an exact projection and an on-manifold starting point, both [ \Pi(a+\eta g) \quad\text{and}\quad \Pi(a+\eta P_Tg) ] have the same first-order expansion, (a+\eta P_Tg+O(\eta^2)). The additional filter should therefore be particularly useful for finite steps where normal displacement threatens the validity of the local correction. A controlled sweep over step size, curvature, and normal-gradient magnitude would more directly establish when filtering is necessary and justify its additional computational cost.

Questions for authors
  1. Can you report map-only and Raw+R robot evaluations under the same critic, gating, and trial protocol? These would directly address attribution of the reported gains.
Reviewer_KdoHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RetraQ which improves flow-matching policies by constraining the direction and destination of critic guided updates using the geometry in the pretrained flow. The problem is that critic values along the behavior manifold do not uniquely determine the critics gradietn normal to the manifold. Thus, performing unconstrained action-gradient ascent can be wrong. RetraQ proposes to use a frozen pretrained flow policy to approximate the tangent space gradient filtering and retraction onto the behavior manifold. The paper gives theoretical analysis for the geometric approximation and shows empirical results on popular benchmarks such as LIBERO and OGBench, including real-robot experiments.

Critical strengths
  • Theoretically interesting. OOD-aware policy improvement for applications such as offline RL is an important problem, and paper provides an interesting approach.
  • Extensive empirical evaluation across simulation benchmarks and real world robot manipulation results. The real-robot results are interesting and shows how RetraQ can improve a pretrained flow policy in a highly practical setting.
Critical weaknesses
  • Missing relevant and competitive baselines. Offline RL approaches such as ReFORM [1] and DSRL [2] consider the OOD offline RL problem and achieve competitive results on benchmarks such as OGBench, but are missing from the experimental comparisons.
  • The gains from Jacobian filtering appear small compared to that given by the retraction. For example, in Table 1, RetraQ only improves on Raw+R by ~2-4 points whose differences are small relative to cross-seed variability. I wonder if more convincing evidence can be shown to show that filtering contributes something meaningful beyond retraction, e.g. on the real robot where the direct Raw+R and map-only ablations are not reported.

[1] S. Zhang et al. "ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulation." ICLR, 2026.

[2] A. Wagenmaker et al. "Steering Your Diffusion Policy with Latent Space Reinforcement Learning." CoRL, 2025.

Questions for authors
  • Get results for baselines [1] and [2]?
  • What advantages can this retraQ formulation give over something such as support-constrained approach?
  • RetraQ, Raw+R, and map-only guidance on real robot task?
  • Does flow matching training formally guarantee a more reliable Jacobian? Prop. 4 indicates that the accuracy of the tangent-space approximation depends on errors in the velocity field's input derivatives, which are not directly constrained by standard flow-matching training. Could the authors provide additional evidence that the learned Jacobian remains a reliable tangent-space approximation in realistic robot action spaces? In particular, is there any observed relationship between flow-matching training quality, the accuracy of the Jacobian approximation, and downstream policy improvement?
3 comments
AuthorsOpenReview ↗

Response to Reviewer KdoH (1/2): Baseline Comparisons and Differences from Support-Constrained Methods

We sincerely thank the reviewer for careful review and constructive suggestions. In response to your feedback, we have added the requested real-robot ablations (Map-only and Raw+R) and evaluated additional baselines (e.g., DSRL). We address your specific questions below, and detail the complete experimental results in our two General Response comments.

Q1. Comparisons with additional baseline methods

To ensure a rigorous and fair comparison, we focused our additional baseline experiments on the real robot, prioritizing methods that can improve the same pretrained base policy (π0.5\pi_{0.5}) under an identical evaluation protocol.

DSRL. Following your suggestion, we added a real-robot comparison with DSRL [2] on the plug insertion task. DSRL is a highly relevant baseline that improves a generative policy by learning a steering policy over its input noise, while keeping the base policy frozen. We evaluated both methods at the 30-minute and 1-hour online-training checkpoints:

MethodAfter 30 minutesAfter 1 hour
DSRL86.67%93.33%
RetraQ (Ours)98.33%100.00%

RetraQ achieves higher observed success at both checkpoints. This result extends our evaluation to latent-space policy improvement on real hardware, demonstrating that RetraQ's refinement-based updates are more sample-efficient than latent-noise steering.

ReFORM. We agree that ReFORM [1] is a relevant offline approach. However, its standard formulation requires the base BC flow to be trained with a bounded noise source, and then learns a reflected-flow noise generator within that support. Because our pretrained policy uses a standard Gaussian source, evaluating ReFORM would require retraining the BC policy from scratch. This would confound the comparison, as any performance differences would stem from both the base policy and the improvement method. Therefore, we prioritized DSRL to maintain a strict, same-checkpoint comparison.

Q2. Advantages relative to support-constrained approaches

While RetraQ shares the high-level goal of support-constrained methods (e.g., ReFORM, DSRL) to improve the policy while reducing critic extrapolation, it differs fundamentally in where the constraint is applied. Instead of acting in the latent noise space, RetraQ operates directly in the action space using the local geometry already captured by the pretrained flow. This formulation offers three distinct practical advantages:

(1) Zero extra training for test-time refinement. RetraQ can directly improve an existing pretrained flow without training any new geometry models or noise policies. While ReFORM requires retraining the base BC flow with a bounded source, and DSRL requires training a separate latent-steering policy, test-time RetraQ only requires the frozen pretrained flow and a critic. As shown in our new real-robot ablations (General Response 2/2), without any online learning or distillation, RetraQ boosts the success rate of a frozen BC policy from 55.0% to 75.0% on average.

(2) Direct control over the update direction, not just the destination. Standard support constraints only ensure the final action lands in-distribution. However, they do not resolve the issue that normal components of the critic gradient remain undetermined by on-manifold data (Prop. 1) and can severely destabilize optimization (Eq. 15). RetraQ explicitly addresses this: the Jacobian of the flow filters the gradient before the step, and the denoising map corrects it afterward. This explicit directional filtering is highly efficient, requiring only a single Jacobian-vector product (JVP) through the one-step map, avoiding the need to differentiate through the full sampling trajectory.

(3) Versatility across learning paradigms. Because RetraQ directly outputs a refined action, the exact same operator can be used flexibly: it can be executed directly at test time, or used as a high-quality regression target for both offline and online policy distillation, unifying these paradigms under a single mechanism.

References

[1] S. Zhang et al. “ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulation.” ICLR, 2026.

[2] A. Wagenmaker et al. “Steering Your Diffusion Policy with Latent Space Reinforcement Learning.” CoRL, 2025.

AuthorsOpenReview ↗

Response to Reviewer KdoH (2/2): Real-Robot Ablations and Jacobian Reliability

Q3. Real-robot map-only and Raw+R ablations

We have conducted the requested Map-only and Raw+R controls on the real-robot plug insertion task. All methods use the identical frozen BC policy and evaluation protocol. The complete comparison with all existing baselines is provided in General Response (2/2).

In summary, the real-robot results confirm that critic guidance provides most of the performance boost, while the controlled studies show that Jacobian filtering acts as an essential stabilizer for safe, large-step optimization.

Q4. Does flow-matching training guarantee a reliable Jacobian?

We agree with the reviewer that a small standard flow-matching loss alone does not guarantee an accurate Jacobian. The objective supervises velocity values rather than their input derivatives. Proposition 4 makes this distinction explicit. Let e(x)=vθˉ(s,x,tg)−v∗(s,x,tg)e(x)=v_{\bar\theta}(s,x,t_g)-v^*(s,x,t_g) denote the velocity error. JR(a)−JRopt(a)=tg(1−tg) ∂xe(tga).J_R(a)-J_R^{\mathrm{opt}}(a) = t_g(1-t_g)\,\partial_x e(t_ga). Under the assumptions of Proposition 3, for an on-manifold action aa, ∥JR(a)−PT(a)∥≤Cσg2+tg(1−tg)∥∂xe(tga)∥.\|J_R(a)-P_T(a)\|\leq C\sigma_g^2+t_g(1-t_g)\|\partial_x e(t_ga)\|.

Thus, accurate tangent filtering requires small derivative error in addition to the population-level geometric approximation. The standard training loss does not by itself establish this condition. We therefore examine the learned Jacobian directly where the reference geometry is available.

1. Direct measurements of tangent-filtering accuracy

For the learned D=16D=16 flow in Table 8, we measure normal gain and tangent error using the known manifold geometry. Normal gain measures the RMS response of JRJ_R to unit normal directions, and tangent error measures the RMS change it introduces to unit tangent directions. Both have an ideal value of zero. For off-manifold queries, the reference spaces are evaluated at the known closest point.

Query pointsMedian normal gain ↓Median tangent error ↓
On the manifold0.03410.0597
Actor samples0.03340.0470
Perturbed points0.03420.0604

Interpretation: The learned Jacobian effectively suppresses normal directions by ~97% while only distorting tangent directions by 5–6%. This confirms it acts as a highly reliable filter both on and near the manifold.

2. Training quality, Jacobian accuracy, and downstream improvement

In a controlled study, we trained flows on a 2D manifold in R16\mathbb{R}^{16} where the optimal denoiser and Jacobian are known exactly. By evaluating checkpoints across training (General Response 1/2), we track how the flow-matching loss, Jacobian error (∥JR−JRopt∥\|J_R-J_R^{\mathrm{opt}}\|), and true-objective improvement evolve together.

Improvement is normalized by the ideal geometric update: 1 means matching the optimal gain, 0 means no improvement, and a negative value means the objective degrades.

Training stepsFM loss ↓Jacobian error ↓RetraQ normalized improvement ↑Raw+R normalized improvement ↑
1k0.3800.90−0.88−1.08
4k0.3610.79−0.42−0.75
8k0.3540.550.15−0.29
16k0.3450.150.920.67
32k0.3400.080.920.93
  • Standard training yields an accurate Jacobian without derivative supervision: Even without explicit Jacobian regularization, the Jacobian error plummets monotonically from 0.90 to 0.08, allowing RetraQ to reach 92% of the optimal gain by half the full budget.

  • Lower Jacobian error rigorously tracks improvement: Across all the checkpoints, Jacobian error has a strong −0.97 Spearman correlation with RetraQ's normalized improvement, confirming a empirical link between derivative accuracy and downstream success.

  • Filtering makes refinement robust to imperfect flows: RetraQ consistently outperforms Raw+R at every budget up to 16k steps. It becomes beneficial much earlier (at 8k, where Raw+R still degrades the objective). Even with severely under-trained flows (1k–4k) where both methods degrade the objective, RetraQ limits the degradation. While the two methods converge at 32k in this specific setting, RetraQ maintains a massive advantage at the full budget (0.89 vs. 0.19) when the normal critic gradient points toward the curvature centre, a direction the data cannot rule out (Prop. 1). See General Response 1/2 for details.

Summary. While standard flow matching lacks a theoretical guarantee for an accurate tangent filter, our measurements confirm it reliably learns one in practice. The resulting accurate Jacobian directly enables downstream improvement, with filtering proving most valuable when the learned geometry is imperfect or the critic gradient is adversarial.

Thank you for the detailed response and new experimental results. I think the new ablations + DSRL comparison strengthens the submission. It is good that the authors make the distinction between the lack of theoretical guarantee for the accurate tangent filter and what they measure in practice. All of my concerns have been addressed.

I raised my score to 3. Good luck!

Reviewer_7oodOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RetraQ, a manifold-constrained critic guidance method for improving flow-based policies in reinforcement learning. The main idea is to use the pretrained flow model to filter critic gradients and retract updated actions toward the behavior manifold. The paper includes theoretical analysis, algorithmic development, simulation studies, and real-world robotic experiments.

Overall, I think this is a strong and well-executed paper. While I do not find the main idea fundamentally new, I appreciate the completeness of the work, from theoretical development and algorithm design to extensive empirical studies and real-world hardware validation. The paper makes a meaningful contribution to improving flow-based policies, and I am inclined to support its acceptance.

Critical strengths

The paper addresses an important problem in critic-guided policy improvement. The proposed method is intuitive and well motivated, and the theoretical and algorithmic developments are reasonably solid.

The work is quite complete, covering theoretical analysis, algorithm development, extensive empirical evaluations, and real-world hardware experiments. The results are generally strong, and the paper is clearly presented.

Critical weaknesses

My main concern is the level of fundamental novelty. The idea of using denoising models for manifold projection and tangent-space guidance has been studied in the existing literature. The proposed method extends these ideas to flow-based policies and critic-guided reinforcement learning, but the main concept seems relatively intuitive given the prior work.

The ablation studies suggest that much of the improvement comes from the retraction operation, while the additional benefit of Jacobian-based gradient filtering is relatively modest. The real-world experiments are also limited to one manipulation task, although the results are encouraging.

Questions for authors

Could the authors further clarify the main conceptual novelty compared with existing denoising-based manifold optimization methods, particularly Landing with the Score?

Could the authors provide more discussion on when the Jacobian-based gradient filtering is particularly important, given that retraction alone already provides much of the empirical improvement?

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2): The Role of Jacobian Filtering

We thank the reviewers for their careful and constructive reviews. A shared question is why and when Jacobian filtering matters. Here, we summarize controlled experiments comparing RetraQ with Raw+R to assess the contribution of filtering. We report the 240 additional real-robot plug-insertion trials in General Response (2/2): Additional Real-Robot Experiments. Reviewer-specific questions are addressed in the individual responses.

To assess the additional contribution of filtering, we compare RetraQ (R(a+ηJRg)R(a+\eta J_Rg)), with Raw+R(R(a+ηg)R(a+\eta g)).

1. Filtering improves robustness to large normal perturbations

We test on a two-dimensional torus in R16\mathbb{R}^{16} with explicit manifold geometry. Starting from actions on the manifold, we apply perturbations perpendicular to it and gradually increase their magnitude. We apply two methods to the perturbed actions, and measure how far the resulting actions lie from the manifold.

The table reports the median distance to the manifold after the update (lower is better). nn is a unit vector perpendicular to the manifold at aa. Perturbation magnitudes are expressed in units of the noise scale σg\sigma_g. For example, in the 1σg1\sigma_g column, the perturbed action is a+σgna+\sigma_g n.

Normal perturbation magnitude1σg1\sigma_g2σg2\sigma_g3σg3\sigma_g4σg4\sigma_g6σg6\sigma_g
Raw+R0.0140.0230.0750.4900.550
RetraQ0.0200.0200.0200.0210.021

Interpretation. While both methods handle small perturbations well, Raw+R degrades sharply as normal perturbations increase. In contrast, RetraQ consistently keeps actions close to the manifold. This highlights our key insight: filtering effectively reduces large normal perturbations before retraction, which is more robust than relying on retraction alone for full correction.

2. Filtering allows larger stable steps and more progress per update

We evaluate repeated action refinement using the same torus as A.1. The true objective rewards progress along the manifold and penalizes deviations (initial mean objective: 0.03, maximum: 2.0). To simulate critic inaccuracies, we use a critic that matches the true objective on the manifold but includes an additional normal-gradient component, gNg_N.

For each method, we tune the step size η∈(0,0.4]\eta \in (0, 0.4] and select the largest stable value (defined as closely tracking an exact geometric update across iterations). Each table entry lists RetraQ / Raw+R.

Normal-gradient magnitudeNormal-gradient directionLargest stable tested η\etaObjective after 10 updatesObjective after 40 updates
∥gN∥=1\lVert g_N\rVert=1Mostly flat normal0.4 / 0.41.34 / 1.361.97 / 1.98
∥gN∥=4\lVert g_N\rVert=4Mostly flat normal0.4 / 0.21.34 / 0.991.96 / 1.75
∥gN∥=4\lVert g_N\rVert=4Toward the tube centreline0.4 / 0.051.31 / 0.381.93 / 0.97

Interpretation. When the normal gradient gNg_N is weak, both methods tolerate large steps (η=0.4\eta=0.4) and perform similarly. However, under stronger normal gradients, Raw+R becomes unstable and is forced to use smaller step sizes (e.g., η=0.05\eta=0.05). In contrast, RetraQ effectively filters out these disruptive gradients, remaining stable at the maximum tested step size (η=0.4\eta=0.4) across all conditions. Filtering therefore supports larger stable steps and greater progress per update in these experiments.

3. Filtering improves the objective at matched refinement displacement

We also compare the methods over the same number of refinement–distillation rounds. In a separate experiment on two curved two-dimensional sheets in R16\mathbb{R}^{16}, both methods use the same frozen reference flow and critic for 30 refinement–distillation rounds. Each round refines the actions and distills them into the policy.

MetricRaw+RRetraQ
Refinement–distillation rounds3030
Total refinement displacement1.5021.501
Final true objective ↑0.77440.8272
Objective gain per unit refinement displacement ↑0.25010.2872
Median distance to the manifold ↓0.01950.0183
95th-percentile distance to the manifold ↓0.07260.0284

Interpretation. RetraQ achieves 14.8% more true-objective gain per unit displacement. The median distances to the manifold are similar, but the 95th-percentile distance is 61% lower with RetraQ. Thus, filtering improves the objective and limits large deviations without increasing the total displacement.


We have incorporated the additional experiments and clarifications into the revised manuscript. Revised text is marked in blue, and updated figures are identified in their captions.

AuthorsOpenReview ↗

General Response (2/2): Additional Real-Robot Experiments

We added the requested map-only and Raw+R controls on real-robot plug insertion task. Together with the existing evaluations, the table below reports 720 real-robot trials across seven methods. Every critic-guided method is evaluated with both critic variants, using 60 trials per method and critic.

All methods use the same frozen BC policy, fork-state detector, and evaluation protocol. The policy and critics remain fixed throughout evaluation, with no online learning or policy distillation. Raw+R and RetraQ use the same critic within each comparison. Map-only applies the denoising map R(a)R(a) without a critic step.

Test-time methodCritic 1Critic 2Overall success
BC——55.0%
QGF58.3%56.7%57.5%
Q-VGM0.0%58.3%29.2%
FlowDPG63.3%66.7%65.0%
Map-only R(a)R(a) (new)——58.3%
Raw+R R(a+ηg)R(a+\eta g) (new)73.3%66.7%70.0%
RetraQ R(a+ηJRg)R(a+\eta J_Rg)76.7%73.3%75.0%

Interpretation. These results clearly separate the contributions of all three components. First, applying the denoising map alone yields only a modest improvement over BC (58.3% vs. 55.0%), indicating that denoising alone does not account for the larger gains achieved by the guided methods. Second, adding critic guidance with Raw+R increases overall success to 70.0%, demonstrating the importance of optimizing actions toward higher-value outcomes. Finally, RetraQ further improves overall success to 75.0%, consistently outperforming Raw+R under both critics. This confirms that Jacobian filtering adds crucial value by actively constraining critic updates to manifold-aligned directions before retraction.


The revised manuscript includes these additional controls alongside the existing test-time guidance results. Revised text is marked in blue, and updated figures and tables are identified in their captions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.