- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RetraQ, a critic-guidance method for pretrained flow-matching policies. Its key motivation is that critic values on a behavior manifold determine tangential derivatives but leave normal derivatives unconstrained. RetraQ therefore uses a frozen flow policy’s denoising map to approximate two geometric operations: its Jacobian filters the critic gradient, and the map itself corrects the updated action. The resulting actions can be executed directly or used as targets for policy distillation and offline-to-online learning.
The paper provides local geometric analysis, controlled experiments with known manifolds, simulation results, and a real-robot plug-insertion experiment. The approach is well motivated and practically appealing. However, I have concerns about whether the empirical gains establish the proposed geometric mechanism, particularly the contribution of Jacobian filtering beyond endpoint denoising.
Critical strengths
- A clear and relevant motivation.
- A simple implementation that reuses the pretrained policy.
- Promising online results. RetraQ achieves higher mean success than the evaluated baselines on the three LIBERO suites. The real-robot results also demonstrate improvements, both at test time and during online learning.
- Transparent discussion of limitations. The paper reports high-dimensional filtering failures, repeated offline-distillation degradation, and uncertainty in several comparisons.
Critical weaknesses
- The real-robot results do not sufficiently separate critic guidance from the denoising map’s own effect.
Appendix C.1 reports a median map-induced displacement of (|R(a)-a|_2=1.62), approximately 65 times the deployed critic-induced displacement target of (0.025). The calibration measures movement relative to (R(a)), rather than relative to the original action. Thus, the map itself contributes a substantial action modification independently of the critic.
This does not imply that the smaller critic-induced correction is unimportant. However, it makes map-only (R(a)) and Raw+R (R(a+\eta g)) essential controls on the robot. The map-only results on the toy problem and OGBench do not establish its effect on plug insertion. Without these controls, it remains unclear how much of the robot improvement requires critic guidance and how much specifically requires Jacobian filtering.
- The incremental benefit of explicit gradient filtering is not yet firmly established.
Endpoint denoising accounts for most of the improvement in the component ablations. On LIBERO, filtering adds approximately 1.9–4.3 percentage points over Raw+R, but the three-seed results do not establish a statistically reliable advantage. On OGBench, Raw+R slightly exceeds RetraQ, while the RoboMimic confidence interval for their difference includes zero.
This distinction also matters theoretically. For an exact projection and an on-manifold starting point, both [ \Pi(a+\eta g) \quad\text{and}\quad \Pi(a+\eta P_Tg) ] have the same first-order expansion, (a+\eta P_Tg+O(\eta^2)). The additional filter should therefore be particularly useful for finite steps where normal displacement threatens the validity of the local correction. A controlled sweep over step size, curvature, and normal-gradient magnitude would more directly establish when filtering is necessary and justify its additional computational cost.
Questions for authors
- Can you report map-only and Raw+R robot evaluations under the same critic, gating, and trial protocol? These would directly address attribution of the reported gains.
Response to Reviewer KdoH (1/2): Baseline Comparisons and Differences from Support-Constrained Methods
We sincerely thank the reviewer for careful review and constructive suggestions. In response to your feedback, we have added the requested real-robot ablations (Map-only and Raw+R) and evaluated additional baselines (e.g., DSRL). We address your specific questions below, and detail the complete experimental results in our two General Response comments.
Q1. Comparisons with additional baseline methods
To ensure a rigorous and fair comparison, we focused our additional baseline experiments on the real robot, prioritizing methods that can improve the same pretrained base policy (π0.5) under an identical evaluation protocol.
DSRL. Following your suggestion, we added a real-robot comparison with DSRL [2] on the plug insertion task. DSRL is a highly relevant baseline that improves a generative policy by learning a steering policy over its input noise, while keeping the base policy frozen. We evaluated both methods at the 30-minute and 1-hour online-training checkpoints:
RetraQ achieves higher observed success at both checkpoints. This result extends our evaluation to latent-space policy improvement on real hardware, demonstrating that RetraQ's refinement-based updates are more sample-efficient than latent-noise steering.
ReFORM. We agree that ReFORM [1] is a relevant offline approach. However, its standard formulation requires the base BC flow to be trained with a bounded noise source, and then learns a reflected-flow noise generator within that support. Because our pretrained policy uses a standard Gaussian source, evaluating ReFORM would require retraining the BC policy from scratch. This would confound the comparison, as any performance differences would stem from both the base policy and the improvement method. Therefore, we prioritized DSRL to maintain a strict, same-checkpoint comparison.
Q2. Advantages relative to support-constrained approaches
While RetraQ shares the high-level goal of support-constrained methods (e.g., ReFORM, DSRL) to improve the policy while reducing critic extrapolation, it differs fundamentally in where the constraint is applied. Instead of acting in the latent noise space, RetraQ operates directly in the action space using the local geometry already captured by the pretrained flow. This formulation offers three distinct practical advantages:
(1) Zero extra training for test-time refinement. RetraQ can directly improve an existing pretrained flow without training any new geometry models or noise policies. While ReFORM requires retraining the base BC flow with a bounded source, and DSRL requires training a separate latent-steering policy, test-time RetraQ only requires the frozen pretrained flow and a critic. As shown in our new real-robot ablations (General Response 2/2), without any online learning or distillation, RetraQ boosts the success rate of a frozen BC policy from 55.0% to 75.0% on average.
(2) Direct control over the update direction, not just the destination. Standard support constraints only ensure the final action lands in-distribution. However, they do not resolve the issue that normal components of the critic gradient remain undetermined by on-manifold data (Prop. 1) and can severely destabilize optimization (Eq. 15). RetraQ explicitly addresses this: the Jacobian of the flow filters the gradient before the step, and the denoising map corrects it afterward. This explicit directional filtering is highly efficient, requiring only a single Jacobian-vector product (JVP) through the one-step map, avoiding the need to differentiate through the full sampling trajectory.
(3) Versatility across learning paradigms. Because RetraQ directly outputs a refined action, the exact same operator can be used flexibly: it can be executed directly at test time, or used as a high-quality regression target for both offline and online policy distillation, unifying these paradigms under a single mechanism.
References
[1] S. Zhang et al. “ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulation.” ICLR, 2026.
[2] A. Wagenmaker et al. “Steering Your Diffusion Policy with Latent Space Reinforcement Learning.” CoRL, 2025.