acceptodds
Under review as a conference paper at ICLR 2027

Adjoint Guidance Flow: Amortized Critic Guidance for VLA Policies

12 views

Abstract

Flow-based Vision-Language-Action (VLA) policies are typically trained by behavior cloning and thus do not explicitly optimize long-term task return. Critic guidance steers generation toward higher-value actions, but existing methods differentiate the critic through a one-step surrogate of the sampler and back-propagate a critic ensemble at every flow step. In contrast, here we propose Adjoint Guidance Flow (AGF), which amortizes trajectory-aware critic guidance into a lightweight guidance network while preserving the pretrained VLA policy. Specifically, we formulate critic-guided flow generation as a deterministic optimal control problem, whose optimal guidance is a costate that carries the terminal critic gradient back through the remaining flow, and regress the guidance network onto this costate while keeping both the VLA and critic frozen. This design provides favorable memory and throughput scaling during training, and inference needs one guidance-network forward pass per step, without the critic ensemble, back-propagation, or adjoint computation. Across LIBERO, RoboCasa, and LIBERO-Pro, AGF consistently improves pretrained VLAs, remains competitive with critic-guidance and policy-fine-tuning baselines, and is the most robust method when a single guidance strength is deployed across tasks. Compared with QGF, AGF runs faster per guidance step with fewer parameters, with comparable and even better performance, showing that critic guidance can be trajectory-aware and lightweight.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, optimal control, guided diffusion sampling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gTdpOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Adjoint Guidance Flow (AGF), which amortizes trajectory-aware critic guidance into a lightweight guidance network while keeping the pretrained flow-based VLA policy frozen.

Critical strengths

1.AGF formulates critic-guided flow generation as a deterministic optimal control problem and characterizes the optimal guidance as a costate that propagates the terminal critic gradient through the remaining flow trajectory.

2.By learning a separate lightweight guidance network, AGF preserves the pretrained VLA and requires only one guidance-network forward pass per flow step, eliminating the critic ensemble, back-propagation, and adjoint computation during inference.

3.Experiments across LIBERO, RoboCasa, LIBERO-Pro, three flow-based VLAs, and real-robot tasks show that AGF consistently improves pretrained policies while achieving competitive performance and substantially lower inference cost than existing critic-guidance methods.

Critical weaknesses
  1. The paper lacks a ablation comparing the same guidance network trained with trajectory-wise costates versus one-step QGF gradients, making the benefit of trajectory-aware supervision unclear.

  2. AGF requires a separate critic ensemble and guidance network for every task, limiting its scalability to large-scale multi-task deployment.

  3. Since the performance difference between AGF and QGF is not substantial, the paper should analyze when the inference-cost savings offset the additional cost of training a task-specific guidance network.

Questions for authors
  1. Can the authors compare the same guidance network trained with trajectory-wise costates versus one-step QGF gradients to isolate the benefit of trajectory-aware supervision?

  2. Since AGF trains a separate critic ensemble and guidance network for each task, can the authors quantify the resulting cost and scalability limitations in multi-task deployment, and, if possible, evaluate whether either component can be shared across tasks?

  3. Can the authors report the complete offline training cost and the break-even number of action chunks or deployment episodes required for AGF’s inference savings to offset its additional guidance-network training cost?

3 comments
AuthorsOpenReview ↗

Response by authors

W1. Missing comparison with QGF-style guidance network

We thank the reviewer for pointing out our missing comparison. Accordingly, we train a small guidance network of AGF by changing only the training target to QGF gradient, by preserving all the other settings such as network architecture, training recipes, and critic functions.

As shown in Tables G1a–c of the General Comments, Distilled QGF falls below live QGF on average, while AGF (M=4) performs on par on the other suites and avoids the drop on LIBERO-Spatial. AGF variants also achieve higher mean success than Distilled QGF at almost all tested weights, and AGF (M=4) shows the smallest drop under LOTO (Table G1b). Moreover, both the QGF target and Distilled QGF are less aligned with the exact adjoint, with the gap growing at noisier flow steps (Table G1c).

These findings support the importance of trajectory-aware adjoint targets for preserving effective guidance under amortization, beyond merely replacing online critic-gradient computation with a lightweight network.

W2. Limited scalability due to task-specific critic and guidance and effect of shared guidance

Per your request, we first test whether the trained guidance network can be shared across different tasks. Specifically, for a given target task, we apply the critic and guidance network trained on each of the other LIBERO-Goal tasks with a fixed guidance weight. As shown in Table R1, cross-task guidance falls significantly below own-task guidance (−4.2, 95% CI [−7.4, −1.0]) and stays at the base level without degrading it. This indicates that the gain of AGF comes from task-specific critic information.

Table R1. Cross-task transfer on LIBERO-Goal (SmolVLA, AGF M=4, w=4), averaged over the other nine source tasks.

MethodSuccess (%)
Base76.6
Own-task AGF81.0
Cross-task AGF76.8

We agree that per-task training limits scalability. As described in Section 4, we follow prior works [1,2] that train a critic per task, and the critic-guidance methods we compare against share this setting. We consider a shared task-conditioned critic beyond the scope of this work since our focus is on amortizing critic guidance given the same critic function.

Regarding the resulting cost, in our real-robot setting, collecting 50 rollout episodes takes approximately 2–3 hours including resets, critic training takes 2 hours 40 minutes, and guidance-network training takes 2 hours 15 minutes, about 7–8 hours per task. This cost and the number of stored guidance networks grow linearly with the number of tasks.

Importantly, the critic ensemble is needed only during training, not deployment. Deployment requires only the corresponding lightweight guidance network, so per-task inference computation does not increase with the total number of supported tasks.

Our current method is therefore most directly applicable to a fixed set of repeatedly executed tasks, while extending AGF toward shared multi-task guidance remains an important direction for future work.

W3. Training-inference cost comparison

Offline training and real-time inference operate under different computational constraints. AGF removes the critic ensemble and backward computation from deployment, reducing the guidance overhead of every action chunk, while its one-time training cost can be amortized across multiple robots performing the same task.

Importantly, AGF also achieves 84.0% real-robot success versus 68.0% for QGF (p = 0.004), using simulation-selected guidance strengths without on-robot tuning. This demonstrates a significant performance advantage under our deployment protocol, beyond inference efficiency.

For reference, the complete offline cost in our real-robot setting is about 7–8 hours per task (see W2), of which AGF requires approximately 2 hours 15 minutes of guidance-network training. Since AGF and QGF share the same critic and rollout dataset, their critic-training and data-collection costs cancel in the comparison. AGF reduces guidance overhead from 31.2 to 11.2 ms/chunk, yielding a time-equivalent break-even point of 405k chunks, or approximately 56 hours of cumulative active operation assuming two chunks per second, excluding resets and downtime.

This calculation compares computational time rather than end-to-end task completion time or monetary cost, which also depend on hardware and deployment conditions. Importantly, AGF additionally improves real-robot success from 68.0% with QGF to 84.0% (p = 0.004) under our deployment protocol, using simulation-selected guidance strengths without on-robot tuning. Thus, AGF offers both reduced online guidance overhead and an observed success-rate improvement in this real-robot setting.

Reference

[1] Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning, arXiv, 2026

[2] Q-Flow: Stable and Expressive Reinforcement Learning with Flow-based Policy, ICML, 2026

Thank you for the detailed response. I believe a score of 3 is appropriate for this paper, and I will therefore maintain my rating. Good luck!

Thank you for your follow-up response. We wish you all the best as well!

Reviewer_H2MxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Adjoint Guidance Flow (AGF), a method designed to reduce the computational overhead of critic guidance in flow-based Vision-Language-Action (VLA) policies while accounting for the influence of the remaining flow trajectory on the final generated action. Existing methods such as QGF primarily rely on local critic gradients evaluated at Tweedie estimates to guide action generation, requiring repeated critic backpropagation at each flow step. To address these limitations, AGF formulates critic-guided flow generation as a deterministic optimal control problem, propagates terminal critic gradients to intermediate flow states through adjoint recursion, and trains a separate lightweight guidance network to predict the corresponding costates. Once trained, AGF guides the pretrained VLA without modifying its parameters or requiring critic backpropagation during inference. Instead, it uses a single guidance-network forward pass per flow step while preserving the ability to adjust guidance strength at inference time. The authors evaluate AGF on three VLA backbones, SmolVLA, π₀.₅, and MolmoAct2, across LIBERO, RoboCasa, LIBERO-Pro, and real-robot manipulation tasks. The results show that AGF improves the pretrained policies, achieves broadly comparable performance to QGF in most simulation settings, reduces guidance computation costs, and exhibits favorable robustness to guidance-strength selection.

Critical strengths
  1. A practically meaningful design combining trajectory-aware guidance with lightweight deployment. AGF combines trajectory-aware adjoint supervision with a separate guidance network, avoiding repeated critic backpropagation during inference while keeping the pretrained VLA frozen. Unlike methods that require updating the original policy parameters, this design preserves the flexibility to adjust guidance strength at inference time and offers practical advantages for deployment.
  2. Broad experimental coverage with valuable real-robot validation. The authors evaluate AGF on three different VLA backbones across multiple simulation benchmarks and provide real-robot experiments. In particular, AGF achieves a success rate of 76% on the Open-PnP-Close task, compared with 44% for the base policy and 48% for QGF. The use of paired evaluations and statistical significance tests further strengthens the experimental evidence.
  3. Convincing guidance-level computational efficiency and robustness. AGF achieves approximately 3.6× faster guidance computation per flow step than QGF. The authors also evaluate training memory, throughput, guidance-strength sensitivity, and leave-one-task-out calibration. These experiments provide meaningful evidence for the efficiency and robustness of the proposed guidance mechanism.
Critical weaknesses
  1. Missing key ablations to disentangle trajectory-aware adjoint supervision from guidance amortization. AGF combines trajectory-aware adjoint supervision with a lightweight guidance network, but the experiments do not isolate their individual contributions. Both AGF (M=1) and AGF (M=4) use trajectory-aware supervision, leaving unclear whether the benefits arise from the adjoint targets or guidance amortization itself. A crucial missing baseline is Distilled QGF, which trains the same guidance network using QGF's local critic gradients under identical conditions. This comparison is necessary to establish the independent value of trajectory-aware adjoint supervision.
  2. Insufficient analysis of the particle-averaged adjoint mechanism. Although the M=1 versus M=4 comparison and guidance-strength sweep suggest improved robustness, they do not isolate the effects of Gaussian smoothing and particle averaging. Moreover, the unbiasedness result applies to Gaussian-smoothed Jacobians rather than the original flow dynamics. The authors should clarify this distinction and provide ablations over particle count M and perturbation scale σ, along with guidance-target prediction accuracy or directional alignment on held-out flow states.
  3. Limited evidence of effectiveness on stronger base policies. Some evaluated base policies have relatively low success rates, such as 33.6% for SmolVLA on LIBERO-10 and 44.0% for π₀.₅ on RoboCasa Atomic, where AGF yields improvements of only 3.2 and 2.6 percentage points, respectively. Although other evaluated settings include stronger base performance, the results do not establish whether AGF consistently benefits stronger, well-fine-tuned policies. Evaluating different policy checkpoints or analyzing gains relative to initial success rates would help clarify the method's applicability.
Questions for authors

Q1. Have the authors considered distilling QGF's local critic gradients into a guidance network with the same architecture, inputs, training data, and computational budget as AGF? If not, how can the independent necessity of trajectory-aware adjoint supervision be established relative to conventional critic-gradient distillation? Q2. Could the authors provide ablations with different particle counts M and perturbation scales σ, and report the guidance network's prediction error or directional alignment with adjoint targets on held-out flow states? Q3. Could the authors further analyze how AGF's performance gains depend on the initial success rate of the base policy? In particular, does AGF provide consistent improvements when applied to stronger, more extensively fine-tuned VLA policies?

4 comments
AuthorsOpenReview ↗

Response by authors (1/2)

We appreciate the reviewer's time and effort in reviewing our work.

W1. Missing comparison with QGF-style guidance network

We agree that the comparison with a small guidance network trained to predict QGF's local critic gradient is worthy. Accordingly, we train a small guidance network of AGF by changing only the training target to QGF gradient, by preserving all the other settings such as network architecture, training recipes, and critic functions.

As shown in Tables G1a-c of the General Comments, Distilled QGF falls below live QGF on average, while AGF (M=4M=4) performs on par on the other suites and avoids the drop on LIBERO-Spatial. AGF variants also achieve higher mean success than Distilled QGF at almost all tested weights, and AGF (M=4) shows the smallest drop under LOTO (Table G1b). Moreover, both the QGF target and Distilled QGF are less aligned with the exact adjoint, with the gap growing at noisier flow steps (Table G1c).

These findings support the importance of trajectory-aware adjoint targets for preserving effective guidance under amortization, beyond merely replacing online critic gradient computation with a lightweight network.

We will include the controlled Distilled QGF comparison and the guidance-alignment analysis in the revised manuscript.

W2. Insufficient analysis of the particle-averaged adjoint mechanism

We thank the reviewer for the careful reading. Lemma 1 applies to the costate propagated with the Gaussian-smoothed Jacobian, not to the original pointwise costate. Smoothing introduces a bias that vanishes as σ\sigma →\to 0 (Section 3.3, Appendix A.2), and increasing MM reduces Monte Carlo variance but does not remove this bias. We have revised Section 3.3 to clarify this distinction.

To separate the two effects, we vary MM and σ\sigma on LIBERO-Goal with SmolVLA (10 tasks, 1k rollout updates). Table R1a analyzes the training targets themselves, by their Monte Carlo error and smoothing bias. Table R1b evaluates the guidance network trained on these targets, by its alignment with the original pointwise costate on held-out flow states.

Table R1a shows that particle averaging reduces only the Monte Carlo error of the target, while the smoothing bias is determined by σ\sigma and remains small at our setting. Table R1b shows that training on perturbed targets brings the guidance network closer to the original pointwise costate on held-out states, while further increasing MM or σ\sigma brings no significant additional improvement in direct paired comparisons. The alignment gain is thus already present at M=1M=1 with nonzero perturbations, whereas increasing MM primarily improves target estimation accuracy. With the full training budget, AGF (M=4M=4) outperforms the pointwise (σ\sigma=0) variant on all five LIBERO and RoboCasa suites (Table 2).

We have incorporated the theoretical clarification and additional analyses into the revised appendix.

Table R1a. Particle Monte Carlo error (top, varying MM at σ=0.02\sigma=0.02) and estimated smoothing bias (bottom, varying σ\sigma) of the targets (10 tasks × 32 held-out states).

MC error vs. smoothed target (σ\sigma=0.02)M=1M=2M=4M=8M=16log-log slope
Relative RMS error4.31%2.95%2.14%1.51%1.12%−0.49
Estimated smoothing bias vs. original costateσ\sigma=0.0050.010.020.050.10.2
Median0.18%0.21%0.42%2.07%6.83%17.6%
RMS0.33%0.47%1.13%4.92%13.2%29.3%

Error is relative to a large-M smoothed target, and bias is between a reference smoothed adjoint, estimated with M=128 particles for this analysis only, and the original pointwise adjoint.

Table R1b. Alignment with the original pointwise costate (LIBERO-Goal + SmolVLA, 10 tasks, 1k updates).

σ\sigmaMHeld-out cosΔ\DeltapTrain cos
010.626——0.710
0.0210.640+0.0150.020.705
0.0220.641+0.0160.060.713
0.0240.641+0.0160.050.720
0.0280.633+0.0070.410.711
0.0540.642+0.0160.060.721
0.140.643+0.0170.050.721
AuthorsOpenReview ↗

Response by authors (2/2)

W3. Limited evidence of effectiveness on stronger base policies.

The reviewer is kindly reminded that AGF already improves settings with high base success rates (SR), such as LIBERO-Object with SmolVLA, MolmoAct2 on LIBERO-Pro, and the real-robot pnp-plate task (Tables 2 and 3, Appendix C.4).

We also analyzed the correlation between base SR and AGF improvement across 98 tasks in our simulation experiments. To avoid false negative correlation from shared base - SR estimation noise, we split each task's 50 evaluation episodes into two disjoint halves by seed. A half estimates base SR and the other half measures AGF gain. Gains are averaged over the full guidance-weight grid (0.25-4.0) to avoid weight-selection bias, so they measure the average response across strengths.

As shown in Table R2, we find no statistically significant Spearman correlation within any of the three benchmarks, and the pooled correlation across all 98 tasks is close to zero. In other words, we find no evidence that the gains of AGF depend on base-policy strength.

We have added the per-benchmark analysis to Appendix C.10 and clarified the scope of our findings.

Table R2. Spearman rank correlations between base SR and AGF gain

Policy / BenchmarkAGF (M=1)AGF (M=4)
SmolVLA / LIBERO (40)+0.07 (p=0.66)+0.15 (p=0.34)
π0.5\pi_{0.5} / RoboCasa (18)+0.32 (p=0.19)+0.14 (p=0.59)
MolmoAct2 / LIBERO-Pro (40)−0.29 (p=0.07)−0.21 (p=0.18)
Pooled (98)−0.01 (p=0.91)+0.06 (p=0.57)

Thank you for the detailed rebuttal and the additional experiments. I appreciate the authors' efforts to address my concerns.

I acknowledge the meaningful improvements demonstrated in the real-robot experiments, and the correlation analysis across 98 tasks is a useful addition.

However, Table 2 shows that under suite-level calibration, AGF improves MolmoAct2 on LIBERO-Pro by only 0.8–1.6 percentage points over the base policy across the four suites. Furthermore, AGF does not consistently outperform QGF in these settings.

Therefore, although the real-robot results demonstrate the practical potential of AGF, the relatively modest simulation improvements on stronger base policies remain a limitation of the empirical evidence. I would encourage the authors to acknowledge this distinction and avoid overstating the generality of the performance gains.

We thank the reviewer for raising the score and constructive comments. We agree that the gains of AGF over the base policy is modest and AGF does not consistently outperform QGF on simulation. As emphasized in the paper, AGF's key advantage lies in achieving comparable performance at substantially lower inference cost, running 3.6× faster than QGF. We will revise the paper to more clearly acknowledge the limitations, highlight the efficiency advantage, and avoid overstating the generality of the performance improvements.

Reviewer_n8GtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a way to improve a pretrained robot policy without changing its weights or repeatedly running a value model during deployment. Existing critic-guidance methods use a value model to adjust the action while it is being generated, but this requires extra gradient calculations at each generation step. The proposed method, Adjoint Guidance Flow (AGF), first calculates how changes at an intermediate step affect the value of the final action. It then trains a small network to predict this guidance directly. The authors also average nearby sensitivity estimates to make the training signal more stable. Experiments on LIBERO, RoboCasa, LIBERO-Pro, and two real-robot tasks show improvements over the base policies and broadly comparable simulation performance to QGF, with cheaper guidance at inference. The reported 3.6-times speedup applies to the guidance step, not to the full robot-policy inference.

Critical strengths
  1. The method is well-motivated and the overall design is easy to understand. Keeping the pretrained policy unchanged while learning a separate guidance network provides a useful middle ground between running a critic at every step and fine-tuning the full policy. Figures 1 and 2 clearly explain what is trained and what is needed during deployment.

  2. The comparison covers several relevant alternatives, including direct critic guidance, selecting among sampled actions, and policy fine-tuning. The authors use the same critics and paired evaluation episodes for the guidance methods.

  3. The paper considers a practical issue on how much guidance to apply when one setting has to work across tasks. The leave-one-task-out analysis is more informative than reporting only the best value for each task. The real-robot experiment also transfers the guidance strength from simulation instead of selecting it on the robot. In the longer task, the reported success rate increases from 44% for the base policy to 76% for AGF over 25 paired episodes, providing encouraging evidence beyond simulation.

Critical weaknesses
  1. The experiments do not fully separate the benefits of following the complete generation process from the benefits of learning a small guidance network. Compared with QGF, AGF changes both the training target and how guidance is delivered. Could the authors train the same small network to predict QGF-style guidance and compare it with AGF under the same data and training budget?

  2. The current method is task-specific despite being built on general-purpose robot policies. Each task needs its own critic ensemble and guidance network. The leave-one-task-out experiment transfers the guidance strength, not an untrained guidance network to a new task. It would be useful to report the full per-task preparation cost, including critic-data collection and training, and to compare against a shared guidance network conditioned on the task. This would clarify when the one-time training cost is worthwhile and how the method would be managed in a robot expected to perform many tasks.

  3. Some details of the real-robot critic input need clarification. Appendix B.3 states that the robot usually executes only 15-20 actions before a new chunk arrives, while the critic is trained on sequences padded to 29 actions by repeating the final action. It is not clear whether the same treatment is used when evaluating candidate chunks and calculating guidance, or how the model is prevented from relying on actions that will not actually be executed.

Questions for authors

The statistical tests measure evaluation uncertainty, but not training uncertainty. Appendix C.4 states that all results use a single training seed. The simulation differences from QGF are generally small. If it is possible for authors to have any insights on the reproducible of the results presented in the paper?

2 comments
AuthorsOpenReview ↗

Response by authors

We thank the reviewer for the time and effort in reviewing our work.

W1. Missing comparison with QGF-style guidance network

Per your suggestion, we train the same small guidance network of AGF with only the training target changed to the QGF gradient, preserving all other settings such as network architecture, training recipe, and critic.

As shown in Tables G1a–c of the General Comments, Distilled QGF falls below live QGF on average, while AGF (M=4) performs on par on the other suites and avoids the drop on LIBERO-Spatial. AGF variants also achieve higher mean success than Distilled QGF at almost all tested weights, and AGF (M=4) shows the smallest drop under LOTO (Table G1b). Moreover, both the QGF target and Distilled QGF are less aligned with the exact adjoint, with the gap growing at noisier flow steps (Table G1c).

These results support that the AGF further improves performance and robustness over using QCF target. We have incorporated these comparisons into Appendix C.8.

W2. Limited scalability due to task-specific critic and guidance.

In our real-robot setting, the full per-task preparation takes about 7-8 hours, consisting of collecting 50 rollout episodes (approximately 2-3 hours including resets), critic training (2 hours 40 mins), and guidance-network training (2 hours 15 mins). Importantly, the critic ensemble is only required during training and is not needed at deployment, where the learned guidance network provides efficient inference-time guidance. Our current approach is therefore most suitable for settings with a limited set of frequently repeated tasks, where the one-time preparation cost can be amortized over repeated executions.

We acknowledge that both preparation cost and the number of guidance networks scale linearly with the number of tasks. As described in Section 4, we follow prior works [1,2] and train a critic per task. We consider a shared task-conditioned critic beyond the scope of this work since our focus is on amortizing critic guidance given the same critic function, although extending AGF toward reusable guidance across multiple tasks is an important direction for future work.

W3. Clarification on real-robot critic input

We thank the reviewer for identifying this implementation detail. Under asynchronous execution, only the executed actions are recorded, so the critic and the AGF guidance network are trained on executed actions padded to 29 steps. At inference, however, QGF used all 29 predicted actions as the critic input. We overlooked this mismatch, which does not arise in simulation.

Following the reviewer's point, we rerun QGF and AGF using only the executed range at inference. Since the number of executed steps is unknown during denoising, we keep the first 15 actions, since a new chunk is requested every 0.5s at 30 Hz (Appendix B.3), and pad them to 29 steps as in training. On open-pnp-close (Table R1), AGF retains its success rate and QGF changes only marginally, so the gain of AGF does not come from non-executed actions.

Table R1. Real-robot success rate (%) on open-pnp-close (25 paired episodes) with the original inference input and with only the executed range. Original denotes the setting of Table 3.

Inference inputBaseQGFAGF
Original44.048.076.0
First 15 actions, padded to 2944.052.076.0

Q1. Training uncertainty

The major difference between QGF and AGF lies in how guidance is computed. In other words, both methods share the same critic functions and the base policies, and the training uncertainty specific to AGF comes from the guidance network. To quantify this, we retain the guidance network with two additional seeds on libero-goal with SmolVLA, keeping the critic and the base policy fixed, and evaluate each seed under the same protocol as in the main paper.

As shown in Table R2, AGF exhibits consistent performance across the three training seeds, with all three runs outperforming QGF under our task-level calibration protocol.

These results provide evidence that the observed improvement is reproducible with respect to guidance-network initialization and training randomness in this setting. We will report the per-seed results in the revised manuscript.

Table R2. Success rate (%) on LIBERO-Goal + SmolVLA across guidance-network training seeds.

MethodSuite-levelTask-level
Base76.676.6
QGF80.681.4
AGF, seed 0 (Table 2)81.083.2
AGF, seed 180.083.0
AGF, seed 281.283.4
AGF, mean ± SD80.7 ± 0.683.2 ± 0.2

Reference

[1] Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning, arXiv, 2026

[2] Q-Flow: Stable and Expressive Reinforcement Learning with Flow-based Policy, ICML, 2026

I thank the authors for the efforts on the rebuttal. I will keep the current score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General comment by authors (1/2)

We are sincerely grateful to all the reviewers for their time and effort in reviewing our work. We are encouraged by the comments that our work is well-motivated and practically meaningful (n8Gt, H2Mx), experiments cover extensive comparisons and settings (n8Gt, H2Mx) including valuable real-robot case (H2Mx), with substantially lower inference cost and compettive performance (gTdp).

Before addressing each reviewer's comments individually, we clarify two major issues raised across the reviews: the independent contribution of trajectory-aware adjoint supervision beyond guidance amortization, and the preparation cost and scalability limitations of task-specific guidance.

G1. Comparison with a small guidance network that is trained with QGF-guidance without adjoint variable.

We thank the reviewers for identifying the missing comparison with conventional critic-gradient distillation. To isolate the contribution of the supervision target, we introduce Distilled QGF, which trains the same lightweight guidance network using QGF's local critic gradients, while keeping the network architecture, critic ensemble, training procedure, and other settings unchanged. We evaluate this baseline on LIBERO with SmolVLA and examine both task success and guidance-direction alignment.

The table G1a shows that Distilled QGF is learnable and improves over the base policy, but falls below live QGF on average, with the largest drop on libero-spatial suite. AGF (M=4) provides a further improvement. Table G1b shows that this difference is not tied to a particular guidance strength by introducing the smallest performance drop with LOTO analysis.

We also evaluate how closely the supervision targets and trained guidance networks align with the exact trajectory adjoint computed from the learned critic on held-out LIBERO-Goal flow states (Table G1c). While AGF's trajectory-aware supervision is derived from the adjoint formulation in optimal control theory, QGF's local critic-gradient target does not account for the full generation trajectory. Consequently, the QGF target shows lower directional alignment with the exact adjoint, with the gap becoming larger at noisier flow steps. This difference persists after training: the learned AGF guidance network achieves higher cosine similarity than Distilled QGF (0.682 vs. 0.515). We have added this analysis to Appendix C.8 of the revised paper.

These results support the value of trajectory-aware adjoint supervision for learning guidance directions closer to the full-trajectory objective, while the final task-performance gains reflect the combined effects of adjoint supervision, perturbation, and particle averaging.

Table G1a. Success rate (%) on LIBERO + SmolVLA under suite-level calibration. QGF and AGF are from Table 2.

MethodGoalObjectSpatialLIBERO-10Avg.
Base76.689.671.633.667.9
QGF (live)80.693.673.036.070.8
Distilled QGF80.893.469.236.870.1
AGF (M=1, σ=0)80.292.472.236.270.3
AGF (M=4, σ=0.02)81.093.474.636.871.5

Table G1b. Mean success rate (%) over the 40 LIBERO tasks at each fixed guidance weight and under each calibration protocol (base 67.9).

Methodw=0.25w=0.5w=1w=2w=4Suite-levelLOTO
QGF (live)67.868.470.168.769.470.868.5
Distilled QGF67.567.567.369.468.870.168.1
AGF (M=1, σ=0)66.768.168.770.269.570.368.7
AGF (M=4, σ=0.02)68.868.768.770.271.271.570.7

Table G1c. Cosine similarity with the exact adjoint across flow time t (LIBERO-Goal, 10 tasks, 320 held-out states, 5k updates). t=1 is pure noise, and generation proceeds from t=1 to t=0.

Targett=0.1t=0.3t=0.5t=0.7t=0.9All (t>0)
AGF (M=4, σ=0.02)1.0001.0001.0001.0001.0000.9999
Distilled QGF0.9110.7850.6850.5940.5100.674
Trained networkt=0.1t=0.3t=0.5t=0.7t=0.9All (t>0)
AGF (M=1, σ=0)0.7910.7560.7090.6420.5560.676
AGF (M=4, σ=0.02)0.7930.7660.7230.6490.5530.682
Distilled QGF0.7530.6350.5290.4300.3460.515
AuthorsOpenReview ↗

General comment by authors (2/2)

G2. Task-specific requirements of critic and gradient network.

We note that our per-task critic ensemble and guidance network setting follows recent works [1, 2] and is shared by the critic-guidance methods we compared against. Accordingly, the current critic cannot be shared directly, as the same observation-action pair may have different expected returns under different goals.

In our real-robot setting, the additional preparation cost per task, excluding base-policy fine-tuning, is approximately 7–8 hours: 2–3 hours for collecting 50 rollout episodes (including resets), 2 hours and 40 minutes for critic training, and 2 hours and 15 minutes for guidance-network training.

Importantly, the critic ensemble is required only during training and is not needed at deployment. Each task uses its corresponding lightweight guidance network, so we would like to emphasize that the inference cost for an individual task does not increase with the total number of supported tasks. These considerations indicate that the current AGF formulation is most suitable for a fixed set of frequently repeated tasks, where the one-time preparation cost can be amortized across repeated executions.

However, both the overall preparation cost and the storage required for task-specific guidance networks scale linearly with the number of tasks, and we agree that the future direction of the critic guidance on VLA should be extended toward a general guidance across multiple tasks. Thereofre, we explicitly acknowledge this limitation and identify shared multi-task guidance as an important direction for future work.

Reference

[1] Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning, arXiv, 2026

[2] Q-Flow: Stable and Expressive Reinforcement Learning with Flow-based Policy, ICML, 2026

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.