acceptodds
Under review as a conference paper at ICLR 2027

QPILOTS: Efficient Test-Time Q-Steering for Flow Policies

Abstract

Flow-matching and diffusion policies are expressive action generators, but optimizing them with reinforcement learning (RL) is difficult: backpropagating the critic's action gradient through a multi-step denoising process is unstable, so existing methods distill the policy into a one-step actor or repeatedly fine-tune it as the critic improves. We show that inference-time steering alone, without ever training the policy against the critic, can outperform these fine-tuning approaches. Our method, QPILOTS, steers each denoising step of a flow policy: it maps the noisy intermediate action to an estimate of the clean action, where the critic is reliable, and adds the critic gradient computed there, rescaled to the magnitude of the base velocity. On a standard offline-to-online RL benchmark, QPILOTS achieves the best aggregate performance across 50 tasks, above baselines that fine-tune the policy against the critic. We also steer a frozen, pretrained VLA model, outperforming prior inference-time approaches in aggregate over six simulated manipulation tasks and on one real-robot task. We ablate key design choices against practicality and performance.

Primary area:
reinforcement learning
Keywords:
Offline to Online Reinforcement Learning, Flow Matching, Policy Steering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VRsiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents QPILOT, which harnesses the whole denoising process instead of just on one-step setting in diffusion-based policy's RL. The proposed method is claimed to help diffusion-based policies' RL out-perform their fine-tuning.

Critical strengths
  1. The idea of steering denoising in all steps, is clearly shown and illustrated, and easy to follow.
  2. Comprehensive experiments are provided to justify the proposed method.
Critical weaknesses
  1. Experiment Fairness. QPILOT is tested on >1 NFE, while some of the comparisons are done on 1 NFE. It would be good to add comparisons with the same NFE when possible, and ablate QPILOT on NFEs.
  2. 1-NFE benefits. Most robotics tasks are latency sensitive. However, if I understand correctly, the QPILOT would have 0 impact on 1-NFE model. It would be interesting to see whether RL with QPILOT and then distill to 1 NFE would help.
Questions for authors

See critical weaknesses. Besides: (1) It will be great if all 50 task results in Table 1 can be released.

9 comments

We thank the reviewer for the review, for recognizing the comprehensiveness of our experiments, and for the suggestion to distill QPILOTS into a one-step policy, which worked remarkably well as a complement to steering: the student learns from QPILOTS' steered actions and keeps most of its gains. We ran additional experiments that directly address each concern: (1) on NFE fairness, nine of the eleven Table 1 baselines, including the strongest (QAM-E), use the same 10-step sampler as QPILOTS, and our new NFE ablation (§5.2, Fig. 3) shows that at a matched 3 steps QPILOTS-U outperforms QAM-E by 43 online points (75% vs. 32% on ten tasks); (2) for one-step deployment, we distilled QPILOTS-U into a one-step student that runs 5.6–5.9× faster per action, keeps about 95% of its teacher's success, and outperforms every one-step method, including QAM's own distilled variant, and even 10-step QAM-E (85% vs. 82% online on 16 tasks); and (3) we release per-task results for all 50 tasks and all 13 methods (App. Table 19). All changes are shown in blue in the revised submission.

W1: Fairness: "QPILOT is tested on >1 NFE, while some of the comparisons are done on 1 NFE ... add comparisons with the same NFE ... and ablate QPILOT on NFEs."

(NFE: network evaluations per action.) Only two Table 1 baselines act with a single network pass: ReBRAC (a Gaussian actor) and FQL (a distilled one-step actor). The other nine use the same 10-step Euler sampler as QPILOTS. The baseline closest in compute is CGQL, which also takes a critic gradient at every step, and QPILOTS-U is 22 points higher online:

MethodInference per action50-task success, off →\to on
ReBRAC1 actor pass40 →\to 64
FQL1 actor pass36 →\to 82
QAM-E (strongest baseline)10 flow steps + learned action edit46 →\to 85
CGQL10 flow steps + critic gradient at every step30 →\to 67
QPILOTS-U10 flow steps + critic gradient at 9 steps44 →\to 89

NFE-matched comparison and ablation. The QAM authors released QAM-E runs with 1, 3, 20 and 30 flow steps on one task per domain. We trained QPILOTS-U with K=3K = 3 steps on all ten of these tasks (4 seeds), keeping the α\alpha tuned at 10 steps. Mean success on these ten tasks, off →\to on:

Flow steps KKQAM-EQPILOTS-U
13 →\to 28see W2
313 →\to 3248 →\to 75
10 (paper setting)40 →\to 8644 →\to 94

Going from 10 to 3 steps, QAM-E loses 54 online points and fails on most of these tasks, while QPILOTS-U loses 19. Each steered step adds one backward pass, so QPILOTS-U at K=3K = 3 costs 3 flow evaluations plus 2 gradient passes. §5.2 now has this as a fourth design axis: Fig. 3 plots success against KK on the eight of these tasks that also have the distilled student (below), and App. F.3 (Fig. 8) gives the per-task curves and the K=3K = 3 mean over all 20 tuning tasks.

W2: "Most robotics tasks are latency sensitive ... QPILOT would have 0 impact on 1-NFE model ... see whether RL with QPILOT and then distill to 1 NFE would help."

We agree that steering a standard flow policy has little effect at one step. Run for a single step, a flow-matching policy outputs roughly the average data action for the state. On OGBench's play datasets, which mix many behaviors, that average is often not a valid action. That is why the 10-step flow baselines also collapse at one step (QAM-E: 3 →\to 28 at K=1K = 1).

Try distilling to 1 NFE. We thank the reviewer for this suggestion, which works remarkably well as a complement to QPILOTS rather than a replacement: the student needs QPILOTS-U as its teacher during training, and in exchange for one-step inference it gives up about 5 online points (85 vs. 90). For a fixed base policy and critic, the QPILOTS-U sampler maps each noise input to one action, so a one-step student can regress the steered action for the same noise. This is FQL's one-step distillation with the steered sampler as teacher. We trained the student online alongside QPILOTS-U; it is never used to act or to compute critic targets. On 16 tasks across 8 domains (4 seeds), the student keeps about 95% of its teacher's success at one network pass per action. It outperforms every one-step method, including QAM-F, QAM's own variant that distills its flow into a one-step actor, and even beats 10-step QAM-E (App. F.3, Table 11):

16 tasks, success off →\to on1 network pass10 flow steps
ReBRAC45 →\to 65
FQL37 →\to 79
QAM-F (QAM distilled to one step)46 →\to 79
QAM-E43 →\to 82
QPILOTS-U: distilled student / its teacher50 →\to 8553 →\to 90

Steering during training and distilling for deployment therefore gives a fast policy at deployment time. At batch size 1 on one NVIDIA L4 GPU, the student takes 0.27 ms per action, against 1.56 ms for its QPILOTS-U teacher (5.9× faster) and 0.53 ms for the unsteered 10-step base flow, on humanoidmaze-large; on cube-triple the student is 5.6× faster than its teacher (0.28 vs. 1.59 ms). We are running the student on all 50 tasks with 8 seeds and aim to add it as a full variant to Table 1 by the camera-ready.

Q1: "It will be great if all 50 task results in Table 1 can be released."

App. Table 19 now lists all 50 tasks for all 13 methods of Table 1, and we post it as a follow-up comment. Baselines are averaged over the 12 seeds released by the QAM authors, ours over 8 seeds. Per-task learning curves are in App. Fig. 11.

I thank the authors for the rebuttal and further evidences! For W1, I thank the author for listing the NFEs, and I suggest including them in the main results. I notice that the 10 NFE methods also have some diffs. It would be nice to list the latency of each of them, given the same compute environment, to further justify the comparison.

Reviewer_dBDDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes QPILOTS, an inference-time method for steering flow policy, which outperforms fine-tuning baselines. At each flow sampling step, QPILOTS uses a one-step estimate of the clean action to calculate the critic gradient, and adds the rescaled gradient to the base velocity for sampling. The method demonstrates superior performance on standard offline-to-online RL benchmarks and in VLA steering evaluations compared to both finetuning and inference-time baselines. Extensive ablation studies and supporting experiments further validate the effectiveness of QPILOTS.

Critical strengths
  1. The paper presents extensive experiments, including evaluation on 50 standard offline-to-online RL tasks, VLA steering experiments and various ablation studies. The experiment setup is also clearly discussed in detail.
  2. The presentation of the paper is clear and easy to follow.
Critical weaknesses
  1. The methodological contribution of this paper is limited. The core components of the algorithm including the one-step clean action estimate and the use of critic gradient to guide action sampling in flow policy are all well-established.
  2. The point estimate is biased, as discussed in the paper. However, the paper only compares against a learned posterior sampler (Meta Flow Map) to demonstrate that the point estimate is sufficient. The learned sampler can introduce additional approximation error and may not be the best variant for this comparison. Experiments on policy performance using more accurate posterior samples for estimating Vt(xt)V_t(x_t) in Equation 4 should be included (e.g., sampling with standard multi-step sampling and using BPTT to calculate the gradient, and sampling multiple Monte-Carlo particles with the equivalent SDE). This would make the potential accuracy-efficiency tradeoff more explicit.
Questions for authors
  1. Why is UF-OPS not included as a baseline in the experiments, given its close relevance to the proposed method?
  2. How are the tasks in LIBERO selected? The paper only mentions that these tasks are "from the larger LIBERO-90 suite on which π0.5’s zero-shot success rate is between 5 and 50 percent". Are these all the tasks that satisfy this condition?
  3. Minor issues. In the 'RL with Diffusion and Flow Policies' paragraph, the policy gradient algorithms such as FPO should not be listed under 'Back-Propagating Through Time'. Figures 6 and 7 are oversized and could be better formatted.
11 comments

We thank the reviewer for the thorough review, for recognizing the breadth of our experiments, and for the excellent suggestion to test more accurate posterior estimators. We ran additional experiments that directly address each concern: (1) exact-gradient estimators (BPTT, and 8 SDE particles) close the offline gap to QPILOTS-M but do not beat the Tweedie point estimate after online training (91 and 92 vs. 93 on OGBench; 85.4 vs. 86.8 for BPTT on LIBERO), at 2.8–4.1× its latency, so the point estimate is sufficient and the learned MFM sampler is not limited by its approximation error; (2) we added UF-OPS as a baseline, and QPILOTS-U outperforms it on OGBench (38% vs. 2% after offline training) and on 10 of 11 LIBERO tasks (86.8% vs. 69.5%); (3) we ran every remaining LIBERO-90 task that meets our selection criterion, and over all 11, QPILOTS-U raises π0.5\pi_{0.5} from 25.5% to 86.8%, vs. 74.4% for DSRL and 40–52% for Best-of-20, EXPO and CGQL-LinEx; and (4) we clarify that our contribution is identifying which design choices make critic-gradient steering work in online RL, where the critic keeps changing, and showing that it then outperforms state-of-the-art fine-tuning. All changes are shown in blue in the revised submission.

W1: Novelty: "the one-step clean action estimate and the use of critic gradient to guide action sampling in flow policy are all well-established." We agree that the Tweedie clean-action estimate (DPS, universal guidance) and critic-gradient guidance are established, and we credit them and UF-OPS. Our contribution is to show which choices make this recipe work in online RL, where the critic is a TD estimate that keeps changing, and that the result can outperform state-of-the-art RL fine-tuning (89% vs. 85% for QAM-E, the strongest fine-tuning method, after online training on 50 OGBench tasks). Furthermore, the same recipe steers a frozen 3B-parameter VLA (π0.5\pi_{0.5}) online: over all 11 qualifying LIBERO-90 tasks it raises success from 25.5% to 86.8%, against 74.4% for DSRL, and on a real robot it reaches 38/50 against 28/50. DSRL (Wagenmaker et al., 2025) is a leading method for improving a frozen diffusion or flow VLA with online RL, and it is the strongest baseline in our VLA evaluation, ahead of best-of-NN, EXPO, classifier guidance, and UF-OPS (69.5%).

W2: Is the biased point estimate sufficient when compared against more accurate posterior estimators than the learned Meta Flow Map ("using BPTT to calculate the gradient, and sampling multiple Monte-Carlo particles with the equivalent SDE")? While meta-flow-maps have demonstrated effective conditional posterior sampling in the image generation space, it is a valid point that the MFM is not the most straightforward method for isolating the effect of the posterior quality sample. To this end, we follow the reviewer’s excellent suggestion and implement a version of our method where we use backprop-through-time (BPTT) to obtain the posterior estimates most directly.

We added both estimators the reviewer suggested, with everything else held fixed to QPILOTS-U:

  • BPTT: integrate the remaining ODE steps from xtx_t to x1x_1 with one deterministic particle and backpropagate ∇Q(s,x1)\nabla Q(s, x_1) through the chain.
  • SDE-8: 8 differentiable particles from the equivalent SDE (exponential integrator), with the gradient checked against a 1024-particle reference.
EstimatorOGBench offlineOGBench onlineLIBERO, 11 tasksLatency vs. QPILOTS-U
QPILOTS-U (Tweedie point estimate)389386.81.0×1.0\times
QPILOTS-M (learned MFM posterior)4395––
BPTT429185.4 (a)2.8×2.8\times
SDE-84292–4.1×4.1\times

(a) On LIBERO, BPTT backpropagates through one SDE particle, at 3.2×3.2\times the latency of QPILOTS-U. Latency is per π0.5\pi_{0.5} action chunk, batch 1, all methods on the same GPU (App. F.5, Table 12).

The more accurate estimators help offline, by about 4 points, matching QPILOTS-M, but not after online training: BPTT and SDE-8 are 1–2 points below the point estimate on OGBench, and BPTT is 1.4 points below on LIBERO, at 2.8–4.1× the latency. BPTT is also unstable online on cube-triple (46 and 60, vs. 82 and 85 for QPILOTS-U), which SDE particles avoid only at the highest cost. On the reviewer's concern about the learned sampler: QPILOTS-M matches or beats both more exact estimators, so its approximation error is not what limits it. Our reading is that exact posterior gradients matter while the critic is fit only to offline data; once the critic learns from the steered policy's rollouts, the bias of the point estimate stops limiting performance. This agrees with our diagnostic that E[Q(s,X1)∣Xt]\mathbb{E}[Q(s, X_1) \mid X_t] and Q(s,x^1)Q(s, \hat x_1) differ by at most 0.10.1 posterior standard deviations (App. F.7). Table 2 in §5.2 (Axis 2) now includes both estimators; per-task tables and learning curves are in App. F.1 (Table 8, Fig. 5).

Q1: "Why is UF-OPS not included as a baseline in the experiments, given its close relevance to the proposed method?" We added UF-OPS (Attarian et al., 2026). It fits a Monte Carlo return verifier once and then steers with Tweedie-estimate guidance or best-of-NN selection. The verifier is never updated from the steered policy's rollouts.

OGBench. With a verifier trained on the offline dataset, plus Tweedie guidance and best-of-10, UF-OPS reaches 2% on the 20 tasks after offline training, against 38% for QPILOTS-U (per task in App. Table 8). On these sparse, long-horizon tasks, returns of mostly failed episodes give almost no useful gradient.

LIBERO. We followed the authors' pipeline: 750 unsteered π0.5\pi_{0.5} rollouts per task (about as many episodes as one of our online runs collects), one reward-model verifier trained on all 11 tasks, and evaluation with best-of-10 and with guidance strengths 0.1,0.2,0.3\\{0.1, 0.2, 0.3\\}, 150 episodes per setting. We report UF-OPS's best setting on each task (App. F.2, Table 9).

Task1226283138435960647779Mean
π0.5\pi_{0.5} (zero-shot)10642143212195036303225.5
UF-OPS (frozen verifier)843187226485699967718569.5
DSRL9444100297390949622829474.4
QPILOTS-U925210051919882998910010086.8

UF-OPS improves on the zero-shot policy on every task but stays well below the methods whose critics keep learning from the policy's own rollouts. QPILOTS-U is above UF-OPS on 10 of 11 tasks and ties on task 60.

Reviewer_ddxUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes QPILOTS that improves flow policies by applying Q-gradient guidance during action generation.

Critical strengths

The developed method is theory rooted and the implementation is relatively simple and the reported improvements demonstrate practical potentials.

Critical weaknesses
  1. Policy Consistency in Q-learning The KL-regularized formulation is appealing, but how it connects to a monotonic policy-improvement guarantee remains somehow unclear. Unlike TRPO, QUILTS does not evaluate the advantage of a fixed reference policy or propose explicitly constrain the KL divergence of the resulting policy. Moreover, the critic and guided policy evolve jointly, so standard fixed-policy Bellman contraction arguments do not directly establish convergence of the overall procedure.
  2. Gap Between the KL Objective and Practical Implementation The practical algorithm uses approximate Q-gradients with magnitude normalization, which seems not fully consistent with the theoretical objective, as the authors also acknowledge. Could the authors provide at least some empirical evidence showing whether the proposed kl objective is approximately satisfied during test time?
  3. Generality of Experimental Improvements It seems that most of the improvement over QAM-E on OGBench comes from humanoidmaze-large (59% vs. 16%), while the performance on other domains is only moderately imprved. Also, only six selected LIBERO tasks and a single real-robot task were evaluated, so more experiments might be needed to better support the generality of the results.
Questions for authors
  1. could the authors clarify which policy the learned Q-function is intended to evaluate? More preferablly to show by theory the performance of the monotonic improvement of the policy
  2. could the authors provide empirical evidence of the actual KL divergence between the base and guided policies?
8 comments

We thank the reviewer for the careful review, for recognizing the practical potential of QPILOTS, and for the insightful questions about its theoretical properties. We ran additional experiments and analysis that directly address each concern: (1) on OGBench, QQ evaluates the current steered policy, since TD targets bootstrap with fresh actions from the steered sampler (Alg. 2); (2) we prove that exact soft value iteration, a KL-regularized variant of QPILOTS with a fixed base policy, converges geometrically to a unique critic whose policy is at least as good as the base policy, and bound how far QPILOTS' own critic can be from it (App. C); (3) continuous-time KL estimates on humanoidmaze show that the steered policy reaches the value gain of the KL-optimal policy with about 25% more KL, and that the gradient rescaling the reviewer asked about reaches comparable value gains with 2.3–6× less mean KL than the unrescaled drift the theory prescribes (App. D); and (4) QPILOTS-U is within 3 points of the best method in 8 of 10 OGBench domains (all 50 per-task results in App. Table 19), and on all 11 qualifying LIBERO tasks it reaches 86.8% vs. 74.4% for DSRL. All changes are shown in blue in the revised submission.

W1/Q1: Which policy does QQ evaluate, and is improvement guaranteed?

Which policy. Write πβ\pi_\beta for the base flow policy and πα\pi^\alpha for the policy QPILOTS deploys: πβ\pi_\beta steered by the critic gradient with coefficient α\alpha. On OGBench, the TD target bootstraps with a fresh next action a′∼πα(⋅∣s′)a' \sim \pi^\alpha(\cdot \mid s') drawn from the steered sampler run with the target networks (Alg. 2), not with the action stored in the replay buffer. So QQ estimates QπαQ^{\pi^\alpha}, the value of the deployed policy. Each critic update changes the steering, while the policy being steered, πβ\pi_\beta, stays the same. (On LIBERO, running the 3B VLA inside every critic update is too slow, so the critic uses SARSA targets, which bootstrap with the stored next action; §5.3 and App. F.6.)

Theory. The reviewer is right that a TRPO-style monotonic-improvement argument does not apply. TRPO keeps each new policy close to the previous one, whereas QPILOTS keeps every policy close to the same base policy πβ\pi_\beta. The standard setting that fits this structure is KL-regularized RL (Geist et al., 2019): maximize the return minus a per-step penalty τ−1KL(π∥πβ)\tau^{-1}\mathrm{KL}(\pi \Vert \pi_\beta), where τ>0\tau > 0 controls how far the policy may move from πβ\pi_\beta. This is the setting of SAC's soft policy iteration, with the entropy bonus replaced by a KL penalty toward πβ\pi_\beta (as in the value-penalty variant of BRAC; Wu et al., 2019). QPILOTS' own target omits the penalty, like BRAC's policy-regularization variant; part (c) below bounds the difference. For a given critic QQ, the best policy in this setting has a closed form, the tilted policy πQ∝πβ,eτQ\pi_Q \propto \pi_\beta\\, e^{\tau Q}: the base policy reweighted toward high-QQ actions. QPILOTS' steering is designed to sample from this tilted policy. If it does so exactly, the policy is determined by the critic, so it suffices to analyze the critic updates. Like SAC's analysis, we assume each critic update is computed exactly, without function-approximation error, and prove (Proposition 1, App. C):

Proposition. Let TτT_\tau be the KL-regularized (soft) Bellman backup, TτQ=r+γ,Es′[τ−1log⁡Ea∼πβeτQ(s′,a)]T_\tau Q = r + \gamma\\, \mathbb{E}_{s'}\big[\tau^{-1} \log \mathbb{E}_{a \sim \pi_\beta} e^{\tau Q(s',a)}\big]. The bracketed term equals the expected QQ of πQ\pi_Q at s′s' minus its KL penalty. (a) Sampler. In continuous time and under the regularity assumptions stated in App. C, the steered flow samples exactly πQ\pi_Q when it uses exact guidance gradients (no Tweedie approximation) and the steering scale σt\sigma_t that the theory prescribes, without our gradient rescaling. We call this the σt\sigma_t drift. (b) Convergence of the regularized variant. Each application of TτT_\tau shrinks the max-norm distance between any two critics by a factor γ\gamma. Repeated backups therefore converge geometrically to a unique critic Qτ⋆Q^\star_\tau. The corresponding policy πτ⋆=πQτ⋆\pi^\star_\tau = \pi_{Q^\star_\tau} maximizes the KL-regularized return, and its ordinary return is at least that of πβ\pi_\beta. As in SAC, alternating evaluation and reweighting also improves the regularized return at every iteration. (c) Distance to QPILOTS. QPILOTS' target bootstraps with the expected QQ of πQ\pi_Q but does not subtract the KL penalty. Any critic that this target leaves unchanged is within γκ/(τ(1−γ))\gamma\kappa / (\tau(1-\gamma)) of Qτ⋆Q^\star_\tau, where κ\kappa is the largest per-state KL(πQ∥πβ)\mathrm{KL}(\pi_Q \Vert \pi_\beta). As τ→0\tau \to 0, κ=O(τ2)\kappa = O(\tau^2) uniformly over such critics, so the gap is O(τ)O(\tau) and vanishes as steering weakens. (The bound applies to any such critic; it does not by itself show that one exists.)

The deployed algorithm differs from the analyzed variant in two ways. It rescales the steering gradient, so there is no fixed τ\tau, and its target omits the KL penalty. We therefore do not claim a monotonic-improvement guarantee for it. Part (c) bounds the effect of the missing penalty, and the KL estimates below quantify the effect of rescaling. We find that running the principled variant to a competitive performance requires a rather large number of base-policy samples in every TD target, making it impractical.

W2/Q2: "The practical algorithm uses approximate Q-gradients with magnitude normalization ... could the authors provide empirical evidence of the actual KL divergence between the base and guided policies?" Is the KL objective approximately satisfied?

We answer two questions. (1) How far does the steered policy actually move from πβ\pi_\beta? (2) Does it solve the objective of §4, max⁡πEπ[Q(s,⋅)]\max_\pi \mathbb{E}_{\pi}[Q(s,\cdot)] subject to KL(π∥πβ)≤δ(s)\mathrm{KL}(\pi \Vert \pi_\beta) \le \delta(s)? The exact solution is the tilted policy πβeτ(s)Q/Z\pi_\beta e^{\tau(s) Q}/Z from above, with τ(s)\tau(s) set so that the KL budget δ(s)\delta(s) is met.

Measurement. Both policies (πα\pi^\alpha, πβ\pi_\beta) are flows: πβ\pi_\beta integrates the base velocity ut=vθu_t = v_\theta, and πα\pi^\alpha integrates the steered velocity ut=vθ+αgtu_t = v_\theta + \alpha g_t (Eq. 8). Log-densities therefore follow from the instantaneous change of variables, log⁡π(a)=log⁡p0(x0)−∫01∇⋅ut(xt),dt\log \pi(a) = \log p_0(x_0) - \int_0^1 \nabla\cdot u_t(x_t)\\,dt. We integrate the flow and its divergence numerically (RK4, 64 steps), computing the divergence exactly from the Jacobian, and obtain log⁡πβ\log \pi_\beta of a steered action by integrating the base flow backward from it. We estimate the KL between the unclipped continuous-time policies; the deployed sampler uses 10 Euler steps followed by clipping. We then estimate KL(πα∥πβ)\mathrm{KL}(\pi^\alpha \Vert \pi_\beta) by Monte Carlo over 256 steered actions per state, with no density model fit to samples. At α=0\alpha=0, where the true KL is 0, the estimate is −0.003-0.003 to −0.004-0.004 nats. For (2), in each state we find the tilted policy πβeτQˉ/Z\pi_\beta e^{\tau \bar Q}/Z whose E[Qˉ]\mathbb{E}[\bar Q] equals that of πα\pi^\alpha, where Qˉ\bar Q is the pessimistic ensemble critic that the sampler steers with (§4). By the closed form above, no policy reaches that value with less KL. The ratio of its KL to the KL of πα\pi^\alpha therefore shows how close πα\pi^\alpha is to solving the objective; a ratio of 1 means it solves it exactly. Setup: humanoidmaze-large tasks 1 and 4, 64 states, 4 seeds, end of offline training; gain is the mean over states, KL and ratio are medians (full table with means: App. D, Table 4).

QPILOTS-U (task 1 / task 4)α=0.05\alpha = 0.050.10.2 (tuned)0.4
Gain in E[Qˉ]\mathbb{E}[\bar Q] (in base-policy std. of Qˉ\bar Q)0.38 / 0.370.75 / 0.741.51 / 1.493.14 / 3.09
(1) KL spent, KL(πα∥πβ)\mathrm{KL}(\pi^\alpha \Vert \pi_\beta) [nats]0.09 / 0.080.34 / 0.321.32 / 1.295.19 / 5.30
(1) Share of the total KL held by the 10% of states with the largest KL (uniform = 10%)19% / 22%15% / 16%14% / 14%14% / 13%
(2) KL of the tilted policy with the same gain [nats]0.07 / 0.060.27 / 0.261.06 / 1.004.14 / 4.04
(2) Ratio (1 = objective solved exactly)0.81 / 0.830.79 / 0.800.79 / 0.770.79 / 0.76

(1) How much does the policy move? About 1.3 nats per state at the tuned α=0.2\alpha = 0.2 (second row). Each doubling of α\alpha multiplies the KL by about 4 (task 1: 0.09, 0.34, 1.32, 5.19 nats), so KL grows smoothly, as ≈33α2\approx 33\alpha^2. The KL is also spread evenly across states. For each α\alpha separately, we rank the 64 states by their KL and take the 10% with the largest KL (6 states); they hold only 13–22% of the total KL (third row), close to the 10% they would hold if every state had the same KL. So α\alpha plays the role of the per-state KL budget δ(s)\delta(s).

(2) Is the objective satisfied? For the value gain QPILOTS-U achieves, the KL-optimal tilted policy would need an estimated 1.0–1.06 nats instead of 1.3, a ratio of about 0.8 at every α\alpha. The unrescaled σt\sigma_t drift (the schedule of part (a); both drifts here use the same Tweedie gradient approximation) is further from the objective: for the same value gain it spends 2.3×2.3\times (task 1) to ≈6×\approx 6\times (task 4) more mean KL, concentrated on a few states (the 10% of states with the largest KL hold 58% (task 1) and 92% (task 4) of it, vs. 14% for QPILOTS-U), because its step size scales with the critic-gradient magnitude, which varies widely across states. So on these checkpoints, the rescaling the reviewer points to reaches comparable value gains with less KL than the drift the theory prescribes.

On cube-triple (25-dimensional action chunks) the ratio is lower (0.41–0.45), so we state these findings for humanoidmaze.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.