acceptodds
Under review as a conference paper at ICLR 2027

CoVE-PPO: Change-of-Variables Exact-Density PPO for Flow Policies

Abstract

Flow models transport probability density through deterministic maps, creating expressive multi-modal distributions attractive for robotic manipulation. Computing the standard policy gradient requires differentiating through action log-density, which is available through change of variables. Prior methods avoid the exact density citing Jacobian-trace estimation noise, cost, or numerical instability. We revisit these objections, and show that change of variables is tractable and effective. Sharing trace estimation probes across the ratio makes the estimator's error scale with the Jacobian difference rather than the Jacobian, and as few as 1 probe matches the full trace's learning performance. Much of the instability comes from a stiff forward solve, and can be fixed by placing integration nodes within the solver's stability region. We introduce CoVE-PPO, which optimizes the change-of-variables density of the executed action, with no surrogate or augmented variable: in a general form for any velocity field, and in EDC, a parameterization whose divergence is a bounded forward network output giving stable solves and free divergence. Across state- and vision-based robot manipulation, we show our method matches or exceeds on-policy baselines at matched or higher sample efficiency.

Primary area:
reinforcement learning
Keywords:
flow matching, reinforcement learning fine-tuning, proximal policy optimization, continuous normalizing flows, exact likelihood, robot manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xjPjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoVE-PPO, a reinforcement learning fine-tuning framework for flow-matching policies based on action probability densities computed through the change-of-variables (CoV) formula. Unlike existing approaches that rely on surrogate likelihood ratios or treat intermediate denoising steps as additional decisions, CoVE-PPO directly estimates action log-densities by solving the flow ODE in reverse and integrating the divergence of the velocity field, enabling PPO-based policy optimization. To improve computational efficiency and numerical stability, the authors share random probes in the Hutchinson trace estimator between the old and new policies to reduce the variance of likelihood-ratio estimation. They also analyze the stiffness of flow ODEs and adjust the placement of numerical integration nodes based on the spectral properties of the velocity-field Jacobian to improve ODE solver stability. Furthermore, the authors introduce an Explicit-Diagonal Coupling (EDC) flow parameterization that allows the velocity-field divergence to be computed directly from network outputs without additional Jacobian differentiation, along with a Split-EDC variant that enables exact density computation for the discrete flow. Experiments on simulated robotic manipulation tasks from Robomimic and MimicGen compare CoVE-PPO with existing policy optimization methods, while ablation studies examine the effects of probe sharing, numerical integration strategies, and flow parameterization.

Critical strengths
  1. The paper addresses a clear and meaningful problem: making PPO optimization with change-of-variables action densities practical for flow policies. Its motivation and presentation clearly connect the computational and numerical obstacles to the proposed solutions, including probe sharing, adjusted integration nodes, and EDC parameterization. This contributes to my positive assessment because the paper demonstrates a viable optimization approach and identifies concrete mechanisms that make it practical in the evaluated settings.

  2. The key designs have explicit mathematical foundations and corresponding empirical support. The probe-sharing analysis relates log-ratio estimation noise to differences between the old and new policy Jacobians, while the EDC construction enables direct divergence computation. Targeted ablations complement evaluations across state- and vision-based manipulation tasks. This connection between analysis and experiments increases my confidence that the reported improvements have identifiable technical causes and that the findings can inform subsequent research.

Critical weaknesses
  1. The experiments provide limited evidence about the computational and expressivity trade-offs as model size, action dimensionality, and action horizon increase. The evaluation primarily uses single-task policies with relatively short action chunks. Reverse integration and trace-related differentiation may become more costly at larger scales, while EDC imposes structural constraints whose impact on more complex policy distributions remains unclear. This limits my assessment of the approach’s broader practical value, since its favorable performance–cost trade-off has yet to be demonstrated beyond the evaluated settings.
  2. The empirical support for some central mechanisms has limited coverage. The main probe-sharing ablation is conducted only on Square, and the numerical integration ablation uses the same task with a single seed. These results leave uncertainty about whether the observed effectiveness of single-probe estimation and adjusted integration nodes is robust across tasks and training seeds. This matters to my overall assessment because these mechanisms underpin the practicality of direct CoV-based PPO; limited evidence of their robustness reduces confidence in the generality of the paper’s central findings.
Questions for authors
  1. Appendix B reports computational costs at matched network sizes. How do update time and peak memory scale with action dimensionality, action horizon, and network size for the general and EDC variants, based on available measurements or complexity analysis?

  2. Beyond matching initial BC success rates, what evidence clarifies EDC’s expressivity–efficiency trade-off, particularly the impact of the diagonal bound α on BC fitting and final RL performance?

  3. Beyond the ablations in Figures 3–4 and the diagnostics in Appendix E, what evidence supports the continued reliability of single-probe estimation and the pretrained integration mesh as the policy changes during PPO fine-tuning?

3 comments

Thank you for the review. We address each point below; the General Response (GR) has the full details.

Ablation robustness (W2).

  • Probe sharing: We extend the ablation to Square, Can and Tool Hang with 3 seeds each (Fig. 3). The shared single probe, four probe and exact trace arms are indistinguishable within seed spread on all three tasks. FFJORD-style and independent probes collapse on the two additional tasks.
  • Integration: We extend the ablation to Tool Hang and run 3 seeds on all settings (Fig. 4). The stiffness-graded mesh again matches a 200-NFE uniform grid using 20–24 NFE, while a uniform grid at the same cost does not.

See GR-B.

Reliability during fine-tuning (Q3). We measured both mechanisms during training (GR-A, App. H and F.2). Every 5 PPO iterations we compared the log-ratio against an exact trace reference and re-measured stiffness on the current policy.

  • Single probe: The error stays at a constant level throughout training. It is about a quarter of a typical log-ratio, which is not small, but as Fig. 3 shows it does not slow learning.
  • Mesh: Stiffness grows over training as the policy narrows, but stays trainable on the fixed mesh. The margin built into the mesh leaves room for this growth. We did not need to re-derive the mesh in our experiments, though this remains an option.

This matches the analysis in Section 3.3. Shared-probe noise depends on how much the Jacobian changes within one PPO update, which the trust region keeps small at every point in training, not on the total change since BC. See GR-A.

Computational scaling (W1, Q1). We measure time and memory against action dimension and network width (GR-C, App. I).

  • Action dimension and horizon: We lengthen the chunk to give DD from 28 to 112. The single probe, four probe and EDC variants stay roughly flat in memory and time. The exact trace grows with DD, and becomes infeasible as the action dimension increases. Training with one shared probe matches performance as dimension size increases.
  • Network size: Memory grows roughly linearly with width for every method, including FPO.
  • Complexity: With NN field evaluations per solve, scoring costs NN forward passes for EDC, N+N⋅PN + N \cdot P JVPs for the general field, and N+N⋅DN + N \cdot D JVPs for the exact trace. Memory is dominated by activations stored for backprop through the NN evaluations, so it scales with width and NN, not DD.

Our chunk lengths of 4–8 match the settings of the baselines we compare against. We leave VLA-scale fine-tuning for future work, and the revised Limitations reflect this.

EDC expressivity and α\alpha (Q2). In Appendix E, we show BC training curves for a sweep of α\alpha compared to a general flow. More restrictive bounds (lower α\alpha) can reduce BC success, and less restrictive bounds lead to a stiffer flow. Stiffness, hμh\mu, scales linearly with α\alpha. For our work, we chose α\alpha such that the flow would not be stiff, and that still maintained BC performance. Stiffer EDC flows with a higher bound could potentially benefit from the same stiffness mesh as the general field, however, we found our choice of α\alpha to be effective for both BC and RL fine-tuning.

We also show RL curves for the same sweep of α\alpha. We train on Square from performance matched BC checkpoints, and compare training curves and final performance. We observe that the stiffest flow, or highest α\alpha, has lower final performance. The least stiff flow starts lower, but eventually reaches performance near more moderate values of α\alpha.

We hope these additions address your concerns about robustness and scaling, and we are happy to answer further questions.

As the discussion period ends today, we'd like to check whether our revisions addressed your concerns. If any remain, we are happy to clarify. Thank you again for your time.

Thank you for the response. My main concern remains the trade off between policy expressivity and computational cost. The general variant still requires backpropagation through a reverse ODE solve, and EDC reduces that cost by restricting the velocity field, so I’m keeping my rating at 2.

Reviewer_iZ2oOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies PPO fine-tuning of flow policies using change-of-variables density. The authors use shared probes to reduce trace-estimation noise, address flow stiffness, and introduce EDC to make density computation cheaper. These designs are useful; the numerical results show competitive performance and efficiency. However, the evaluation remains limited to task-specific policies, without validation on VLA models. Comparisons with closely related methods such as GenPO and PolicyFlow are missing. The scope of the exact-density and some notation need clearer clarification.

Critical strengths
  1. Sharing probes between the old and new policies is a simple and useful design. It reduces trace-estimation noise and allows training with only one probe, with performance close to using the full trace.
  2. The paper provides a useful analysis of flow stiffness and its effect on training stability. Placing integration nodes according to stiffness helps reduce the number of network evaluations while keeping training stable.
  3. The EDC design makes divergence available directly from the network output, reducing the cost of density computation. It supports a split solver with an exactly invertible discrete mapping.
Critical weaknesses
  1. The evaluation in Section 4 is limited to task-specific policies, without experiments on VLA models. The scalability to larger policies and higher-dimensional action spaces remains unverified.
  2. Section 2 discusses GenPO and PolicyFlow, but Section 4.1 does not include them as experimental baselines.
  3. The likelihood-ratio error under the solver settings used in the main experiments is not directly quantified. Although the authors explain the numerical approximations, their practical effect on training remains unclear.
  4. The notation in Section 3 and Appendix D is not fully consistent, makes parts of the method difficult to follow.
  5. Please refer to my question for the details, many thanks.
Questions for authors
  1. The experiments presented in Section 4 are still limited in numerical scale, and there is no evaluation over VLA models. It is not clear whether the proposed method can scale to larger policies and higher-dimensional action spaces.

  2. GenPO and PolicyFlow are discussed in Section 2, but the authors are not considered the corresponding comparisons in Section 4. The achievable advantage over these existing related methods is still unclear.

  3. The authors explain the numerical approximations and introduce split-EDC to address discrete-map exactness. Please quantify the likelihood-ratio error under the solver settings used in the main experiments and discuss its effect on training.

  4. The notation in Section 3 and Appendix D is a little confusing, such as the use of (d) and (D), and the meaning of Eq. (21). Please check the consistency and provide a notation table for better presentation.

2 comments

Thank you for the review. References to GR are to the General Response.

VLA models and scale (W1, Q1). We evaluate in the same setting as our baselines, and leave multitask and VLA policies for future work; the revised Limitations now state this. To address scaling as far as we can within the discussion period, GR-C measures time, memory and estimator error with action dimension up to 112 and network width up to 4096. The exact trace becomes infeasible at the largest dimension, while the shared probe estimator and EDC stay roughly flat in cost. Training with one shared probe matches the performance of four shared probes as action dimension size increases.

GenPO and PolicyFlow (W2, Q2). Both are designed and evaluated primarily for training from scratch, while we fine-tune BC-pretrained policies. GenPO executes the average of a doubled action through an invertible map whose Jacobian determinant is θ\theta-independent, so the entropy of the augmented variable cannot change during training. PolicyFlow approximates a ratio from velocity differences at interpolation points, using Gaussian noise added at the action endpoint. Neither GenPO nor PolicyFlow provides a BC-initialization procedure, and while we focus on fine-tuning BC-initialized policies, future work could compare GenPO, PolicyFlow, and our method from scratch. Our method optimizes the density of the actions produced by the flow directly, rather than an approximation through velocity variation or the density of an augmented variable.

Likelihood-ratio error (Q3). We show through our experiments that the ratio error does not affect learning, and quantify it in Appendix H.

  • General field with one probe: The error is constant throughout training. Fig. 3 shows it does not slow learning. Fig. 12 quantifies it, and shows error relative to typical ratios.
  • Split-EDC: It is exact for the executed discrete policy and performs comparably to EDC (Fig. 8), so we do not believe discretization error limits performance.

Notation (Q4). We now use DD for the flow dimension throughout, unify the superscripts in Eqs. 14, 20 and 21, and add a notation table in App. K.

Eq. 21 is the log-density of the action under the split solver. It is the base log-density at the inverted latent, minus hh times the sum over steps of three terms: half the A-block diagonal at the first substep, the B-block diagonal, and half the A-block diagonal at the third substep. The A block is evaluated twice because the B substep changes its input in between.

As the discussion period ends today, we'd like to check whether our revisions addressed your concerns. If any remain, we are happy to clarify. Thank you again for your time.

Reviewer_kyD7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that you can just run PPO on flow policies with the exact change-of-variables density of the executed action, instead of the surrogates people have been using (FPO-style ratios, denoising-chain likelihoods, augmented-variable flows). Three things make this work: (1) reusing the same Hutchinson probes for the old and new log-density inside the PPO ratio, so the estimator noise scales with the Jacobian change and one probe is enough; (2) diagnosing the reported instability as stiffness in the forward solve, fixed by placing RK4 nodes based on the measured Jacobian spectrum of the pretrained flow; and (3) an EDC parameterization, a coupling-style velocity whose Jacobian diagonal is a bounded network output, so the divergence comes for free. A split-EDC variant makes the discrete and continuous densities match exactly. Experiments fine-tune BC policies on Robomimic (state and pixel) and four MimicGen tasks against DPPO, ReinFlow, FPO/FPO++ and RLDT, with ablations on probe sharing and discretization.

Critical strengths

It goes back and actually checks the reasons people gave for avoiding the exact density, and finds that they mostly don't hold up. The probe-sharing trick is simple, and Figure 3 makes the point very clearly: independent single probes fail, while shared single probes train about as well as the exact trace. The stiffness explanation (narrow action dims, e.g. gripper, make the flow-matching optimum stiff near t=1, and the forward solve is the real problem) is also useful beyond this paper. Overall it gives a cleaner and less biased alternative to surrogate-ratio methods, which I think the community would find useful.

Critical weaknesses

The individual pieces aren't very new. Sharing probes across the ratio is essentially common random numbers. EDC is close to affine coupling / hollow networks (which the paper acknowledges), and split-EDC is basically RealNVP trained with flow matching. So the contribution is mostly the empirical message that the exact density works fine for PPO, which puts more weight on the experiments.

The experiments are a bit thin and not fully apples-to-apples. Much of the gain seems to come from EDC rather than the exact density itself: the general-field variant is clearly behind EDC on several tasks (state Transport and Tool Hang, pixel Square and Transport), and on state Transport it is on par with FPO++ on the same general flow (Fig. 7). Starting BC success also differs noticeably across methods in Fig. 1 (e.g. RLDT starts around 0.6 on Square vs. roughly 0.35-0.45 for the others) despite the checkpoint matching. Main results use 3 seeds, and the discretization/stiffness study (Fig. 4) is a single seed on a single task. Main curves evaluate CoVE-PPO with stochastic sampling, while DPPO/ReinFlow use their own eval protocol, and the deterministic comparison is only against FPO++. The "exact density" story is also softened by quite a few stabilizers in practice (large clip 0.5, dual clip, zeroing samples with |log r| > 3, a stiffness mesh calibrated once on the pretrained policy and reused across tasks). Finally, everything is on small chunks (4-8 steps) with small policies, while the motivation is large flow policies like pi0, so it's unclear how probe noise and stiffness scale.

Questions for authors
  1. How does the method hold up with larger action chunks / higher-dimensional actions (e.g. chunk 16-50)? Does one shared probe still suffice there?
  2. How much do the stabilizers matter? What happens without the |log r| > 3 mask, or with a standard clip like 0.2?
  3. The stiffness mesh is computed from the pretrained policy, but the flow changes during RL. Do you see stiffness drift during fine-tuning, and does the mesh need recomputing?
  4. Does the EDC restriction (and the alpha bound) cost anything at the BC stage before RL? Matching BC performance through checkpoint selection could hide that.
3 comments

Thank you for the detailed review. References to GR are to the General Response.

Novelty. Probe sharing is a variation on a common variance reduction technique (common random numbers). EDC is close to hollow and affine coupling networks. Split-EDC is RealNVP-style coupling trained with flow matching. Our contributions are demonstrating effective CoV fine-tuning, and showing why the usual objections to the exact density do not hold for PPO, with an explanation for each:

  • The ratio, not the density, sets the estimator noise. That is what makes one probe enough, and it enables low-cost change-of-variables computation.
  • The instability comes from a stiff forward solve, which node placement fixes and better reverse solves cannot.

We also see the split-EDC framing as new. It unifies the discrete-map and continuous change-of-variables densities, so for split-EDC the objective is exact for the executed policy.

EDC versus exact density. Some RL gain comes from EDC, but Fig. 7 is an important counterpoint. Fig. 7 compares CoVE and FPO++ on the same flow, and in Fig. 1 FPO and FPO++ already run on our EDC flow. On the EDC flow, CoVE outperforms FPO++ on all three tasks. On the general flow, CoVE outperforms on Square and Tool Hang, while FPO++ is marginally ahead on Transport. EDC improves both objectives, so its benefit is not exclusive to ours. The general field shows the exact density works without an architectural restriction, and EDC in practice shows faster performance gains. The CoV density outperforms on a matched flow base in 5 of 6 runs.

BC start. Checkpoint selection and evaluation is noisy, and we acknowledge this. On Square RLDT's checkpoint is higher than the others' starts, as we did not apply the same matched start selection rule to RLDT. This has been corrected in the updated draft.

Evaluation protocol. Diffusion and flow based methods have different evaluation modes available, and we match them as closely as we can. Diffusion based methods have noise added during denoising, and ReinFlow adds injected noise mid flow. Mid flow noise and the initial Gaussian (z0z_0) can be set to zero for evaluation, however if all noise is removed the policy loses multimodality, so most techniques are evaluated with the full initial Gaussian rather than without.

  • DPPO and ReinFlow: DPPO uses either DDIM with no added noise, or DDPM without their added exploration noise floor and a noise free last step. ReinFlow uses the full stochastic base distribution, but removes their learned added noise during evaluation. We match ReinFlow's evaluation mode, and are comparable to the DDIM setting for DPPO. Their Robomimic training uses DDPM sampling, which we match to avoid unfair comparison.
  • FPO and FPO++: These methods have no mid-flow noise, FPO uses the full initial Gaussian while FPO++ sets the full initial Gaussian to zero for evaluation.
  • CoVE: We primarily evaluate CoVE with the full stochastic base distribution, which is the same policy RL optimizes. App. A shows that setting the full initial Gaussian to zero helps on Transport and hurts on Tool Hang. We compare this mode against FPO++ because it is its stated evaluation mode, and it is most comparable to our method.

(continued below)

Seeds. We agree more seeds would tighten the main curves. We put the discussion-period compute into the ablations that previously had one seed, which now have 3 seeds across tasks (GR-B).

Stabilizers (Q2). The log ratio mask is part of our method, but the large clip threshold is not a stabilizer: a larger clip allows more policy movement per update. 0.5 is a tuned value for our method, and it is too aggressive for FPO, which collapses with it. 0.2 limits policy movement, and learns marginally slower, or has a lower early rise. In terms of success rate difference, clip 0.5 − clip 0.2, on Square at 1M the difference is 0.07, and at 5M it is −0.01, and for Tool Hang it is 0.08 and 0.01 at 3M.

For dual clip and the log ratio mask, we show with a new ablation in Appendix J their effects. We show dual clip is removable without a noticeable change in performance, but the log ratio mask can have large performance impacts on the general field's training.

Larger chunks (Q1). GR-C lengthens the chunk to give DD from 28 to 112. The single probe, four probe and EDC variants stay roughly flat in cost, while the exact trace scales poorly, and we are unable to train larger dimension sizes. Four probes estimate the trace more accurately at high DD, but learning speed is matched between one and four probe runs as the dimension size increases.

Stiffness drift (Q3). Yes, we observe drift (GR-A). Stiffness grows over training as the policy narrows, but stays trainable on the fixed mesh, and a single mesh sufficed in our experiments. Re-deriving the mesh later in training is a cheap option if needed.

On reuse across tasks: each Robomimic task has its own mesh. For MimicGen, one mesh built on Threading is reused, and it stays at 2.0–2.4 on the other tasks (Table 1).

EDC at BC (Q4). We show BC training curves, that show success rates over training for a variety of α\alpha values, which should help clarify the effect of the bound on BC performance.

Thank you for your detailed response and additional experiments. The expanded ablation studies and the EDC comparison have addressed most of my concerns, and I will maintain a weak accept rating.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response (1/2)

We thank the reviewers. They found the problem clear and meaningful (xjPj), probe sharing a simple and useful design (iZ2o), and the method a cleaner, less biased alternative to surrogate-ratio methods (kyD7). All three found probe sharing, the stiffness diagnosis and EDC useful. The shared questions were whether the numerical approximations stay reliable during RL, how the method scales, what EDC's restriction costs, and whether the ablations hold beyond one task and seed. We answer these here with new experiments. Each individual reply summarizes the parts relevant to that reviewer. The revised PDF marks changes in blue, and a separate "Summary of revisions" comment lists them.

A. Do the single shared probe and the fixed mesh stay reliable during fine-tuning? (xjPj Q3, kyD7 Q3, iZ2o Q3)

We ran general CoVE-PPO with one probe, EDC, and split-EDC on Square and Tool Hang. Every 5 iterations, we saved the policy before and after one update and re-scored a fresh 1,024-sample rollout against the exact trace on the same mesh, which gives the probe error.

The single-probe log-ratio differs from the exact trace by about a quarter of its value, and does not increase during training in the examined tasks. The absolute error being about a quarter of the value may be relatively large, but as we show empirically this error does not lead to slower learning. This is now shown in Fig. 12 in App. H.

For the fixed mesh, we re-measured the 99th-percentile Jacobian spectral radius µ on the current policy, using the protocol that built the mesh. Fig. 11 in App. F.2 shows that stiffness grows over training on our tasks as the policy narrows to the optimal policy distribution, but remains trainable. We note that providing a margin in the chosen stiffness mesh leaves room for this growth, and while not needed in our experiments, re-deriving a mesh later in training remains an option.

Summary: The single probe adds noise at a constant level throughout training. The mesh drifts toward its stability limit through training, however neither shows up in learning.

B. Ablations now cover more tasks and 3 seeds (xjPj W2, kyD7)

We extend the probe sharing ablation to 3 tasks, Square, Can, and Tool Hang. Fig. 3 is updated. The shared single probe, four probe, and exact trace arms are indistinguishable within seed spread on all tasks. Other probe variations, FFJORD style or independent probes collapse on the two additional environments, demonstrating the claim holds consistently across environments.

For our mesh ablations, we extend to an additional task and run 3 seeds on all settings. Fig. 4 is updated. The pattern again holds across seeds and environments, showing that instability is derived primarily from the forward integration of the flow. Reverse integration needs less resolution when forward integration is more accurate to the underlying flow, as shown by the success rate curves.

Summary: Additional tasks and seeds support our conclusion, on both probe sharing and stiffness analysis.

(continued in General Response 2/2)

AuthorsOpenReview ↗

General Response (2/2)

C. Scaling with action dimension, chunk length and width (all reviewers)

We leave VLA fine tuning and scaling this method for future work, reflected in the revised Limitations. To answer the cost and estimator questions directly, we trained Square with larger chunk lengths, giving action dimensions from 28 to 112, and profiled up to dimension 896 with all else equal to earlier runs. For larger networks, we run the method at wider network sizes.

In App. I, we show that probe sharing single probes and 4 probes learn at larger action dimensions, and single probe training remains as effective as more expensive four probe runs. At the larger dimensions, exact trace runs are infeasible, and so the 4 shared probe runs serve as a measure for whether or not more trace estimation accuracy helps. We also describe how the method scales in memory and time cost.

  • Action dimension: Single, four probe, and EDC variants, memory usage and time cost increase modestly with dimension, solely for the increased network size required. We show memory and time cost as action dimension increases in Appendix I. For exact trace memory and time cost scales with dimension size, rendering training infeasible. We show memory growth, and note that time cost scales with N⋅DN \cdot D.
  • Single-probe error: We measure error directly, and compare training runs with 1 and 4 shared probes as action dimension increases. We show in Fig. 12 that probe estimation error does not grow with time, and in Table 3 that its relative error does not grow with DD. As expected, the 4 probe setting has greater accuracy at higher dimensions. However, training performance is consistent between 1 and 4 probes, with both learning similarly.
  • EDC: It has no estimator error, and its update time grows comparably to other methods as dimensions increase.
  • Width: We measured peak memory per minibatch of 1000 at D=28D=28, for a 3 layer actor. Memory grows roughly linearly with width for every method.
    • EDC vs FPO: At matched field evaluations, EDC and FPO cost the same, since both are forward passes. As run, CoVE-EDC uses 2.3–2.7× the memory and about 2.9× the step time of FPO on the same network.
    • General field: At matched evaluations, general CoVE-PPO with single probe costs 2.3–2.5× the memory of FPO on the plain flow MLP, due to the JVPs for trace estimation.
  • Complexity: With NN field evaluations per solve, scoring costs NN forward passes (EDC), N+N⋅PN + N \cdot P JVPs (general field) or N+N⋅DN + N \cdot D JVPs (exact trace). Memory is dominated by activations kept for backprop through the NN evaluations, which scale with width and NN, not DD for our variants.

D. What does EDC's restriction and α\alpha cost? (xjPj Q2, kyD7 Q4)

In Appendix E, we show BC training curves for a sweep of α\alpha compared to a general flow. More restrictive bounds (lower α\alpha) can reduce BC success, and less restrictive bounds lead to a stiffer flow. Stiffness, hμh\mu, scales linearly with α\alpha. For our work, we chose α\alpha such that the flow would not be stiff, and that still maintained BC performance. Stiffer EDC flows with a higher bound could potentially benefit from the same stiffness mesh as the general field, however, we found our choice of α\alpha to be effective for both BC and RL fine-tuning.

We also show RL curves for the same sweep of α\alpha. We train on Square from performance matched BC checkpoints, and compare training curves and final performance. We observe that the stiffest flow, or highest α\alpha, has lower final performance. The least stiff flow starts lower, but eventually reaches performance near more moderate values of α\alpha.

AuthorsOpenReview ↗

Summary of revisions

Changes are marked in blue in the revised PDF. Section and figure numbers refer to the revised version.

New experiments and appendices

  • App. E (new), EDC restriction and α\alpha: BC training curves for α∈{2,4,8,16}\alpha \in \{2, 4, 8, 16\} against the general field, under both the RK4 deployment sampler and 5-step Euler (Fig. 9). RL curves on Square for the same sweep, each starting from a BC checkpoint trained with that α\alpha (Fig. 10). [kyD7 Q4, xjPj Q2]
  • App. F.2 (new), stiffness during training: hμh\mu on the fixed mesh, re-measured every 5 iterations on Square and Tool Hang for the general field, EDC and split-EDC, 3 seeds (Fig. 11). [kyD7 Q3, xjPj Q3]
  • App. H (new), probe error during training: single shared-probe log-ratio error against the exact trace across training, shown with the median ∣log⁡r∣|\log r|, on Square and Tool Hang, 3 seeds (Fig. 12). [xjPj Q3, iZ2o Q3]
  • App. I (new), scaling: [all reviewers]
    • Training with P=1P=1 and P=4P=4 shared probes at chunk lengths 4, 8 and 16 (D=28D = 28–112112), 3 seeds (Fig. 13).
    • Probe error by chunk length (Table 3).
    • Update time and peak memory for D=28D = 28–896896 (Tables 4–5).
    • Memory and runtime for widths 512–4096 (Table 6).
    • Methods covered: EDC, the general field with P=1P=1 and P=4P=4, FPO, and the exact trace.
  • App. J (new), stabilizer ablation: the ∣log⁡r∣>3|\log r| > 3 mask and dual clip each removed, for EDC and the general field, on Square and Tool Hang, 3 seeds (Fig. 14). [kyD7 Q2]
  • App. K (new), notation tables. [iZ2o Q4]

Updated experiments in the main text

  • Fig. 3 (probe sharing): extended from Square to Square, Can and Tool Hang, 3 seeds each. [xjPj W2, kyD7]
  • Fig. 4 (discretization and stiffness mesh): extended from one seed on Square to 3 seeds on Square and Tool Hang. [xjPj W2, kyD7]
  • Fig. 1: RLDT on Square now starts from a BC checkpoint chosen by the same matching rule as the other methods. [kyD7]

Text and notation

  • Notation: DD is now used for the flow dimension throughout (previously dd in places), with D=HdaD = H d_a defined in Sec. 3.2. The trace estimator is written with XX, the quadrature weights with ωn\omega_n, and the probe indexing and the variance V(X)V(X) are stated explicitly. The split-EDC substep notation is unified: hh replaces τ\tau and Δt\Delta t, and jkA,1,jkB,jkA,2j^{A,1}_k, j^{B}_k, j^{A,2}_k are used consistently across Eqs. 14, 20 and 21. A sentence after Eq. 21 explains why the A block is evaluated twice. [iZ2o Q4]
  • Sec. 3.4 / App. F: the stability condition is stated in terms of the spectral radius μ\mu and the target τ\tau. The log-ratio mask is described precisely: the gradient of any sample with ∣log⁡r∣>3|\log r| > 3 is zeroed.
  • Sec. 4.2–4.3: the text is updated for the added tasks and seeds.
  • Sec. 5 (Limitations): now states that from-scratch training, VLA-scale fine-tuning and multitask fine-tuning are left for future work. [iZ2o W1, xjPj W1]
  • Minor corrections to the derivation in App. F.1 and the block partition in App. L.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.