acceptodds
Under review as a conference paper at ICLR 2027

Path-Integral Value-Control Matching with Applications to Smoothing and Game

Abstract

Stochastic optimal control (SOC) provides a computational framework for a broad range of machine-learning problems. Policy-based solvers learn feedback controls directly, but often require costly full-horizon rollouts and may suffer from unstable likelihood-ratio weights during off-policy training. Local path-integral value learning enables short-trajectory updates and experience replay, but its scalar objective does not directly supervise the value gradient that determines the control. This limitation is particularly relevant to control-sensitive applications such as Bayesian smoothing and mean-field games. Building on Path-Integral Value Matching (PI-VM), we propose Path-Integral Value-Control Matching (PI-VCM), which adds direct control supervision while retaining local value learning and replay. We spatially differentiate the local path-integral value recursion to derive a corresponding control recursion, allowing the same short trajectories to provide both value and control targets for a shared value network. Affine sensitivity caching and local likelihood correction further enable off-policy experience replay without repeated adjoint propagation. Experiments demonstrate improved control accuracy over baselines on SOC benchmarks and higher-dimensional instances, together with competitive performance on Bayesian smoothing and mean-field games.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Stochastic Optimal Control, Nonlinear Smoothing, Mean-field Game

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GaZpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Path-Integral Value-Control Matching (PI-VCM), an extension of Path-Integral Value Matching (PI-VM) that jointly supervises a value function and its gradient-derived control.

The authors differentiate the existing local path-integral value recursion to construct control targets from short stochastic trajectories. They also introduce affine sensitivity caching, which allows adjoint targets to be updated when the continuation value changes without recomputing backward sensitivities along stored trajectories.

The method is evaluated on stochastic optimal control benchmarks, including problems with up to 200 state dimensions, as well as Bayesian smoothing and mean-field games. The experiments report improved control accuracy over PI-VM and competitive performance against several existing methods.

Critical strengths

Improved control accuracy across multiple benchmarks. The paper demonstrates substantial improvements in control accuracy over the original PI-VM method. In particular, the control MSE decreases from 2.557×10−32.557\times10^{-3} to 2.30×10−42.30\times10^{-4} on the 200-dimensional quadratic OU benchmark. The experiments on Bayesian smoothing and mean-field games provide additional evidence that directly supervising the gradient-derived control can be useful beyond standard stochastic control benchmarks.

A practical framework for reusing stochastic trajectories. The combination of local path-integral value-control targets, likelihood-corrected off-policy replay, and affine sensitivity caching provides a coherent computational framework. In particular, the affine representation of the adjoint permits previously collected trajectories to be reused when the terminal continuation gradient changes. This may reduce repeated sensitivity calculations during iterative value-function learning.

Critical weaknesses

Limited methodological novelty beyond existing value matching and derivative supervision. Under the assumed control-affine dynamics and quadratic control costs, the optimal control is already explicitly characterized by the HJB first-order condition, u∗=−λ−1σ⊤∇V∗u^\ast=-\lambda^{-1}\sigma^\top\nabla V^\ast. The proposed control target is obtained by differentiating the existing PI-VM Bellman recursion and applying this optimality relation. Jointly regressing the value function and its gradient-derived control therefore appears closely related to Sobolev or derivative-supervised Bellman learning. Although the construction is mathematically valid, I find it difficult to identify a fundamentally new control-learning principle beyond augmenting PI-VM with gradient supervision. The improvements over value-only PI-VM demonstrate the usefulness of this additional supervision, but do not by themselves establish its methodological novelty.

Insufficient evidence for the computational advantages of affine sensitivity caching. The affine identity as=Ks:τ∇V(τ,Xτ)+cs:τa_s=K_{s:\tau}\nabla V(\tau,X_\tau)+c_{s:\tau} follows from the linearity of the adjoint equation with respect to its terminal condition. While caching KK and cc may accelerate repeated target updates, this requires additional computation and storage, including O(d2)O(d^2) memory per trajectory for a dense sensitivity matrix. The paper does not provide a sufficiently direct comparison of end-to-end computation time and memory consumption against recomputing the same control targets without caching. Consequently, it is difficult to determine whether affine replay offers a substantial computational advantage over standard sensitivity calculations. The high-dimensional experiments primarily use a structurally simple quadratic OU model, leaving the computational benefits for more general, densely coupled nonlinear systems uncertain.

Questions for authors

The paper is mathematically reasonable and reports encouraging empirical improvements. However, I remain unconvinced that the proposed approach represents a sufficiently substantial methodological advance over existing techniques.

  1. Methodological novelty of value-control matching

At the population level, the control target in Eq. (12b) follows from differentiating PI-VM’s local value recursion and applying the standard HJB feedback relation. The joint objective in Eq. (17) therefore admits a Sobolev-type interpretation, supervising the value and its control-relevant spatial derivatives.

Could the authors clarify which aspects of the control-target construction constitute the main methodological advance beyond this derivative-supervision principle?

I appreciate the distinction made in Appendix C.3 between the proposed empirical control target and direct differentiation of a finite-sample off-policy Monte Carlo value estimator. Does this distinction provide measurable statistical or computational benefits, such as lower gradient-estimation error, reduced variance, or lower differentiation cost?

Where both constructions are applicable, a comparison against an estimator obtained by differentiating through the proposal trajectories and their likelihood factors, using the same continuation function, collection policy, and trajectory budget, would help isolate the contribution of the proposed target construction.

  1. Computational benefits of affine sensitivity caching

The affine representation in Eq. (23) allows the adjoint to be refreshed when the terminal gradient changes, but the underlying affine dependence follows from the standard linear adjoint equation.

Could the authors quantify the computational advantages of explicitly caching KK and cc relative to recomputing the adjoint using conventional backward sensitivity calculations?

In particular, comparisons of wall-clock time, peak memory usage, and performance under matched computational budgets would be helpful. Since constructing and storing dense sensitivity matrices incurs nontrivial costs, it would also be useful to identify the dimensional regimes and replay frequencies for which the proposed caching method becomes advantageous.

Clarifying these two points would materially affect my assessment of the paper's novelty and practical significance.

5 comments
AuthorsOpenReview ↗

Response to Reviewer GaZp (1/3)

Methodological contribution and control-target construction (W1, Q1)

At the population level, the control target satisfies the standard HJB first-order condition, and Eq. (17) can be viewed as joint supervision of the value and its gradient, related to Sobolev training (Czarnecki et al., 2017), VGG-Flow and Efficient Adjoint Matching.

Sobolev training, however, assumes that derivative labels are given; in SOC, the optimal control has no ready-made labels, and they must be constructed from finitely many short trajectories collected off-policy. The construction determines whether the labels are reliable, whether they can enter experience replay, and how costly they are to refresh. Our contribution is such a construction and its application to Bayesian smoothing and MFGs, in three respects.

  1. Differentiate under the reference measure, then change measure. The adjoints in the control target are computed along collected paths using only the Jacobian of the reference drift, and Girsanov weights correct for off-policy collection; the collection policy is never differentiated (Appendix C.3). Directly differentiating a finite-sample off-policy value estimator, by contrast, goes through the sampled trajectories, the collection policy and the likelihood factors. PI-VCM control labels therefore need only first derivatives and do not require a differentiable collection policy.
  2. Affine sensitivity replay. After a value-network update, the control labels change only through the continuation gradient ∇V(τ,Xτ)\nabla V(\tau,X_\tau), and can be refreshed with the cached KK, cc or by adjoint recomputation along the stored paths, without recollection. This allows gradient supervision to be combined with short-segment bootstrapping and experience replay; AM, by contrast, uses full-horizon on-policy trajectories with terminal gradients and re-solves the whole adjoint at every update.
  3. An application framework for smoothing and MFGs. Section 5 casts smoothing and MFGs as observation-dependent and population-dependent SOC problems that PI-VCM can train directly, and specifies how to train and deploy them.
    • Smoothing. The observation likelihood enters the drift and cost through a pathwise transformation, without time derivatives of the observation path. At deployment, the learned value determines the posterior reweighting of the initial distribution and its gradient determines the posterior drift, so both the value and the control must be accurate, which is exactly the setting joint supervision targets.
    • MFGs. For a fixed population flow, the best response is an SOC problem; PI-VCM serves as the inner solver and is warm-started across outer iterations. After a population update, only the replay buffer needs to be cleared and refilled, at small cost.

Within this framework, replacing the inner solver with AM or SOCM, or using AM, SOCM, RE and LV under the same observation transformation, gives clearly worse results than PI-VCM.

AuthorsOpenReview ↗

Response to Reviewer GaZp (2/3)

Methodological contribution and control-target construction (W1, Q1, continued)

To test whether the distinction in Appendix C.3 brings measurable benefits, we implemented, as the reviewer suggested, an estimator that directly differentiates the finite-sample value estimator, using the same initial points, Brownian increments, continuation value function, collection policy and number of paths M as PI-VCM. The two finite-sample constructions differ as follows.

PI-VCM, Eq. (12b)Direct differentiation
Control targetuA=−∑iwiaiu_A=-\sum_i w_i a_i, where aia_i is the reference-drift adjoint along the collected pathuB=−∇xV^(x)u_B=-\nabla_x \hat V(x), V^=−log⁡1M∑ie−Ci(x)\hat V=-\log\frac1M\sum_i e^{-C_i(x)}
What is differentiatedOnly the reference-drift Jacobian DxbD_x b; the collection policy uˉ\bar u is not differentiatedThe sampled trajectories, the collection policy uˉ\bar u and the likelihood factors, introducing DxuˉD_x\bar u; second derivatives of the value network when uˉ=−∇Vθ\bar u=-\nabla V_\theta
Experience replayLabels depend on the value network only through the continuation gradient ∇V(τ,Xτ)\nabla V(\tau,X_\tau); stored segments can be refreshed directly with the cached KK, cc or by adjoint recomputationStored segments cannot be refreshed directly and must be resimulated with the current policy

Accuracy of a single control target. We compare the mean squared error of the two control targets on Dense LQ and Reaction–diffusion. Dense LQ is defined in the common response "Time and memory: caching vs. adjoint recomputation"; its exact optimal control follows from the Riccati equation. On Reaction–diffusion, the continuation value function is the value network trained by PI-VCM, and the collection policy is the same network or an earlier one, mimicking stale data in the replay buffer; the ground truth is estimated from 65,536 branches under the reference measure. The table reports the MSE ratio of direct differentiation to PI-VCM with M=8; a ratio below 1 means direct differentiation is more accurate.

Collection policyDense LQ, d=50Reaction–diffusion
Reference measure, i.e. zero control (the two targets coincide)11
Near optimal: exact optimal feedback / current value network0.000410.17–0.18
Severe deviation: random MLP perturbation 1.0 / network at update 1005.90.61–0.62

When the collection policy is near optimal, the pathwise derivative through the collection policy in direct differentiation acts as a control variate and yields lower error; this advantage shrinks as the deviation grows, and on Dense LQ under severe deviation PI-VCM is more accurate. We therefore do not claim that the PI-VCM control target has lower variance. Computationally, direct differentiation must keep the computation graph of the collected trajectories and take second derivatives of the value network; on Reaction–diffusion, constructing the same batch of control targets takes about 10× the memory of PI-VCM.

AuthorsOpenReview ↗

Response to Reviewer GaZp (3/3)

Methodological contribution and control-target construction (W1, Q1, continued)

Training comparison. On Reaction–diffusion, we keep the paper's network, replay settings and learning-rate schedule, and change only the control target and the way data are reused.

MethodTime per update (ms)Training time (min)Final path KL ↓
Direct differentiation, recollection at every update1,17998.80.517
PI-VCM, experience replay10016.80.164

Direct differentiation must recollect with the current policy at every update and is too slow, so it was trained only up to update 5,000, the end of the constant-learning-rate phase; PI-VCM completes all 10,000 updates in less than one fifth of the time and reaches a lower path KL.

After training, both control targets reach a path KL far below that of matched, value-only PI-VM (see the common response "Matched ablation"), so control supervision itself brings the main gain. Our construction makes this supervision usable with off-policy replay: direct differentiation must recollect with the current policy at every update and take second derivatives of the value network, whereas our control labels need only first derivatives, and stored segments can be refreshed directly after value-network updates. As a result, PI-VCM has the shortest training time and the best final accuracy.

Computational benefits of the affine sensitivity cache (W2, Q2)

End-to-end time and peak memory of explicit caching versus conventional adjoint recomputation are given in the common response "Time and memory: caching vs. adjoint recomputation". Below we add the accuracy under matched computational budgets and the dependence of the cache's benefit on the reuse ratio.

On Dense LQ with d=20, each of the three implementations is trained for 20,000 updates.

ImplementationTraining time (s)Control L2 at 500 s ↓
Cache491–4964.3–6.7e-5
Adjoint recomputation534–5403.8–6.8e-5
Per-step resampling1,665–1,6971.0–1.2e-4

The cache and adjoint recomputation produce identical control targets, and their errors at 500 s fall in the same range; at equal wall-clock time, both have about half the error of per-step resampling. The benefit of the cache also depends on the reuse ratio ρ, i.e. the average number of times each sample is used. The table below varies the replay refresh interval on Dense LQ with full-batch collection; entries are in ms per update.

Refresh interval / reuse ratio ρd=20 cached=20 adjoint recomputationd=50 cached=50 adjoint recomputation
1 / ≈112990.422194.6
5 / ≈541.939.963.739.1
25 / ≈2526.428.431.226.6
100 / ≈9824.926.426.825.8
400 / ≈39022.427.824.524.6

The higher the reuse ratio, the more the cache benefits; the higher the dimension, the higher the reuse ratio the cache needs to come out ahead.

These results show that the cache is faster when each trajectory is reused a number of times comparable to the dimension; otherwise, one can switch to the numerically equivalent adjoint recomputation, and both keep the benefit of experience replay.

Reviewer_a3LrOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Path-Integral Value-Control Matching (PI-VCM), a stochastic optimal control (SOC) solver that bridges the gap between policy-based control matching and local path-integral value learning (PI-VM). While PI-VM enables low-variance short-trajectory updates and off-policy replay, supervising only scalar values Vθ(s,x)V_\theta(s, x) does not guarantee accurate spatial gradients ∇xVθ(s,x)\nabla_x V_\theta(s, x), which directly determine the deployed feedback control uθ(s,x)=−λ−1σs⊤∇xVθ(s,x)u_\theta(s, x) = -\lambda^{-1}\sigma_s^\top \nabla_x V_\theta(s, x). PI-VCM resolves this by spatially differentiating the local path-integral value recursion on [s,τ][s, \tau] via a backward "lean adjoint" ODE, allowing the same short off-policy trajectory segments to jointly supervise both the scalar value target V^(s,x)\widehat{V}(s, x) and the gradient-derived control target u^(s,x)\widehat{u}(s, x) of a single value network. Because the segment adjoint is affine in the endpoint value gradient, a~s=Ks:τ∇xVˉτ(Xτ)+cs:τ\tilde{a}_s = K_{s:\tau}\nabla_x \bar{V}_\tau(X_\tau) + c_{s:\tau}, the authors introduce affine sensitivity caching of (Ks:τ,cs:τ)(K_{s:\tau}, c_{s:\tau}) alongside local Girsanov likelihood corrections Γs:τuˉ\Gamma^{\bar{u}}_{s:\tau}, enabling experience replay to refresh control targets in O(d2)O(d^2) via a single matrix-vector product without re-solving backward ODEs. Across analytic SOC benchmarks up to d=200d=200 dimensions, nonlinear Bayesian smoothing tasks (reaction–diffusion, double-well, Lorenz–63), and mean-field games (synthetic benchmarks and real-world crowd forecasting), PI-VCM substantially reduces control L2L_2 error—up to 11.1×11.1\times at d=200d=200 and 18×18\times in mean-field best-response gaps over PI-VM—while achieving superior posterior and population distribution recovery.

Critical strengths
  • Enabling off-policy experience replay for local control matching: While prior methods either perform on-policy full-horizon control matching (e.g., Adjoint Matching) or off-policy replay for scalar values only (PI-VM), PI-VCM is the first to combine direct control supervision with short-horizon off-policy replay. By differentiating under the reference dynamics prior to the Girsanov measure change and exploiting the affine structure of the lean adjoint ODE, the proposed affine sensitivity cache refreshes bootstrapped control targets in O(d2)O(d^2) via a matrix-vector product without re-solving backward ODEs during replay.

  • Theoretical clarity and variance reduction: The paper provides a clean variance-reduction proof (via the law of total covariance and the delta method) showing how short-segment bootstrapping suppresses the variance of both exponential path weights and weighted adjoints, alongside transparent analyses of continuation perturbation errors and cache validity.

Critical weaknesses

Weaknesses

  • Limited evaluation on complex, high-dimensional nonlinear problems and missing comparisons with key Girsanov- and matching-based baselines: Although the paper claims improved scalability and broad applicability, the experimental evaluation is largely restricted to low-dimensional synthetic systems, with d≤20d \le 20 for all nonlinear smoothing and mean-field game tasks and d=2d = 2 on the Lyon pedestrian dataset where gains are marginal. The only higher-dimensional experiment, at d=200d = 200 in Table 2, is conducted on Quadratic OU Easy, where the drift is linear and completely decoupled across coordinates with A=0.2IA = 0.2I. Crucially, the paper does not evaluate PI-VCM on the challenging high-dimensional nonlinear benchmarks standard in recent SOC literature—such as multimodal sampling via Schrödinger–Föllmer processes, molecular transition-path sampling, or reward fine-tuning of diffusion/flow models. Furthermore, the paper omits comparisons with Hua et al. (2024), which directly leverages Girsanov's theorem to enable simulation-free/adjoint-free SOC training via off-policy reference expectations and detached on-policy gradients without solving backward adjoint ODEs. While the paper compares against Stochastic Optimal Control Matching (SOCM; Domingo-Enrich et al., 2024) and Adjoint Matching (AM; Domingo-Enrich et al., 2025) on the toy Ornstein–Uhlenbeck benchmarks, these primary matching-based baselines are completely omitted from the nonlinear Bayesian smoothing and mean-field game evaluations.
  • Prohibitive O(d3)O(d^3) computational and O(d2)O(d^2) memory overhead of affine sensitivity caching, with no wall-clock runtime comparisons or cache-retention in MFGs: To avoid re-running backward ODEs during replay, PI-VCM integrates and stores a dense sensitivity matrix K∈Rd×dK \in \mathbb{R}^{d \times d} for every branch in the replay buffer. As acknowledged in Appendix B.5 and D, this requires O(Md2)O(M d^2) memory per bundle and O(Md3)O(M d^3) matrix-matrix multiplication cost per ODE step during trajectory collection for general nonlinear drifts with dense Jacobians. This cubic scaling is masked in the d=200d = 200 OU experiment because the drift Jacobian is a constant scalar multiple of the identity, and no wall-clock training time or memory comparisons against PI-VM, AM, or adjoint-free methods (Hua et al., 2024) are reported. Moreover, as noted in Section 4.3, any change to the reference drift or running cost invalidates the cached sensitivities and requires clearing and repopulating the replay buffer. In mean-field games, where the population distribution updates the cost and drift at every outer iteration, the off-policy replay buffer must be repeatedly discarded, largely negating the sample-efficiency advantage of experience replay.

References

  • Mengjian Hua, Mathieu Laurière, and Eric Vanden-Eijnden. "An Efficient On-Policy Deep Learning Framework for Stochastic Optimal Control." arXiv preprint arXiv:2410.05163, 2024.
  • Carles Domingo-Enrich, Jiequn Han, Brandon Amos, Joan Bruna, and Ricky T. Q. Chen. "Stochastic Optimal Control Matching." Advances in Neural Information Processing Systems (NeurIPS), 37:112459–112504, 2024.
  • Carles Domingo-Enrich, Michal Drozdzal, Brian Karrer, and Ricky T. Q. Chen. "Adjoint Matching: Fine-Tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control." International Conference on Learning Representations (ICLR), 2025.
Questions for authors
  1. Have the authors examined the distribution of the Girsanov reweighting factor, and whether that factor substantially increases the variance of the estimator in practice? The factor is exponential and can blow up when the collection policy is far from the reference.

  2. On the OU tasks, even the weaker baselines look strong on the first two examples. Could the authors add wall-clock time versus control error? That comparison would be useful because PI-VCM is off-policy and could be faster than the on-policy methods in the benchmark set.

4 comments
AuthorsOpenReview ↗

Response to Reviewer a3Lr (1/3)

High-dimensional nonlinear experiments (W1)

Longer Reaction–diffusion chains. We extend Reaction–diffusion smoothing to coupled chains with d=20, 50 and 100; a larger d means a longer chain at the same coupling strength. We compare PI-VCM, matched PI-VM, Adjoint Matching (AM) and SOCM by path KL, defined as in the common response; lower is better.

dZero-control priorMatched PI-VMSOCMAMPI-VCM
2014052.6 ± 5.8150 ± 233.8 ± 0.10.34 ± 0.03
50239Numerical overflow251 ± 910.3 ± 0.81.38 ± 0.11
100555Numerical overflow500 ± 4020.3 ± 0.36.6 ± 2.6

PI-VCM attains the lowest path KL in all three dimensions and also trains faster: AM solves the adjoint along the whole path at every update and completes far fewer updates than PI-VCM in the same time, while SOCM relies on importance weights over whole paths, which degenerate, so it barely learns.

Particle and multimodal sampling. On the 8-dimensional DW4 particle system, i.e. the Boltzmann distribution of 4 planar particles under a pairwise double-well potential, we compare PI-VCM, PI-VM, VM and Adjoint Sampling. All methods are trained for 3,000 rounds, and results are averaged over the last six evaluations. Configuration W₂ is minimized over particle permutations and rigid transformations; distance W₂² is the squared W₂ distance between the interparticle-distance distributions.

MethodConfiguration W₂ ↓Distance W₂² ↓Energy W₂ ↓
Adjoint Sampling0.52480.03840.3559
VM0.53340.04010.2965
PI-VM0.46960.02780.2979
PI-VCM0.39700.00790.3598

PI-VCM attains the lowest configuration W₂ and distance W₂², and an energy W₂ comparable to Adjoint Sampling. On a 4-mode Gaussian mixture (GMM) and a double-well distribution, the same four methods attain the following position W₂.

MethodGMM position W₂ ↓Double-well position W₂ ↓
Adjoint Sampling1.1930.174
VM0.4530.236
PI-VM0.4850.169
PI-VCM0.4300.130

PI-VCM attains the lowest position W₂ on both targets, and on the double-well target all four methods cover both wells.

These results show that PI-VCM scales to higher-dimensional nonlinear coupled systems and applies to particle and multimodal sampling.

Additional baselines: Hua et al. (2024), SOCM and AM (W1)

We added the method of Hua et al. (2024) on the three OU tasks, with the same training budget and evaluation as Table 1. The control L2 errors are as follows.

MethodOU Linear ↓OU Quadratic (easy) ↓OU Quadratic (hard) ↓
Hua et al. (2024)0.001197 ± 0.0004390.010252 ± 0.0002680.159071 ± 0.004308
REINFORCE0.000938 ± 0.0004250.016159 ± 0.0004100.208436 ± 0.020531
PI-VM0.000008 ± 0.0000010.000107 ± 0.0000100.000788 ± 0.000065
PI-VCM0.000002 ± 0.0000000.000061 ± 0.0000050.000657 ± 0.000079

At λ=1, which all OU tasks use, the parameter gradient of this method coincides with that of REINFORCE, so their errors are of the same order. The method avoids the backward adjoint solve, but its error is far higher than that of PI-VM and PI-VCM.

For smoothing and MFGs, SOCM and AM are run with the original authors' code.

MethodDouble well path KL ↓Lorenz–63 path KL ↓Reaction–diffusion global RMSE ↓
RE0.4562 ± 0.23900.0384 ± 0.00890.0219 ± 0.0050
LV0.5223 ± 0.24570.0562 ± 0.03380.0278 ± 0.0054
AM0.4556 ± 0.23830.0376 ± 0.00910.0206 ± 0.0041
SOCM0.5131 ± 0.28530.2417 ± 0.0330Barely learns
PI-VCM0.0220 ± 0.00050.0221 ± 0.00560.0060 ± 0.0009

PI-VCM is best on all three tasks; with the baseline learning rate raised to 1.5×10⁻³, the baselines remain an order of magnitude above PI-VCM on Double well. On Reaction–diffusion, SOCM barely learns because its importance weights over whole paths degenerate.

In MFGs, we replace only the inner best-response solver with SOCM or AM and keep all other settings unchanged.

MethodTwo-exit routing BR gap ↓Transport–congestion BR gap ↓Non-potential LQ Bellman advantage ↓
AM2.74 ± 0.570.0764 ± 0.00830.00065 ± 0.00012
SOCM4.05 ± 0.090.0894 ± 0.02400.00105 ± 0.00032
PI-VCM0.0145 ± 0.01010.00176 ± 0.001520.00024 ± 0.00014

PI-VCM attains the smallest gap on all three tasks, and increasing the inner budget of AM and SOCM fivefold leaves their results essentially unchanged.

These results show that PI-VCM achieves much lower error than Hua et al. (2024) on OU, and also outperforms SOCM and AM in smoothing and MFGs.

AuthorsOpenReview ↗

Response to Reviewer a3Lr (2/3)

Cache cost and wall-clock time (W2, Q2)

We agree that, for dense Jacobians, explicit caching requires O(Md2)O(Md^2) memory per bundle and O(Md3)O(Md^3) matrix operations per backward step during collection. The common response "Time and memory: caching vs. adjoint recomputation" compares explicit caching with adjoint recomputation, which gives identical control targets, and with per-step resampling. On dense systems from d=100 onward the cache is slower or runs out of memory; adjoint recomputation then keeps experience replay and remains clearly faster than per-step resampling. On coupled systems such as Pairwise, the cache is still clearly faster.

Regarding wall-clock time, we recorded the control error versus wall-clock time of each method on the OU tasks on the same hardware. Taking the control error of official PI-VM at the end of its training as the threshold, the table reports the wall-clock time in minutes at which each method's error first falls below this threshold; "–" means the error never falls below the threshold within the full training budget.

MethodOU LinearOU Quadratic (easy)OU Quadratic (hard)
Official PI-VM7.810.018.6
AM18.6––
SOCM19.0––
SOCM-Adjoint13.8––
PI-VCM4.28.651.6

LVAR, CE, VAR and REINFORCE do not reach the threshold on any of the three tasks.

PI-VCM is the fastest on OU Linear and OU Quadratic (easy), where most on-policy methods never reach the threshold.

For the smoothing tasks, we recorded on the same hardware the time each method takes to complete training under its own training budget.

MethodDouble well training time (s)Lorenz–63 training time (s)
AM2,1022,734
LV2,4763,250
SOCM3,0394,474
RE5,4306,735
PI-VCM94154

PI-VCM needs only a few minutes on both tasks, while all baselines are more than an order of magnitude slower.

These results show that PI-VCM reaches the threshold accuracy faster than on-policy methods on OU Linear and OU Quadratic (easy), and on the smoothing tasks it completes training in a small fraction of their time while attaining lower path KL.

Experience replay in MFGs (W2)

We measured the overhead of clearing and refilling the replay buffer after each population update, and compared this with two alternatives: per-step resampling, and retaining samples across outer iterations while recomputing their costs under the new population. Refilling accounts for only 5.2% and 1.4% of the training time on Transport–congestion and Two-exit routing, respectively. The three strategies are compared in the same set of reruns, with all other settings identical.

StrategyTransport–congestion BR gap ↓Time (s)Two-exit routing BR gap ↓Time (s)
Clear and refill0.0018 ± 0.001519.70.027 ± 0.016129
Per-step resampling0.0013 ± 0.000673.72.63 ± 0.45534
Retain and recompute0.0014 ± 0.001219.52.62 ± 0.46128

On Transport–congestion, the three strategies give similar BR gaps, but per-step resampling is much slower. On Two-exit routing, the population collapses onto a single exit under both per-step resampling and retaining samples, whereas clearing and refilling recovers near-equilibrium exit fractions.

These results show that retaining old samples across outer iterations converges to a wrong equilibrium on Two-exit routing, so clearing and refilling after each population update is necessary; its overhead is small, and experience replay still avoids per-step resampling, saving most of the training time.

AuthorsOpenReview ↗

Response to Reviewer a3Lr (3/3)

Distribution of the Girsanov weights (Q1)

Under the default training settings, we measured the normalized effective sample size (ESS) of the Girsanov weights, whose ideal value is 1. Dense LQ and Pairwise are defined in the common response "Time and memory: caching vs. adjoint recomputation"; Dense tanh adds the nonlinear term tanh⁡(WX)\tanh(WX) to the Dense LQ drift.

TaskMean ESS ↑1% quantile of ESS ↑
Dense LQ d=200.9980.994
Dense tanh d=500.9880.957
Pairwise d=320.950.74
Reaction–diffusion0.990.95–0.97

The ESS is close to 1 on every task.

What enters the value target is not the Girsanov factor alone but its product with e−Ce^{-C}, where CC is the segment cost plus the continuation value. If the collection policy equals the optimal control and the continuation value is exact, this product is the same for every path, so the estimator has zero variance however far the collection policy is from the reference. The relevant deviation is therefore from the optimal control rather than from the reference, which we test next.

On 10-dimensional OU Linear, we also measured the variance of the PI value target at the same space–time point t=0, x=0 when the collection policy deviates from the optimal control by an offset of magnitude δ, varying the segment length H and the number of paths N per estimate. The table reports the variance for N=8; δ=0 corresponds to the optimal control.

δH=8H=16H=32H=100
03.5e-78.4e-72.4e-63.7e-5
0.50.00260.00520.0110.037
10.0100.0220.0460.18
20.0460.100.241.03
40.240.581.385.23
81.373.156.8522.9

The variance grows rapidly with δ and H, and short segments reduce it by more than an order of magnitude.

These results show that, under the default settings, the Girsanov weights do not substantially increase the estimator variance; the variance grows mainly with the deviation of the collection policy from the optimal control and with the segment length, and short segments with H=8–16 keep it under control.

Thank the authors for the additional ablations done in this short discussion period and these address my concerns on the scability of the method. Though I would like to see how the method performs harder and more complex experiments (e.g. image fine-tuning), I understand it's hard to get such done in such a short time period. Therefore, I would like to raise my evaluation on this submission.

Reviewer_2WAvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Path-Integral Value-Control Matching (PI-VCM), extending PI-VM with direct supervision of the control induced by a value network. By differentiating the local path-integral recursion, the method constructs value and control targets from the same short trajectory segments. An affine sensitivity cache allows control targets to be refreshed during experience replay without repeatedly solving the backward adjoint equation.

The theoretical analysis establishes the off-policy recursion, the affine caching identity, and variance reduction results under exact continuation functions. Experiments cover analytic stochastic optimal control benchmarks, dimensional scaling up to 200 dimensions, Bayesian smoothing, mean-field games, and pedestrian-density prediction.

Critical strengths
  1. The control supervision has a coherent mathematical foundation. The local value and control targets use the same trajectory weights and compatible continuation information. The affine dependence of the adjoint on the endpoint gradient also provides a principled basis for replay. This directly addresses the relevant problem that accurate scalar value predictions need not imply accurate controls.

  2. The experiments directly evaluate control accuracy and include a close baseline. Table 1 compares against analytic optimal controls. On Quadratic OU Easy, PI-VCM reduces the reported mean control error from PI-VM’s 0.0001070.000107 to 0.0000330.000033. Table 2 also reports consistently lower mean errors across the tested dimensions. These evaluations provide meaningful evidence beyond training-loss improvements.

Critical weaknesses
  1. The experimental configurations and practical benefits of caching are insufficiently documented. Appendix G.2 and Table 6 mainly report PI-VCM’s training budgets, while corresponding baseline architectures, sampling schedules, replay settings, and tuning budgets are not fully specified in the PDF. PI-VM is a reasonable value-only comparison, but it is unclear whether the reported implementation uses otherwise identical settings. Moreover, the paper does not directly compare caching against repeated adjoint propagation in training time or peak memory. This matters because dense caching requires quadratic storage and potentially cubic computation during collection. The high-dimensional experiments use separable, isotropic quadratic OU problems, leaving the cost-benefit tradeoff in more general coupled systems uncertain.
  2. The smoothing experiments do not isolate the contribution of the added control supervision. Table 3 includes several relevant smoothing baselines but omits PI-VM. Consequently, the improvements support the applicability of the overall method, but do not directly establish the benefit of the proposed extension in this application. Performance is also mixed: Variational achieves lower mean errors on both Lorenz–63 metrics. A matched PI-VM comparison would strengthen the connection between the paper’s motivation and its application results.
Questions for authors
  1. Clarification and validation: Is the reported PI-VM baseline equivalent to disabling the control loss in the same implementation, with matching architectures, sampling schedules, optimization budgets, and tuning opportunities? Could you provide representative runtime and memory comparisons, including cache construction?
  2. Additional validation of the current contribution: On at least one smoothing task, does PI-VCM improve over PI-VM under matched settings? This would materially strengthen the evidence for direct control supervision in the proposed applications.
  3. Clarification of the MFG evaluation: When computing the best-response gaps in Table 4, is the fixed population the final relaxed population iterate or the distribution generated by deploying the final learned control? If it is the former, could you quantify the discrepancy between these distributions? This distinction affects whether the reported gap measures equilibrium quality or only optimality against a fixed candidate population.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 2WAv (1/2)

Experimental configuration and matched PI-VM (W1, Q1)

The PI-VM in Table 1 uses the original authors' implementation and is not equivalent to disabling the control loss in our implementation. Matched PI-VM sets the control-loss weight to 0 in the same implementation; the three configurations differ as follows.

SettingPI-VCMMatched PI-VMOfficial PI-VM on OU
Control-loss weight3 for Double well and Lorenz–63; 30 for Reaction–diffusion0None
Network, replay, M, H, budget, learning-rate scheduleAs in the paperSame as PI-VCMOfficial configuration

The ablation results under this setting are given in the common response "Matched ablation".

We will add to the appendix of the revised version the network architectures, optimization schedules, trajectory collection, replay settings, training budgets and hyperparameter selection rules for all methods. All methods follow the same hyperparameter selection procedure, and the OU baselines use the default hyperparameters of the official PI-VM repository.

Time and memory of caching (W1, Q1)

Under the same training and replay settings, we compared explicit caching, adjoint recomputation and per-step resampling on OU, Pairwise and Dense LQ, with timings that include periodic collection and cache construction; see the common response "Time and memory: caching vs. adjoint recomputation". The two replay implementations give identical control targets. The cache is faster on OU Quadratic (easy) and Pairwise; on dense systems from d=100 onward it is slower than adjoint recomputation or runs out of memory, and adjoint recomputation then keeps the benefit of experience replay.

Control supervision in smoothing (W2, Q2)

As shown in the common response "Matched ablation", PI-VCM attains lower path KL than matched PI-VM on all three smoothing tasks, and on Reaction–diffusion matched PI-VM fails to learn an effective control.

Regarding the advantage of Variational on Lorenz–63, we additionally report path-space metrics on two smoothing tasks.

MethodDouble well path KL ↓Double well ESS ↑Lorenz–63 path KL ↓Lorenz–63 ESS ↑
Variational0.528 ± 0.0050.480.0268 ± 0.00090.954
PI-VCM0.022 ± 0.00050.960.0221 ± 0.00560.961

The Lorenz–63 posterior is close to Gaussian and is already well approximated by linear-Gaussian guidance: in Table 3, Variational has lower marginal W1 and unobserved-state RMSE than PI-VCM, yet the two have comparable path KL. The former two metrics compare only the per-time marginals of individual coordinates and the posterior mean of the unobserved coordinates, whereas path KL compares the joint law of entire paths, including temporal correlations and dependence across coordinates. The Double well posterior is bimodal, and affine guidance cannot represent both modes; its path KL is nearly the same as that of Ad-hoc. PI-VCM attains a path KL more than an order of magnitude lower, and its marginal W1 in Table 3 is also clearly lower.

These results show that Variational is better only on the marginal metrics for the near-Gaussian Lorenz–63 posterior, and is clearly worse than PI-VCM on the bimodal posterior.

AuthorsOpenReview ↗

Response to Reviewer 2WAv (2/2)

Evaluation of the MFG experiments (Q3)

In Table 4, the fixed population is the candidate population from the final relaxation iterate.

On three tasks, we recomputed the best-response gap (BR gap) with the population fixed to the distribution generated by deploying the final learned control: Transport–congestion uses the Fokker–Planck response density of the learned control, Non-potential LQ uses the empirical mean flow of the deployed paths, and Two-exit routing uses the exit fractions of the deployed particles.

PI-VCM on Transport–congestion and Non-potential LQ

Task and metricCandidate population ↓Deployed population ↓Discrepancy between the two distributions
Transport–congestion BR gap0.00176 ± 0.001520.00318 ± 0.00293Density L1 distance 0.126
Non-potential LQ Bellman advantage0.000237 ± 0.0001430.000266 ± 0.000203Maximum mean-flow difference 0.026

Under the deployed population, the gaps of PI-VCM increase only slightly and remains better than all baselines in Table 4.

Two-exit routing (48 outer iterations for PI-VCM and PI-VM)

MethodCandidate population ↓Deployed population ↓Sliced W1 between the two distributions
PI-VCM0.0083 ± 0.00690.017 ± 0.0110.033
PI-VM0.252 ± 0.0100.251 ± 0.0050.024
DEDA-FP1.85 ± 0.211.93 ± 0.180.294
APAC-Net3.46 ± 0.563.34 ± 0.350.213
DPI6.53 ± 0.306.51 ± 0.301.481
MFDGM7.09 ± 0.047.20 ± 0.131.305

The last column quantifies the discrepancy between the candidate and deployed populations: it is small for PI-VCM and PI-VM, and much larger for DPI and MFDGM. The ranking of the six methods is identical under both evaluations.

The BR gap under the deployed population measures how far the deployed population itself is from equilibrium. Under this evaluation, PI-VCM remains close to equilibrium and outperforms all baselines.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response to All Reviewers

Matched ablation: the effect of control supervision

To isolate the effect of control supervision, we set the control-loss weight to 0 in the same PI-VCM implementation, keeping the network, experience replay, M, H, training budget and learning-rate schedule unchanged. Path KL is the KL divergence between the controlled and the posterior path laws, estimated with Girsanov weights over 16,384 paths; it requires no reference samples, and lower is better.

Path KL ↓Matched PI-VMPI-VCM
Double well0.0487 ± 0.01610.0220 ± 0.0005
Lorenz–630.246 ± 0.0160.0221 ± 0.0056
Reaction–diffusion76 ± 140.176 ± 0.013
Reaction–diffusion chain, d=2052.6 ± 5.80.34 ± 0.03
Reaction–diffusion chain, d=50Numerical overflow1.38 ± 0.11

PI-VCM attains lower path KL on every task. On Reaction–diffusion, matched PI-VM fails to learn an effective control, and on the d=50 chain its training overflows numerically. Under the same implementation and training budget, control supervision therefore improves all three smoothing tasks, and value-only supervision becomes harder to train as the chain grows longer and higher-dimensional.

Time and memory: caching vs. adjoint recomputation

We compare three implementations under the same training and replay settings. Their costs are as follows.

ImplementationAt collectionAt each updateReplay storage
CacheForward simulation; backward accumulation of the d×d matrix KK and vector ccOne matrix–vector product K∇V(τ)+cK\nabla V(\tau)+cMd2Md^2; MdMd for diagonal problems
Adjoint recomputationForward simulation; store the state at every stepH vector–Jacobian products along the stored path to recompute the adjoint backwardMHdMHd
Per-step resamplingNo experience replayResimulate from t=0t=0 and solve the adjoint backward0

The two added coupled tasks are SOC problems of the same form as the OU benchmarks: the controlled SDE is dX=(b(X)+u) dt+dWdX=(b(X)+u)\,dt+dW with a quadratic control cost, but the drift Jacobian is dense. Dense LQ is a densely coupled linear–quadratic regulator: the drift is b(X)=AXb(X)=AX with a dense random matrix AA, the running and terminal costs are dense positive-definite quadratic forms, and the exact optimal control follows from the Riccati equation. Pairwise is a control problem for an interacting particle system: n planar particles follow overdamped Langevin dynamics under a pairwise potential, and the control steers them to a given target configuration; the drift is the negative gradient of the pairwise potential, and its Jacobian is a dense, state-dependent Hessian.

On all three problem classes below, the cache and adjoint recomputation give identical endpoint states and value targets, and the relative L2 difference between their control targets is below 10⁻⁷. The table reports the steady-state average time per update in milliseconds, including periodic collection and cache construction.

TaskdCacheAdjoint recomputationPer-step resampling
OU Quadratic (easy)2026.736.0126
OU Quadratic (easy)20031.343.6159
Pairwise6445.3201.5323
Dense LQ5025.327.580.9
Dense LQ10038.328.986.7
Dense LQ200Out of memory37.2113

Peak memory is reported below in GiB; bold marks the lower of the two replay implementations.

TaskdCacheAdjoint recomputationPer-step resampling
OU Quadratic (easy)200.540.890.33
OU Quadratic (easy)2002.896.420.97
Pairwise642.932.502.33
Dense LQ501.820.490.40
Dense LQ1007.020.820.61
Dense LQ200Out of memory1.350.88

The cache is faster on OU Quadratic (easy) and Pairwise, by about 4.5× on Pairwise. In dense systems, cache memory grows as d2d^2, and from d=100 onward the cache is slower than adjoint recomputation or runs out of memory. For dense Jacobians, building the cache costs O(d3)O(d^3) per backward step, whereas adjoint recomputation costs O(d2)O(d^2) per step at every reuse, so the cache is faster when each trajectory is reused a number of times comparable to the dimension; for high-dimensional dense systems, one can switch to adjoint recomputation, which keeps the benefit of experience replay and remains clearly faster than per-step resampling.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.