- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Path-Integral Value-Control Matching (PI-VCM), an extension of Path-Integral Value Matching (PI-VM) that jointly supervises a value function and its gradient-derived control.
The authors differentiate the existing local path-integral value recursion to construct control targets from short stochastic trajectories. They also introduce affine sensitivity caching, which allows adjoint targets to be updated when the continuation value changes without recomputing backward sensitivities along stored trajectories.
The method is evaluated on stochastic optimal control benchmarks, including problems with up to 200 state dimensions, as well as Bayesian smoothing and mean-field games. The experiments report improved control accuracy over PI-VM and competitive performance against several existing methods.
Critical strengths
Improved control accuracy across multiple benchmarks. The paper demonstrates substantial improvements in control accuracy over the original PI-VM method. In particular, the control MSE decreases from to on the 200-dimensional quadratic OU benchmark. The experiments on Bayesian smoothing and mean-field games provide additional evidence that directly supervising the gradient-derived control can be useful beyond standard stochastic control benchmarks.
A practical framework for reusing stochastic trajectories. The combination of local path-integral value-control targets, likelihood-corrected off-policy replay, and affine sensitivity caching provides a coherent computational framework. In particular, the affine representation of the adjoint permits previously collected trajectories to be reused when the terminal continuation gradient changes. This may reduce repeated sensitivity calculations during iterative value-function learning.
Critical weaknesses
Limited methodological novelty beyond existing value matching and derivative supervision. Under the assumed control-affine dynamics and quadratic control costs, the optimal control is already explicitly characterized by the HJB first-order condition, . The proposed control target is obtained by differentiating the existing PI-VM Bellman recursion and applying this optimality relation. Jointly regressing the value function and its gradient-derived control therefore appears closely related to Sobolev or derivative-supervised Bellman learning. Although the construction is mathematically valid, I find it difficult to identify a fundamentally new control-learning principle beyond augmenting PI-VM with gradient supervision. The improvements over value-only PI-VM demonstrate the usefulness of this additional supervision, but do not by themselves establish its methodological novelty.
Insufficient evidence for the computational advantages of affine sensitivity caching. The affine identity follows from the linearity of the adjoint equation with respect to its terminal condition. While caching and may accelerate repeated target updates, this requires additional computation and storage, including memory per trajectory for a dense sensitivity matrix. The paper does not provide a sufficiently direct comparison of end-to-end computation time and memory consumption against recomputing the same control targets without caching. Consequently, it is difficult to determine whether affine replay offers a substantial computational advantage over standard sensitivity calculations. The high-dimensional experiments primarily use a structurally simple quadratic OU model, leaving the computational benefits for more general, densely coupled nonlinear systems uncertain.
Questions for authors
The paper is mathematically reasonable and reports encouraging empirical improvements. However, I remain unconvinced that the proposed approach represents a sufficiently substantial methodological advance over existing techniques.
- Methodological novelty of value-control matching
At the population level, the control target in Eq. (12b) follows from differentiating PI-VM’s local value recursion and applying the standard HJB feedback relation. The joint objective in Eq. (17) therefore admits a Sobolev-type interpretation, supervising the value and its control-relevant spatial derivatives.
Could the authors clarify which aspects of the control-target construction constitute the main methodological advance beyond this derivative-supervision principle?
I appreciate the distinction made in Appendix C.3 between the proposed empirical control target and direct differentiation of a finite-sample off-policy Monte Carlo value estimator. Does this distinction provide measurable statistical or computational benefits, such as lower gradient-estimation error, reduced variance, or lower differentiation cost?
Where both constructions are applicable, a comparison against an estimator obtained by differentiating through the proposal trajectories and their likelihood factors, using the same continuation function, collection policy, and trajectory budget, would help isolate the contribution of the proposed target construction.
- Computational benefits of affine sensitivity caching
The affine representation in Eq. (23) allows the adjoint to be refreshed when the terminal gradient changes, but the underlying affine dependence follows from the standard linear adjoint equation.
Could the authors quantify the computational advantages of explicitly caching and relative to recomputing the adjoint using conventional backward sensitivity calculations?
In particular, comparisons of wall-clock time, peak memory usage, and performance under matched computational budgets would be helpful. Since constructing and storing dense sensitivity matrices incurs nontrivial costs, it would also be useful to identify the dimensional regimes and replay frequencies for which the proposed caching method becomes advantageous.
Clarifying these two points would materially affect my assessment of the paper's novelty and practical significance.
5 comments
Response to Reviewer GaZp (2/3)
Methodological contribution and control-target construction (W1, Q1, continued)
To test whether the distinction in Appendix C.3 brings measurable benefits, we implemented, as the reviewer suggested, an estimator that directly differentiates the finite-sample value estimator, using the same initial points, Brownian increments, continuation value function, collection policy and number of paths M as PI-VCM. The two finite-sample constructions differ as follows.
| PI-VCM, Eq. (12b) | Direct differentiation | |
|---|---|---|
| Control target | , where is the reference-drift adjoint along the collected path | , |
| What is differentiated | Only the reference-drift Jacobian ; the collection policy is not differentiated | The sampled trajectories, the collection policy and the likelihood factors, introducing ; second derivatives of the value network when |
| Experience replay | Labels depend on the value network only through the continuation gradient ; stored segments can be refreshed directly with the cached , or by adjoint recomputation | Stored segments cannot be refreshed directly and must be resimulated with the current policy |
Accuracy of a single control target. We compare the mean squared error of the two control targets on Dense LQ and Reaction–diffusion. Dense LQ is defined in the common response "Time and memory: caching vs. adjoint recomputation"; its exact optimal control follows from the Riccati equation. On Reaction–diffusion, the continuation value function is the value network trained by PI-VCM, and the collection policy is the same network or an earlier one, mimicking stale data in the replay buffer; the ground truth is estimated from 65,536 branches under the reference measure. The table reports the MSE ratio of direct differentiation to PI-VCM with M=8; a ratio below 1 means direct differentiation is more accurate.
| Collection policy | Dense LQ, d=50 | Reaction–diffusion |
|---|---|---|
| Reference measure, i.e. zero control (the two targets coincide) | 1 | 1 |
| Near optimal: exact optimal feedback / current value network | 0.00041 | 0.17–0.18 |
| Severe deviation: random MLP perturbation 1.0 / network at update 100 | 5.9 | 0.61–0.62 |
When the collection policy is near optimal, the pathwise derivative through the collection policy in direct differentiation acts as a control variate and yields lower error; this advantage shrinks as the deviation grows, and on Dense LQ under severe deviation PI-VCM is more accurate. We therefore do not claim that the PI-VCM control target has lower variance. Computationally, direct differentiation must keep the computation graph of the collected trajectories and take second derivatives of the value network; on Reaction–diffusion, constructing the same batch of control targets takes about 10× the memory of PI-VCM.
Response to Reviewer GaZp (3/3)
Methodological contribution and control-target construction (W1, Q1, continued)
Training comparison. On Reaction–diffusion, we keep the paper's network, replay settings and learning-rate schedule, and change only the control target and the way data are reused.
| Method | Time per update (ms) | Training time (min) | Final path KL ↓ |
|---|---|---|---|
| Direct differentiation, recollection at every update | 1,179 | 98.8 | 0.517 |
| PI-VCM, experience replay | 100 | 16.8 | 0.164 |
Direct differentiation must recollect with the current policy at every update and is too slow, so it was trained only up to update 5,000, the end of the constant-learning-rate phase; PI-VCM completes all 10,000 updates in less than one fifth of the time and reaches a lower path KL.
After training, both control targets reach a path KL far below that of matched, value-only PI-VM (see the common response "Matched ablation"), so control supervision itself brings the main gain. Our construction makes this supervision usable with off-policy replay: direct differentiation must recollect with the current policy at every update and take second derivatives of the value network, whereas our control labels need only first derivatives, and stored segments can be refreshed directly after value-network updates. As a result, PI-VCM has the shortest training time and the best final accuracy.
Computational benefits of the affine sensitivity cache (W2, Q2)
End-to-end time and peak memory of explicit caching versus conventional adjoint recomputation are given in the common response "Time and memory: caching vs. adjoint recomputation". Below we add the accuracy under matched computational budgets and the dependence of the cache's benefit on the reuse ratio.
On Dense LQ with d=20, each of the three implementations is trained for 20,000 updates.
| Implementation | Training time (s) | Control L2 at 500 s ↓ |
|---|---|---|
| Cache | 491–496 | 4.3–6.7e-5 |
| Adjoint recomputation | 534–540 | 3.8–6.8e-5 |
| Per-step resampling | 1,665–1,697 | 1.0–1.2e-4 |
The cache and adjoint recomputation produce identical control targets, and their errors at 500 s fall in the same range; at equal wall-clock time, both have about half the error of per-step resampling. The benefit of the cache also depends on the reuse ratio ρ, i.e. the average number of times each sample is used. The table below varies the replay refresh interval on Dense LQ with full-batch collection; entries are in ms per update.
| Refresh interval / reuse ratio ρ | d=20 cache | d=20 adjoint recomputation | d=50 cache | d=50 adjoint recomputation |
|---|---|---|---|---|
| 1 / ≈1 | 129 | 90.4 | 221 | 94.6 |
| 5 / ≈5 | 41.9 | 39.9 | 63.7 | 39.1 |
| 25 / ≈25 | 26.4 | 28.4 | 31.2 | 26.6 |
| 100 / ≈98 | 24.9 | 26.4 | 26.8 | 25.8 |
| 400 / ≈390 | 22.4 | 27.8 | 24.5 | 24.6 |
The higher the reuse ratio, the more the cache benefits; the higher the dimension, the higher the reuse ratio the cache needs to come out ahead.
These results show that the cache is faster when each trajectory is reused a number of times comparable to the dimension; otherwise, one can switch to the numerically equivalent adjoint recomputation, and both keep the benefit of experience replay.
Response to Reviewer GaZp (1/3)
Methodological contribution and control-target construction (W1, Q1)
At the population level, the control target satisfies the standard HJB first-order condition, and Eq. (17) can be viewed as joint supervision of the value and its gradient, related to Sobolev training (Czarnecki et al., 2017), VGG-Flow and Efficient Adjoint Matching.
Sobolev training, however, assumes that derivative labels are given; in SOC, the optimal control has no ready-made labels, and they must be constructed from finitely many short trajectories collected off-policy. The construction determines whether the labels are reliable, whether they can enter experience replay, and how costly they are to refresh. Our contribution is such a construction and its application to Bayesian smoothing and MFGs, in three respects.
Within this framework, replacing the inner solver with AM or SOCM, or using AM, SOCM, RE and LV under the same observation transformation, gives clearly worse results than PI-VCM.