acceptodds
Under review as a conference paper at ICLR 2027

Path Integral Value Matching: Regularized Control of Arbitrary Markov Processes

Abstract

KL-regularized control of continuous-time Markov processes seeks an optimal path distribution that balances expected path cost against deviation from a reference Markov process. Many existing solvers directly optimize control policies, but often require process-specific formulations and costly full-horizon rollouts, with off-policy reuse limited by accumulated likelihood-ratio variability. We propose Path Integral Value Matching (PI-VM), a process-agnostic value-based control method. Our method builds on a common path-integral representation of the value, which determines the optimal controlled dynamics through a Doob's -transform once the reference process is fixed. However, direct path-integral estimation still requires full-horizon trajectory simulation and can suffer from high variance. By truncating and marginalizing the path integral, we obtain a temporal recursion for the value function. PI-VM learns this recursion through temporal-difference learning on short trajectories, while experience replay and likelihood correction support off-policy training. The same learning objective applies to Euclidean and manifold diffusions, continuous-time Markov chains, and finite-activity jump processes. Experiments show that PI-VM achieves the best reported performance on multiple tasks across diverse process families and performs well in image fine-tuning.

Primary area:
generative models
Keywords:
Stochastic Optimal Control, General Markov Process

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cqPDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PI-VM, a value-based approach to KL-regularized control of continuous-time Markov processes. By combining short-horizon trajectory sampling, bootstrapped continuation values, likelihood correction, and experience replay, PI-VM learns a scalar value function from which optimal controls can be recovered through the Doob transform. The method is evaluated on OU diffusions, manifold diffusions, jump processes, CTMCs, and reward-guided image generation. Its main contribution lies in the proposed value-learning estimator and its application across different stochastic processes, rather than in new fundamental control theory.

Critical strengths
  1. The paper presents a unified value-learning objective applicable to multiple stochastic processes, while leaving process-specific simulation and control recovery separate. This provides a relatively simple and extensible framework for stochastic control.

  2. The temporal recursion and off-policy likelihood correction are mathematically well motivated. The paper also carefully distinguishes theoretical guarantees under exact continuation values from the limitations of practical neural approximations.

Critical weaknesses
  1. Prior work has already explored off-policy value matching, truncated trajectories, and continuation estimates. The paper does not directly compare against Du et al.'s off-policy VM or sufficiently isolate the benefits of its specific path-integral estimator.

  2. Training budgets differ across methods, particularly for manifold and jump/CTMC experiments. Without matched computational budgets or convergence curves, it is difficult to attribute the reported improvements solely to the proposed algorithm.

  3. Jump and CTMC experiments measure induced-drift error rather than discrepancies in the full transition dynamics. Moreover, PI-VM achieves a worse regularized objective than VM and the uncontrolled baseline on the DiT aesthetic task, despite obtaining higher rewards.

  4. The ablations primarily examine branch counts, trajectory lengths, and network architectures, without isolating the effects of experience replay, behavior guidance, or likelihood correction. The theoretical variance results also do not directly establish the practical benefits of learned continuation values.

Questions for authors
  1. How does the proposed estimator improve upon Du et al.'s off-policy Value Matching in terms of bias, variance, or computational efficiency? Could the authors provide a direct comparison under matched simulation budgets?

  2. Can the authors report total computational costs, including trajectory collection and replay refresh, along with performance-versus-compute curves? How many independent training seeds were used, and what do the reported error bars represent?

  3. Why do CE, VAR, SOCM, and SOCM-A report identical results on the hard OU task? How is AM implemented for non-differentiable JPEG rewards, and which components of the original Hard-SOC method are retained or modified?

  4. What are the respective contributions of experience replay and likelihood correction to the reported improvements? How does performance change with increasing behavior-policy mismatch or replay-data staleness?

  5. How are the ground-truth optimal controls and state distributions computed for the jump, CTMC, and manifold benchmarks? Could the authors report transition-rate/kernel errors or actual control objectives to complement the induced-drift metric?

  6. The DiT experiments use different CFG settings during training and evaluation. Which reference process defines the learned value and likelihood corrections, and do the main conclusions hold when training and evaluation use consistent CFG settings?

2 comments
AuthorsOpenReview ↗

Response to Reviewer cqPD (1/2)

We thank the reviewer for their suggestions. We added comparisons with Du, clarified implementations and reference solutions, and evaluated replay contributions and consistent CFG settings.

R3.1 Comparison with Du et al.'s off-policy Value Matching (W1, Q1)

With unit regularization and Du's κ=1\kappa=1, let zθ=σ⊤∇Vθz_\theta=\sigma^\top\nabla V_\theta, behavior control aa, and shared endpoint term BjB_j (equal to g(XTj)g(X_T^j) for complete trajectories). Define

C_j&=B_j+\int_t^s\!\left(f+\tfrac12\|a\|^2\right)dr +\int_t^s a^\top dW^a,\\ D_j&=\tfrac12\int_t^s\|a+z_\theta\|^2dr +\int_t^s(a+z_\theta)^\top dW^a. \end{aligned}$$ Du's residual is $R_j^{\rm Du}=V_\theta(t,x)-C_j+D_j$, with target $Y_j^{\rm Du}=C_j-D_j$. PI-VM uses $\widehat V_N^{\rm PI}=-\log(\frac1N\sum_j e^{-C_j})$. Under the exponential-martingale conditions, $\mathbb E_a[e^{-D_j}-1\mid t,x]=0$, giving the calibrated target $$\begin{aligned} \widehat V_N^{\rm cal} &=-\log\frac{\sum_j e^{-C_j}}{\sum_j e^{-D_j}} =-\log\sum_j w_j e^{-Y_j^{\rm Du}},\\ w_j&=\frac{e^{-D_j}}{\sum_k e^{-D_k}}. \end{aligned}$$ **This is a weighted soft-min of Du's per-path targets and reduces to Du's target at $N=1$**. On-policy ($a=-z_\theta$), $D_j=0$, so the original PI-VM is an equally weighted soft-min with the same single-branch residual as Du. Du differentiates through $z_\theta$ along the path, whereas PI-VM stops target gradients, so residual equivalence does not imply identical updates. **ReAct-VM results on OU benchmarks** | Task | Final control MSE | Training time (ms/update) | | --- | --- | --- | | Linear | $(1.43\pm0.403)\times10^{-5}$ | $79.6\pm4.9$ | | Easy | $(1.05\pm0.118)\times10^{-4}$ | $42.2\pm3.5$ | | Hard | $(2.36\pm0.291)\times10^{-3}$ | $44.0\pm3.2$ | PI-VM combines multiple branches, continuation values, and replay to incorporate more trajectory information, which can improve stability and accuracy. Under the respective reported configurations (Table 2), PI-VM updates faster, has lower Linear/Hard errors, and similar Easy accuracy. Its Hard error is about one-third of ReAct-VM's. We will add the derivation, comparisons, and timing conventions to the revised manuscript. ### **R3.2 Total compute, convergence, and statistical reporting (W2, Q2)** To address budget differences on manifold and Jump/CTMC tasks, we continued Hard-SOC/DAM training. The General Response reports settings, results, and time/memory comparisons. PI-VM retains an accuracy advantage on most tasks. R2.2 explains trajectory-collection and replay timing. Ablations, the Du comparison, and continued training use 5 independent training seeds (endpoint mean and sample SD). Original Jump/CTMC results use sample SD, manifold results population SD, and image experiments one training seed. We will state these conventions per table and distinguish training from evaluation-sampling uncertainty in the revision. ### **R3.3 Baseline implementations and identical results on Hard OU (Q3)** CE, VAR, SOCM, and SOCM-A stall at the same incorrect Hard OU solution, giving identical control errors. AM uses a smoothed JPEG reward for gradients. Hard-SOC retains bridge sampling, terminal adjoints, vector transport, and regression updates, with a tangent-projected controller-only implementation and no Schrodinger bridge corrector. We will specify networks, step sizes, budgets, and implementation details in the revision. ### **R3.4 Jump/CTMC metrics and construction of ground-truth references (W3, Q5)** We added the regularized objective gap $\Delta J=J(P_\theta)-J(P^\star)$, where $$J(P)=\mathbb E_P[C]+\lambda D_{\mathrm{KL}}(P|P^0)$$ This evaluates recovered path laws beyond induced drift, capturing jump activity and transition rates that drift error alone can miss. Results are in the General Response. Jump/CTMC references use independent semigroup/matrix-exponential computations, Sphere a Legendre heat-equation solver, and Stiefel a spectral solver. We will add the metric, solver settings, and accuracy checks to the revised manuscript.
AuthorsOpenReview ↗

Response to Reviewer cqPD (2/2)

R3.5 The regularized objective on DiT aesthetics (W3)

Image fine-tuning requires a reward–quality trade-off. DiT also reports J=C−ηE[r]J=C-\eta\mathbb E[r], with KL-regularized control cost CC. For a fixed reference and weights, lower JJ indicates a better overall control objective. PI-VM improves both reward and JJ on both JPEG tasks.

R3.6 Replay, behavior guidance, likelihood correction, and data staleness (W4, Q4)

Separately disabling replay, behavior guidance, or likelihood correction increases mean control error on all three OU tasks. The complete method has the lowest mean error in each task, showing that all three components help training with learned continuation values. Replay also gives a 13.3–17.5-fold speedup at the same update count. Mismatch diagnostics are in R2.5. Replay staleness has not been varied independently. We will add these ablations and their scope to the revised manuscript.

Component ablation: control L2L^2 (units of 10−310^{-3})

SettingLinearEasyHard
Complete method0.179±0.0150.179\pm0.0151.053±0.1881.053\pm0.188204.249±50.217204.249\pm50.217
Without replay0.317±0.0300.317\pm0.0305.267±1.5635.267\pm1.563421.671±154.717421.671\pm154.717
Without likelihood correction27.052±27.16827.052\pm27.1685.833±0.6985.833\pm0.698372.278±17.014372.278\pm17.014
Without behavior guidance1.924±0.3851.924\pm0.3851.255±0.0701.255\pm0.070378.992±31.003378.992\pm31.003

R3.7 Training/evaluation CFG and the reference process (Q6)

Training "CFG0" uses conditional prediction without CFG extrapolation, defining the reference drift for values and stored likelihood corrections. CFG2 evaluation uses ϵuncond+2(ϵcond−ϵuncond)\epsilon_{\rm uncond}+2(\epsilon_{\rm cond}-\epsilon_{\rm uncond}), changing that drift, so the original value cannot be interpreted as the exact CFG2 value. With CFG0 in both training and evaluation, PI-VM still has lower JJ on both JPEG tasks and higher JJ on aesthetics. The conclusions therefore persist without the CFG switch. We will add this comparison and clarify reference processes in the revision.

Training and evaluation at CFG0: regularized objective JJ (lower is better)

TaskVMPI-VM
Compression42.38142.38125.00225.002
Incompression−87.403-87.403−93.542-93.542
Aesthetic−44.632-44.632−38.395-38.395
Reviewer_8NwoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Path Integral Value Matching (PI-VM), a value-based framework for KL-regularized control of continuous-time Markov processes.

The method builds on the Gibbs variational principle and the Doob (h)-transform, which characterize optimal path distributions and their associated controlled dynamics.

The central algorithmic contribution is a temporal recursion for the path-integral value function. Instead of estimating full-horizon exponential costs, PI-VM constructs short-segment temporal-difference targets using bootstrapped continuation values. The method additionally supports off-policy training through likelihood-ratio correction and experience replay.

The authors instantiate the framework for Euclidean diffusions, manifold diffusions, continuous-time Markov chains, finite-activity jump processes, and jump-diffusions.

Experiments report favorable control errors on synthetic benchmarks and improved reward-guided adaptation of pretrained image generators.

The work is interesting and technically motivated. However, several aspects of the experimental methodology currently prevent a sufficiently strong conclusion about its comparative effectiveness.

Critical strengths
  1. Strengths

S1. A useful unified perspective on entropy-regularized stochastic control.

The formulation provides a common value-learning objective across substantially different process families.

Although the Gibbs variational representation and Doob transform are classical, using them as the foundation of a reusable temporal-difference learning procedure is a meaningful algorithmic direction.

S2. The mathematical derivations are generally well motivated.

The conditional-expectation recursion in Eqs. (8)–(9) follows naturally from the Markov property.

The off-policy formulation in Eqs. (11)–(12) is consistent with a change of measure under the stated absolute-continuity assumptions.

Proposition 4.1 gives an informative characterization of importance-weight variance through the chi-squared divergence between the optimal and behavior path laws.

The appendices also appropriately distinguish exact-continuation guarantees from the behavior of learned approximations.

S3. Broad empirical coverage.

The evaluation includes several distinct stochastic-process families rather than focusing exclusively on Euclidean diffusion.

The reported improvements on quadratic OU problems, manifold benchmarks, and image-generation tasks are potentially significant.

In particular, Table 3 suggests promising scaling behavior in the tested high-dimensional OU settings.

Critical weaknesses
  1. Weaknesses

W1. Substantial differences in experimental budgets complicate the comparisons

My main concern is that several baseline comparisons use considerably different training configurations.

For example, Table 10 reports the following manifold configurations:

Sphere: PI-VM uses 20,000 epochs with batch size 3,000, whereas Hard-SOC uses 600 epochs with batch size 500.

Stiefel, β = 0.01: PI-VM uses 10,000 epochs with batch size 3,000, whereas Hard-SOC uses 1,000 epochs with batch size 600.

Stiefel, β = 0.5: PI-VM uses 20,000 epochs with batch size 3,000, whereas Hard-SOC uses 1,000 epochs with batch size 600.

The nominal products of epochs and batch sizes differ by factors of 200, 50, and 100, respectively.

These factors should not be interpreted as FLOP ratios, because the methods incur different costs per sample. Nevertheless, they illustrate why comparing final errors alone is insufficient.

Similar concerns apply to the jump and CTMC experiments, where PI-VM uses more updates and larger batches than DAM.

Moreover, the reported per-update times in Tables 2–3 do not establish end-to-end efficiency when replay initialization, trajectory collection, branch simulation, and buffer refresh are also required.

Appendix K provides useful generator-evaluation counts for image generation, but these reveal that the computational advantage is task-dependent. For instance, PI-VM requires more training hours than VM on CIFAR incompression and aesthetics.

The paper would benefit considerably from accuracy-versus-total-compute comparisons, preferably including generator evaluations, wall-clock time, and memory consumption.

Without such comparisons, the extent to which the reported improvements arise from the learning objective rather than differences in computational allocation remains uncertain.

W2. Induced-drift error is insufficient to establish optimality for jump processes and CTMCs

Tables 4 and the corresponding appendices evaluate jump-process and CTMC performance using the squared error between induced drifts.

However, the induced drift

[ b_\nu(x)=\int (y-x)\nu(x,dy) ]

does not uniquely determine the jump kernel.

For example, at a state with symmetric transitions to (+1) and (-1), simultaneously increasing both transition rates can preserve the induced drift while substantially changing jump activity, path distributions, and KL costs.

Consequently, a small induced-drift error does not necessarily imply accurate recovery of the optimal jump rates or the optimal path-space control law.

The qualitative terminal-distribution comparisons in Figure 1 are encouraging but do not fully resolve this concern.

I recommend reporting at least one metric directly sensitive to the jump kernel or controlled path law, such as rate error, path-space KL divergence, or the achieved regularized control objective.

This would substantially strengthen the claims of process-independent optimal control.

W3. Image-generation improvements are not uniform under the actual regularized objective

The DiT experiments provide interesting evidence that PI-VM can improve terminal rewards.

However, reward improvements do not necessarily imply better solutions to the KL-regularized control problem.

Table 6 provides an important example.

On the aesthetic objective, PI-VM achieves a higher reward than VM, but its regularized objective is worse:

Base: (J=-51.588)

VM: (J=-48.820)

PI-VM: (J=-41.227)

Since the objective is minimized, PI-VM performs worse than both comparisons on this task.

The authors acknowledge the increased control cost, which is appreciated. Nevertheless, the distinction between reward maximization and improvement of the stated optimization objective deserves greater emphasis.

In addition, the image examples suggest that JPEG-based rewards can favor substantial visual changes. Independent image-quality or semantic-preservation metrics would help establish whether these improvements remain meaningful beyond the particular reward functions.

W4. The practical advantages of the proposed temporal recursion and off-policy correction require more direct validation

The temporal recursion is mathematically natural, but it is closely connected to classical Feynman–Kac and entropy-regularized Bellman formulations.

Therefore, the principal novelty appears to lie in the particular finite-branch regression target, its integration with replay, and its applicability across process families.

It would be useful to isolate these components more clearly from the underlying classical identities and from existing Value Matching approaches.

In particular, the experiments do not sufficiently characterize the effects of behavior-policy mismatch and importance-weight variability.

Proposition 4.1 characterizes variance under an exact continuation value, whereas practical PI-VM uses a learned continuation and finite numbers of branches.

Appendices C and D appropriately acknowledge that the temporal operator is nonexpansive rather than strictly contractive, and that small value-function errors do not necessarily imply small gradient or diffusion-control errors.

These qualifications are important.

A targeted empirical investigation of effective sample size, continuation error, and sensitivity to behavior-policy mismatch would make the practical implications of the theoretical analysis more convincing.

W5. Some reproducibility details remain underspecified

Although the appendices contain substantial implementation information, several details would help readers interpret the quantitative results:

The number of independent runs and the meaning of the reported ± values are not clearly specified.

Hardware specifications and end-to-end computational costs are insufficiently documented for the synthetic benchmarks.

The procedure used to compute or approximate the reference optimal controls, particularly for the manifold benchmarks, should be described more explicitly.

Results displayed as (0.00000\pm0.00000) should be reported with enough precision to distinguish genuinely negligible error from rounding.

These omissions are especially relevant when numerical differences between methods are very small.

Questions for authors
  1. Questions for the Authors

Q1. Computational fairness.

Can the authors provide error-versus-compute curves for PI-VM and the principal baselines, particularly Hard-SOC and DAM?

It would be helpful to include the total number of simulated transitions or generator evaluations, rather than relying primarily on update counts and per-update timings.

Q2. Recovery of jump dynamics.

Can the authors evaluate the learned CTMC transition rates and jump kernels directly?

For example, how closely do the learned rates approximate the optimal rates under an occupancy-weighted error metric, and how does the achieved regularized objective compare with the optimum?

Q3. Empirical validation of off-policy correction.

How sensitive is PI-VM to the discrepancy between behavior and reference processes?

Could the authors report effective sample sizes or empirical importance-weight variance for different behavior-policy mismatches and segment lengths?

A focused comparison with replay and likelihood correction individually disabled would also clarify their respective contributions.

Q4. Image-generation objective.

Why does PI-VM incur a substantially worse regularized objective on DiT aesthetics despite improving the reward?

Is this primarily caused by the value approximation, the control parameterization, or the particular reward scale and regularization strength?

Q5. Reproducibility.

How many independent random seeds were used, what do the reported uncertainty values represent, and how were the manifold ground-truth controls computed?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 8Nwo

We thank the reviewer for their suggestions. We added continued baseline training, control-objective evaluations, and off-policy diagnostics, and clarified costs and statistical conventions.

R2.1 Continued baseline training (W1, Q1)

The General Response reports continued-training settings and five-seed L2L^2 and ΔJ\Delta J results. We will include these in the revised manuscript.

R2.2 Training time, trajectory collection, replay, and memory (W1, Q1)

Tables 2–3 report total training time divided by epochs, including trajectory collection, branch simulation, and replay refresh. Per-epoch times and median GPU/CPU memory are in the General Response. PI-VM achieves higher accuracy with lower per-epoch costs on most tasks.

Image-training times are lower than VM on all three DiT tasks, but higher on CIFAR Incompression and Aesthetics. We will clarify hardware, timing scope, and memory statistics in the revised manuscript.

R2.3 Evaluation of jump kernels and CTMC transition rates (W2, Q2)

We added the regularized objective gap ΔJ=J(Pθ)−J(P⋆)\Delta J=J(P_\theta)-J(P^\star), where

J(P)=EP[C]+λDKL(P∣P0)J(P)=\mathbb E_P[C]+\lambda D_{\mathrm{KL}}(P|P^0)

This measures path-law discrepancies, including jump activity and transition rates missed by induced-drift error. Results are in "Additional results from continued baseline training" in the General Response.

R2.4 The regularized objective and image quality on DiT aesthetics (W3, Q4)

Image fine-tuning requires a reward–quality trade-off. DiT also reports J=C−ηE[r]J=C-\eta\mathbb E[r], with KL-regularized control cost CC. For a fixed reference and weights, lower JJ indicates a better overall control objective. PI-VM improves both reward and JJ on both JPEG tasks.

R2.5 Continuation-value error and behavior-policy mismatch (W4, Q3)

Our contribution is finite-branch soft-min regression with short-segment replay. See R3.1 for the relation to Du and R3.6 for component ablations. OU Linear diagnostics show that bounded continuation errors are not amplified in one PI update, but constant offsets propagate unchanged. Correction lowers absolute bias in the settings below, while strong mismatch reduces ESS and can worsen RMSE. These one-step diagnostics do not guarantee neural control accuracy. We will add the experiments and their scope to the revised manuscript.

Continuation-value error: V~H=VH⋆+ϵcos⁡z\widetilde V_H=V_H^\star+\epsilon\cos z, H=8H=8, with exact Euler-optimal behavior sampling. Here zz is a standardized projection under the optimal endpoint distribution.

Maximum input error ϵ\epsilonPopulation value shiftMean starting-value shift N=8N=8Mean starting-value shift N=64N=64
0.10.059630.059630.05982±0.000100.05982\pm0.000100.05971±0.000100.05971\pm0.00010
0.50.275460.275460.28001±0.000560.28001\pm0.000560.27634±0.000570.27634\pm0.00057
10.483300.483300.50714±0.001390.50714\pm0.001390.48711±0.001400.48711\pm0.00140

Behavior-policy mismatch: Exact continuation, aδ=uGT+δea_\delta=u_{\rm GT}+\delta e, ∥e∥=1\|e\|=1, H=8H=8, N=64N=64. Correction-on/off comparisons share sampled paths.

δ\deltaLikelihood correctionPI value biasRMSEMean ESS (max. 64)
0On(1.52±3.89)×10−6(1.52\pm3.89)\times10^{-6}2.10×10−42.10\times10^{-4}64.00
0Off−0.09863±0.00067-0.09863\pm0.000670.106630.1066358.04
2On0.00383±0.001210.00383\pm0.001210.076710.0767147.19
2Off0.08054±0.000670.08054\pm0.000670.090160.0901658.04
8On0.27267±0.015180.27267\pm0.015180.690470.690477.08
8Off0.61806±0.000670.61806\pm0.000670.619390.6193958.04

R2.6 Seeds, uncertainty, hardware, and reference controls (W5, Q5)

  1. Seeds and error bars. Ablations, the Du comparison, and continued training use 5 independent training seeds, reporting endpoint mean ±\pm sample SD. R2.5 uses 5 sampling seeds. Image experiments use one training seed. We will state these conventions per table and distinguish training, evaluation-sampling, and timing variability in the revision.
  2. Hardware and cost. ReAct-VM timing uses a local RTX 4060 Laptop GPU and independent timing repeats. We will specify hardware, timing scope, and memory statistics per experiment in the revision. Continued-training runs without complete timing records do not support equal-compute claims.
  3. Reference controls. Jump/CTMC use independent semigroup/matrix-exponential computations, Sphere a Legendre heat-equation solver, and Stiefel a spectral solver (R3.4). We will add solver settings and accuracy checks in the revision.
  4. Precision. Fixed decimals can round nonzero means and SDs to zero. We will use scientific notation with sufficient significant digits in the revision, e.g., (1.1±0.2)×10−7(1.1\pm0.2)\times10^{-7} for PI-VM's Jump Partial error instead of 0.00000±0.000000.00000\pm0.00000.

Thanks. Added metrics help, but matched-compute gains remain unproven and aesthetics worsens J. One-step tests do not establish multi-step control accuracy. Concerns remain.

AuthorsOpenReview ↗

Response to Reviewer 8Nwo's Follow-up

Thank you for your follow-up. We summarize the additional evidence on computational efficiency, image fine-tuning, and control accuracy with learned continuation values.

1. Compute and accuracy. The updated comparisons show lower reported training time for PI-VM than the corresponding baselines across the OU, discrete, manifold, and DiT benchmarks:

  • OU: PI-VM, AM, and VM use the same 60,000/60,000/80,000-update budgets on Linear/Easy/Hard. PI-VM's amortized training times are 8/7/7 ms per update, compared with 41/21/21 ms for AM and 90/45/46 ms for VM. These times include trajectory collection, branch simulation, and replay refresh.
  • Discrete and jump processes: Both methods use 24,000 training updates. On the CTMC tasks, estimated total training times are approximately 1.4–2.5 hours for PI-VM versus 8.8–13.3 hours for DAM. On the jump tasks, they are approximately 19–20 minutes versus 10.7–17.8 hours, respectively.
  • Manifolds: On Sphere, Stiefel β=0.01\beta=0.01, and Stiefel β=0.5\beta=0.5, PI-VM's estimated total training times are 28.21/30.13/58.41 minutes, compared with 2.77/3.61/3.65 hours for the continued Hard-SOC runs. PI-VM also achieves lower control error on all three tasks.
  • DiT: PI-VM uses fewer generator evaluations on all three tasks: 25.969M versus 32.000M for VM, including replay initialization and refresh. PI-VM's total training times are 22.08/22.17/39.68 hours, also lower than VM's 24.00/26.17/43.97 hours.

Taken together, these experiments show that our method achieves an accuracy advantage on the vast majority of tasks with lower training-time costs, demonstrating its advantages in both accuracy and computational efficiency.

2. Aesthetics. Relative to VM, PI-VM increases control cost by approximately 8.97 and weighted reward by approximately 1.37. The additional control cost exceeds the reward gain, resulting in a higher JJ. This pattern may reflect reward-model overoptimization.

On the other two DiT tasks—Compression and Incompression—PI-VM improves both reward and JJ over VM, demonstrating better fine-tuning control performance on both tasks.

3. Practical control accuracy with learned continuation values. The full neural-training ablation in R3.6 directly evaluates control accuracy with learned continuation values, finite numbers of branches, and repeated bootstrap updates. The results are reproduced below.

Control error L2\mathcal{L}_2. All entries are multiplied by 10−310^{-3}:

SettingLinearEasyHard
Complete method0.179±0.0150.179\pm0.0151.053±0.1881.053\pm0.188204.249±50.217204.249\pm50.217
Without replay0.317±0.0300.317\pm0.0305.267±1.5635.267\pm1.563421.671±154.717421.671\pm154.717
Without likelihood correction27.052±27.16827.052\pm27.1685.833±0.6985.833\pm0.698372.278±17.014372.278\pm17.014
Without behavior guidance1.924±0.3851.924\pm0.3851.255±0.0701.255\pm0.070378.992±31.003378.992\pm31.003

The complete method achieves the lowest mean control error on all three tasks. Individually disabling replay, likelihood correction, or behavior guidance increases mean control error, showing that each component improves practical training with learned continuation values. R2.5 complements these results by quantifying how strong behavior-policy mismatch reduces ESS and can increase finite-sample RMSE.

These full-training results show that the complete algorithm maintains good control accuracy through repeated bootstrap updates with approximate continuation values and finite-branch sampling. They support its robustness to value-approximation error and sampling noise on the tested OU benchmarks.

In light of this additional evidence, we would appreciate your reconsideration of the paper and an upward revision of your score. Thank you for your careful review and thoughtful discussion.

Reviewer_oPV6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Path Integral Value Matching (PI-VM), a method for KL-regularized control of continuous-time Markov processes. The main idea is to learn a value function using short trajectories instead of estimating the full path integral. The learned value function is then used to construct the controlled process through the Doob transform.

The method uses temporal-difference learning, experience replay, and importance sampling to improve training efficiency. It can be applied to several types of processes, including diffusions, jump processes, and continuous-time Markov chains. The authors evaluate PI-VM on synthetic control problems and image-generation tasks. The results show strong performance on several benchmarks.

Critical strengths

I find the idea of learning a single value function and using it to recover different control processes interesting. Although the path-integral formulation and Doob transform are well-known, the paper provides a practical way to combine them with temporal-difference learning. This makes it possible to apply a similar learning procedure to several types of Markov processes.

The experiments cover a wide range of stochastic processes. PI-VM achieves low control errors on the OU benchmarks and performs well as the state dimension increases. The results on manifold and jump processes are also encouraging. I appreciate that the authors test the method beyond standard diffusion models and include image-generation experiments.

Critical weaknesses
  1. The main contribution of PI-VM is its temporal-difference target in Equation 19. However, taking the logarithm of a finite-sample average introduces a systematic bias, even when the continuation value is exact.

The authors acknowledge this issue in Appendix D.4. However, its impact on the learned control is not fully studied. For example, with only one branch and reference sampling, the expected target becomes the expected segment cost plus the continuation value, rather than the desired soft value update. Therefore, this bias may change the learned control, not just increase the training noise.

Moreover, Equation 8 provides a linear recursion for the partition function Z. It is unclear why the authors choose to learn V directly using the biased logarithmic target instead of learning Z through this linear recursion.

The branch-count ablations are useful, but they do not show how this bias affects learning under different regularization strengths or more difficult reward distributions. This is important because the finite-branch estimator is a central part of the proposed method.

  1. The paper presents PI-VM as a general method for controlling different Markov processes. However, learning a value function does not always make it easy to sample from the corresponding controlled process.

For jump processes, the optimal transition kernel depends on the exponential difference between value estimates. Applying this transformation may require additional computation or approximations.

For example, Appendix K.3 shows that the authors clip the jump-rate multiplier in the CIFAR experiments. This makes the implementation more practical, but the resulting process is no longer the exact Doob transform of the learned value function.

The paper does not clearly measure how much this approximation changes the final distribution or the original regularized control objective.

I believe the unified learning objective is useful. However, more evidence is needed to show that the method can also recover and simulate the intended controlled dynamics reliably across different process types.

Questions for authors
  1. Equation 8 gives a linear recursion for the partition function Z. Have the authors considered learning Z directly with temporal-difference regression?

Could the authors explain the advantage of the logarithmic target in Equation 19 and provide evidence of how its finite-sample bias affects the learned control? In particular, I would be interested in its behavior under smaller values of λ\lambda.

  1. In Appendix K.3, the jump-rate multiplier is clipped during image generation.

How sensitive are the results to this clipping range? Could the authors clarify whether the resulting controlled process remains close to the intended Doob transform, either in terms of its transition dynamics or the regularized control objective?

1 comment
AuthorsOpenReview ↗

Response to Reviewer oPV6

We thank the reviewer for their suggestions. We added V/ZV/Z comparisons, branch-count ablations across regularization strengths, and jump-rate clipping statistics to address estimator bias and sampling approximations.

R1.1 Learning VV versus learning ZZ directly (W1, Q1)

Learning ZZ directly avoids the logarithmic bias when the continuation value is fixed, but does not guarantee better control accuracy and may also make training unstable. We completed V/ZV/Z comparisons on Linear, Easy, and Hard OU. These results support our choice of VV regression.

TaskLearning VV: control L2L^2Learning ZZ directly: control L2L^2
Linear OU0.000179±0.0000150.000179\pm0.00001583.17±38.1283.17\pm38.12
Easy OU0.001053±0.0001880.001053\pm0.0001880.01185±0.004010.01185\pm0.00401
Hard OU0.204±0.0500.204\pm0.0500.975±0.0680.975\pm0.068

R1.2 Branch-count ablations across regularization strengths and harder rewards (W1, Q1)

We added branch-count ablations on Hard OU with λ=1,0.1\lambda=1,0.1 and on the more difficult HardHard task. Further increases in branch count yield limited gains in both easier and harder settings. Thus, branch count affects final control accuracy, but 4–8 branches are sufficient for good accuracy, with controllable systematic error.

Branch-count ablation: control L2L^2 (all entries are multiplied by 10−310^{-3})

Task and regularizationM=1M=1M=4M=4M=8M=8M=32M=32
Hard, λ=1\lambda=11.316±0.1561.316\pm0.1560.814±0.1260.814\pm0.1260.694±0.0230.694\pm0.0230.653±0.2220.653\pm0.222
Hard, λ=0.1\lambda=0.10.554±0.0140.554\pm0.0140.371±0.0090.371\pm0.0090.310±0.0700.310\pm0.0700.318±0.1300.318\pm0.130
HardHard, λ=1\lambda=119.986±3.25419.986\pm3.25410.895±1.41210.895\pm1.41210.866±1.08110.866\pm1.08110.362±1.24510.362\pm1.245

R1.3 Jump-rate clipping and recovery of controlled dynamics (W2, Q2)

We clip the jump-rate multiplier to [1/2,2][1/2,2] to enable thinning with a fixed upper bound. The low clipping frequencies below indicate that this approximation has little effect on the final results. We will add the clipping implementation and activation frequencies to the revised manuscript.

Clipping frequency among proposed jumps

TaskPure jumpHybrid
Compression0.142%0.142\%0.575%0.575\%
Incompression0.334%0.334\%0.136%0.136\%
Aesthetics1.265%1.265\%0.854%0.854\%

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their suggestions. We have completed additional experiments and derivations. We will include the results and clarifications in the revised manuscript. Below, we address the main concerns shared by the reviewers.

Additional results from continued baseline training

We extended Hard-SOC training from 600 to 60,000 epochs on Sphere and from 1,000 to 3,000 epochs on Stiefel, and DAM training from 12,000 to 24,000 updates on Jump/CTMC. The tables below report control errors and regularized objective gaps. We will add the continued-training settings and results to the revised manuscript.

Control L2L^2 error after continued baseline training

TaskHard-SOC/DAMPI-VM
Sphere2.94083754±0.148430472.94083754\pm0.148430470.01622443±0.00351891\mathbf{0.01622443\pm0.00351891}
Stiefel β=0.01\beta=0.010.00002925±0.000003130.00002925\pm0.000003130.00000059±0.00000044\mathbf{0.00000059\pm0.00000044}
Stiefel β=0.5\beta=0.50.01971108±0.007309640.01971108\pm0.007309640.00315083±0.00018657\mathbf{0.00315083\pm0.00018657}
Jump Partial0.00000337±0.000000060.00000337\pm0.000000060.00000011±0.00000002\mathbf{0.00000011\pm0.00000002}
Jump Refresh0.00003564±0.000009960.00003564\pm0.000009960.00001385±0.00000357\mathbf{0.00001385\pm0.00000357}
CTMC Partial0.00000371±0.00000050\mathbf{0.00000371\pm0.00000050}0.00000497±0.000000200.00000497\pm0.00000020
CTMC Refresh0.00006837±0.000036050.00006837\pm0.000036050.00006496±0.00002561\mathbf{0.00006496\pm0.00002561}

Regularized objective gap ΔJ\Delta J

TaskDAMPI-VM
Jump Partial0.00009827±0.000009360.00009827\pm0.000009360.00000472±0.00000040\mathbf{0.00000472\pm0.00000040}
Jump Refresh0.00009303±0.000027970.00009303\pm0.000027970.00003294±0.00000529\mathbf{0.00003294\pm0.00000529}
CTMC Partial0.00011189±0.00002076\mathbf{0.00011189\pm0.00002076}0.00020284±0.000006340.00020284\pm0.00000634
CTMC Refresh0.00017456±0.000092440.00017456\pm0.000092440.00009912±0.00003523\mathbf{0.00009912\pm0.00003523}

Training time and GPU/CPU memory usage

TaskEst. time (s/epoch)Median GPU memory (GiB)Median CPU memory (GiB)
PIVMHard-SOC/DAMPIVMHard-SOC/DAMPIVMHard-SOC/DAM
Sphere0.085\mathbf{0.085}0.1660.1660.6110.6110.564\mathbf{0.564}1.4991.4991.227\mathbf{1.227}
Stiefel β=0.01\beta=0.010.181\mathbf{0.181}4.3324.3320.770\mathbf{0.770}0.8420.8421.5281.5281.224\mathbf{1.224}
Stiefel β=0.5\beta=0.50.175\mathbf{0.175}4.3804.3800.770\mathbf{0.770}0.8420.8421.5461.5461.223\mathbf{1.223}
CTMC Partial0.380\mathbf{0.380}1.3261.3260.645\mathbf{0.645}0.8520.8521.491\mathbf{1.491}1.5501.550
CTMC Refresh0.213\mathbf{0.213}1.9911.9910.631\mathbf{0.631}0.8810.8811.4851.4851.455\mathbf{1.455}
Jump Partial0.048\mathbf{0.048}1.6011.6011.5141.5141.422\mathbf{1.422}1.585\mathbf{1.585}1.6361.636
Jump Refresh0.050\mathbf{0.050}2.6722.6721.419\mathbf{1.419}1.4251.4251.584\mathbf{1.584}1.6321.632

With larger baseline training budgets, PI-VM still achieves better accuracy than Hard-SOC and DAM on most tasks, with faster training and comparable GPU and CPU memory usage.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.