- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PI-VM, a value-based approach to KL-regularized control of continuous-time Markov processes. By combining short-horizon trajectory sampling, bootstrapped continuation values, likelihood correction, and experience replay, PI-VM learns a scalar value function from which optimal controls can be recovered through the Doob transform. The method is evaluated on OU diffusions, manifold diffusions, jump processes, CTMCs, and reward-guided image generation. Its main contribution lies in the proposed value-learning estimator and its application across different stochastic processes, rather than in new fundamental control theory.
Critical strengths
-
The paper presents a unified value-learning objective applicable to multiple stochastic processes, while leaving process-specific simulation and control recovery separate. This provides a relatively simple and extensible framework for stochastic control.
-
The temporal recursion and off-policy likelihood correction are mathematically well motivated. The paper also carefully distinguishes theoretical guarantees under exact continuation values from the limitations of practical neural approximations.
Critical weaknesses
-
Prior work has already explored off-policy value matching, truncated trajectories, and continuation estimates. The paper does not directly compare against Du et al.'s off-policy VM or sufficiently isolate the benefits of its specific path-integral estimator.
-
Training budgets differ across methods, particularly for manifold and jump/CTMC experiments. Without matched computational budgets or convergence curves, it is difficult to attribute the reported improvements solely to the proposed algorithm.
-
Jump and CTMC experiments measure induced-drift error rather than discrepancies in the full transition dynamics. Moreover, PI-VM achieves a worse regularized objective than VM and the uncontrolled baseline on the DiT aesthetic task, despite obtaining higher rewards.
-
The ablations primarily examine branch counts, trajectory lengths, and network architectures, without isolating the effects of experience replay, behavior guidance, or likelihood correction. The theoretical variance results also do not directly establish the practical benefits of learned continuation values.
Questions for authors
-
How does the proposed estimator improve upon Du et al.'s off-policy Value Matching in terms of bias, variance, or computational efficiency? Could the authors provide a direct comparison under matched simulation budgets?
-
Can the authors report total computational costs, including trajectory collection and replay refresh, along with performance-versus-compute curves? How many independent training seeds were used, and what do the reported error bars represent?
-
Why do CE, VAR, SOCM, and SOCM-A report identical results on the hard OU task? How is AM implemented for non-differentiable JPEG rewards, and which components of the original Hard-SOC method are retained or modified?
-
What are the respective contributions of experience replay and likelihood correction to the reported improvements? How does performance change with increasing behavior-policy mismatch or replay-data staleness?
-
How are the ground-truth optimal controls and state distributions computed for the jump, CTMC, and manifold benchmarks? Could the authors report transition-rate/kernel errors or actual control objectives to complement the induced-drift metric?
-
The DiT experiments use different CFG settings during training and evaluation. Which reference process defines the learned value and likelihood corrections, and do the main conclusions hold when training and evaluation use consistent CFG settings?
2 comments
Response to Reviewer cqPD (2/2)
R3.5 The regularized objective on DiT aesthetics (W3)
Image fine-tuning requires a reward–quality trade-off. DiT also reports , with KL-regularized control cost . For a fixed reference and weights, lower indicates a better overall control objective. PI-VM improves both reward and on both JPEG tasks.
R3.6 Replay, behavior guidance, likelihood correction, and data staleness (W4, Q4)
Separately disabling replay, behavior guidance, or likelihood correction increases mean control error on all three OU tasks. The complete method has the lowest mean error in each task, showing that all three components help training with learned continuation values. Replay also gives a 13.3–17.5-fold speedup at the same update count. Mismatch diagnostics are in R2.5. Replay staleness has not been varied independently. We will add these ablations and their scope to the revised manuscript.
Component ablation: control (units of )
| Setting | Linear | Easy | Hard |
|---|---|---|---|
| Complete method | |||
| Without replay | |||
| Without likelihood correction | |||
| Without behavior guidance |
R3.7 Training/evaluation CFG and the reference process (Q6)
Training "CFG0" uses conditional prediction without CFG extrapolation, defining the reference drift for values and stored likelihood corrections. CFG2 evaluation uses , changing that drift, so the original value cannot be interpreted as the exact CFG2 value. With CFG0 in both training and evaluation, PI-VM still has lower on both JPEG tasks and higher on aesthetics. The conclusions therefore persist without the CFG switch. We will add this comparison and clarify reference processes in the revision.
Training and evaluation at CFG0: regularized objective (lower is better)
| Task | VM | PI-VM |
|---|---|---|
| Compression | ||
| Incompression | ||
| Aesthetic |
Response to Reviewer cqPD (1/2)
We thank the reviewer for their suggestions. We added comparisons with Du, clarified implementations and reference solutions, and evaluated replay contributions and consistent CFG settings.
R3.1 Comparison with Du et al.'s off-policy Value Matching (W1, Q1)
With unit regularization and Du's κ=1, let zθ=σ⊤∇Vθ, behavior control a, and shared endpoint term Bj (equal to g(XTj) for complete trajectories). Define
C_j&=B_j+\int_t^s\!\left(f+\tfrac12\|a\|^2\right)dr +\int_t^s a^\top dW^a,\\ D_j&=\tfrac12\int_t^s\|a+z_\theta\|^2dr +\int_t^s(a+z_\theta)^\top dW^a. \end{aligned}$$ Du's residual is $R_j^{\rm Du}=V_\theta(t,x)-C_j+D_j$, with target $Y_j^{\rm Du}=C_j-D_j$. PI-VM uses $\widehat V_N^{\rm PI}=-\log(\frac1N\sum_j e^{-C_j})$. Under the exponential-martingale conditions, $\mathbb E_a[e^{-D_j}-1\mid t,x]=0$, giving the calibrated target $$\begin{aligned} \widehat V_N^{\rm cal} &=-\log\frac{\sum_j e^{-C_j}}{\sum_j e^{-D_j}} =-\log\sum_j w_j e^{-Y_j^{\rm Du}},\\ w_j&=\frac{e^{-D_j}}{\sum_k e^{-D_k}}. \end{aligned}$$ **This is a weighted soft-min of Du's per-path targets and reduces to Du's target at $N=1$**. On-policy ($a=-z_\theta$), $D_j=0$, so the original PI-VM is an equally weighted soft-min with the same single-branch residual as Du. Du differentiates through $z_\theta$ along the path, whereas PI-VM stops target gradients, so residual equivalence does not imply identical updates. **ReAct-VM results on OU benchmarks** | Task | Final control MSE | Training time (ms/update) | | --- | --- | --- | | Linear | $(1.43\pm0.403)\times10^{-5}$ | $79.6\pm4.9$ | | Easy | $(1.05\pm0.118)\times10^{-4}$ | $42.2\pm3.5$ | | Hard | $(2.36\pm0.291)\times10^{-3}$ | $44.0\pm3.2$ | PI-VM combines multiple branches, continuation values, and replay to incorporate more trajectory information, which can improve stability and accuracy. Under the respective reported configurations (Table 2), PI-VM updates faster, has lower Linear/Hard errors, and similar Easy accuracy. Its Hard error is about one-third of ReAct-VM's. We will add the derivation, comparisons, and timing conventions to the revised manuscript. ### **R3.2 Total compute, convergence, and statistical reporting (W2, Q2)** To address budget differences on manifold and Jump/CTMC tasks, we continued Hard-SOC/DAM training. The General Response reports settings, results, and time/memory comparisons. PI-VM retains an accuracy advantage on most tasks. R2.2 explains trajectory-collection and replay timing. Ablations, the Du comparison, and continued training use 5 independent training seeds (endpoint mean and sample SD). Original Jump/CTMC results use sample SD, manifold results population SD, and image experiments one training seed. We will state these conventions per table and distinguish training from evaluation-sampling uncertainty in the revision. ### **R3.3 Baseline implementations and identical results on Hard OU (Q3)** CE, VAR, SOCM, and SOCM-A stall at the same incorrect Hard OU solution, giving identical control errors. AM uses a smoothed JPEG reward for gradients. Hard-SOC retains bridge sampling, terminal adjoints, vector transport, and regression updates, with a tangent-projected controller-only implementation and no Schrodinger bridge corrector. We will specify networks, step sizes, budgets, and implementation details in the revision. ### **R3.4 Jump/CTMC metrics and construction of ground-truth references (W3, Q5)** We added the regularized objective gap $\Delta J=J(P_\theta)-J(P^\star)$, where $$J(P)=\mathbb E_P[C]+\lambda D_{\mathrm{KL}}(P|P^0)$$ This evaluates recovered path laws beyond induced drift, capturing jump activity and transition rates that drift error alone can miss. Results are in the General Response. Jump/CTMC references use independent semigroup/matrix-exponential computations, Sphere a Legendre heat-equation solver, and Stiefel a spectral solver. We will add the metric, solver settings, and accuracy checks to the revised manuscript.