- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies overfitting in variance-controlling guidance (VCG) for diffusion steering. VCG fits a linear drift correction by minimizing the empirical variance of the Feynman–Kac log-weight rate. With limited fitting particles, this can lead to poor generalization beyond the fitting population. The authors propose RCPT, which uses leave-one-out residuals to estimate a scalar shrinkage factor for the VCG correction. Experiments cover synthetic distributions, molecular generation, and CIFAR-10, showing promising results.
Critical strengths
- The paper identifies a relevant finite-particle failure mode of VCG.
- RCPT is simple and computationally inexpensive, requiring no additional model evaluations.
Critical weaknesses
- Limited novelty and unclear advantage over regularized VCG: RCPT is essentially a scalar shrinkage of the fitted VCG coefficients, and the risk decomposition and oracle shrinkage rule largely follow from the quadratic regression objective. More importantly, Eq. (8) already includes ridge regularization, which controls overfitting during fitting and shrinks different eigendirections by different amounts, whereas RCPT applies a single scalar after fitting. Although Table 1 shows that RCPT outperforms LOO Ridge, it remains unclear whether this comes from a better risk estimator or from differences in how regularization is selected. A more controlled comparison with adaptively tuned ridge would help establish the benefit of RCPT.
- The risk estimator may be unreliable for online interacting particles: RCPT estimates the coefficient-estimation penalty from leave-one-out residuals and the ESS of the importance weights (Eq. 10). In online SMC, however, particles are dependent due to resampling and the shared adaptive proposal. The ESS does not account for this dependence, and deleting one particle does not remove its influence on the remaining population. The penalty may therefore be underestimated, so that RCPT retains too much of the VCG update. The paper acknowledges that its frozen-schedule guarantees do not extend to online interacting particles, but this is an important limitation since RCPT is also proposed for online deployment.
Questions for authors
- Can the authors compare RCPT with ridge-regularized VCG across a range of regularization strengths, using the same fitting particles and evaluation protocol on all benchmarks?
- How accurate is the estimated penalty in online SMC, where particles are dependent? In particular, can the authors compare it against the empirical estimation error and check whether particle dependence leads to underestimation?
- In the online QM9 experiment, VCG and RCPT perform similarly. How often does VCG produce harmful updates in online settings, and in which regimes does RCPT give a clear improvement?
4 comments
[2/2] Response to Reviewer CK4J
2. Online penalty accuracy and dependence (W2, Q2)
Your reliability concern is confirmed: the penalty is not quantitatively calibrated across online populations, and the implemented rule does not guarantee online safety. Equal weights and observation-wise deletion can overstate information when resampling leaves correlated copies.
In the controlled experiment, nominal and weight ESS remain 64. Reducing independent ancestors from 64 to 16 increases RCPT Risk/Anchor from 0.2639 to 16.4102. All 100 ensembles are harmful in the latter case. With eight ancestors, the estimated-penalty/reference-error ratio is about and mean amplitude is 0.99997, despite Risk/Anchor 75.7924 (Table L.11). Deleting one copy leaves its relatives in the fitting data.
On RCPT-driven Checker histories (Table L.12):
| Estimated penalty / reference error, 95% CI | Full VCG risk | RCPT risk, 95% CI | Harmful integrated RCPT runs | |
|---|---|---|---|---|
| 8 | 0.1047 [0.0156, 0.2644] | 510.9 | 6.042 [1.533, 14.095] | 7/10 |
| 16 | 0.2131 [0.1017, 0.6544] | 50.42 | 0.977 [0.620, 1.506] | 3/10 |
| 32 | 0.4415 [0.1081, 1.218] | 5.041 | 0.929 [0.540, 1.561] | 2/10 |
| 64 | 0.3149 [0.2042, 0.4477] | 0.7093 | 0.3814 [0.3452, 0.4292] | 0/10 |
Risk is normalized by Anchor. Ratios use run means after time integration. On separately VCG-driven histories at , penalty/error ratios are 3.402, 8.488, and 2.475. QM9 ratios range from 1.5–13.7 at and 0.032–0.263 at , across drivers and reference banks (Table L.13). Thus underestimation occurs, but its direction depends on the fitting population.
Reference coefficient error includes bias and evaluation-law mismatch, rather than only sampling variance. Whole-run intervals condition on finite reference banks. Appendix L.8 reports both banks and leave-one-population-out sensitivity. Some RCPT–VCG contrasts become unresolved under this sensitivity.
The exact fixed-law H/P identity permits dependent fitting data. The iid approximation motivating the proxy and the conditional bound in Theorem 3 do not establish accurate estimation for arbitrary online particles. We therefore evaluate online use by measuring both local harm and terminal generation outcomes.
3. Online harmful updates and a clear-benefit regime (Q3)
Full VCG is frequently locally harmful at low QM9 support, and RCPT reduces that frequency. At , RCPT also improves actual generation over both Anchor and VCG. Local harm means reference residual risk above Anchor on the same states. It is measured at 21 fixed nodes per run with time weights summing to one, separately from terminal failure probability.
Eight runs per driver and budget give these local-harm rates (Table L.13):
| , driver | Bank | Full VCG, 95% CI | RCPT, 95% CI |
|---|---|---|---|
| 18, RCPT | A | 54.4% [49.4, 60.1] | 9.7% [3.8, 16.2] |
| 18, RCPT | B | 57.6% [53.8, 61.3] | 11.6% [6.9, 16.5] |
| 18, VCG | A | 55.7% [50.7, 61.3] | 6.9% [3.1, 10.0] |
| 18, VCG | B | 60.7% [53.8, 67.5] | 10.6% [6.9, 14.4] |
| 144, RCPT | A | 6.2% [4.4, 7.5] | 3.8% [1.6, 5.9] |
| 144, RCPT | B | 10.6% [8.1, 13.1] | 9.1% [5.9, 12.5] |
| 144, VCG | A | 6.2% [4.4, 7.5] | 5.3% [3.4, 6.6] |
| 144, VCG | B | 10.6% [8.1, 13.1] | 9.7% [7.2, 12.2] |
Bank A combines four independent 144-particle systems. Bank B combines two 288-particle systems. Each totals 576 reference particles. The intervals use whole runs.
In the separate eight-run generation comparison at , terminal-weighted MAE is 0.7926 for RCPT, 0.9403 for Anchor, and 0.8238 for VCG. Paired differences are −0.1477 [−0.2827, −0.0038] and −0.0311 [−0.0476, −0.0138]. Terminal-weighted target success improves by +20.94 pp [6.00, 35.58] and +4.53 pp [2.25, 6.69], and mean distinct successes tie VCG at 12.875. Mean amplitude is 0.733 (Table 4). Both endpoints improve over each comparator with paired intervals excluding zero. VCG itself improves on Anchor here, so RCPT improves an already useful correction.
At , RCPT retains mean amplitude 0.945 and all 144 lineages, with 45.50 distinct successes versus VCG's 45.25. At , two RCPT runs lose ancestry, its MAE contrast with VCG is unresolved, and it has fewer distinct successes. The complete eight-run-per-budget sweep includes these outcomes (Appendix L.2). The evidence supports useful online operation in measured regimes, while leaving the reliability limitation intact.
The practical finding and its theoretical explanation remain useful even where another regularizer is competitive: a low fitting residual is insufficient evidence to deploy an FK correction. The matched comparisons show when the proposed low-cost decision rule helps, and the molecular endpoints establish benefits beyond that diagnostic observation.
Thank you for the detailed response and the additional experiments.
The new ridge comparisons and dependence audits address my questions, and the QM9 results give a clear example where RCPT improves over both Anchor and VCG.
At the same time, it shows that the advantage over tuned ridge depends on the regime, and that the penalty can be underestimated in online SMC. I think these limitations should be stated clearly in the main text. I will keep my score.
We are glad that the new ridge comparisons and dependence audits addressed your questions. Following your suggestion, we have revised Section 7 to make both limitations clear in the main text. Thank you again for the constructive review and helpful suggestions.
[1/2] Response to Reviewer CK4J
Thank you for asking for matched regularization comparisons and direct tests of online reliability. We identify when finite fitting reverses the benefit of an FK correction, explain how the error affects sampling, and use that analysis to choose how much correction to deploy without extra model calls. On frozen scaffold and online QM9, RCPT improves generation over both Anchor and VCG. The dependence audits show where the online penalty can fail. References are to the revised manuscript. Intervals are nominal paired 95% intervals over whole fits, runs, or pockets.
1. What RCPT adds to regularized VCG (W1, Q1)
A fitted FK correction can have nearly zero fitting residual yet increase independent residual risk and worsen generated samples. Theorem 1 explains this reversal through P−H. Theorem 4 and Proposition 1 connect the residual to unweighted transport error and local correction-weight inflation. Building on classical shrinkage, our structured deletion-LOO rule uses this deployment risk to choose how much correction to retain, with no additional model calls.
For fixed features before safeguards, ridge scales covariance eigendirections by 1/(cj+η), while RCPT scales the fitted direction by α. For α>0, matching RCPT with a new ridge would require η′=(cj+η)/α−cj for every active eigenvalue cj. In general no single value does so. Which choice works better is an empirical question.
Fixed ridge. We deployed four strengths (10−6,10−4,10−2,0.1) on the same ten paired CIFAR runs. Stronger ridge closes much of the gap, but at least six of seven endpoint means favor RCPT at every strength. At 0.1, ridge reaches 60.219% independent accuracy, 0.2583 KID and 78.9632 FID, versus RCPT's 60.906%, 0.2381 and 78.5020. DLSM-classifier accuracy favors ridge at 0.01 and 0.1 (91.469% and 91.844% vs 91.375%). Table L.7 gives the full results. Anchor (α=0) does better than RCPT on six endpoints and has nearly identical mean FID. Original DLSM is a separate baseline (Table 5).
Checker and scaffold also have strength-by-strength reference-risk comparisons (Figure L.1, Table L.8). On eight scaffold pockets with four fits each, ridge alone has lower mean reference risk than RCPT applied to the same candidate throughout the grid. Fixed-strength molecular deployments cover 10−3,10−2,10−1 on ten scaffold pockets and in a four-seed, three-field QM9 audit (Tables I.11–I.12).
Selected ridge. We minimize deletion loss on the relative grid {0,10−6,10−4,10−2,0.1,1,10,100}, scaled by the training covariance trace per coefficient and added to the numerical ridge. Selection uses fitting data alone. Independent states and terminal outcomes in all four domains are reserved for evaluation (Appendix L.6).
The risk and regret comparisons with Direct LOO use the same fitted direction to isolate amplitude selection. At Checker N=16, RCPT-minus-Direct risk is −0.6331 [−1.3630, −0.0669]. Holding direction and safeguards fixed, the criterion-by-continuity factorial attributes most of the N=16 gain to the structured criterion. Continuity adds resolved benefits at N=8,32 and in all eight saved online budget/driver conditions (Tables L.16–L.17). At N=64, Direct LOO and selected ridge can have lower risk.
Frozen scaffold deployment improves useful yield over both Anchor and VCG. At fit-64 (12 coefficients), unique-useful yield is 27.1% vs 23.6% for Anchor and 20.0% for VCG across 76 pockets. The paired gains are +3.45 pp [0.66, 6.17] over Anchor and +7.07 pp [3.04, 11.10] over VCG (Table 3). Both intervals exclude zero at this larger fitting budget. At fit-16, guarded VCG and Direct LOO have higher means than RCPT (0.2656 and 0.2771 vs 0.2508), with intervals spanning zero. Guarded VCG's recovery shows that safeguards explain much of the gain over raw VCG (Appendix L.4). The matched Checker factorial and the QM9 comparisons provide evidence specific to the selection rule.