acceptodds
Under review as a conference paper at ICLR 2027

Imperfect Verifiers, Exact Targets: Inference-Time Scaling for Continuous Generation

Abstract

Verifier-guided inference-time scaling uses additional computation to score intermediate states and steer generation toward better outputs. However, imperfect intermediate verifiers can distort sampling, and repeated refinement alone does not ensure recovery of the desired reward-tilted distribution. We introduce Continuous-State Verifier-Guided Backtracking (CS-VGB), an inference-time scaling method for sampling from reward-tilted distributions with continuous generative models through repeated denoising and renoising. With correctly paired denoising and renoising transitions, the stationary distribution of CS-VGB at the clean checkpoint equals the desired reward-tilted target, even when intermediate verifiers are inaccurate. Under stated regularity conditions, CS-VGB converges exponentially to the target despite verifier error, and the expected number of updates needed to reach a clean sample grows only linearly with the number of checkpoints. In analytically tractable settings, we empirically demonstrate that CS-VGB recovers the desired reward-tilted distribution despite verifier error, whereas the compared verifier-based sampling methods fail to recover the target under the same verifier errors. We also instantiate the framework with diffusion models and one-step stochastic posterior maps. Across prompt-guided ImageNet generation, facial-attribute generation on CelebA, and multi-robot planning, recurrent CS-VGB search improves reward and task success as inference budgets grow, with image-generation rewards continuing beyond the observed plateaus of denoising-only baselines.

Primary area:
generative models
Keywords:
Inference-time scaling, continuous generative models, verifier-guided backtracking, reward-tilted sampling, Markov chain Monte Carlo

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_jHemOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an innovative method, Continuous State Verifier Guided Backtracking (CS-VGB). It introduces: 1. Bidirectional search among intermediate states to generate samples from a desired reward-tilted terminal distribution; 2. Theoretical guarantees for the proposed method; 3. Empirical studies across multiple domains demonstrating the method's effectiveness. The method works with any pretrained model and alleviates verifier error very effectively. Overall, the paper is well written with rigorous theories and proofs. As a theoretical paper, it also provides promising empirical results in simple 2D data, image data, and robotics data.

Critical strengths

It does not rely on the choice of underlying model or verifier accuracy.

The theorems are relevant to the main paper. The proofs are well structured.

The results clearly show the strengths and effectiveness of the proposed method. The experiments are also diverse across multiple domains. Ablation studies clearly demonstrate the importance of each key component (bidirectionality, backward guidance, reverse insertion, etc.) in the method.

Critical weaknesses

Some concerns about the gap between theory and practice, mostly on the trade-off between high reward and low search time.

Questions for authors

The questions are mostly related to the practical feasibility.

Could you justify the introduction of extra hyperparameters such as directional persistence ϕ\phi, depth weight α\alpha and pool sizes LF,LBL_F,L_B? These parameters add more burden and worsen the trade-off between high reward and low search time. In addition, how invariant are these hyperparameters? In other words, do you need to tune them for different underlying models, tasks, or other conditions?

Have you considered any early stopping criteria for your method? From the results, it seems that the clean outputs are already in a sufficiently high reward state given a low budget. I believe this can alleviate some of the time complexity, especially for time-sensitive tasks where a lower search time is preferred. In addition, although this paper focuses on clean (boundary) states, have you guys observed any improvements in the interior states as well? Perhaps they are also useful byproducts.

Minor question: could you provide more justification or intuition on why reverse insertion is needed and how naive resampling introduces bias?

6 comments
AuthorsOpenReview ↗

(1/4) Response to Official Review by Reviewer jHem

We thank the reviewer for the helpful comments and suggestions. We have addressed them below and hope that our responses provide a clearer understanding of the method and how it can be used in practice.


Weakness (1): Some concerns about the gap between theory and practice, mostly on the trade-off between high reward and low search time.

Question (1): Could you justify the introduction of extra hyperparameters such as directional persistence χ\chi, depth weight α\alpha and pool sizes LF,LBL_F, L_B? These parameters add more burden and worsen the trade-off between high reward and low search time. In addition, how invariant are these hyperparameters? In other words, do you need to tune them for different underlying models, tasks, or other conditions?

Question (2): Have you considered any early stopping criteria for your method? From the results, it seems that the clean outputs are already in a sufficiently high reward state given a low budget. I believe this can alleviate some of the time complexity, especially for time-sensitive tasks where a lower search time is preferred. In addition, although this paper focuses on clean (boundary) states, have you guys observed any improvements in the interior states as well? Perhaps they are also useful byproducts.

Answer. We thank the reviewer for these practical suggestions. Our additional experiments support using shared persistence and depth settings, χ=1\chi=1 and α=0.3\alpha=0.3, as a starting point across tasks, while choosing candidate-pool sizes LFL_F and LBL_B according to the available computation. Under tighter inference budgets, reducing the depth weight or pool sizes may help the search produce a clean output within the available computation. Once an output is available, early stopping can save generation time if some quality loss is acceptable. If more time is available, continued search can also improve the intermediate states, allowing generation from those states to produce better outputs. These results therefore help users decide when to stop and show how intermediate states can be useful for further generation.

On tuning χ\chi and α\alpha. Our guided experiments use χ=1\chi=1 and α=0.3\alpha=0.3 across ImageNet, CelebA, and planning, without tuning these two parameters separately for each task. Based on our theoretical design, we expect both χ\chi and α\alpha to substantially influence the search dynamics through complementary roles: χ\chi controls how persistently the search maintains its direction, while α\alpha controls the balance between denoising toward clean outputs and renoising toward noisier states.

Choosing χ=1\chi=1 satisfies the persistence condition in Proposition 2.1, which gives an expected hitting time of Θ(J)\Theta(J) to the first clean level under its stated assumptions. This motivates strong persistence when obtaining a clean output within a limited budget matters. Consistent with the persistence ablation in Appendix F.6.1, our new sweep in Table R4(a) shows progressively fewer forward-to-backward changes (i.e., switches from denoising to renoising) and more clean-level visits as χ\chi increases. From χ=0\chi=0 to χ=1\chi=1, the mean number of direction changes falls from 58.70 to 24.33, while clean-level visits increase from 12.35 to 16.18. This supports using strong persistence to obtain clean outputs more frequently, alongside the reported practical benefits of momentum in MDM-VGB [1] and AR-VGB [2]. We therefore use χ=1\chi=1 as our default rather than tuning it separately for each task.

Table R4(a). Persistence χ\chi sweep on CelebA at a 4,096-NFE cap. Each setting uses 20 fixed conditions × 3 seeds. Values are mean ± standard deviation across the three seed means.

Persistence χ\chiForward-to-backward changes ↓Clean-level visits ↑
058.70 ± 1.3612.35 ± 0.56
0.349.05 ± 2.4312.93 ± 0.47
0.542.08 ± 1.5113.80 ± 0.28
0.734.80 ± 1.0014.55 ± 0.13
1 (default)24.33 ± 2.0416.18 ± 0.65
AuthorsOpenReview ↗

(2/4) Response to Official Review by Reviewer jHem

For α\alpha, lowering its value favors denoising over renoising by scaling the backward-to-forward mass ratio (Eq. B.17). This encourages clean returns under a tight budget. However, if α\alpha is too small, the search favors denoising so strongly that renoising may not proceed far enough to enable meaningful changes to the current candidate. For example, the chain may repeatedly move between the clean level j=0j=0 and the adjacent level j=1j=1, returning to a clean output before reaching higher noise levels where it has more freedom to change the sample. We therefore use α=0.3\alpha=0.3 as a shared starting point to balance clean returns with deeper renoising, and examine how α\alpha affects the clean-hit rate under a 512-NFE cap on ImageNet in Table R4(b).

Table R4(b). Clean-hit rate on ImageNet under a 512-NFE cap. Results use 20 fixed conditions × 3 seeds. Clean-hit rate is the percentage of runs that return at least one clean image within the NFE cap. α=0.3\alpha=0.3 is our default setting.

α\alpha0.10.20.30.40.61.0
Clean-hit rate (%)100.0100.096.793.351.73.3

As α\alpha decreases, the clean-hit rate increases and reaches 100% at α=0.2\alpha=0.2 in this setting. However, as discussed above, maximizing this rate alone can leave too little renoising for broader exploration, especially in tasks such as multi-robot planning. To balance reliable clean returns with deeper exploration, we therefore kept α=0.3\alpha=0.3 across all three tasks in our guided experiments.

On tuning LFL_F and LBL_B. Whereas χ\chi and α\alpha control directional persistence and the denoising–renoising balance, LFL_F and LBL_B determine how many candidate states are considered before each forward or backward move, trading more alternatives at each step for fewer search steps within the same budget.

Table R4(c) shows that small candidate pools already perform well at the tested medium inference budget (4,096 NFE). Starting from (LF,LB)=(2,2)(L_F,L_B)=(2,2), we increase either the forward or backward pool to 4 and 8 while keeping the other at 2. This compares spending the budget on more candidates per move with leaving more computation for subsequent search steps.

Table R4(c). Candidate-pool size ablation on ImageNet at a 4,096-NFE cap. Entries report mean HPSv2 over 20 fixed conditions × 3 seeds, with mg=4m_g=4 Monte Carlo samples for guidance and J=64J=64 checkpoint intervals.

(LF,LB)(L_F,L_B)(2,2)(2,2)(4,2)(4,2)(8,2)(8,2)(2,4)(2,4)(2,8)(2,8)
HPSv2 ↑46.3145.8344.9646.2545.87

Larger pools require more candidate evaluations at each move, leaving fewer search steps within a fixed inference budget. Although each move considers more alternatives, the resulting reduction in exploration can hurt performance, consistent with the slightly lower mean rewards observed for several larger-pool settings in Table R4(c). These results support starting with small pools and increasing them only when they improve quality within the intended inference budget.

AuthorsOpenReview ↗

(3/4) Response to Official Review by Reviewer jHem

On early stopping and intermediate-state quality. We thank the reviewer for these suggestions, which raise two practical questions: when to stop the search, and whether intermediate states also improve as the search continues. Table R5(a) shows how much time early stopping saves and how it affects output quality. Table R5(b) examines whether later intermediate states produce better clean outputs than earlier states at the same noise level.

For early stopping, we evaluated a patience rule that stops after insufficient reward improvement over a specified period. On ImageNet at the 4,096-NFE preset, this reduces average generation time from 45.63 to 41.01 seconds, saving 10.13%, with decreases of 0.26% in HPSv2 and 2.96% in VQAScore. At the 512 preset, the larger time saving of 21.81% comes with a slightly larger reward decrease of 0.86% and a VQAScore decrease of 7.21%. Thus, early stopping offers a controllable quality–time trade-off, rather than preserving every quality metric. We also remark that our multi-robot planning implementation already stops upon finding a feasible trajectory.

Table R5(a). Patience stopping. Each setting uses 10 conditions × 2 seeds. ΔReward and ΔVQAScore are percentage changes relative to full-budget generation, computed as 100×(stop−full)/full100\times(\mathrm{stop}-\mathrm{full})/\mathrm{full}. Reward is GM for CelebA and HPSv2 for ImageNet. Patience settings were selected on validation data and fixed before evaluation; trigger rate is the fraction of stopping runs in which the rule activates.

Task / NFE presetFull (sec/image)Early stop (sec/image)Time savedΔReward (%)ΔVQAScore (%)Trigger rate
CelebA / 51216.5715.814.60%−0.26%+9.82%20%
CelebA / 1,02423.4522.464.23%−0.07%−6.43%35%
ImageNet / 51212.739.9621.81%−0.86%−7.21%100%
ImageNet / 4,09645.6341.0110.13%−0.26%−2.96%55%

Continued search also improves intermediate-state quality: Table R5(b) shows higher mean completion reward and VQAScore at all six tested task–level combinations. We compare the first visit with the first eligible revisit after 3,072 NFE at the same noise level, estimating the reward of each intermediate state by averaging the terminal rewards of 16 fresh one-step clean completions.

Table R5(b). Quality of clean completions from intermediate states. Each row contains 15 paired states from 5 conditions × 3 seeds at the same noise level, with 16 fresh completions per state. Values compare first → later visits. Reward is GM for CelebA and HPSv2 for ImageNet. Reward improvement is mean ± standard deviation across the three seed means.

Task / levelSearch time: first → later (s)Reward: first → later ↑Reward improvement ± SD ↑VQAScore: first → later ↑
CelebA / j=2j=29.10 → 39.470.616 → 0.983+0.367 ± 0.0240.269 → 0.368
CelebA / j=4j=48.77 → 39.450.541 → 0.979+0.439 ± 0.0370.270 → 0.356
CelebA / j=6j=68.43 → 39.610.500 → 0.976+0.476 ± 0.0380.259 → 0.343
ImageNet / j=4j=411.06 → 35.5135.331 → 44.502+9.171 ± 0.2980.545 → 0.837
ImageNet / j=8j=810.45 → 35.4132.419 → 42.599+10.180 ± 0.6270.436 → 0.839
ImageNet / j=12j=129.84 → 35.5230.096 → 40.380+10.283 ± 0.8360.399 → 0.839

These results indicate that the benefits of continued search extend beyond the final clean output: at the same noise level, later intermediate states generate clean completions with higher mean reward and VQAScore. In other words, search also improves intermediate states, which can then produce better clean outputs. MDM-VGB [1] shows a related benefit: editing an existing sample can match or improve the quality of generation from scratch while requiring fewer adjusted NFEs, especially when the starting sample already has a high reward. Since CS-VGB can likewise start from an existing clean sample and apply renoising and backtracking, sample editing is an interesting potential application, with its efficiency and content preservation left for future evaluation. We thank the reviewer for suggesting this interesting direction.

Reviewer_iNadOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CS-VGB, an inference-time scaling method that performs bidirectional denoising and renoising guided by an imperfect process verifier. Under correctly paired transitions and exact terminal-reward evaluation, its stationary distribution equals the reward-tilted target regardless of process-verifier accuracy, with convergence guarantees under additional assumptions. Synthetic experiments demonstrate target recovery, while image-generation and multi-robot planning experiments demonstrate practical search improvements.

Critical strengths

• Preserves the reward-tilted stationary target despite imperfect process verifiers under the stated conditions. • Provides average-case verifier-error analysis and separately analyzes transition mismatch with an explicit bias bound. • Achieves strong performance against competing approaches at comparable NFE, particularly at larger computational budgets.

Critical weaknesses

• The practical relevance of the theoretical guarantees remains unclear. As acknowledged by the authors, the practical search implementation omits correction factors and uses heuristics outside these guarantees. It would be helpful to explain how informative the convergence assumptions and bounds are for the experimental settings. • Computational comparisons primarily use NFE, which may not capture differences in sequential dependence, parallelization, and verifier overhead. Consequently, the reported NFE advantages do not necessarily establish better wall-clock efficiency.

Questions for authors

• Appendix F.6 reports runtime improvements from lazy evaluation, but not comparisons against competing methods. Could the authors report reward or task success versus end-to-end wall-clock time on comparable hardware, including average generation time per image at representative NFE budgets? • Beyond the transition-mismatch analysis, could the authors discuss the practical consequences of violating the assumptions required for target preservation and convergence, particularly in settings not covered by the theoretical bounds?

5 comments
AuthorsOpenReview ↗

(1/4) Response to Official Review by Reviewer iNad

We appreciate the reviewer's careful reading and constructive questions. We respond to each point below and hope that the additional explanations and results help clarify our approach and its practical implications.


Weakness (1): The practical relevance of the theoretical guarantees remains unclear. As acknowledged by the authors, the practical search implementation omits correction factors and uses heuristics outside these guarantees. It would be helpful to explain how informative the convergence assumptions and bounds are for the experimental settings.

Question (2): Beyond the transition-mismatch analysis, could the authors discuss the practical consequences of violating the assumptions required for target preservation and convergence, particularly in settings not covered by the theoretical bounds?

Answer. We thank the reviewer for asking us to clarify what the theoretical assumptions and bounds tell us about experiments using our practical search algorithms. The theory distinguishes two effects: errors in intermediate-value evaluation can slow convergence without changing the target distribution under the stated assumptions, while omitting the corrections can change the distribution itself. In CS-VGB-G, we retain the sampler's stochastic, value-guided denoising and renoising to find high-reward candidates, while omitting the corrections and returning the best clean candidate collected. Although selecting the best collected candidate does not itself change the internal sampling dynamics, omitting the corrections changes the transition probabilities. We therefore examine whether the resulting search retains the practical benefits of CS-VGB's value-guided exploration: finding high-reward candidates by favoring promising states in both directions, while remaining effective when intermediate-value evaluations are inaccurate.

(1) The theory distinguishes slower convergence from a change in the target. Under the stated pairing and terminal-anchoring conditions, the corrected sampler preserves π0⋆(dx)∝r(x)π0(dx)\pi_0^\star(dx)\propto r(x)\pi_0(dx) despite imperfect intermediate verifiers (Theorem 2.1), while Theorem 2.2 describes how verifier error affects the guaranteed convergence rate under additional assumptions. Thus, intermediate errors can reduce convergence speed without changing the corrected target. Omitting the edge correction ρ\rho, which accounts for transition mismatch between reference denoising and renoising, can instead change the stationary target.

For this latter case, Remark 2.4, supported by Theorem C.5 and Corollary C.4, shows that when the transition mismatch is sufficiently small under the stated assumptions, the analyzed (lazified) CS-VGB chain still converges exponentially to a nearby stationary law even without the ρ\rho correction. Running longer therefore brings the chain closer to that law, but does not necessarily remove its difference from the intended target.

(2) Search retains stochastic recurrent exploration to find high-reward candidates. Building on this distinction, CS-VGB-G retains stochastic recurrent exploration with reverse insertion, but omits the corrections and returns the best clean candidate found. Specifically, these are the edge correction ρ\rho and the proposal importance correction κjd(y∣x):=dKjd(⋅∣x)dGjd(⋅∣x)(y)\kappa_j^d(y\mid x):=\frac{dK_j^d(\cdot\mid x)}{dG_j^d(\cdot\mid x)}(y), where KjdK_j^d and GjdG_j^d denote reference and proposal kernels for d∈F,Bd\in\\{\mathrm F,\mathrm B\\}.

This changes the objective from reproducing a prescribed distribution to discovering good candidates within a finite budget. Since even an exact terminal reward is useful only if such candidates are found, we seek to retain the ability to explore alternative continuations when intermediate evaluations are inaccurate. In this case, the sampling guarantee does not automatically carry over, so we examine whether omitting the corrections preserves useful search performance.

AuthorsOpenReview ↗

(2/4) Response to Official Review by Reviewer iNad

(3) Correction-omitted search remains effective in the analytic benchmark. Following a similar setup to the 2D analytic experiment in Section 3.1, we use a high-dimensional, 24-component Gaussian-mixture benchmark with known reference and target densities pp and qq (Appendix F.6.3). This allows us to evaluate the terminal reward r(x)=q(x)/p(x)r(x)=q(x)/p(x) and both correction factors exactly while introducing errors into intermediate-value evaluations. We then independently switch ρ\rho and κ\kappa on or off, keeping reward-gradient guidance and stochastic candidate selection enabled in all four recurrent variants.

As shown in Table R2(a) (Table F.9 in the revised paper), omitting both corrections retains effective high-reward search: at the larger compute budget, this variant achieves higher mean best reward than both the fully corrected sampler and the forward-only baseline across dimensions from 16 to 16,384. This highlights the distinction between preserving a target distribution and finding high-reward candidates within a finite budget.

Table R2(a). Best terminal reward with and without corrections (λ=3\lambda=3). Forward-only disables backward moves in CS-VGB and forces forward denoising. CS-VGB and forward-only: mean per-trajectory best reward over six verifier-error functions × 64 trajectories. IID: mean of six best-of-64 maxima.

DimensionBoth correctionsOmit ρ\rhoOmit κ\kappaOmit bothForward-onlyIID (best of 64)
1611.0810.6512.5212.068.6312.48
6432.7733.6840.7341.9318.9631.42
25618.3621.4122.4325.8315.5119.59
1,02422.2023.0730.3731.0321.6825.58
4,09624.7926.7132.3537.7118.5124.35
8,19225.7628.5833.3035.6727.0330.67
16,38430.4731.6440.1343.0526.0024.72

The two corrections have different effects on search reward: omitting ρ\rho produces relatively small changes, whereas omitting κ\kappa yields larger gains. This is consistent with the role of κ\kappa, which corrects the change from the reference proposal to the gradient-guided proposal so that guidance can be used while preserving the prescribed reward-tilted target. Preserving that distribution, however, need not maximize the best reward found within a finite budget. These results support our choice to omit κ\kappa in practical search, where the goal is to find high-reward outputs, while retaining guidance in both directions.

Higher reward could nevertheless come at the cost of reduced sampling diversity. To examine this possibility, Table R2(b) (Table F.10 in the revised paper) compares the number of mixture components represented at matched sample counts. Omitting both corrections increases mean coverage at d=16d=16 and d=4,096d=4{,}096, while modestly reducing it at the other tested dimensions. Overall, the similar component counts provide indirect evidence that the approximate sampling dynamics retain much of their sampling diversity even when both corrections are omitted. However, we remark that similar component coverage does not imply that samples follow the target component probabilities.

Table R2(b). Component coverage under correction omission. Same setup as Table R2(a). Each sample is assigned to the target component with the highest posterior membership probability. Coverage counts distinct assigned components out of 24, using samples from 64 independent trajectories per verifier-error function, with sample counts matched across methods. Target IID uses exact target draws. Results are averaged over six verifier-error functions.

DimensionBoth correctionsOmit ρ\rhoOmit κ\kappaOmit bothForward-onlyTarget IID
1617.8316.5017.3318.3315.0018.50
6412.8314.3311.3312.6713.1712.17
25614.1713.5013.8313.0013.0014.00
1,02414.1714.0013.3313.6715.1714.17
4,09614.5014.8316.0015.6716.1715.83
8,19215.0014.3314.1713.3314.3314.00
16,38415.6716.0013.1714.8315.1715.17

We next test whether the practical image-generation algorithm also maintains its performance when intermediate-value evaluations are perturbed.

AuthorsOpenReview ↗

(3/4) Response to Official Review by Reviewer iNad

(4) CS-VGB-G better retains reward and held-out alignment under intermediate-value errors on ImageNet. To test sensitivity to these errors, we add a similar smooth error function ew(xt,t)e_w(x_t,t) (Eq. F.8 of the revised paper) to both methods' intermediate log-values and its gradient to their reward guidance. Increasing its frequency ww produces faster oscillations and stronger gradient perturbations while keeping the error amplitude bounded. The error vanishes at the clean and initial noise levels, while terminal rewards and best-candidate ranking remain unperturbed.

As shown in Table R2(c), CS-VGB-G better retains both optimized reward and held-out alignment as the perturbations vary more rapidly. At w=128w=128, its VQAScore decreases by 1.53%, compared with 23.26% for forward-only; at w=192w=192, its HPSv2 decreases by 9.01%, compared with 16.16%. These results provide empirical evidence of error tolerance in practical search, without extending the exact sampling guarantee to this implementation. The setup and results are provided in Appendix F.6.3, Table F.11, and Figure 7 of the revised paper. Here, forward-only disables backward moves in CS-VGB-G and stops upon reaching the clean level, while recurrent search continues within the same time cap.

Table R2(c). ImageNet sensitivity to intermediate-verifier error frequency. Each entry reports CS-VGB-G (F+B) / forward-only over 20 conditions × 3 seeds. Percentage decreases are relative to each method's zero-added-error control; ww controls error frequency.

Error frequency wwHPSv2 ↑HPS decrease (%) ↓VQAScore decrease (%) ↓
144.50 / 38.870.01 / −0.110.09 / 1.49
844.57 / 38.25−0.14 / 1.471.62 / 3.85
3243.44 / 36.542.39 / 5.881.47 / 8.09
12841.48 / 33.836.80 / 12.871.53 / 23.26
19240.50 / 32.559.01 / 16.168.38 / 17.10
Reviewer_f3aMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the authors first formulate the standard inference time diffusion reward alignment with graph theory and identify the challenge of using imperfect intermediate verifiers can distort sampling. To address this problem, they propose ContinuousState Verifier-Guided Backtracking (CS-VGB), which describes the reward alignment process as graph transitions and establish distributional and convergence guarantees to the correct distribution. They demonstrate the effectiveness of their method with diffusion models and posterior flow maps.

Critical strengths
  1. I think the authors provide a fresh new perspective to new diffusion posterior sampling and reward alignment. I like how the authors describes different noise levels as nodes and denoising/renoising as edges. This is an interesting perspective that I think is worth investigating further in the future.
  2. Because of this ew framework, the authors are able to provide new theoretical results such as distributional preservation and convergence guarantees.
Critical weaknesses
  1. This paper significantly lacks related work review and comparison. In particular, it's missing the literature on diffusion posterior sampling and flow map initial noise optimization:

Diffusion posteiror sampling:

[1] Chung, et al. "Diffusion Posterior Sampling for General Noisy Inverse Problems". ICLR 2023.

[2] He et al. "Manifold Preserving Guided Diffusion". ICLR 2024.

[3] Yu et al. "Freedom: Training-free energy-guided conditional diffusion model". ICCV 2023.

Flow map/distilled model initial noise optimization:

[4] Eyring et al. "Reno: Enhancing one-step text-to-image models through reward-based noise optimization". NeurIPS 2024.

[5] Yoon et al. "$\Psi $-Sampler: Initial Particle Sampling for SMC-Based Inference-Time Reward Alignment in Score Models". NeurIPS 2026.

[6] Min et al. "Origen: Zero-shot 3d orientation grounding in text-to-image generation". NeurIPS 2026.

  1. The benefit of the proposed method mainly shows at high NFE regime, which renders the method impractical.

  2. The abstract is really badly written in the sense that it's very difficult to know this paper is even about diffusion from the abstract.

Questions for authors

It would be great if the authors can answer my concerns above.

6 comments
AuthorsOpenReview ↗

(1/3) Response to Official Review by Reviewer f3aM

We sincerely thank the reviewer for the thoughtful and constructive feedback. Below, we address each point and hope that our explanations and revisions help clarify the scope and contributions of our work.


Weakness (1): This paper significantly lacks related work review and comparison. In particular, it's missing the literature on diffusion posterior sampling and flow map initial noise optimization

Answer. We thank the reviewer for pointing us to these relevant works. We first would like to clarify the focus of our original related-work discussion: our central question is how to preserve a prescribed reward-tilted target during bidirectional exploration with imperfect intermediate verifiers. This led us to focus our related-work discussion in Section 1 and Appendix E on intermediate guidance and recurrent denoising–renoising methods.

To place CS-VGB in this broader context, we discuss below how the suggested works relate to our formulation. We also present additional comparisons with ReNO [1] and Ψ-Sampler [2], which assess the competitiveness of our practical search against alternative ways of using inference computation.

On diffusion posterior sampling and the guidance baselines. We would like to clarify that MFM-G and PDM-G use intermediate-value gradient guidance within a one-step posterior-map framework, which we believe shares the core guidance principle of diffusion posterior sampling, despite differences in value estimation and implementation. Our comparisons therefore examine the additional benefit of recurrent exploration over this form of forward-only generation with intermediate gradient guidance.

To make this connection explicit, DPS [3] addresses inverse problems that recover x0x_0 from a measurement y=A(x0)+ϵy=\mathcal A(x_0)+\epsilon, where A\mathcal A is a known measurement operator and ϵ\epsilon is observation noise. Its diffusion prior corresponds to the reference model's clean distribution π0\pi_0 in our notation, and taking the reward to be the observation likelihood gives

\qquad \pi_0^\star(x_0)\propto r(x_0)\pi_0(x_0), \quad r(x_0)=p(y\mid x_0).$$ Our image-preference and text–image alignment tasks instead specify rewards without an observation model, so applying DPS requires adapting its likelihood guidance to these rewards. While MPGD [4] and FreeDoM [5] address conditional and energy-based objectives that fit the broader reward-tilting formulation above, they share the same principle of using gradient guidance to steer intermediate states toward clean outputs that better satisfy the desired condition. For example, in DPS, the intermediate value becomes $V_t(x_t)=\mathbb E[r(X_0)\mid X_t=x_t]=p(y\mid x_t)$, which it approximates through a denoised estimate before taking its log-gradient. These methods formulate their guidance updates for diffusion models, so extending them to the one-step posterior maps considered here is nontrivial. In particular, **Meta Flow Maps and Posterior Diamond Maps provide one-step clean posterior samples**, requiring adaptations to both value estimation and the guided transitions. Our MFM-G and PDM-G baselines draw clean completions from an intermediate state, estimate expected terminal reward by Monte Carlo, and differentiate that estimate to guide generation, thereby adapting the core gradient-guidance principle of diffusion posterior sampling to the one-step posterior-map setting. Building on this guidance principle, our focus is on how recurrent sampling can improve reward tilting across various **stochastic generators in continuous spaces**. To keep that question clear, we originally centered the related-work discussion on intermediate guidance and recurrent exploration, rather than providing a broad survey of inference-time scaling algorithms. However, we agree with the reviewer that explaining the broader connections would better situate our contribution and clarify how these guidance principles extend to the generators we study. We have therefore expanded the related-work discussion in **Appendix E**, and we appreciate the reviewer bringing these connections to our attention.
AuthorsOpenReview ↗

(2/3) Response to Official Review by Reviewer f3aM

On initial-noise optimization and sampling. These methods use additional computation to find more promising starting points for generation. ReNO [1] updates the initial noise through reward gradients, while Ψ-Sampler [2] constructs reward-aware initial particles and then guides them through SMC; ORIGEN [6] explores the initial latent space using reward-guided Langevin sampling.

In contrast, CS-VGB allocates additional computation to revisiting intermediate states and exploring alternative continuations, rather than optimizing the initial noise. Since both strategies aim to obtain higher-reward outputs, comparing them allows us to assess whether our practical search offers a competitive quality–compute trade-off against strong alternatives for inference-time reward optimization and sampling. Following the reviewer's suggestion, we therefore compare ReNO, an MFM adaptation of the full Ψ-Sampler pipeline, and ORIGEN with CS-VGB in our ImageNet prompt-guided generation setting.

Table R1(a). Comparison with initial-noise optimization and sampling methods on ImageNet (100 conditions × 3 seeds), grouped by similar generation time. Wall-clock times are measured on a single NVIDIA H100.

Method / NFE budgetSeconds/image ↓HPSv2 ↑VQAScore ↑CLIP-IQA ↑
CS-VGB-G / 102413.3341.56 ± 0.070.745 ± 0.0080.950 ± 0.004
ReNO / 102412.7947.33 ± 0.190.622 ± 0.0140.936 ± 0.003
Ψ-Sampler / 204812.5829.84 ± 0.030.550 ± 0.0220.923 ± 0.004
ORIGEN / 102412.6327.28 ± 0.110.515 ± 0.0190.916 ± 0.004
CS-VGB-G / 819253.7646.30 ± 0.070.790 ± 0.0180.970 ± 0.001
ReNO / 409650.9555.24 ± 0.200.650 ± 0.0290.940 ± 0.002
Ψ-Sampler / 819258.8932.25 ± 0.070.643 ± 0.0080.941 ± 0.002
ORIGEN / 409650.9627.99 ± 0.030.568 ± 0.0180.928 ± 0.004

Across the two evaluated mid-NFE settings, at approximately 13 and 51–59 seconds per image, CS-VGB-G shows a larger increase in mean HPS than Ψ-Sampler and ORIGEN while maintaining higher held-out VQAScore and CLIP-IQA in both groups. Its HPS increases by 4.74, compared with 2.41 for Ψ-Sampler and 0.71 for ORIGEN, indicating stronger reward scaling over this measured range.

The comparison with ReNO reveals a different trade-off. ReNO achieves higher HPS, but CS-VGB-G improves both the optimized reward and held-out alignment: HPS rises from 41.56 to 46.30, alongside a VQAScore increase from 0.745 to 0.790. ReNO's HPS rises more sharply, from 47.33 to 55.24, while VQAScore improves only from 0.622 to 0.650 and remains substantially lower at both settings.

This distinction is relevant because ReNO [1] performs reward optimization through initial-noise updates, rather than sampling from a prescribed reward-tilted target. Its paper explicitly discusses the risk of reward hacking during noise optimization and introduces regularization and limited optimization steps to mitigate it. However, even with such regularization, the substantial HPS gains in our experiments did not translate into comparable improvements in held-out alignment and image quality, a pattern consistent with possible reward overoptimization.

The scaling curves in Figure F.5 of the revised paper show that CS-VGB-G improves reward alongside held-out quality over a broader range of computation. The qualitative comparisons in Figure F.7 also illustrate why optimized reward alone is insufficient: despite high HPS, ReNO partially crops the hen in the farmyard example and does not clearly depict the requested shark–buoy interaction in the hammerhead example.


Weakness (2): The benefit of the proposed method mainly shows at high NFE regime, which renders the method impractical.

Answer. Thank you for raising this concern. We agree that limited-budget performance matters and would like to clarify the intended benefit of CS-VGB: making additional computation productive beyond the observed plateau of forward-only guidance, particularly in the high-NFE regime. We believe this is practically important for demanding tasks such as multi-robot planning, where a plateau in success rate can leave difficult instances unsolved even when additional inference computation is available. A method that can turn that additional budget into better solutions therefore remains useful when solution quality takes priority over minimal latency. In this regard, our results demonstrate continued scaling alongside higher reward at fewer NFEs in the comparisons below.

AuthorsOpenReview ↗

(3/3) Response to Official Review by Reviewer f3aM

On why the benefit grows with budget. Once forward-only guidance begins to saturate, spending more computation on the same exploration strategy can yield diminishing returns. CS-VGB provides another way to use that budget by returning to earlier checkpoints, reconsidering decisions made from imperfect intermediate evaluations, and exploring alternative continuations from promising states. This is the additional scaling opportunity we sought, supported by the restart-versus-recurrence ablation in Table 3 of the revised paper. On ImageNet, CS-VGB-G continues improving beyond MFM-G's observed plateau and reaches HPSv2 52.01 versus 45.66 at approximately 2162^{16} NFE, with held-out VQAScore 0.778 versus 0.745 (Figure 5(a)).

This also explains why we did not expect CS-VGB to consistently outperform forward-only guidance at low NFE. At small budgets, forward guidance may still offer substantial improvements, so using the available computation for denoising may be more effective than spending some of it on renoising.

On practical efficiency. Beyond improving at high budgets, CS-VGB-G achieves higher reward with approximately 2× and 5.8× fewer NFEs than PDM-G and MFM-G, respectively, in Figure 5. Since fewer NFEs do not necessarily mean faster generation, we supplement these results with wall-clock time measurements and held-out metrics. In the ImageNet comparison below, generation takes 35.43 seconds versus 203.92 seconds for MFM-G, corresponding to approximately 83% less generation time.

Table R1(b). Reward, computation, and held-out quality in comparisons with forward-only guidance on CelebA and ImageNet. Generation time is measured in seconds per image on a single NVIDIA H100.

TaskMethodReward (GM / HPSv2) ↑Actual NFE ↓Seconds/image ↓VQAScore ↑CLIP-IQA ↑
CelebACS-VGB-G0.948514.629.770.3950.753
CelebAPDM-G0.9291,02311.360.3840.732
ImageNetCS-VGB-G45.464,10435.430.7870.966
ImageNetMFM-G44.9423,959203.920.7410.972

These comparisons support CS-VGB as an efficient option when stronger reward tilting is needed, with higher mean VQA scores in both tasks. For more detailed results and discussion, we kindly refer the reviewer to our response to Reviewer iNad, Weakness (2) / Question (1), and to Appendix F.6.4, Figure F.6, and Table F.12 of the revised paper.


Weakness (3): The abstract is really badly written in the sense that it's very difficult to know this paper is even about diffusion from the abstract.

Answer. Thank you for pointing out a clarity issue we had overlooked. We would like to clarify that CS-VGB is a sampling framework for continuous generative models, with diffusion as one instantiation, which is why we deliberately avoided presenting it as diffusion-specific. More intuitively, it is a sampling harness on top of a stochastic generator in continous domain, such as a diffusion model or a one-step posterior map: the generator proposes states, while CS-VGB uses verifier feedback to organize exploration and backtracking.

However, emphasizing this generality without explaining the relationship clearly enough made the abstract less self-contained. We have therefore revised its opening to name the model families studied and explain how CS-VGB operates on them, before introducing target preservation under imperfect intermediate verifiers. We believe this makes the intended scope easier to understand without narrowing it to diffusion alone. We thank the reviewer again for highlighting the need to make this scope clearer in the abstract.

References

[1] Eyring et al., ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimization, NeurIPS 2024.

[2] Yoon et al., Ψ-Sampler: Initial Particle Sampling for SMC-Based Inference-Time Reward Alignment in Score Models, NeurIPS 2025.

[3] Chung et al., Diffusion Posterior Sampling for General Noisy Inverse Problems, ICLR 2023.

[4] He et al., Manifold Preserving Guided Diffusion, ICLR 2024.

[5] Yu et al., FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model, ICCV 2023.

[6] Min et al., ORIGEN: Zero-Shot 3D Orientation Grounding in Text-to-Image Generation, NeurIPS 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.