acceptodds
Under review as a conference paper at ICLR 2027

Beyond More Samples: Test-Time Scaling of Training-Free Guidance in Generative Models

Abstract

Recent work on test-time scaling has demonstrated that additional computation during inference can improve the quality of generative model outputs. In parallel, training-free guidance has enabled pretrained diffusion and flow-matching models to pursue specific target objectives during sampling without retraining. These two inference-time approaches can complement one another in improving a pretrained sampler's outputs for the target objective. We study their intersection through three recurring scaling strategies applied to guided sampling: (1) averaging local draws, (2) propagating parallel sampling trajectories, and (3) selecting among final samples. Yet alongside inconsistencies reported in past research, we find that these strategies often fail to reliably improve final outputs and can even severely degrade them. The strategy that improves outputs most also changes, even between samplers that share the same checkpoint and objective. In this work, we develop a common characterization of how additional compute acts in each scaling strategy: what increasing the sampling budget actually refines, why that refinement can still fail to improve final outputs, and where to intervene when further scaling stalls. Building on this characterization, we also show that conventional summaries used to monitor these strategies, such as weight concentration or rank agreement, can omit information needed to predict the effects of scaling.

Primary area:
generative models
Keywords:
training-free guidance, test-time scaling, diffusion models, flow matching

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_M9DSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes test-time scaling of training-free guidance through averaging local draws, particle propagation, and output selection. Using established statistical identities, it characterizes what additional samples refine and which components remain unchanged. It also constructs matched counterexamples showing limitations of population summaries. Budget sweeps across 18 released checkpoints and 29 settings, controlled interventions, summary-based predictions, and small-scale gain evaluations examine how these limitations relate to final-output quality and decisions about additional sampling.

Critical strengths

S1. The matched counterexamples establish specific limitations of population summaries.

Proposition 3.4 constructs identical joint distributions of draw weights with opposite effects on gradient MSE relative to the plug-in. Proposition 3.6 constructs identical selector and utility marginals, with the same Kendall's τ, but opposite Best-of-N expected-utility gains. These results establish that the considered summaries cannot determine the corresponding outcomes.

S2. The empirical comparisons are useful for configuring and evaluating existing methods.

Figure 3 shows how output quality changes with K, P, and N, while Table 5 reports guidance strengths and guidance-active windows selected on validation data for the topology-optimization benchmark. Table 13 compares the averaging methods against both the released plug-in and a separately calibrated plug-in. Recalibrating the plug-in improves mean PSNR by 3.06 and 5.85 dB in the two conditions, matching or exceeding the gains of ps+. This highlights the importance of calibrating the baseline before attributing improvements to additional averaging.

Critical weaknesses

W1. The characterization largely formalizes expected behavior and provides limited guidance for choosing among sampling components.

The common characterization is built on established statistical identities, as the paper itself states: the bias–variance decomposition (Lemma 3.1), the Feynman–Kac output law (Lemma 3.2), and the Beta(1, N) order-statistic weighting (Lemma 3.3). These lemmas formalize largely expected behavior: more draws reduce variance but not bias, more particles approximate a fixed target law, and more candidates push selection further into the selector's tail. The theoretical novelty therefore rests mainly on the matched counterexamples. The contribution of the characterization then depends on whether it can guide practice, which the abstract describes as identifying "where to intervene when further scaling stalls." The current evidence for this is limited:

  • For averaging, the draw-replacement, update-length, and sampling-stage controls narrow the explanations for the observed losses (Tables 10 and 12). However, the link from measured gradient properties to the final-utility ordering of projected draws, transition draws, and differentiable continuations remains unestablished. The paper notes that the logged disagreement with the plug-in measures bias only when the plug-in equals the exact guidance, and does not determine which direction improves utility (Appendix A.1.1).
  • For particles, weight permutation and replay controls show that weight assignment affects measurement residual beyond the availability of alternative trajectories. The ODE/SDE comparison also identifies the role of transitions in separating resampled copies. However, weighting improves final utility beyond greedy search in most inverse-imaging cells but provides no additional benefit on the flow-matching stack. These results explain mechanisms in the tested settings, while evidence for using the identified relationships to choose a weighting rule under a given sampler and budget remains limited (Tables 7 and 15).

W2. Evidence for using population measurements to predict gains in new settings is limited.

  • The summary-based predictions in Table 1 do not consistently identify gains. The held-out comparison does not establish a predictive advantage for tail statistics over global rank correlation, and the clone-calibrated separation criterion is evaluated on applicable checkpoints that all show no gain. The paper acknowledges both limitations. These results limit the predictive use of the evaluated measurements.
  • The terminal pilot provides positive evidence on four held-out implementations, detecting 36% of qualifying gains compared with 5.5% for the relative-IQR comparator. The larger particle pilot also improves on the better fixed policy in the evaluated cohort. However, the primary particle pilot does not establish an advantage even before pilot cost, and the larger pilot requires more samples. These results support small-scale gain evaluation, while evidence for transferring the conclusions across budgets and for a benefit after accounting for pilot cost remains limited.
Questions for authors

Q1. Criterion for choosing sampling components

The controlled interventions identify effects of draw construction, weight assignment, and particle separation. Do the reported measurements support a criterion for choosing among alternative draw constructions or weighting rules under a given sampler and budget? If so, which comparisons connect that criterion to improvements in final utility?

5 comments

We thank the reviewer for the detailed review and very valuable feedback. We addressed related weaknesses and questions together.

[W1, Q1] A criterion for choosing draw constructions or weighting rules under a given sampler and budget, and its link to final utility.

[A] Yes: we use small-budget comparisons of final outputs, then scale the better choice. The analysis explains why the considered summaries alone cannot determine the better choice. In the tests below, on held-out or new instances, the small-budget winner was the large-budget winner in 20 of 21 draw-construction settings and, for the weighting rule, in 4 of 4 settings on the measurement residual and 3 of 3 inpainting settings on PSNR. A setting is one task on one model; differences are in units of the plug-in baseline's interquartile range (IQR). Each prediction below was recorded in advance; the exact targets are in Appendix E.6.5.

1. Draw construction (averaging). Compare constructions at K=2K=2 before raising KK, then compare the better one with the plug-in. Why: increasing KK cannot change the construction's mean guidance (Lemma 3.1), and the gradient-error summaries considered in Proposition A.5 do not determine final utility, so we compare final outputs directly. On 24 held-out instances in each of 21 settings of three production models:

PredictionSettingsEffect
Changing the construction at K=2K=2 beats raising KK to 3216 of 21median +0.47 vs +0.10 IQR over projected K=2K=2
The K=2K=2 winner also wins at K=16K=1620 of 21same ranking at 8 times the budget

Here LGD-scale draws won every K=2K=2 comparison; the comparison is what finds the winner when it is not known in advance. The better construction overtakes the tuned plug-in in 5 of 21 settings, so the last step is a comparison with the plug-in.

2. Resample or select (particles). Compare according to the sampler dynamics:

  • Deterministic sampler: copies share the same continuation. Compare particles with independent selection at the same budget and final scoring rule.
  • Stochastic sampler: copies can diverge. Retune guidance strength, compare weighting procedures at small PP (criterion 3), then check their final utility against independent selection.

On 24 new prompts in each of nine reward-guided settings, paired runs share prompt, seed and starting noise:

Budget of 4, deterministic samplerBest-of-44 particles
Median gain over the plug-in (IQR)+0.40+0.27
Settings with a gain over the plug-in9 of 99 of 9
Settings ahead of the other7 of 92 of 9
Paired runs ahead of the other, of 648405 (62.5%)243

When particles return their best final particle instead of their highest-weight one, selection wins 51.7% of pairs, showing that the terminal return rule affects the advantage in the table. On earlier instances, selection was ahead in at least 20 of the 21 settings at every budget from 2 to 32.

3. Weighting rule (particles, stochastic sampler). Compare the rules on the measurement residual at P=4P=4, then check final quality on validation cases. On 64 new FFHQ images, median PSNR gain of the weighted rule over greedy search, with the number of the 192 paired runs it won:

TaskP=4P=4P=64P=64
Inpainting S+1.5 dB (192 of 192)+3.5 dB (192 of 192)
Inpainting M+0.8 dB (185 of 192)+1.9 dB (192 of 192)
Inpainting D+0.7 dB (182 of 192)+1.8 dB (192 of 192)
Phase retrieval+0.2 dB (99 of 192)−0.4 dB (89 of 192)
Prediction, recorded in advanceNeededResult
On the residual, the P=4P=4 winner is the P=64P=64 winner4 of 44 of 4
Four pilot images find that winner85% of draws97%
On PSNR, the P=4P=4 winner is the P=64P=64 winner, inpainting3 of 33 of 3

Phase retrieval is the exception: weighting lowers the residual but PSNR does not clearly improve, which is why the criterion compares final quality and not the residual. The inpainting PSNR ranking transfers from P=4P=4 to P=64P=64, supporting the small-budget comparison in those settings.

[W1-1] Averaging: the link from measured gradient properties to final utility is not established.

[A] We agree. Proposition A.5 gives two cases with identical gradient error at every step and every KK, yet averaging improves final utility in one and worsens it in the other. This motivates our controlled interventions in Section 4.2: rather than infer final utility from gradient-error summaries, we vary draw construction and directly evaluate its effect on final outputs. Section 4.4 then builds on these findings to provide practical criteria for choosing constructions.


[W1-2] Particles: the ODE/SDE comparison and choosing a weighting rule.

[A] We thank the reviewer for pointing out this concern. On re-checking the comparison, we found that our stochastic sampler added noise to the deterministic step without the compensating drift term, leaving residual noise in its outputs and weakening its own baseline (by 5 dB on SiT super-resolution). We corrected the sampler and repeated the experiments; the effect of sampler dynamics persists under the corrected comparison.

The corrected experiments use a standard stochastic sampler (Euler-ancestral). SiT retains the original guidance strength; production models tune it separately for each sampler using the same rule.

Median PSNR gain of particles over the plug-in of the same sampler on SiT, with the number of the 192 paired runs (64 images, 3 seeds) in which particles were better:

SiT, particles minus plug-inSuper-resolutionInpainting
Deterministic (Euler) sampler0.0 dB (97 of 192)0.2 dB (103 of 192)
Submission's stochastic step (noise without its drift correction)1.1 dB (182 of 192)0.6 dB (134 of 192)
Stochastic (Euler-ancestral) sampler1.2 dB (186 of 192)0.7 dB (156 of 192)
  • SiT: corrected particles also end above the deterministic baseline in both restoration settings, in 92% and 86% of 192 paired runs (by 0.7 and 2.1 dB at the median). In the class-reward setting, where the baseline's reward is already near its maximum, they stay below it.
  • Three production models, nine restoration settings (inpainting, box inpainting, colorization), particles improve on their own plug-in in 8 of 9 settings (median +0.13 IQR; deterministic particles: 5 of 9) and on the deterministic plug-in in 7. The gains are small: 1 of 9 exceeds the rerun band of 0.40 IQR, where we predicted 3.

Particles exceed best-of-4 in 4 of 9 settings under the same sampler and in 3 of 9 against deterministic best-of-4. The corrected experiments support the proposed sampler mechanism, with clear utility gains in the two SiT restoration settings and smaller gains on the production models. These results support adapting the particle procedure to the sampler and comparing its gain with independent selection (Sections 4.2 and 4.4). On the weighting rule: on the flow-matching stack the weighted rule and greedy search stay level under the corrected sampler, as in the submission, while on the three FFHQ inpainting tasks, the weighted rule wins by 0.7 to 3.5 dB. The rule is therefore chosen per stack at P=4P=4.

[W2-1] Population measurements do not predict gains in new settings.

[A] We agree, and this limitation is a central finding of our paper. Our counterexamples establish why common population summaries cannot reliably determine scaling gains on their own, and our empirical results demonstrate their limited predictive usefulness. Appendix E.7.1 now reports that always predicting the more common outcome achieves 96%, 80%, and 52% accuracy for averaging, particles, and selection, compared with 79%, 60%, and 74% for our summary-based rules.

This also motivates Section 4.4, where we move beyond population-summary diagnostics to practical decisions based on direct comparisons of final utility and computational cost, rather than claiming a universally reliable predictor from these summaries.


[W2-2] Pilots: transfer across budgets, and benefit after cost.

[A] We agree that the pilot results do not yet establish a reliable, cost-effective deployment rule. We therefore treat these pilots as exploratory diagnostics rather than our main practical recommendations. The terminal pilot's endorsements at N=4N=4 and N=6N=6 persist at larger budgets, but the pilot itself does not repay its cost. The original particle pilot also fails to establish an advantage over fixed policies, while the larger pilot requires additional samples. We clarify these limitations and the cost accounting in Appendix E.7.2.

Instead, the new Section 4.4 provides practical recommendations based on final utility and sampling cost:

  • When the selector measures utility directly: continue only while another candidate's expected utility improvement exceeds its cost, following optimal stopping studied for language-model generation [1]. We fit the rule on earlier instances, freeze it, and evaluate it on new instances.
  • When the selector is a proxy: compare candidate counts on validation instances, measuring the actual final utility of the proxy-selected output at each NN. Choose the budget maximizing mean utility minus candidate cost, then keep it fixed for new instances.
Stopping rule, 24 new prompts in each of nine reward-guided settingsResult
Candidates saved at matched mean utility (median)30%
Settings with higher mean utility at matched average count9 of 9 (median +0.18 IQR)

For proxy selectors, validation-selected fixed budgets achieved net utility within 0.03 IQR of the hindsight-best budget in 6 of 6 settings and matched or exceeded eight candidates in 5, at a cost of 0.02 IQR per candidate.

The stopping test allows at most four candidates and excludes fitting cost; both rules count candidates rather than wall time. For particles, our recommendations instead focus on comparing sampler dynamics, weighting procedures, and final return rules directly, as described in criteria 2 and 3.


References:

[1] Kalayci, et al. “Optimal Stopping vs Best-of-N for Inference Time Optimization”. arXiv 2025.

Reviewer_sKTqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies test-time compute scaling for training-free guidance in diffusion and flow models, focusing on three strategies: averaging local gradient estimates, propagating multiple particles, and selecting the best of multiple generated outputs. The authors develop a unified characterization of what increasing the sample budget improves in each strategy and why this may not translate to better final outputs. They also construct theoretical counterexamples showing that common diagnostics such as effective sample size, particle separation, and rank correlation, can be insufficient to predict scaling benefits. They extensively validate their claims across various checkpoints and tasks.

Critical strengths
  • Understanding how to apply test time compute is an important practical question, particularly given the growing interest in inference-time scaling.
  • Experiments across 18 checkpoints and 29 settings provide useful evidence that scaling benefits are inconsistent across models and tasks.
  • Experiments modifying draw construction, particle dynamics, and selector fidelity help identify mechanisms underlying scaling failures.
  • Counterexamples establish limitations of common diagnostics, including effective sample size and rank correlation, for predicting scaling benefits.
Critical weaknesses
  • The main results rely on standard bias-variance decomposition, SMC convergence, and order statistics. The new counterexamples demonstrate that summary statistics can be misleading, but do not yield new performance guarantees or predictions about when scaling will help.
  • Relevant approaches, including parallel tempering (e.g., CREPE), FK Steering, and few-step reward-tilting methods, are omitted or insufficiently discussed, making it difficult to assess the novelty and scope of the contributions.
  • The paper does not establish a reliable criterion for deciding when to increase sampling compute. For example, the averaging diagnostic correctly predicts no gain in 22/23 cases, but simply always predicting no gain would be correct in 27/28 cases.
Questions for authors
  • What does the proposed unified characterization explain about scaling behavior that existing analyses of the individual methods do not? Are there empirical findings that the framework predicted in advance, rather than explained retrospectively?
  • How sensitive are the main conclusions to guidance-strength calibration and computational cost?
  • How does the framework relate to alternative test-time scaling approaches such as CREPE (parallel tempering), FK Steering, and few-step reward-tilting methods?
6 comments

We thank the reviewer for the careful review and related-work suggestions. We greatly appreciate their recognition of the importance of the problem and our experiments. We addressed related weaknesses and questions together.

[W1-1, Q1-1] Main results rely on standard tools; what does the unified characterization explain that analyses of the individual methods do not?

[A] We agree that the lemmas build on established theory. The central message of our work is that effective test-time scaling depends not only on the amount of inference compute, but also on how that compute is used. Our unified characterization explains why additional sampling can fail to improve final utility and identifies which components to reconsider when scaling stalls. Our theoretical contributions are as follows:

  • Unified characterization and where to intervene. We provide a common framework that distinguishes what additional samples refine from which sampling choices remain unchanged: draw construction fixes the mean guidance (Lemma 3.1), sampler transitions and weights fix the particle target law (Lemma 3.2), and selection gains depend on the selector–utility relationship in the increasingly extreme tail reached at larger NN (Lemma 3.3). This separates increasing sample count from changing the components that govern its usefulness, and identifies draw construction for averaging, sampler dynamics or weighting for particles, and selector fidelity for selection as components to reconsider when scaling stalls.
  • New counterexamples (Propositions 3.4-3.6 and A.5). We establish that the common summaries are insufficient on their own in principle. Identical joint draw-weight distributions can accompany lower or higher gradient MSE than the plug-in (Proposition 3.4); identical asymptotic normalized ESS can accompany exact likelihood reweighting that moves the output mean toward or away from the same posterior mean (Proposition 3.5); and identical selector and utility marginal distributions, with the same Kendall’s τ\tau, can accompany opposite expected Best-of-NN utility gains (Proposition 3.6). Proposition A.5 further shows that even identical per-step gradient MSE can yield opposite final-utility gains.

Section 4 then tests the practical implications of these findings through controlled interventions and evaluates recommendations derived from them.

[W1-2, W3] No new guarantees or predictions; no reliable criterion for deciding when to increase sampling compute.

[A] Our theory shows why the population summaries considered here cannot alone supply such a criterion: the counterexamples above give identical summaries with opposite gains. The reviewer's base-rate comparison confirms it (Appendix E.7.1): always guessing the more common outcome is right in 96%, 80% and 52% of the averaging, particle and selection settings, against 79%, 60% and 74% for our summary rules.

To translate these theoretical findings into practical guidance, we newly added Section 4.4, with concrete recommendations for choosing sampling components and allocating additional compute. Each recommendation was fixed on earlier instances and evaluated on held-out or new ones.

  • Averaging: compare draw constructions at K=2K=2 before adding draws, then compare the better one with the plug-in. Switching at K=2K=2 beat raising KK to 32 in 16 of 21 settings (+0.47 against +0.10 IQR over projected K=2K=2), and the K=2K=2 winner also won at K=16K=16 in 20 of 21.

  • Particles: compare propagation with independent selection.

    • With a deterministic sampler: compare at the same budget and final scoring rule. Best-of-4 beats highest-weight-return particles in 7 of 9 reward-guided settings; against particles returning their best final particle, it wins 51.7% of pairs.
    • With a stochastic sampler: retune guidance strength, compare weighting procedures at small PP using final utility, and check against best-of-NN. Particles improve on their own plug-in in 8 of 9 restoration settings.
  • Selection: choose the candidate budget from final utility and cost.

    • When the selector measures utility directly: continue only while another candidate's expected improvement exceeds its cost. A rule fitted earlier and kept fixed (optimal stopping, as in [1] for language models) used a median 30% fewer candidates at matched mean utility on 24 new prompts per setting, and improved mean utility at matched average count in 9 of 9 settings.
    • When the selector is a proxy: compare candidate counts on validation instances. At each NN, let the proxy select an output and measure its actual utility. Choose the NN maximizing mean utility minus the cost of generating NN candidates, and keep it fixed for new instances. On 24 new images per setting, net utility was within 0.03 IQR of the hindsight-best budget in 6 of 6 settings and matched or exceeded eight candidates in 5, at a cost of 0.02 IQR per candidate.

Both selection rules measure deployment candidate use; fitting cost is excluded, and the stopping rule was tested with at most four candidates. Representative predictions recorded in advance are under Q1-2; 33 of 39 were met.


References:

[1] Kalayci, et al. “Optimal Stopping vs Best-of-N for Inference Time Optimization”. arXiv 2025.

[W2, Q3] Relevant approaches (parallel tempering (e.g., CREPE), FK Steering, and few-step reward-tilting methods) omitted or insufficiently discussed

[A] We thank the reviewer for highlighting some critical related work. Each approach mentioned connects to a component of our analysis, and we have updated the related works section of the paper to make each connection more explicit (Appendix J).

In short, FK Steering and parallel tempering methods like CREPE connect to our particle family through resampling and replica exchange, respectively. Few-step reward-tilting methods illustrate the complementary intervention our analysis motivates: improving how guidance or reward estimates are constructed when increasing sample count alone is ineffective.

  • Parallel tempering (CREPE [2]). CREPE exchanges states between replicas at different diffusion times and generates samples sequentially after burn-in. This connects to our particle perspective through interactions between sampling states, while introducing an exchange-iteration budget. The conceptual connection is our distinction between approximating a chosen target and how that target relates to final utility.
  • FK Steering [3]. FK Steering fits directly within our particle formulation: sampler transitions and weighting potentials determine the law approximated as the particle count increases (Lemma 3.2). Our particle arm uses its difference potential. Its reported gains with stochastic sampling motivate our experiments on when intermediate resampling improves final utility beyond independent selection at the same budget.
  • Few-step reward tilting. These methods, such as flow-map methods (FMTT [4], FMRG [5]), improve guidance by using a better prediction of the final image to evaluate rewards, not only by scaling more samples. This actually illustrates the kind of change our analysis motivates for averaging when adding draws stops helping. In our experiments, changing how the draws are made at K=2K=2 outperformed keeping the original construction and using 16 times as many draws (K=32K=32) in 16 of 21 settings.

References:

[2] He, et al. “CREPE: Controlling Diffusion with Replica Exchange”. ICLR 2026.

[3] Singhal, et al. “A General Framework for Inference-time Scaling and Steering of Diffusion Models”. ICML 2025.

[4] Sabour, et al. “Test-time Scaling of Diffusions with Flow Maps”. arXiv 2025.

[5] Huang, et al. “How to Guide Your Flow: Few-Step Alignment via Flow Map Reward Guidance”. arXiv 2026.

Reviewer_VnxQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines 3 test-time scaling strategies for training-free guidance in diffusion and flow models: averaging, particles, and selection, while building one framework around each strategy’s “consumed population” (the K draws, P particles, or N samples). Extra budget improves what is estimated: for Averaging, more draws reduce variance, but the draw construction fixes the mean guidance; for Particles, more particles approximate the target law more faithfully, but the sampler and weighting rule fix that law; for Selection, more candidates push selection into the selector’s top tail, so gains continue only while the selector stays reliable there. The paper provides a common characterization of what extra sampling changes are and why they can fail, together with counterexamples showing population summaries can be uninformative. It also shows empirical evidence of heterogeneous returns and controlled interventions.

Critical strengths

This paper provides some interesting insights into characterizing different scaling strategies in a principled way, under a unified framework. The paper reveals some useful properties of the existing strategies and provides some helpful guidance into what factors to look into when a scaling experiments doesn’t work. The paper also suggest that the ESS estimator is not very reliable with experimental support.

Critical weaknesses

The theories presented in the paper are primarily descriptive: they explain why scaling may not reliably improve outcomes and why population summaries can be unreliable, but they stop short of offering actionable guidance for practitioners. For example, the theory does not suggest how to address scaling stalls or provide recommendations for better estimators than ESS.

While the paper includes supporting experiments, many of the results are not particularly surprising. The heterogeneity of returns across scaling settings shown in Section 4.1 is well known in the literature, and the intervention experiments in Section 4.2 do not fully substantiate the paper’s stronger claims. In particular, the intervention in the Particles (SiT) experiments involves changes to additional factors, making it difficult to isolate the effect; moreover, the generalizability of these experiments appears limited.

Given the abundance of AI theory papers, I believe a more impactful theory should generate clear, actionable predictions and practical tools for a broad range of scenarios. Incorporating these aspects would make the paper much stronger.

Questions for authors
  1. Can the authors do some experiments that show that the theory they establish helps improve the existing testing-time scaling methods, for example, how to adjust hyperparameters when a scaling stalls or under what conditions we should use which scaling method, and show that the predictions / recommendation from the theory does result in the predicted improvements in practice?
  2. Can the authors suggest a better estimator than ESS, as the paper establishes that it’s not very reliable? (This is less crucial compared to 1, and can be treated as future work when needed.)
5 comments

We thank the reviewer for their careful review and highly constructive suggestions for strengthening the paper. We addressed related weaknesses and questions together.

[W1, W3, Q1] Under what conditions we should use which scaling method, and do the recommendations give the predicted improvements in practice?

[A] Thank you for encouraging us to make the practical consequences explicit. Following the reviewer's suggestion, we conducted additional experiments and added Section 4.4, which provides concrete, actionable recommendations for each scaling family, supported by prospective evaluations on three production models: SD3.5-Large, FLUX.2-klein and Qwen-Image. We summarize the recommendations below; Section 4.4 explains their motivation, procedures, and experimental results.

  • Averaging: compare draw constructions at small KK before increasing the count, then compare the better one with the plug-in. Lemma 3.1 explains why more draws cannot change a construction's mean guidance; which construction improves final quality is an empirical question.

  • Particles: compare resampling with independent selection.

    • With a deterministic sampler: copies follow the same continuation. Compare particles with independent selection at the same budget and final scoring rule.
    • With a stochastic sampler: copies can explore different continuations. Retune guidance strength, compare weighting procedures at small PP using final utility, and check the better procedure against independent selection at the same budget.
  • Selection: choose the candidate budget from final utility and cost.

    • When the selector measures utility directly: generate another candidate only while its estimated improvement exceeds its cost. We fit this stopping rule on earlier instances and evaluate it on new ones, building on optimal stopping studied for LLM inference [1].
    • When the selector is a proxy: compare candidate counts on validation instances. At each NN, let the proxy select an output and measure its actual utility. Choose the NN with the highest mean utility minus the cost of generating NN candidates, then keep it fixed for new instances.

Empirical results. We evaluated these recommendations on held-out or new instances across the three production models. The table summarizes how often each recommendation was supported and its effect on final utility.

FamilyComparisonSettingsObserved effect
AveragingChange construction at K=2K=2 vs scale projected draws to K=32K=3216 of 21Median gains +0.47 vs +0.10 IQR
ParticlesBest-of-4 vs 4 particles, deterministic sampler7 of 9Selection wins 62.5% of 648 paired runs against highest-weight return
SelectionAdaptive stopping vs fixed budgets9 of 9Higher utility at matched average count; 30% median candidate saving at matched mean utility
SelectionValidation-selected fixed budget6 of 6Net utility within 0.03 IQR of the best fixed budget in hindsight

A setting denotes a task–model pair. Gains are measured in units of the plug-in baseline's interquartile range (IQR), with projected draws at K=2K=2 serving as the reference for the averaging comparison. Particle comparisons use paired instances and initial noise.

Each decision was fixed on earlier instances and tested on held-out or new ones, with its target recorded before the scored outcomes were inspected. Protocols are in Appendix E.6.5; the complete prediction record, including the 6 of 39 missed targets, is in Appendix E.7.1.

Scope.

  • Selection and particles: nine reward-guided settings at a budget of four; both improve on the plug-in in 9 of 9 (median +0.40 and +0.27 IQR). When particles return their best final particle, selection wins 51.7% of pairs, identifying the terminal return rule as an important part of the comparison. On earlier instances, selection was ahead in at least 20 of the 21 settings at every budget from 2 to 32.
  • Draw construction: the better construction also overtakes the tuned plug-in in 5 of 21 settings, so the last step is a comparison with the plug-in.
  • Selection budget: the stopping rule was tested with at most four candidates and counts samples, not wall time. At a cost of 0.02 IQR per candidate, the fixed budgets average 4.0 candidates and retain about three quarters of the quality gain of eight.

For further details: Section 4.4 presents all three decisions; Appendix E.6.5 gives the results per setting.


References:

[1] Kalayci, et al. “Optimal Stopping vs Best-of-N for Inference Time Optimization”. arXiv 2025.

[W2-1] Heterogeneity of returns across scaling settings is well known.

[A] We agree that heterogeneous scaling returns are not, by themselves, a surprising finding. Our contribution is to characterize why additional samples can fail to improve final utility, identify which components can change that outcome, and test whether modifying them improves final outputs. The matched counterexamples also show that identical population summaries can accompany opposite changes in estimation error or the utility of the selected output. To strengthen the empirical evidence and assess generality beyond the original testbeds, we extended the interventions to three production models (SD3.5-Large, FLUX.2-klein, and Qwen-Image), across averaging, particle propagation, and selection.

[W2-2] The SiT intervention changes more than one factor.

[A] Thank you for pointing this out. We corrected the stochastic sampler and repeated the comparison. With the corrected stochastic sampler, particles still outperform the plug-in using the same sampler in both SiT restoration tasks and in 8 of 9 production-model restoration settings.

On re-checking, our stochastic sampler added noise to the deterministic step without the drift term that compensates for it. Its outputs kept residual noise and its own baseline was worse (by 5 dB on SiT super-resolution). The paper now names that step a noise-injected control (Appendix E.4); its three SiT settings stay in the counts of Sections 4.1 and 4.3, each compared with that step's own baseline.

We reran with a standard stochastic sampler (Euler-ancestral). SiT retains the original guidance strength; production models tune it separately for each sampler using the same rule.

Median PSNR gain of particles over the plug-in of the same sampler on SiT, with the number of the 192 paired runs (64 images, 3 seeds) in which particles were better:

SiT, particles minus plug-inSuper-resolutionInpainting
Deterministic (Euler) sampler0.0 dB (97 of 192)0.2 dB (103 of 192)
Submission's stochastic step (noise without its drift correction)1.1 dB (182 of 192)0.6 dB (134 of 192)
Stochastic (Euler-ancestral) sampler1.2 dB (186 of 192)0.7 dB (156 of 192)
  • SiT: corrected particles also end above the deterministic baseline in both restoration settings, in 92% and 86% of 192 paired runs (by 0.7 and 2.1 dB at the median). In the class-reward setting, where the baseline's reward is already near its maximum, they stay below it.
  • Three production models, nine restoration settings, predictions recorded first: particles improve on their own plug-in in 8 of 9 settings (median +0.13 IQR; deterministic particles: 5 of 9) and on the deterministic plug-in in 7. The gains are small: 1 of 9 exceeds the rerun band of 0.40 IQR, where we predicted 3.

Particles exceed best-of-4 in 4 of 9 settings under the same sampler and in 3 of 9 against deterministic best-of-4. The corrected experiments support the proposed sampler mechanism, with clear utility gains in the two SiT restoration settings and smaller gains on the production models. These results support adapting the particle procedure to the sampler and comparing its gain with independent selection (Sections 4.2 and 4.4). In SD3.5's reward settings the predicted particle gain on best-of-4 did not appear (1 of 3).

[W2-3] Limited generality of the experiments; intervention experiments do not fully substantiate the paper’s stronger claims.

[A] We extended the Section 4.2 interventions beyond their original testbeds:

  • Averaging (draw construction): Originally evaluated on two FFHQ settings, the intervention now outperforms increasing KK in 16 of 21 production-model settings. The construction ranking persists at K=16K=16 in 20 of 21 settings and under guidance-strength changes in 8 of 8 comparisons.
  • Selection (selector fidelity): Originally examined in topology optimization, the intervention now extends to production text-to-image models. On stored candidate pools, selection gains grow with NN in 12 of 12 settings where the selector directly measures utility. In nine restoration settings where it serves as a proxy, selecting with a reference-based quality oracle yields larger gains (+0.51 vs +0.29 IQR at N=16N=16), illustrating the importance of selector fidelity.

These extensions complement the broader evaluation across five testbeds, 21 models, and 29 budget-sweep settings. Together with the corrected particle experiments above, they test whether changing components left unchanged by additional sampling improves final utility when scaling stalls. Section 4.4 turns these findings into practical sampling decisions and evaluates them prospectively.

[Q2] A better estimator than ESS?

[A] We appreciate this suggestion. ESS measures weight concentration, but does not reliably predict whether additional sampling improves final utility. Our characterization opens up two promising directions:

  • Utility-aware particle diagnostics: capture the relationship between particle weights, trajectory quality, and final utility, rather than weight concentration alone.
  • Direct gain estimation: estimate the expected marginal improvement in final utility from additional sampling, accounting for evaluation cost. For selection, this connects to optimal stopping studied for language-model generation [1]. In guided generation, this rule reaches the same average utility with a median 30% fewer candidates across nine reward-guided settings.

Developing reliable, well-calibrated, and computationally inexpensive estimators across samplers and tasks remains an important direction for future work.


References:

[1] Kalayci, et al. “Optimal Stopping vs Best-of-N for Inference Time Optimization”. arXiv 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Contributions

Dear Reviewers and Area Chair,

We sincerely thank all reviewers for their time, thoughtful feedback, and constructive suggestions. We have carefully considered the reviewers' concerns and substantially expanded the manuscript.

As test-time scaling becomes increasingly prevalent in generative modeling, we believe a solid understanding of what it means to scale, why scaling does not reliably improve final outcomes, and how to efficiently allocate finite test-time compute has become critical. Our work addresses these questions through a unified characterization of three scaling families in training-free guidance. We further establish formal limitations of commonly used population diagnostics and develop practical recommendations for identifying where to intervene when scaling stalls.

We would like to summarize how the revisions address three recurring concerns raised during the review process.

  1. Practical Implications and Actionable Guidance

A common concern across the reviews was that, while our characterization explains limitations of scaling, its practical implications were insufficiently established. We took this concern seriously and introduced a new Section 4.4, translating the characterization into practical recommendations for more effective use of test-time compute, providing a more constructive and actionable conclusion to our findings.

These recommendations are rooted in the components our characterization identifies as worth reconsidering when scaling stalls: comparing draw constructions before increasing the averaging budget, accounting for sampler dynamics and weighting rules while evaluating particle propagation against independent selection, and choosing selection budgets based on final utility and sampling cost, with different procedures for direct-utility and proxy selectors. We also conducted prospective evaluations on held-out or new instances to test these recommendations, with detailed results and limitations reported in Section 4.4 and Appendix E.6.5.

  1. Theoretical Novelty of the Characterization

We recognize the concerns that Lemmas 3.1–3.3 underlying our characterization build on established mathematical principles, and do not claim novelty for individual identities. However, we believe the unified characterization constitutes a theoretical contribution in its own right, beyond the established properties of the individual methods.

Our population–operator framework provides a novel and unified analytical perspective on three distinct scaling families that have traditionally been studied through separate mathematical foundations. By distinguishing what additional sampling refines from the procedure-level components it leaves unchanged, we identify a common principle underlying their different scaling behaviors: additional sampling refines intermediate quantities, but its effect on final utility depends on choices that increasing the budget leaves unchanged. We believe identifying this shared structure across otherwise distinct scaling strategies is itself a meaningful theoretical contribution, alongside our new matched counterexamples (Propositions 3.4–3.6 and A.5), which formally establish limitations of commonly used population diagnostics and per-step error measurements.

  1. Corrected Sampler and Expanded Experimental Validation

We appreciate the concerns regarding the stochastic-sampler comparison. Upon re-examination, we identified and corrected the missing drift term and repeated the experiments using a standard Euler-ancestral sampler. We are glad the concerns were addressed as the effect of sampler dynamics persists under the corrected comparisons.

We further expanded the controlled interventions across three production models, added sampler-specific guidance calibration and guidance-strength robustness checks, and strengthened computational-cost accounting and comparisons with calibrated baselines. Together with our original budget sweeps across 18 checkpoints and 29 settings, we believe this extensive empirical investigation constitutes a significant analytical and empirical contribution, going beyond documenting heterogeneous scaling returns to systematically examining the mechanisms governing scaling effectiveness.

Lastly, we recognize opportunities for further research and for improving the presentation of our contributions. We plan to articulate the novelty and significance of our work more coherently in the final manuscript. Nevertheless, we believe our unified characterization, formal counterexamples, extensive empirical analysis, and practical recommendations offer valuable insights to the test-time scaling community, both in understanding why additional inference compute may fail and how it can be used more effectively.

We sincerely thank the reviewers and Area Chair again for their careful consideration of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.