- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper analyzes test-time scaling of training-free guidance through averaging local draws, particle propagation, and output selection. Using established statistical identities, it characterizes what additional samples refine and which components remain unchanged. It also constructs matched counterexamples showing limitations of population summaries. Budget sweeps across 18 released checkpoints and 29 settings, controlled interventions, summary-based predictions, and small-scale gain evaluations examine how these limitations relate to final-output quality and decisions about additional sampling.
Critical strengths
S1. The matched counterexamples establish specific limitations of population summaries.
Proposition 3.4 constructs identical joint distributions of draw weights with opposite effects on gradient MSE relative to the plug-in. Proposition 3.6 constructs identical selector and utility marginals, with the same Kendall's τ, but opposite Best-of-N expected-utility gains. These results establish that the considered summaries cannot determine the corresponding outcomes.
S2. The empirical comparisons are useful for configuring and evaluating existing methods.
Figure 3 shows how output quality changes with K, P, and N, while Table 5 reports guidance strengths and guidance-active windows selected on validation data for the topology-optimization benchmark. Table 13 compares the averaging methods against both the released plug-in and a separately calibrated plug-in. Recalibrating the plug-in improves mean PSNR by 3.06 and 5.85 dB in the two conditions, matching or exceeding the gains of ps+. This highlights the importance of calibrating the baseline before attributing improvements to additional averaging.
Critical weaknesses
W1. The characterization largely formalizes expected behavior and provides limited guidance for choosing among sampling components.
The common characterization is built on established statistical identities, as the paper itself states: the bias–variance decomposition (Lemma 3.1), the Feynman–Kac output law (Lemma 3.2), and the Beta(1, N) order-statistic weighting (Lemma 3.3). These lemmas formalize largely expected behavior: more draws reduce variance but not bias, more particles approximate a fixed target law, and more candidates push selection further into the selector's tail. The theoretical novelty therefore rests mainly on the matched counterexamples. The contribution of the characterization then depends on whether it can guide practice, which the abstract describes as identifying "where to intervene when further scaling stalls." The current evidence for this is limited:
- For averaging, the draw-replacement, update-length, and sampling-stage controls narrow the explanations for the observed losses (Tables 10 and 12). However, the link from measured gradient properties to the final-utility ordering of projected draws, transition draws, and differentiable continuations remains unestablished. The paper notes that the logged disagreement with the plug-in measures bias only when the plug-in equals the exact guidance, and does not determine which direction improves utility (Appendix A.1.1).
- For particles, weight permutation and replay controls show that weight assignment affects measurement residual beyond the availability of alternative trajectories. The ODE/SDE comparison also identifies the role of transitions in separating resampled copies. However, weighting improves final utility beyond greedy search in most inverse-imaging cells but provides no additional benefit on the flow-matching stack. These results explain mechanisms in the tested settings, while evidence for using the identified relationships to choose a weighting rule under a given sampler and budget remains limited (Tables 7 and 15).
W2. Evidence for using population measurements to predict gains in new settings is limited.
- The summary-based predictions in Table 1 do not consistently identify gains. The held-out comparison does not establish a predictive advantage for tail statistics over global rank correlation, and the clone-calibrated separation criterion is evaluated on applicable checkpoints that all show no gain. The paper acknowledges both limitations. These results limit the predictive use of the evaluated measurements.
- The terminal pilot provides positive evidence on four held-out implementations, detecting 36% of qualifying gains compared with 5.5% for the relative-IQR comparator. The larger particle pilot also improves on the better fixed policy in the evaluated cohort. However, the primary particle pilot does not establish an advantage even before pilot cost, and the larger pilot requires more samples. These results support small-scale gain evaluation, while evidence for transferring the conclusions across budgets and for a benefit after accounting for pilot cost remains limited.
Questions for authors
Q1. Criterion for choosing sampling components
The controlled interventions identify effects of draw construction, weight assignment, and particle separation. Do the reported measurements support a criterion for choosing among alternative draw constructions or weighting rules under a given sampler and budget? If so, which comparisons connect that criterion to improvements in final utility?
5 comments
[W1-1] Averaging: the link from measured gradient properties to final utility is not established.
[A] We agree. Proposition A.5 gives two cases with identical gradient error at every step and every , yet averaging improves final utility in one and worsens it in the other. This motivates our controlled interventions in Section 4.2: rather than infer final utility from gradient-error summaries, we vary draw construction and directly evaluate its effect on final outputs. Section 4.4 then builds on these findings to provide practical criteria for choosing constructions.
[W1-2] Particles: the ODE/SDE comparison and choosing a weighting rule.
[A] We thank the reviewer for pointing out this concern. On re-checking the comparison, we found that our stochastic sampler added noise to the deterministic step without the compensating drift term, leaving residual noise in its outputs and weakening its own baseline (by 5 dB on SiT super-resolution). We corrected the sampler and repeated the experiments; the effect of sampler dynamics persists under the corrected comparison.
The corrected experiments use a standard stochastic sampler (Euler-ancestral). SiT retains the original guidance strength; production models tune it separately for each sampler using the same rule.
Median PSNR gain of particles over the plug-in of the same sampler on SiT, with the number of the 192 paired runs (64 images, 3 seeds) in which particles were better:
| SiT, particles minus plug-in | Super-resolution | Inpainting |
|---|---|---|
| Deterministic (Euler) sampler | 0.0 dB (97 of 192) | 0.2 dB (103 of 192) |
| Submission's stochastic step (noise without its drift correction) | 1.1 dB (182 of 192) | 0.6 dB (134 of 192) |
| Stochastic (Euler-ancestral) sampler | 1.2 dB (186 of 192) | 0.7 dB (156 of 192) |
- SiT: corrected particles also end above the deterministic baseline in both restoration settings, in 92% and 86% of 192 paired runs (by 0.7 and 2.1 dB at the median). In the class-reward setting, where the baseline's reward is already near its maximum, they stay below it.
- Three production models, nine restoration settings (inpainting, box inpainting, colorization), particles improve on their own plug-in in 8 of 9 settings (median +0.13 IQR; deterministic particles: 5 of 9) and on the deterministic plug-in in 7. The gains are small: 1 of 9 exceeds the rerun band of 0.40 IQR, where we predicted 3.
Particles exceed best-of-4 in 4 of 9 settings under the same sampler and in 3 of 9 against deterministic best-of-4. The corrected experiments support the proposed sampler mechanism, with clear utility gains in the two SiT restoration settings and smaller gains on the production models. These results support adapting the particle procedure to the sampler and comparing its gain with independent selection (Sections 4.2 and 4.4). On the weighting rule: on the flow-matching stack the weighted rule and greedy search stay level under the corrected sampler, as in the submission, while on the three FFHQ inpainting tasks, the weighted rule wins by 0.7 to 3.5 dB. The rule is therefore chosen per stack at .
[W2-1] Population measurements do not predict gains in new settings.
[A] We agree, and this limitation is a central finding of our paper. Our counterexamples establish why common population summaries cannot reliably determine scaling gains on their own, and our empirical results demonstrate their limited predictive usefulness. Appendix E.7.1 now reports that always predicting the more common outcome achieves 96%, 80%, and 52% accuracy for averaging, particles, and selection, compared with 79%, 60%, and 74% for our summary-based rules.
This also motivates Section 4.4, where we move beyond population-summary diagnostics to practical decisions based on direct comparisons of final utility and computational cost, rather than claiming a universally reliable predictor from these summaries.
[W2-2] Pilots: transfer across budgets, and benefit after cost.
[A] We agree that the pilot results do not yet establish a reliable, cost-effective deployment rule. We therefore treat these pilots as exploratory diagnostics rather than our main practical recommendations. The terminal pilot's endorsements at and persist at larger budgets, but the pilot itself does not repay its cost. The original particle pilot also fails to establish an advantage over fixed policies, while the larger pilot requires additional samples. We clarify these limitations and the cost accounting in Appendix E.7.2.
Instead, the new Section 4.4 provides practical recommendations based on final utility and sampling cost:
- When the selector measures utility directly: continue only while another candidate's expected utility improvement exceeds its cost, following optimal stopping studied for language-model generation [1]. We fit the rule on earlier instances, freeze it, and evaluate it on new instances.
- When the selector is a proxy: compare candidate counts on validation instances, measuring the actual final utility of the proxy-selected output at each . Choose the budget maximizing mean utility minus candidate cost, then keep it fixed for new instances.
| Stopping rule, 24 new prompts in each of nine reward-guided settings | Result |
|---|---|
| Candidates saved at matched mean utility (median) | 30% |
| Settings with higher mean utility at matched average count | 9 of 9 (median +0.18 IQR) |
For proxy selectors, validation-selected fixed budgets achieved net utility within 0.03 IQR of the hindsight-best budget in 6 of 6 settings and matched or exceeded eight candidates in 5, at a cost of 0.02 IQR per candidate.
The stopping test allows at most four candidates and excludes fitting cost; both rules count candidates rather than wall time. For particles, our recommendations instead focus on comparing sampler dynamics, weighting procedures, and final return rules directly, as described in criteria 2 and 3.
References:
[1] Kalayci, et al. “Optimal Stopping vs Best-of-N for Inference Time Optimization”. arXiv 2025.
We thank the reviewer for the detailed review and very valuable feedback. We addressed related weaknesses and questions together.
[A] Yes: we use small-budget comparisons of final outputs, then scale the better choice. The analysis explains why the considered summaries alone cannot determine the better choice. In the tests below, on held-out or new instances, the small-budget winner was the large-budget winner in 20 of 21 draw-construction settings and, for the weighting rule, in 4 of 4 settings on the measurement residual and 3 of 3 inpainting settings on PSNR. A setting is one task on one model; differences are in units of the plug-in baseline's interquartile range (IQR). Each prediction below was recorded in advance; the exact targets are in Appendix E.6.5.
1. Draw construction (averaging). Compare constructions at K=2 before raising K, then compare the better one with the plug-in. Why: increasing K cannot change the construction's mean guidance (Lemma 3.1), and the gradient-error summaries considered in Proposition A.5 do not determine final utility, so we compare final outputs directly. On 24 held-out instances in each of 21 settings of three production models:
Here LGD-scale draws won every K=2 comparison; the comparison is what finds the winner when it is not known in advance. The better construction overtakes the tuned plug-in in 5 of 21 settings, so the last step is a comparison with the plug-in.
2. Resample or select (particles). Compare according to the sampler dynamics:
On 24 new prompts in each of nine reward-guided settings, paired runs share prompt, seed and starting noise:
When particles return their best final particle instead of their highest-weight one, selection wins 51.7% of pairs, showing that the terminal return rule affects the advantage in the table. On earlier instances, selection was ahead in at least 20 of the 21 settings at every budget from 2 to 32.
3. Weighting rule (particles, stochastic sampler). Compare the rules on the measurement residual at P=4, then check final quality on validation cases. On 64 new FFHQ images, median PSNR gain of the weighted rule over greedy search, with the number of the 192 paired runs it won:
Phase retrieval is the exception: weighting lowers the residual but PSNR does not clearly improve, which is why the criterion compares final quality and not the residual. The inpainting PSNR ranking transfers from P=4 to P=64, supporting the small-budget comparison in those settings.