acceptodds
Desk rejected at ICLR 2027

Scale-Local Information Governs Sample Synthesis from Noisy Smoothed Scores

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Statistically equivalent targets can have asymptotically different sampling costs under the same score interface. We establish this through an oracle that returns the full normalized smoothed-score vector at one shared scale, with independent Gaussian observation noise and a precision cap. For geometrically spaced Fourier and spatial families, constant target capacity coexists with an optimal cost of queries at precision and fixed, sufficiently small average total variation error. Preserving the entire statistical experiment, we then construct Blackwell-equivalent spatial targets with a diverging ratio of optimal costs at the same dimension, precision, and accuracy. Matching bounds across all precision levels explain this separation at fixed relative TV accuracy in a local perturbation regime under a uniform prior. Precision determines the feasible coordinate block length; at high precision, the optimal gain over coordinate thinning is of the order of its square root. A blockwise converse quantifies the dependence required by an accurate joint output. Shared activation attains the bound by matching coordinate marginals, cancelling single-coordinate error terms, and controlling the remaining interactions. Nonadaptive samplers match lower bounds allowing adaptive queries and random stopping.

Primary area:
learning theory
Keywords:
Equivalent Targets, Different Sampling Costs: Precision and Scale Geometry of Noisy Scores

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_T1b3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the expected number of queries needed to sample, at prescribed TV accuracy, from a family P_θ, using an oracle that returns a normalized smoothed score at one scale shared by all coordinates, corrupted by fresh Gaussian noise of precision at most B. Theorem 1 gives matching rates in three precision regimes for two product families. The main result, Theorem 3, shows that two Blackwell-equivalent placements of m spatial coordinates have optimal costs differing by a factor \sqrt{log m}.

Critical strengths

The upper and lower bounds match, and the lower bounds cover adaptive protocols with random stopping. The block-wise witness converse and the chi^2 cancellation under shared activation are careful arguments that may apply elsewhere.

Critical weaknesses
  1. No realistic experiments, and no motivation for the ICLR community. The paper frames its results as statements about the cost of score-based sampling but provides no evidence that they hold in practice. The paper contains no experiments with a trained diffusion model or with realistic data such as images. In particular, it does not test whether the effect it predicts, that sampling cost depends on how the data’s features are spread across scales, appears in practice. Even within the model, no simulation shows at what dimension the \sqrt{log m} separation becomes observable. The paper also does not explain why the problem matters to the ICLR community or which design choice the results inform.

  2. Limiting target class. Theorem 3 concerns m rescaled copies of one weakly perturbed one-dimensional mixture, padded with known coordinates independent of θ. All targets are products, which excludes the dependence across scales that characterizes practical multiscale data. The lower bounds extend to larger target classes, but the separation does not, because its upper bound relies on product structure.

Questions for authors

see my comments above

2 comments
AuthorsOpenReview ↗

Weakness 2: Why the controlled product construction is informative

Thank you for asking about the role of the target class. The product construction enables a controlled comparison of oracle access with the full statistical content of the target held fixed. The coordinates are independent conditional on the hidden parameter, so the separation identifies a source of sampling difficulty that is present even without dependence between target coordinates.

The rescaled copies make this control exact. Dividing each active coordinate by its public radius and discarding inactive coordinates yields the same independent canonical factors in both layouts,

νθr=1+εθr2N(1,1)+1−εθr2N(−1,1).\nu_{\theta_r}=\frac{1+\varepsilon\theta_r}{2}\mathcal{N}(1,1)+\frac{1-\varepsilon\theta_r}{2}\mathcal{N}(-1,1).

Conversely, public rescaling and independently generated inactive base coordinates reconstruct either target. Both transformations are independent of the hidden signs. They establish Blackwell equivalence, preserving every statistical decision problem based on a target sample (Section 5 and Appendix K).

Padding provides a second control by keeping the ambient dimension identical. Conditional on a public query action, the inactive response coordinates have a known law, including their Gaussian noise, and can be simulated internally. Appendix K thus accounts for the full returned vector while keeping information about the hidden signs confined to the active coordinates.

Product structure also makes the attaining sampler and its joint error explicit. The construction combines exact coordinate synthesis with independent block outputs. Shared activation saves queries within a block and introduces dependence among its output coordinates. Section 4 and Appendix I expand the product densities, match the coordinate marginals to cancel the terms involving one coordinate, and control the remaining interactions to obtain joint TV accuracy. For the clustered layout in Theorem 3, a feasible integer block length has order ℓB(m)=min⁡{m,1+log⁡(1+εB)}\ell_B^{(m)}=\min\{m,1+\log(1+\varepsilon\sqrt B)\} in the stated high precision regime. This gives a saving of order ℓB(m)\sqrt{\ell_B^{(m)}} over coordinate thinning at the required accuracy.

The extension to a larger class has two distinct parts. With the same oracle and the original average TV requirement on our hard subfamily, its lower bound remains valid for expected queries averaged under the uniform prior on that subfamily. A matching separation for dependent targets additionally requires an attaining construction for that class under the chosen accuracy and cost criterion.

AuthorsOpenReview ↗

Weakness 1: Theoretical contribution, sampling design, and finite dimension

Thank you for asking us to clarify the contribution and its relevance to diffusion sampling. This is a theoretical paper on the query complexity of sampling from noisy scores. For two explicit target families, we characterize how scale layout and observation precision determine sampling cost through matching lower bounds and explicit sampling constructions under an oracle with fresh independent Gaussian observation noise. Experiments with trained image models would address the complementary empirical question of whether and how this mechanism manifests in learned samplers, beyond the scope of the present theoretical study.

Evaluating a vector at a common noise level is central to diffusion sampling, and reducing the number of evaluations is the objective of methods such as DPM-Solver. Section 1 connects our normalized score to noise prediction through −σsθ,σ(x)=E[Z∣X+σZ=x]-\sigma s_{\theta,\sigma}(x)=\mathbb{E}[Z\mid X+\sigma Z=x].

Theorem 3 isolates the effect of scale layout on sampling cost. Dividing spatial coordinate jj by its public radius rjr_j changes a common smoothing scale σ\sigma into the relative scale σ/rj\sigma/r_j. Under the theorem's precision and spacing conditions, clustering active radii lets one query serve several coordinates, whereas sufficient separation prevents that saving. This changes the optimal query cost while preserving the complete target experiment, ambient dimension, precision cap, and average TV requirement under the uniform prior. The lower bounds allow adaptive queries and random stopping, so the result applies beyond a particular solver.

Two existing results make the design connections concrete. Proposition 14 allows a separate smoothing scale in each coordinate of the Fourier construction. With this modified interface, sufficiently large constant normalized precision depending only on the fixed spacing parameter qq permits exact generation in one query. Corollary 13 considers both target families without a precision cap: for each positive relative price of cumulative observation precision, a suitable precision held fixed across queries attains the optimal order of the combined cost of calls and precision. Thus the analysis addresses the choice of a shared noise scale and the allocation of observation quality. Representation choices also have a practical precedent in the wavelet conditional model of Guth et al. (2022).

For the finite dimension question, Section 5 gives a concrete asymptotic separation. Its parameter sequence fixes q>1q>1 and a sufficiently small relative TV tolerance κ\kappa, takes α=1\alpha=1, ε=m−1/2\varepsilon=m^{-1/2} and B=m3B=m^3, and sets G=⌈Csep(q,κ)[1+log⁡(1+m)]⌉G=\lceil C_{\rm sep}(q,\kappa)[1+\log(1+m)]\rceil. Both layouts have ambient dimension D=1+(m−1)GD=1+(m-1)G. For sufficiently large mm, their optimal costs have orders m/log⁡m\sqrt{m/\log m} and m\sqrt m, with comparison constants depending on q,κq,\kappa. These asymptotic rates do not specify the first dimension at which the gap becomes numerically visible. Figure 1 illustrates the theory schematically. Appendix I supplies the clustered construction's expected cost ⌈m/L⌉min⁡{1,ΓεL}\lceil m/L\rceil\min\{1,\Gamma\varepsilon\sqrt L\}, where the integer LL must satisfy its precision feasibility condition and Γ\Gamma is chosen for the prescribed joint TV error.

Reviewer_ct2MOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

It is quite hard for me to even understand what this paper is about. I think this is an artifact of using AI to generate large parts of the main text, since there doesn't seem to be much coherence or a broad theme or narrative in the paper.

Critical strengths

None

Critical weaknesses

As said in the summary, I cannot really understand what the main message of this paper is, and can't judge its importance. The presentation seems to be quite poor and reminiscent of current AI writing style without providing a clear narrative or broader picture. I read the abstract, the introduction, tried to go through theorem statements and the notation definition, and could not connect anything to a broad message or intuition.

Questions for authors

Please see above.

2 comments
AuthorsOpenReview ↗

Weakness: Key idea for our paper.

Thank you for explaining that the paper's main message and its importance were difficult to follow. Our finding is that generating a sample can require different amounts of work even when the sample contains exactly the same information. In the examples we construct, the difference comes from how the target is represented and how its information can be reached through score queries.

This matters because it identifies a source of sampling difficulty that remains even after the algorithm has been optimized. Understanding sampling efficiency therefore requires looking at the target's representation as well as the sampler itself. Our result gives a reason to study representation choices alongside algorithm design when investigating how to reduce the number of score evaluations.

We establish this through a controlled comparison. We construct two targets whose samples can be converted exactly into one another using known rescalings and independent randomness, without knowing the hidden quantities that define the targets. Neither target therefore asks the sampler to produce more statistical information. For each pair, we keep the dimension, query rules, precision and required sampling accuracy the same.

We then examine what each call can help the sampler generate. A call uses one smoothing scale for the whole vector. In one target, nearby scales allow several coordinates to benefit from the same call. In the other, the same unknown quantities are placed at sufficiently separated scales, so this saving is lost at the chosen precision. This explains how the same information can be easier or harder to use.

To establish that this is an unavoidable difference, we prove limits on how few calls any accurate sampler can make on average. These limits also cover algorithms that adjust their later calls using earlier responses. We then construct samplers that match those limits in how their costs grow. The two sides together establish the best achievable cost for each target, up to fixed factors. One target becomes increasingly more expensive than the other as the examples grow. The difference therefore survives improvements to the sampling algorithm under the same query rules.

The AI use statement on p. 10 reports assistance with wording and proofreading. We take responsibility for making the research question, its significance and the connection between the results clear.

Thank you for your response. However, I believe the submission requires a major rewriting to properly frame the narrative and contributions. Therefore, I have to keep my score.

Reviewer_Mf6GOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper claims to construct "statistically equivalent targets" which have different costs to sample from under noisy queries to their score functions, and then derives statistical rates for these targets.

Critical strengths

None.

Critical weaknesses

The main problem is incomprehensible. It is not clear what "statistically equivalent targets" means. The statement appears to be a certain counterexample, but it is not clear what the significance of this would be. The exposition and theorem statements are overly terse and technical, with little interpretation. There appear to be several different type of score queries, but as the query model is unclear, their role in the problem is incomprehensible.

Questions for authors

Explain the main problem in a self-contained way, without relying on undefined technical terms such as "statistically equivalent."

2 comments
AuthorsOpenReview ↗

Weakness: What the separation establishes and how the results fit together

Thank you for asking what the separation establishes and how the results fit together. The spatial construction shows that scale layout can change optimal sampling cost even when exact sample conversions preserve all information about the hidden parameters.

The generator knows a family of target distributions but not the hidden signs selecting its member. At a point chosen by the generator, each query returns the full score vector of the target after Gaussian smoothing, multiplied by the chosen smoothing scale, with independent Gaussian observation noise. The score is the gradient of the log density. All coordinates use one smoothing scale per call, and a precision cap limits the quality of each response. We minimize expected calls at a prescribed total variation error averaged over the uniform sign prior. Total variation measures the largest discrepancy in probabilities assigned to the same event.

In Section 5, equivalence has an explicit operational meaning. The hidden signs change the weights of Gaussian mixtures placed at public coordinate radii. Dividing each coordinate carrying a sign by its radius produces the same standardized mixture in both layouts. Coordinates without hidden signs have known distributions. Rescaling the standardized coordinates and independently generating the latter coordinates converts samples exactly between the layouts in either direction, without knowing the signs. This is Blackwell equivalence: it preserves the complete statistical experiment, rather than just one numerical measure of information.

Theorem 3 quantifies why this matters. Fix the geometric ratio q>1q>1 and a sufficiently small error constant κ>0\kappa>0. Its concrete example uses mm hidden signs, perturbation strength ε=m−1/2\varepsilon=m^{-1/2}, precision cap B=m3B=m^3, the same ambient dimension of order mlog⁡mm\log m, and error tolerance δ=κ\delta=\kappa. With the separation of active scales specified in the theorem, for sufficiently large mm, the optimal expected query counts are Tclustered=Θ ⁣(m/log⁡m),Tseparated=Θ(m).T_{\rm clustered}=\Theta\!\left(\sqrt{m/\log m}\right),\qquad T_{\rm separated}=\Theta(\sqrt m). Their ratio grows as log⁡m\sqrt{\log m} although the targets admit the exact sample conversions above. The lower bounds allow adaptive queries and random stopping, and explicit nonadaptive samplers attain their order. Thus the result identifies a difference in optimal sampling cost under the same observation resources.

The mechanism is the interaction of layout with the common scale. Converting between the layouts rescales different coordinates by different factors. Applied to a commonly smoothed target, that conversion produces different smoothing scales across coordinates, whereas one legal query uses a single scale. In standardized coordinates, radius rjr_j turns the shared scale σ\sigma into relative scale σ/rj\sigma/r_j. At a given precision, nearby active scales can share a query. Sufficiently separated active scales lose that saving. Increasing precision can widen the group served by a query, so this is a quantitative constraint on usable responses, not a claim that other coordinates have zero response.

The other results explain the cost rather than merely supplying another example. Corollary 2 first shows that the maximum mutual information between the hidden signs and one target sample, over all priors, can remain constant while query cost grows. Theorem 3 preserves the whole experiment, excluding loss of detail in that single information measure as an explanation. Theorem 1 and Sections 3 and 4 give matching bounds across precision levels. Repetition accumulates weak responses, coordinate thinning uses stronger responses less frequently, and shared block activation serves several coordinates while controlling joint output error. The lower bound measures the parameter dependence an accurate output must retain, and the constructions supply it at matching cost.

There is one main query model. Fourier and spatial are two target families, and the three procedures above use that same model. Appendix L tests how much of each response must be observed: one threshold bit per coordinate still attains the same cost orders. Appendix M instead removes the precision cap and charges both calls and the sum of the precisions used. Appendix N varies the shared scale, spacing, normalization, or precision cap separately in the Fourier construction. Under each specified modification, suitable precision permits exact generation in one call. Its Table 2 places fixed range quantization and general observation channels alongside the main oracle as comparisons. These extensions distinguish the observation resources responsible for the separation.

AuthorsOpenReview ↗

Question: The sampling problem, allowed observations, and cost

Thank you for requesting a direct statement of the problem. A generator knows a family of probability distributions, but an unknown sign vector θ\theta selects the particular target PθP_\theta. The generator must produce a random output with approximately that distribution, using its own randomness and the responses to allowed queries. It does not need to recover the signs first.

A query chooses a point xx, one Gaussian smoothing scale σ>0\sigma>0 for all coordinates, and a precision 0<b≤B0<b\le B, where BB is a fixed cap. Let pθ,σp_{\theta,\sigma} be the target density after adding Gaussian noise with covariance σ2I\sigma^2 I. The response is Y=σ∇log⁡pθ,σ(x)+b−1/2G,Y=\sigma\nabla\log p_{\theta,\sigma}(x)+b^{-1/2}G, where GG is a fresh standard Gaussian vector. The score ∇log⁡pθ,σ\nabla\log p_{\theta,\sigma} describes how the smoothed log density changes with location. The factor σ\sigma normalizes it across scales. The separate observation noise has variance 1/b1/b per coordinate, so larger bb gives a more accurate response. The oracle is defined by the target distribution and does not give the generator a target sample.

One full vector response costs one call. Later query choices and the time to stop may depend on earlier responses. Accuracy concerns the distribution of the final output: averaged over uniformly chosen sign vectors, its total variation distance from PθP_\theta must be at most a fixed tolerance δ\delta. Total variation is the largest difference in probabilities assigned to the same event. The optimization problem is to achieve this accuracy with the smallest expected number of calls, averaging over the signs, observation noise, and algorithmic randomness.

We then compare two target families indexed by the same signs. For every possible sign vector, a known procedure can turn a sample from either family into an exact sample from the other, without knowing those signs. That precise sample conversion is what we mean by statistical equivalence. The question is whether such families must have comparable minimum query costs when the dimension, query rules, precision cap, and accuracy requirement are the same. This separates the information represented by a target sample from the resources needed to generate it through the permitted observations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.