- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper considers screening a finite collection of prompt-induced synthetic distributions using a small sample from a target population. For each candidate, it compares a smoothed empirical distributional discrepancy with discrepancies computed from simulated calibration batches sharing a synthetic reference sample. Inverting the resulting randomized rank tests produces a retained set of candidates, with abstention when that set is empty. Theorem 1 establishes exact candidate-wise null retention under the sampling assumptions, and Theorem 3 claims an asymptotic power bound. Simulations and experiments on OpinionQA and MovieLens evaluate distributional agreement and downstream estimation.
Critical strengths
-
The basic finite-sample validity argument is clean and useful. Under the equality null, the target batch and simulated calibration batches are identically distributed conditional on the shared synthetic reference. Their discrepancy scores are therefore exchangeable, and randomized ranking gives the stated candidate-wise calibration (Section 4.1, Theorem 1). Reuse of the reference is handled by conditioning rather than incorrectly treating all scores as unconditionally independent. The distinction between retaining a correct candidate and certifying that every retained candidate is close to the target is important and is stated in Remark 1.
-
The application includes meaningful controls and abstention behavior. Screening prompt distributions is relevant when synthetic population data could otherwise introduce misleading precision. The OpinionQA evaluation compares synthetic controls beyond a real-only estimator, and the MovieLens study includes configurations where all candidates are rejected (Table 6). Allowing the procedure to decline augmentation is a useful part of the proposed workflow and provides evidence of behavior beyond always selecting a prompt.
Critical weaknesses
-
The power proof does not establish the claim for the globally defined score as written. Equation (5), p. 16, integrates the smoothed divergence over all of R^d, and Theorem 3 uses the corresponding population Gaussian-smoothed KL divergence. However, Proposition 9 establishes the density-ratio and denominator bounds used in Lemmas 5–6 only for evaluation points in compact Ω (pp. 27–28). Compact support of the original observations does not provide a positive denominator lower bound over all of R^d after Gaussian smoothing. Page 23 explicitly acknowledges this tail issue, but Appendix D.6 nevertheless invokes Proposition 9 as sufficient for the concentration argument. This leaves a central claimed contribution unproved. A tail-aware analysis, or an explicitly defined compact-domain score with matching theory and implementation, is needed. The basic rank-validity theorem remains supported; the unresolved issue is the power argument for the implemented score.
-
The multiple-question experiment is inconsistent with the stated candidate-retention rule. Appendix A.2 (and Corollary 11) states the retention rule, for Bonferroni, as "retained only if no question-level null is rejected"; no separate rule is stated for Holm, and the only rule under which Holm controls the familywise error rate across questions as asserted in Remark 2 is the same no-rejections rule. For the same vector of T p-values, both Bonferroni and Holm reject at least one null exactly when the minimum p-value is at most α/T. They must therefore retain the same candidates under this rule. Nevertheless, Appendix H.4 and Figure 13 report different retained-set sizes and systematically improved divergences for Holm. Different Monte Carlo draws or a different aggregation rule could produce different outputs, but would require an explicit account and would not by themselves establish the claimed advantage of the correction. The text on p. 42 treats the agreement of the two corrections at (N, n, m) = (600, 20, 40) as an exception to "their broader difference across the displayed settings"; under the stated rule, agreement must hold at every configuration, so the exception-versus-rule framing is inverted. This directly affects the experimental evidence for the multiple-question screening procedure.
Questions for authors
- Is the implemented KL discrepancy integrated over R^d or a bounded region? How are the Gaussian-tail terms controlled in Theorem 3, given the domain of Proposition 9 and the qualification on p. 23?
- What exact aggregation rule and paired question-level p-values produced Figure 13? Under the stated no-rejections rule, how can Bonferroni and Holm retain different candidates, and were the same question-level p-values (same calibration batches and tie-breaking draws) used for both corrections?
3 comments
Thank you for the response and clarifications. I agree that the Gaussian-envelope argument offers a plausible way to repair the power proof, and I appreciate your explanation of the Holm/Bonferroni discrepancy. However, I still have concerns after checking the revised manuscript.
The proof in Appendix D.6 (p. 26) still relies on Proposition 9, whose bounds apply only on the compact set Ω. The full-space sensitivity and bias arguments needed for Theorem 3 are still missing. Showing that the KL terms are integrable is not enough to establish those bounds. I would need to see the promised derivation before considering this concern resolved.
The implementation clarification also leaves a gap: the experiments integrate over the pooled sample range and use Silverman's bandwidth, while Theorem 3 assumes full-space integration and a fixed bandwidth. Even a repaired proof needs to explain how it applies to this implementation, or clearly limit the theorem's scope. My positive assessment of the exchangeability argument in Theorem 1 is unchanged.
For multiple testing, I accept that different raw p-values across separate runs explain the different retained sets. But this also means that the comparison cannot support an advantage of Holm under the stated retention rule. Section 5.2.2 and Appendix H.4 still make that claim, and Figure 15 retains the comparison. The text should be corrected, with a paired check using identical p-values to confirm identical candidate retention.
The response helps clarify both issues, but the revised paper has not yet resolved them. I therefore keep my rating.
Response to followup of Reviewer JF6L
Thank you for identifying the gaps in the power proof and the multiple testing.
1. The role of Assumption 1 in the power proof.
The proof of Theorem 3 invokes Assumption 1 through Proposition 9 to verify Condition 1 and the empirical bounds in Lemmas 5 and 6 on . The population ratio bound in Condition 1 permits unbounded supports. With Condition 1 and direct control of empirical sensitivity and bias, the same power argument yields the separation scale under the stated cutoff and gap conditions. Appendix D.7 derives the full-space Gaussian sensitivity bounds and and bias . The power analysis uses a fixed bandwidth and full-space integration. The experiments use Silverman's rule and the pooled sample range. Theorem 1's validity argument rests on conditional exchangeability.
2. A fixed-bandwidth example with unbounded support.
Take and with the Cauchy kernel . For ,
Averaging these bounds gives a global population ratio bound. The empirical averages of are bounded with exponentially high probability, which puts the empirical densities divided by in a fixed compact interval. Clipping to this interval in the proof allows bounded differences and a Taylor bias bound. The original score then satisfies
\le 2e^{-c r_{n,N}t^2}+e^{-c_0 n}+e^{-c_0 N},\qquad t > 0 .$$ **The score concentration scale is preserved up to exponentially small exceptional terms,** yielding consistency at this fixed alternative. **3. Holm and Bonferroni.** We removed the superiority claim and retained the figure as a descriptive comparison of separate runs.
Response to Reviewer JF6L
Thank you for identifying the support/proof distinction and the implications of the multiple-testing rule.
W1. Does compact support conflict with Gaussian smoothing or the full-space KL proof?
Compact support concerns the original distributions, not their smoothed densities. Assumption 1 requires supp(Pt)⊆supp(Psθ)⊆Ω for compact Ω. With the common fixed Gaussian bandwidth σ>0 in Eq. (6), both population and empirical smoothed densities are positive throughout Rd. This is consistent with Eq. (5).
Proposition 9 gives bounds on Ω, not the full space. Compactness and kernel positivity give uniform upper and positive lower density bounds, and hence two-sided ratio bounds, on Ω. These also hold for empirical densities and single-observation replacements within Ω.
The full-space argument requires Gaussian-tail control. Bounded kernel centers yield common Gaussian upper and lower envelopes. Although density ratios may grow in the tails, density-weighted KL terms remain integrable. Under independent sampling and fixed bandwidth, these envelope estimates give O(1/n) and O(1/N) single-observation changes and an O(1/n+1/N) bias relative to the population Gaussian-smoothed KL, with constants depending on Ω,σ,d. We will extend the Gaussian-tail estimates from Theorem 4's proof (Appendix D.1) to make the concentration and bias steps explicit in Appendix D.6.
The target-to-reference score is numerically integrated from the minimum to the maximum of the pooled target and synthetic reference samples, with bandwidth chosen by Silverman's rule. We will explicitly distinguish this computation from the fixed-bandwidth, full-space analysis. Theorem 1's finite-sample validity relies on conditional exchangeability, not the compact-support assumption used in the power analysis.
W2. Can Holm and Bonferroni retain different candidates with the same raw p-values?
Identical raw p-values imply identical candidate retention. For a fixed candidate θ, let pjθ=1−Qjθ, j=1,…,T. Both Holm and Bonferroni operate across the T questions, not the m calibration replicates; a candidate is retained only if no question is rejected.
Bonferroni retains the candidate exactly when minjpjθ>α/T. Holm uses the same first threshold: it either stops without rejection or rejects at least one question, excluding the candidate. Thus, both induce the same candidate-level global p-value:
pglobal,B=pglobal,H=min1,Tminjpj,retain iff pglobal>α.
Their adjusted p-values and question-level rejections can nevertheless differ.
Figure 13 reports separate runs, with different raw p-value inputs. These differences reflect between-run variation, not an intrinsic screening advantage of Holm under our retention rule. We will make this distinction explicit in the figure caption and the multiple-output discussion.
With valid marginal p-values, both procedures control familywise error across questions for each fixed candidate without requiring question independence. Neither provides simultaneous control across candidates nor certifies retained candidates. The tests assess marginal distributions, not the full joint rating distribution.