- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes synthetic-powered directional testing (SPDT), which uses independent synthetic data to select a testing direction and combines a directional test with an existing base test. The authors establish type-I error control and power guarantees for the combined procedure. The method is evaluated through simulations and applications to LLM safety evaluation, treatment effect heterogeneity, and agent safety auditing. I currently lean toward rejection, but would reconsider my rating if these concerns are satisfactorily addressed.
Critical strengths
-
The paper addresses an interesting problem: using synthetic information to improve hypothesis testing without relying on its distributional correctness.
-
The framework is simple and applicable to several testing problems.
-
The experiments cover different settings, including Gaussian mean testing and real-world applications.
Critical weaknesses
-
The main theoretical guarantees are relatively elementary. Type-I error control follows from the union bound, while base-power preservation follows directly from the OR combination. The contribution beyond combining existing valid tests is not entirely clear.
-
Power preservation is established relative to the level- base test, rather than the base test at the same total significance level . Thus, the claimed protection against misleading synthetic directions is weaker than it may initially appear.
-
Some empirical results appear inconsistent with the stated combination rule. This raises concerns about the implementation or reporting of the proposed method.
Questions for authors
1. In Table 1 (Page 8), the directional test rejects in 99.1% of subsamples, whereas SPDT rejects in only 91.6%. Similarly, Table 2 reports 61.4% for the directional test but 34.2% for SPDT. Since SPDT rejects whenever either component rejects, shouldn't its rejection frequency be at least as large as that of each component? Are different significance levels being used, or is there an error in the reported results?
2. Proposition 2 guarantees that SPDT preserves the power of the level- base test. However, at a fixed total level of 0.05, the relevant comparison is arguably the level-0.05 base test rather than the level-0.04 base. How often does SPDT actually lose power relative to the level-0.05 base when the synthetic direction is inaccurate?
3. Propositions 1–3 follow directly from combining two valid tests using an OR rule. What is the main theoretical contribution beyond this construction? In particular, can the authors characterize when SPDT is more powerful than the level- base test, rather than the level- test?
4. The experiments mainly use and . How sensitive are the power gains to this allocation? For example, would perform better when the synthetic direction is strongly informative? Is there a principled way to choose the allocation before observing the real data?
5. In Section 4.2, the directional test uses a normal approximation with only 50 real queries in each subsample. Appendix D.4 also reports some inflation in the directional test's type-I error. Could the authors clarify whether the observed power improvement remains substantial when the directional component is calibrated more conservatively?
6 comments
In Section 4.2, the directional test uses a normal approximation with only 50 real queries in each subsample. Appendix D.4 also reports some inflation in the directional test's type-I error. Could the authors clarify whether the observed power improvement remains substantial when the directional component is calibrated more conservatively?
Great question! To address this, we have run additional experiments in Appendix D.11 which replaces the normal-based directional component with a one-sided Gaffke test that has finite-sample validity. The observed improvement remains substantial: SPDT rejects in 80.3% of subsets, compared with 14.5% for both Base (5%) and GESPI (5%). We note this is smaller than the original normal-based SPDT rejection frequency, while retaining a substantial gain under finite-sample-valid calibration to resolve the Type-1 error inflation issue. We observe that the Gaffke directional component’s false positive-rate ranges from 0-0.44% at epsilon=1%, compared with the 1.11-2.84% for the original normal component. The resulting SPDT false-positive rate ranges from 0.04-2.54%, below the total 5% level throughout the tested settings.
Clarification points
Some empirical results appear inconsistent with the stated combination rule. This raises concerns about the implementation or reporting of the proposed method. In Table 1 (Page 8), the directional test rejects in 99.1% of subsamples, whereas SPDT rejects in only 91.6%. Similarly, Table 2 reports 61.4% for the directional test but 34.2% for SPDT. Since SPDT rejects whenever either component rejects, shouldn't its rejection frequency be at least as large as that of each component? Are different significance levels being used, or is there an error in the reported results?
We apologize for the confusion and thank you for raising this clarification point. Yes, different significance levels are being used here. For Tables 1 and 2, the standalone directional tests are evaluated at 5%, while the directional component inside SPDT uses 1%. We had used 5% for the standalone tests to compare methods under the same total type-1 error budget, to provide a fair comparison with SPDT’s combined 5% budget. We have updated the manuscript to make this clear and have also added the results for the standalone directional tests at 1%, so that this relationship can be checked directly.
Power preservation is established relative to the level-\alpha base test, rather than the base test at the same total significance level \alpha+\epsilon. Thus, the claimed protection against misleading synthetic directions is weaker than it may initially appear.
Thank you for raising this clarification point! Indeed, Proposition 2 states preservation relative to the level-\alpha base test; it does not claim preservation relative to the level \alpha+\epsilon base test, though our experiments (Figure 2 and applications) compare SPDT against the level-\alpha+\epsilon base test. We have updated the manuscript (Lines 16-17, Line 59, Lines 180-181) to make this qualification more explicit.
We thank the reviewer again for their time and consideration. We are more than happy to address any other concerns they might have.
Thanks for the response. I maintain my original rating.
Dear Reviewer bZrg,
Thank you for your feedback! We address your concerns below:
To address this question, we have added Propositions 5 and 6, which characterize when SPDT is more powerful than the alpha+epsilon base test in the Gaussian setting. (Full proofs are in Appendix F.2 and G.5). In short, Proposition 5 shows that for a given dimension d and signal strength r, there exists a unique cutoff c_* such that SPDT improve over base test iff c > c_. For example, Table 14 illustrates this cutoff at d=100 and r=3, which is c_=0.153. The base test has 15.82% power. SPDT has 15.10% power at c=0.10 and 29.50% at c=0.50. Proposition 6 gives a sufficient condition for improvement over a range of signal strengths, stating that whenever c > constant x d^{-1/4}, there exists a range of signal strengths (i.e. r) where SPDT will achieve higher power than the base test at alpha+epsilon.
To quantify how often SPDT loses power under inaccurate guidance, we compared its exact power with that of the 5% base test across our Gaussian simulation grid in Appendix D.1. With the 4%/1% allocation, SPDT had lower power with orthogonal or negatively aligned guidance, with a median loss of 1.93 percentage points and maximum loss of 3.85 percentage points.
We would also like to clarify that our contribution and novelty is to develop a method to use synthetic data for reliable statistical inference. While there is a large literature on synthetic data, as discussed in our paper, this problem remains unresolved up to now, existing methods either fail to control the type I error (e.g., simple pooling), or are too conservative (e.g., GESPI). To solve this problem, we leverage techniques from directional testing, which have been developed for other purposes, and their connection to synthetic data for this problem setup remains underexplored. Thus we agree with the reviewer that our method leverages simple existing techniques, but we would like to highlight our new contribution, which is to develop methods for leveraging synthetic data in a trustworthy way, which we show to have good empirical promise. Thus in a sense, our observations could be viewed as recognizing the perhaps surprising effectiveness of using directional tests for synthetic data.
Additional experiments and ablations
Thank you for this suggestion! To address this concern, we have added additional experiments holding \alpha+\epsilon=0.05 and varying the budget allocation. We evaluate these splits across different levels of synthetic guidance. We observe that our main conclusions hold under these different splits across both aligned and opposite synthetic guidance (see Appendix D.7 in the updated manuscript for details).
In order to choose the optimal allocation in a principled way before observing the real data, we would need to know the alignment of the target signal with the supplied direction, which synthetic data cannot identify under distribution shift. This would necessitate additional assumptions about the relationship between synthetic and real effects or by assuming access to independent validation data. This is a challenge noted in existing works and still an open question for the area. For instance, a related tuning challenge arises in GESPI, which treats its additional error tolerance as application-dependent and recommends heuristic choices informed by synthetic-data quality [1]. Developing allocation rules that adapt to guidance quality under explicit assumptions on the synthetic-target alignment is an important direction for future work. We have updated the draft to add this discussion in Section 2.4 (Lines 216-221).
[1] General Synthetic-Powered Inference. Bashari, et. al, 2025.