acceptodds
Under review as a conference paper at ICLR 2027

Synthetic-Powered Directional Testing

Abstract

Many data-driven decisions rely on tests of composite hypotheses, where alternatives may depart from the null in multiple directions and thus a uniformly most powerful test may not exist. We propose synthetic-powered directional testing (SPDT), a framework that safely leverages synthetic data to increase power in composite hypothesis testing problems. The key idea is to use synthetic data to identify a plausible direction in which to concentrate power, and then combine a directional test with an existing base test. We show that SPDT can yield substantial power gains when the chosen direction is informative, while retaining base power even when it is misleading. At the same time, the procedure inherits the validity guarantees of its component tests within a prespecified total type-I error budget. We demonstrate the benefits of SPDT through simulations and applications spanning LLM safety and treatment effect heterogeneity detection.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Hypothesis testing, statistical inference, synthetic data

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_bZrgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes synthetic-powered directional testing (SPDT), which uses independent synthetic data to select a testing direction and combines a directional test with an existing base test. The authors establish type-I error control and power guarantees for the combined procedure. The method is evaluated through simulations and applications to LLM safety evaluation, treatment effect heterogeneity, and agent safety auditing. I currently lean toward rejection, but would reconsider my rating if these concerns are satisfactorily addressed.

Critical strengths
  1. The paper addresses an interesting problem: using synthetic information to improve hypothesis testing without relying on its distributional correctness.

  2. The framework is simple and applicable to several testing problems.

  3. The experiments cover different settings, including Gaussian mean testing and real-world applications.

Critical weaknesses
  1. The main theoretical guarantees are relatively elementary. Type-I error control follows from the union bound, while base-power preservation follows directly from the OR combination. The contribution beyond combining existing valid tests is not entirely clear.

  2. Power preservation is established relative to the level-α\alpha base test, rather than the base test at the same total significance level α+ε\alpha+\varepsilon. Thus, the claimed protection against misleading synthetic directions is weaker than it may initially appear.

  3. Some empirical results appear inconsistent with the stated combination rule. This raises concerns about the implementation or reporting of the proposed method.

Questions for authors

1. In Table 1 (Page 8), the directional test rejects in 99.1% of subsamples, whereas SPDT rejects in only 91.6%. Similarly, Table 2 reports 61.4% for the directional test but 34.2% for SPDT. Since SPDT rejects whenever either component rejects, shouldn't its rejection frequency be at least as large as that of each component? Are different significance levels being used, or is there an error in the reported results?

2. Proposition 2 guarantees that SPDT preserves the power of the level-α\alpha base test. However, at a fixed total level of 0.05, the relevant comparison is arguably the level-0.05 base test rather than the level-0.04 base. How often does SPDT actually lose power relative to the level-0.05 base when the synthetic direction is inaccurate?

3. Propositions 1–3 follow directly from combining two valid tests using an OR rule. What is the main theoretical contribution beyond this construction? In particular, can the authors characterize when SPDT is more powerful than the level-(α+ε)(\alpha+\varepsilon) base test, rather than the level-α\alpha test?

4. The experiments mainly use α=0.04\alpha=0.04 and ε=0.01\varepsilon=0.01. How sensitive are the power gains to this allocation? For example, would α=ε=0.025\alpha=\varepsilon=0.025 perform better when the synthetic direction is strongly informative? Is there a principled way to choose the allocation before observing the real data?

5. In Section 4.2, the directional test uses a normal approximation with only 50 real queries in each subsample. Appendix D.4 also reports some inflation in the directional test's type-I error. Could the authors clarify whether the observed power improvement remains substantial when the directional component is calibrated more conservatively?

6 comments

Dear Reviewer bZrg,

Thank you for your feedback! We address your concerns below:

The main theoretical guarantees are relatively elementary. Type-I error control follows from the union bound, while base-power preservation follows directly from the OR combination. The contribution beyond combining existing valid tests is not entirely clear. Propositions 1–3 follow directly from combining two valid tests using an OR rule. What is the main theoretical contribution beyond this construction? In particular, can the authors characterize when SPDT is more powerful than the level-(\alpha+epsilon) base test, rather than the level-\alpha test? Proposition 2 guarantees that SPDT preserves the power of the level-\alpha base test. However, at a fixed total level of 0.05, the relevant comparison is arguably the level-0.05 base test rather than the level-0.04 base. How often does SPDT actually lose power relative to the level-0.05 base when the synthetic direction is inaccurate?

To address this question, we have added Propositions 5 and 6, which characterize when SPDT is more powerful than the alpha+epsilon base test in the Gaussian setting. (Full proofs are in Appendix F.2 and G.5). In short, Proposition 5 shows that for a given dimension d and signal strength r, there exists a unique cutoff c_* such that SPDT improve over base test iff c > c_. For example, Table 14 illustrates this cutoff at d=100 and r=3, which is c_=0.153. The base test has 15.82% power. SPDT has 15.10% power at c=0.10 and 29.50% at c=0.50. Proposition 6 gives a sufficient condition for improvement over a range of signal strengths, stating that whenever c > constant x d^{-1/4}, there exists a range of signal strengths (i.e. r) where SPDT will achieve higher power than the base test at alpha+epsilon.

To quantify how often SPDT loses power under inaccurate guidance, we compared its exact power with that of the 5% base test across our Gaussian simulation grid in Appendix D.1. With the 4%/1% allocation, SPDT had lower power with orthogonal or negatively aligned guidance, with a median loss of 1.93 percentage points and maximum loss of 3.85 percentage points.

We would also like to clarify that our contribution and novelty is to develop a method to use synthetic data for reliable statistical inference. While there is a large literature on synthetic data, as discussed in our paper, this problem remains unresolved up to now, existing methods either fail to control the type I error (e.g., simple pooling), or are too conservative (e.g., GESPI). To solve this problem, we leverage techniques from directional testing, which have been developed for other purposes, and their connection to synthetic data for this problem setup remains underexplored. Thus we agree with the reviewer that our method leverages simple existing techniques, but we would like to highlight our new contribution, which is to develop methods for leveraging synthetic data in a trustworthy way, which we show to have good empirical promise. Thus in a sense, our observations could be viewed as recognizing the perhaps surprising effectiveness of using directional tests for synthetic data.

Additional experiments and ablations

The experiments mainly use \alpha=0.04 and \epsilon=0.01. How sensitive are the power gains to this allocation? For example, would \alpha=\epsilon=0.025 perform better when the synthetic direction is strongly informative? Is there a principled way to choose the allocation before observing the real data?

Thank you for this suggestion! To address this concern, we have added additional experiments holding \alpha+\epsilon=0.05 and varying the budget allocation. We evaluate these splits across different levels of synthetic guidance. We observe that our main conclusions hold under these different splits across both aligned and opposite synthetic guidance (see Appendix D.7 in the updated manuscript for details).

In order to choose the optimal allocation in a principled way before observing the real data, we would need to know the alignment of the target signal with the supplied direction, which synthetic data cannot identify under distribution shift. This would necessitate additional assumptions about the relationship between synthetic and real effects or by assuming access to independent validation data. This is a challenge noted in existing works and still an open question for the area. For instance, a related tuning challenge arises in GESPI, which treats its additional error tolerance as application-dependent and recommends heuristic choices informed by synthetic-data quality [1]. Developing allocation rules that adapt to guidance quality under explicit assumptions on the synthetic-target alignment is an important direction for future work. We have updated the draft to add this discussion in Section 2.4 (Lines 216-221).

[1] General Synthetic-Powered Inference. Bashari, et. al, 2025.

In Section 4.2, the directional test uses a normal approximation with only 50 real queries in each subsample. Appendix D.4 also reports some inflation in the directional test's type-I error. Could the authors clarify whether the observed power improvement remains substantial when the directional component is calibrated more conservatively?

Great question! To address this, we have run additional experiments in Appendix D.11 which replaces the normal-based directional component with a one-sided Gaffke test that has finite-sample validity. The observed improvement remains substantial: SPDT rejects in 80.3% of subsets, compared with 14.5% for both Base (5%) and GESPI (5%). We note this is smaller than the original normal-based SPDT rejection frequency, while retaining a substantial gain under finite-sample-valid calibration to resolve the Type-1 error inflation issue. We observe that the Gaffke directional component’s false positive-rate ranges from 0-0.44% at epsilon=1%, compared with the 1.11-2.84% for the original normal component. The resulting SPDT false-positive rate ranges from 0.04-2.54%, below the total 5% level throughout the tested settings.

Clarification points

Some empirical results appear inconsistent with the stated combination rule. This raises concerns about the implementation or reporting of the proposed method. In Table 1 (Page 8), the directional test rejects in 99.1% of subsamples, whereas SPDT rejects in only 91.6%. Similarly, Table 2 reports 61.4% for the directional test but 34.2% for SPDT. Since SPDT rejects whenever either component rejects, shouldn't its rejection frequency be at least as large as that of each component? Are different significance levels being used, or is there an error in the reported results?

We apologize for the confusion and thank you for raising this clarification point. Yes, different significance levels are being used here. For Tables 1 and 2, the standalone directional tests are evaluated at 5%, while the directional component inside SPDT uses 1%. We had used 5% for the standalone tests to compare methods under the same total type-1 error budget, to provide a fair comparison with SPDT’s combined 5% budget. We have updated the manuscript to make this clear and have also added the results for the standalone directional tests at 1%, so that this relationship can be checked directly.

Power preservation is established relative to the level-\alpha base test, rather than the base test at the same total significance level \alpha+\epsilon. Thus, the claimed protection against misleading synthetic directions is weaker than it may initially appear.

Thank you for raising this clarification point! Indeed, Proposition 2 states preservation relative to the level-\alpha base test; it does not claim preservation relative to the level \alpha+\epsilon base test, though our experiments (Figure 2 and applications) compare SPDT against the level-\alpha+\epsilon base test. We have updated the manuscript (Lines 16-17, Line 59, Lines 180-181) to make this qualification more explicit.

We thank the reviewer again for their time and consideration. We are more than happy to address any other concerns they might have.

Thanks for the response. I maintain my original rating.

Reviewer_UeTNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes synthetic-powered directional testing (SPDT), which uses independent auxiliary data to select a promising direction and combines a directional test with a base test. The union rule controls type-I error within the allocated total budget and preserves the base test’s rejections at its allocated level. Simulations and three applications illustrate power gains when the auxiliary direction is informative.

Critical strengths
  1. The framework is simple and broadly applicable, using auxiliary data to guide testing while evaluating the evidence on real observations.

  2. The core guarantees are transparent and valid when the component tests are correctly calibrated, without requiring the auxiliary and real distributions to match.The evaluation covers several testing settings and three applications, with extensive simulation repetitions and explicit reporting of misleading directions and calibration limitations.

Critical weaknesses
  1. The methodological novelty appears limited. The framework combines existing directional tests with a base test, while its main guarantees follow directly from the union bound and inclusion of rejection regions. The contribution beyond this combination needs stronger justification.The empirical evaluation does not sufficiently isolate the benefit of learning directions from synthetic data. In the LLM application, the comparison also changes from coordinatewise Bonferroni testing to an aggregated statistic. Equal-weight and prespecified-direction controls are needed to establish the value of synthetic guidance.

  2. Small-sample calibration is problematic in some settings. Appendix D.2 reports null rejection rates of 7.39% and 7.24% at a nominal 5% level with n = 200 and a larger nuisance model. These failures are particularly relevant to the paper’s motivation of limited real data. Power preservation applies to the base test at its allocated 4% level, rather than the 5% baseline with the same total budget. The authors acknowledge this distinction, but its practical consequences deserve greater emphasis and sensitivity analysis for the budget allocation.

Questions for authors
  1. How much of the application-level gain remains when the learned direction is replaced by an equal-weight or prespecified direction, keeping the testing procedure and error budget unchanged?

  2. Can the authors improve small-sample calibration and show whether the reported power advantages remain after controlling actual type-I error at comparable levels?

  3. How sensitive are the conclusions to the 4%/1% budget split, especially when auxiliary guidance is weak or misleading?

7 comments

Dear Reviewer UeTN,

Thank you for your feedback! We address your concerns below:

The methodological novelty appears limited. The framework combines existing directional tests with a base test, while its main guarantees follow directly from the union bound and inclusion of rejection regions. The contribution beyond this combination needs stronger justification.

As suggested, we clarify our contribution beyond using simple methods. Our main contribution and novelty is to develop a method to use synthetic data for reliable statistical inference. While there is a large literature on synthetic data, as discussed in our paper, this problem remains unresolved up to now, as existing methods either fail to control the type I error (e.g., simple pooling), or are too conservative (e.g., GESPI). To solve this problem, we leverage techniques from directional testing, which have been developed for other purposes, and their connection to synthetic data for this problem setup remains underexplored. Thus we agree with the reviewer that our method leverages simple existing techniques, but we would like to highlight our new contribution, which is to develop methods for leveraging synthetic data in a trustworthy way, which we show to have good empirical promise. Thus in a sense, our observations could be viewed as recognizing the perhaps surprising effectiveness of using directional tests for synthetic data.

Additional experiments and ablations

The empirical evaluation does not sufficiently isolate the benefit of learning directions from synthetic data. In the LLM application, the comparison also changes from coordinatewise Bonferroni testing to an aggregated statistic. Equal-weight and prespecified-direction controls are needed to establish the value of synthetic guidance. How much of the application-level gain remains when the learned direction is replaced by an equal-weight or prespecified direction, keeping the testing procedure and error budget unchanged?

Great point! To address this, we have added an ablation experiment in Appendix D.9, where we replace the synthetic-learned direction with a fixed equal-weight direction as suggested. We observe that although equal-weight aggregation provides some improvement, synthetic-learned weights still produce substantially more rejections. We have added this discussion in Section 4.2 (Lines 385-389) to isolate the value of synthetic guidance from aggregation.

Small-sample calibration is problematic in some settings. Appendix D.2 reports null rejection rates of 7.39% and 7.24% at a nominal 5% level with n = 200 and a larger nuisance model. These failures are particularly relevant to the paper’s motivation of limited real data. Can the authors improve small-sample calibration and show whether the reported power advantages remain after controlling actual type-I error at comparable levels?

Thank you for this suggestion! To address this, we have added additional experiments in Appendix D.10, which improves small-sample calibration in the specified setting (n=200, d=10, sixteen nuisance slopes). For linear DTS, we calibrate the statistic by Monte Carlo simulation under the Gaussian linear null. For logistic DST, we use conditional randomization under the simulation’s known Gaussian covariate distribution.

We observe that recalibration reduces the Type 1 error for linear SPDT from 7.39% to 4.98% and for logistic SPDT from 7.24% to 4.804%. The corresponding Type 1 error rates for the 5% base test are 4.86% and 4.904%, respectively. At these comparable type-I error rates, the power advantage persisted under aligned guidance in both models. Linear SPDT achieved 57.05% power versus 42.10% for the base test, a gain of 14.95 percentage points. Logistic SPDT achieved 56.05% versus 42.30%, a gain of 13.75 points. Under opposite guidance, recalibration left SPDT’s power unchanged in both models, leaving conclusions the same.

Power preservation applies to the base test at its allocated 4% level, rather than the 5% baseline with the same total budget. The authors acknowledge this distinction, but its practical consequences deserve greater emphasis

As suggested, we have expanded our emphasis on its practical consequences to make this qualification more explicit (Lines 16-17, Line 59, Lines 180-181). To further address this, we have also added Propositions 5 and 6, to characterize when SPDT is more powerful than the alpha+epsilon base test in the Gaussian setting. (Full proofs are in Appendix F.2 and G.5). In short, Proposition 5 shows that for a given dimension d and signal strength r, there exists a unique cutoff c_* such that SPDT improve over base test iff c > c_. For example, Table 14 illustrates this cutoff at d=100 and r=3, which is c_=0.153. The base test has 15.82% power. SPDT has 15.10% power at c=0.10 and 29.50% at c=0.50. Proposition 6 gives a sufficient condition for improvement over a range of signal strengths, stating that whenever c > constant x d^{-1/4}, there exists a range of signal strengths (i.e. r) where SPDT will achieve higher power than the base test at alpha+epsilon.

Sensitivity analysis for the budget allocation. How sensitive are the conclusions to the 4%/1% budget split, especially when auxiliary guidance is weak or misleading?

As suggested, we have (1) expanded our emphasis and discussion on the practical consequences in Section 2.4 (Lines 210-221) and (2) added a sensitivity analysis, holding \alpha+\epsilon=0.05 and varying the budget allocation. We observe that our main conclusions hold under these different splits across both aligned and opposite (misleading) synthetic guidance (See Appendix D.7 in the updated manuscript for details).

In practice, the allocation should be fixed independently of the real testing outcomes. A power-optimal choice would depend on the target signal and its alignment with the supplied direction, which synthetic data cannot identify under distribution shift. This would necessitate additional assumptions about the relationship between synthetic and real effects or by assuming access to independent validation data. This is a challenge noted in existing works and still an open question for the area. For instance, a related tuning challenge arises in GESPI, which treats its additional error tolerance as application-dependent and recommends heuristic choices informed by synthetic-data quality [1]. Developing allocation rules that adapt to guidance quality under explicit assumptions on the synthetic-target alignment is an important direction for future work. We have updated the draft to add this discussion in Section 2.4 (Lines 216-221).

[1] General Synthetic-Powered Inference. Bashari, et. al, 2025.

We thank the reviewer again for their time and consideration. We are more than happy to address any other concerns they might have.

Dear authors ,thanks for you reply! I think the format of your rebuttal has not been organized so clearly. I can not capture your information fastly and clearly and I think my concerns has not been solved so efficiently.So I can just maintain my score.

Reviewer_DPsaOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work proposed a synthetic-powered directional testing framework that safely leverages synthetic data to increase power in composite hypothesis testing problems. The key idea is to use synthetic data to identify a plausible direction in which to concentrate power, and then combine a directional test with an existing base test. Overall, the paper is well-written, and the problem is interesting. The procedure is simple and the theory is solid. I enjoyed reading it a lot.

Critical strengths
  1. The testing procedure is easy to implement with gauranteed power gain.
  2. The framework can be adopted in many problems involving smooth loss functions, e.g., CATE estimation and inference.
  3. The compariosn with GESPI is interesting and informative.
Critical weaknesses
  1. The choice of the directional budget needs more discussions.
  2. The syntehtic data is very different from the target data; and thus the additional Type I error control, ϵ\epsilon, may lead to loss of power in certain alternative.
Questions for authors
  1. I feel that a good choice of the directional budget depends on how far the synthetic data it is w.r.t. the target data. Related to this, I have two questions: 1) is there a notion of optimal choice of the directional budget; 2) how should be set the directional budget in practice? Although Section 2.4 provides some discussions on the possible trade-offs, it is hard to directly guide practical implementation without a systematic procedure.

  2. Proposition 2 compares the power of the SPDT to a base test with Type I error controlled at α\alpha. How about it compares to a base test with Type I error controlled at α+ϵ\alpha+\epsilon. As it discussed, GESPI is less powerful compared with a base test with Type I error controlled at α+ϵ\alpha+\epsilon; and the SPDT does not have this constraint. However, can authors show that there is at least one alternative where their power is larger than a base test with Type I error controlled at α+ϵ\alpha+\epsilon; and at least one alternative where their power is less than a base test with Type I error controlled at α+ϵ\alpha+\epsilon?

  3. It would be interesting to see how sample size of the sythetic data affects the testing framework. I feel that the direction can come from anything; even without sythetic data, user can specify an alternative direction of interest. Thus, the logic/motivation of using synthetic data is somewhat weak in the current presentation. Some results related to the power given certain distribution shift between the synthetic data and the target data will strengthen the paper a lot.

4 comments

Dear Reviewer DPsa,

Thank you for your feedback! We are happy to hear that you enjoyed reading our paper a lot! We address your concerns below:

The synthetic data is very different from the target data; and thus the additional Type I error control, ϵ\epsilon , may lead to loss of power in certain alternative. Proposition 2 compares the power of the SPDT to a base test with Type I error controlled at α\alpha. How about it compares to a base test with Type I error controlled at α+ϵ\alpha+\epsilon. As it discussed, GESPI is less powerful compared with a base test with Type I error controlled at α+ϵ\alpha+\epsilon; and the SPDT does not have this constraint. However, can authors show that there is at least one alternative where their power is larger than a base test with Type I error controlled at α+ϵ\alpha+\epsilon; and at least one alternative where their power is less than a base test with Type I error controlled at α+ϵ\alpha+\epsilon?

Thank you for this suggestion! Indeed, in Proposition 2, we only claim baseline power protection with a base test at α\alpha. To address this suggestion, we have added in Appendix F.2 concrete examples of an alternative where the power is larger than a α+ϵ\alpha+\epsilon base test and an alternative where the power is less than a α+ϵ\alpha+\epsilon base test, as suggested. For a Gaussian mean model, we fix the dimension d = 100, signal strength r = 3, and allocation α\alpha = 0.04 and ϵ\epsilon = 0.01. The level-0.05 base test has 15.82% power for every target direction at this signal strength. Let c denote the alignment defined in Line 236 Equation 4, which corresponds to the cosine of the angle between the synthetic direction and the target signal. Holding the synthetic direction fixed, SPDT has 14.23% power when the target signal is orthogonal to it (c = 0) and 29.50% power when the target signal is moderately aligned with it (c = 0.5). Changing only the target direction therefore gives the two alternatives.

To further address your suggestion, we have also added Propositions 5 and 6, which characterize when SPDT is more powerful than the α+ϵ\alpha+\epsilon base test in the Gaussian setting. (Full proofs are in Appendix F.2 and G.5). In short, Proposition 5 shows that for a given dimension d and signal strength r, there exists a unique cutoff c_* such that SPDT improve over base test iff c > c_. For example, Table 14 illustrates this cutoff at d=100 and r=3, which is c_=0.153. The base test has 15.82% power. SPDT has 15.10% power at c=0.10 and 29.50% at c=0.50). Proposition 6 gives a sufficient condition for improvement over a range of signal strengths, stating that whenever c > constant x d^{-1/4}, there exists a range of signal strengths (i.e. r) where SPDT will achieve higher power than the base test at α+ϵ\alpha+\epsilon.

The choice of the directional budget needs more discussions. I feel that a good choice of the directional budget depends on how far the synthetic data it is w.r.t. the target data. Related to this, I have two questions: 1) is there a notion of optimal choice of the directional budget; 2) how should be set the directional budget in practice? Although Section 2.4 provides some discussions on the possible trade-offs, it is hard to directly guide practical implementation without a systematic procedure.

To address this point, we have added additional experiments holding α+ϵ\alpha+\epsilon=0.05 and varying the budget allocation. We evaluate these splits across different levels of synthetic guidance. We observe that our main conclusions hold under these different splits across both aligned and opposite synthetic guidance (see Appendix D.7 in the updated manuscript for details).

In practice, the allocation should be fixed independently of the real testing outcomes. A power-optimal choice would depend on the target signal and its alignment with the supplied direction, which synthetic data cannot identify under distribution shift. This would necessitate additional assumptions about the relationship between synthetic and real effects or by assuming access to independent validation data. This is a challenge noted in existing works and still an open question for the area. For instance, a related tuning challenge arises in GESPI, which treats its additional error tolerance as application-dependent and recommends heuristic choices informed by synthetic-data quality [1]. Developing allocation rules that adapt to guidance quality under explicit assumptions on the synthetic-target alignment is an important direction for future work. We have updated the draft to add this discussion in Section 2.4 (Lines 216-221).

[1] General Synthetic-Powered Inference. Bashari, et. al, 2025.

I feel that the direction can come from anything; even without sythetic data, user can specify an alternative direction of interest.

We apologize for the confusion! We have updated the draft to more clearly state that synthetic data is one source of external guidance, rather than a requirement of the framework. As described in Section 2.1 (Lines 78-79), auxiliary information may come from generated data or independent real data, and its distribution need not match the target distribution. Indeed, a direction specified through domain knowledge is also admissible.

It would be interesting to see how sample size of the sythetic data affects the testing framework. Some results related to the power given certain distribution shift between the synthetic data and the target data will strengthen the paper a lot.

Thank you for this suggestion! To address this, we have added an additional experiment in Appendix D.8 to report performance varying levels of the synthetic data sample size, while varying levels of distribution shift between the synthetic and target distribution. We observe that as the synthetic sample grew, SPDT power increased from 86.0% to 96.5% under full alignment and from 74.7% to 79.2% under partial alignment. Under opposite alignment, SPDT power stayed between 80.6% and 80.8% across all evaluated sample sizes. Thus, more synthetic data improved power even when the treatment-effect pattern partially transferred, but offered little benefit when the pattern was opposite, as expected.

Our existing two-sample testing simulation in Appendix Section D.3 also examines synthetic sample size jointly with distributional mismatch. We compare 500 vs. 4,500 auxiliary observations per group, while varying the alignment between the auxiliary and target population contrasts from aligned to orthogonal and opposite. These experiments show that additional synthetic observations can reduce uncertainty in estimating the auxiliary direction, while distribution shift determines whether that direction is informative for the target. Increasing synthetic sample size alone does not guarantee better target alignment or uniformly higher power under arbitrary shift, as expected. We have updated the manuscript (Section 4.1 Lines 320-323) to highlight this distinction and sample-size and alignment experiments more clearly in the main text.

We thank the reviewer again for their time and consideration. We are more than happy to address any other concerns they might have.

I would like to thank authors for addressing my concerns. In this paper, the methodology is simple, and the theory used is elementary. However, I appreciate the proposed idea and believe that it will be practically useful; and thus I move to acceptance.

Thank you for confirming that our response has addressed your concerns! We sincerely appreciate your valuable time and feedback. Best wishes

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.