acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Two-Sample Testing via Dependence-Aware Witness Aggregation

Abstract

Two-sample testing is a fundamental problem in statistics and machine learning. Existing tests are typically powerful only against particular classes of alternatives. Since the underlying discrepancy is unknown in practice, relying on a single test can lead to substantial power loss. We introduce an adaptive two-sample testing framework that combines complementary procedures through a unified witness-function perspective. Our key observation is that many seemingly heterogeneous tests can be represented by witness functions whose expectation differences reveal discrepancies between two distributions. This representation enables their signals to be aggregated within a common framework. To handle the dependence among witness statistics evaluated on the same data, we develop a dependence-aware aggregation procedure. We establish asymptotic Type-I error control and show that the proposed test is consistent whenever at least one constituent witness carries sufficiently strong signal. Extensive experiments across diverse distributional shifts demonstrate that our method consistently delivers competitive or superior power across regimes while maintaining valid Type-I error control.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Two-sample testing, witness functions, adaptive testing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XGBQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper combines MMD, ED, KFDA, and PWD through learned witness functions, using covariance adjustment to account for dependence among witness statistics evaluated on the same testing sample. It establishes asymptotic Type-I error control and consistency when at least one witness carries sufficiently strong signal. Numerical experiments demonstrate competitive or superior power across diverse distributional shifts while maintaining Type-I error control.

Critical strengths

The unified witness perspective provides a useful framework for combining complementary procedures, including MMD, ED, KFDA, and PWD, while accounting for dependence among their statistics. Theorem 1 provides a quantitative Gaussian approximation bound under stated conditions. This relaxes the exponential-moment requirement used for the corresponding bound in Hu & Lin (2026) in certain regimes.

Critical weaknesses
  1. The methodological novelty over Hu & Lin (2026) requires further clarification. While the relaxed moment conditions provide a useful theoretical contribution, Max-Cov has the same form as their Equation (10). The main methodological addition appears to be combining witnesses from different procedures, and the significance of this extension needs stronger justification.

  2. The theoretical guarantees hold for a fixed number of witnesses and require a uniformly nondegenerate correlation matrix. It remains unclear whether they extend to larger collections that contain highly correlated or redundant witnesses. Moreover, the pseudoinverse used in implementation does not carry these guarantees over to settings where population nondegeneracy fails.

Questions for authors
  1. The benefit of combining witnesses from different procedures needs clearer justification. In similar regimes, Hu & Lin (2026) already aggregate PWD witnesses generated under different parameter configurations using the same form of statistic. Could a collection containing only PWD witnesses, matched in size to the proposed collection, achieve comparable power? What gains can be attributed specifically to including witnesses from other procedures? The comparison should use the same number of witnesses. The current PWD baseline uses nine witnesses, while the proposed aggregation seems to use twelve, so the gains may partly come from using more witnesses.

  2. The construction and selection of the witness collection needs clearer guidance. Why were MMD, ED, KFDA, and PWD chosen? A paragraph or two explaining this choice may help. Furthermore, what properties must a new witness have to be included, and how can Assumption 2 be verified for the combined collection? Could the selection or aggregation of candidate witnesses be optimized for test power in broader regimes?

  3. The implementation requires further assessment of numerical stability and computational cost. How are eigenvalues close to zero handled when computing the pseudoinverse? How sensitive are Type-I error and power to this choice? Could the authors also report how runtime changes as the number of witnesses increases?

  4. The sample-splitting strategy in Appendix B.2 needs clearer justification and further assessment of its effect on test performance. Why was the sample splitting ratio set to 0.5, and how sensitive are Type-I error and power to this ratio and the random split? Could this ratio be optimized to improve test power while maintaining Type-I error control?

  5. The code for reproducing the experiments does not appear to be available. Could the authors provide anonymized code during the rebuttal period?

5 comments
AuthorsOpenReview ↗

Response to Reviewer XGBQ (1)

We thank the reviewer for the appreciation of the value of our unified witness-based framework and theoretical contributions. Below we address the weaknesses (W) and questions (Q) raised by the reviewer.

W1. Novelty relative to Hu and Lin [1]

Hu and Lin [1] develop neural Wasserstein two-sample tests and aggregate PWD witnesses across hyperparameter configurations to adapt to unknown sparsity levels and projection dimensions. In contrast, our framework addresses the broader challenge of test selection when the underlying discrepancy structure is unknown. We adopt a general witness-based perspective that accommodates witnesses derived from different discrepancy measures and testing principles, including constructions beyond the conventional IPM framework. By aggregating complementary witnesses from heterogeneous procedures, including MMD, ED, KFDA, and PWD, our method extends adaptivity beyond a single discrepancy family, aiming to achieve robust power across diverse alternatives. While our Max-Cov statistic builds on the covariance-whitened maximum aggregation of [1], we also investigate two alternative aggregation statistics, Max-Var and Hotelling, within the same general framework.

We extend the theoretical analysis of [1] in three aspects. First, we establish the asymptotic null distribution and a Berry--Esseen type bound under weaker normalized fourth-moment conditions, without restricting the witness learning to PWD. Second, we derive a generic consistency guarantee whenever at least one constituent witness has sufficiently strong signal relative to its estimation errors in Theorem 2, and specialize this result to the witnesses considered in our experiments in Appendix G of the revised version (Appendix D of original submission). Third, we show that even when the positive-definiteness assumption is violated, the pseudoinverse-based statistics TnT_n and Tn′T_n^{\prime} achieve asymptotically conservative Type-I error control, see Appendix C. Together, these results extend both the applicability and theoretical understanding of covariance-whitened witness aggregation.

We have revised the Introduction to make these distinctions explicit and more accurately position our work relative to the existing literature.

W2. Theoretical guarantees with correlated or redundant witnesses

The nondegeneracy of the correlation matrix is a standard regularity condition for whitening, which is used to establish the asymptotically pivotal null distribution. However, it is not necessary for conservative Type-I error control. In Appendix C of the revised manuscript, we extend our theoretical analysis to show that, the pseudoinverse-based statistics TnT_n and Tn′T_n^{\prime} yield asymptotically conservative Type-I error control even when the population covariance matrix is singular. Alternatively, permutation calibration provides finite-sample Type-I error control without requiring nondegenerate correlation matrix.

We further provide numerical experiments to examine the effects of redundant collinear witnesses in Appendix D.4. Specifically, we introduce an exact duplicate of the MMD witness into the aggregation procedure. As shown in Table R1, the empirical Type-I error rates and power remain largely unchanged across the considered settings, suggesting that the inclusion of an identical witness does not substantially affect the performance of our procedure.

Table R1: Comparison of empirical rejection rates with and without a duplicated witness.

ModelCollinearCollinearOriginalOriginal
Max-CovHotellingMax-CovHotelling
Null
Gaussian0.0630.0600.0670.070
Gaussian mixture0.0430.0400.0670.060
Alternative
A, r=1r=10.9630.9840.9700.987
A, r=30r=301.0001.0001.0001.000
B, τ=0.1\tau=0.10.9430.9470.9430.957
B, τ=0.7\tau=0.70.9971.0001.0001.000
C, τ=0.1\tau=0.11.0001.0001.0001.000
C, τ=0.7\tau=0.70.2070.3070.2170.353
D, β=0.2\beta=0.20.6600.7800.6700.803
D, β=0.5\beta=0.50.9971.0000.9971.000
E, β=0.2\beta=0.20.8770.9300.8930.937
E, β=0.5\beta=0.51.0001.0001.0001.000
AuthorsOpenReview ↗

Response to Reviewer XGBQ (2)

Q1. Benefits of combining witnesses from different procedures

Thank you for the comment. As suggested, we provide comparisons with PWD baseline containing 12 witnesses, obtained by including an additional sparsity parameter to match the size of our proposed collection. As shown in Table R2, the PWD baseline still exhibits substantial power loss under several alternatives, particularly Model A with r=30r=30 and Model B with τ=0.7\tau=0.7. In contrast, our method achieves more robust power across diverse alternatives, demonstrating the benefit of aggregating witnesses from heterogeneous procedures rather than simply increasing the number of witnesses. These additional experiments are included in Appendix D.6 of the revised manuscript.

Table R2: Empirical rejection rates of different methods.

ModelPWD (12)PWD (9)Max-CovMax-VarHotelling
Null
Gaussian0.0630.0630.0670.0470.070
Gaussian mixture0.0600.0730.0670.0570.060
Alternative
A, r=1r=10.9330.9800.9700.6370.987
A, r=30r=300.3830.3831.0000.9871.000
B, τ=0.1\tau=0.10.9500.9500.9430.9530.957
B, τ=0.7\tau=0.70.1930.1301.0000.9431.000
C, τ=0.1\tau=0.10.9971.0001.0001.0001.000
C, τ=0.7\tau=0.70.2700.2500.2170.5570.353
D, β=0.2\beta=0.20.6770.7000.6700.5100.803
D, β=0.5\beta=0.50.9930.9970.9970.9701.000
E, β=0.2\beta=0.20.8370.8330.8930.2770.937
E, β=0.5\beta=0.50.9901.0001.0000.4631.000

Q2. Construction and selection of the witness collection

We provide further clarification on the choice of witnesses, the requirements for incorporating new witnesses, the assessment of Assumption 2, and potential strategies for optimizing witness selection. We have also added the relevant discussion to the revised manuscript, see Section 5 and Appendix F.3.

Choice of witnesses. We selected MMD, ED, KFDA, and PWD to represent complementary approaches to two-sample testing. MMD captures discrepancies through kernel mean embeddings, ED through pairwise distances, KFDA through covariance-aware discrimination in an RKHS, and PWD through learned Wasserstein projections with different structural constraints. These methods emphasize different aspects of distributional differences and may exhibit complementary power across alternatives. Our goal is not to identify an optimal fixed collection, but to demonstrate that heterogeneous witnesses can be combined within a unified adaptive testing framework.

Requirements for new witnesses. Our framework is highly flexible in incorporating new witnesses, provided they are learned independently of the testing sample. For the Gaussian calibration, the moment and nondegeneracy conditions in Assumption 2 ensure asymptotic validity. In practice, the empirical correlation matrix and standardized fourth moments provide useful diagnostics, although they cannot rigorously verify the population assumptions (see our response below on assessment of Assumption 2). Moreover, our extended theoretical analysis establishes asymptotic Type-I error control for the pseudoinverse-based Max-Cov statistic, even when the population covariance matrix is singular. Alternatively, permutation calibration provides finite-sample Type-I error control under the null exchangeability assumption, without requiring moment or nondegeneracy conditions. For consistency, at least one constituent witness must provide sufficiently strong discriminative signal, as characterized in Theorem 2.

Assessment of Assumption 2. In practice, the empirical correlation matrix and standardized fourth moments provide useful diagnostics, although they cannot rigorously verify the population assumptions. If the estimated correlation matrix R^\hat R is nearly singular, one possible remedy is backward elimination, as suggested in [1]. Specifically, let R^−j\hat R_{-j} be the sample correlation after excluding the jj-th witness. We remove the witness indexed by j′=arg⁡max⁡jλmin(R^−j)j^\prime= \arg\max_{j}\lambda_{min}(\hat R_{-j}) and continue this process recursively until the retained collection is sufficiently well conditioned.

Potential optimization of witness selection. The witness collection could be further optimized using an independent validation sample to select candidates based on their estimated discriminative strength and dependence structure. Such selection may improve power by retaining informative and complementary witnesses. Importantly, selection must be performed independently of the final testing sample to preserve the validity of our existing calibration. Developing a systematic power-optimal selection strategy is an interesting extension beyond the scope of the present work.

AuthorsOpenReview ↗

Response to Reviewer XGBQ (3)

Q3. Numerical stability and runtime

We conduct additional experiments to assess the numerical stability of the pseudoinverse computation in Appendix D.7 and examine how runtime changes as the number of witnesses increases in Appendix D.9.

Numerical stability. We thank the reviewer for raising this concern. In our implementation, we compute the pseudoinverse using PyTorch's torch.linalg.pinv() with its default tolerance, which treats eigenvalues below a numerical threshold as zero. To assess numerical stability, we conduct additional experiments with different truncation thresholds: 1e-17, 2.66e-15 (default), 1e-13, 1e-10, 1e-8, 1e-6 and 1e-4. As shown in Table R3, the empirical Type-I error rates and power remain stable for thresholds up to 1e-10. Larger thresholds generally lead to more conservative Type-I error rates and reduced power in some settings. Overall, the results suggest that the proposed procedures are reasonably robust to the truncation threshold, provided it is reasonably small.

Table R3: Sensitivity analysis of empirical rejection rates to the truncation thresholds.

ModelMethod1e-172.66e-151e-131e-101e-81e-61e-4
Null
GaussianMax-Cov0.0670.0670.0670.0670.0570.0530.047
Hotelling0.0700.0700.0700.0700.0570.0400.027
Gaussian mixtureMax-Cov0.0670.0670.0670.0670.0530.0500.037
Hotelling0.0600.0600.0600.0600.0470.0300.027
Alternative
A, r=1r=1Max-Cov0.9700.9700.9700.9700.9700.9700.970
Hotelling0.9870.9870.9870.9870.9870.9800.980
A, r=30r=30Max-Cov1.0001.0001.0001.0001.0001.0001.000
Hotelling1.0001.0001.0001.0001.0001.0001.000
D, β=0.2\beta=0.2Max-Cov0.6700.6700.6700.6700.6630.6570.583
Hotelling0.8030.8030.8030.8030.7870.7230.670
D, β=0.5\beta=0.5Max-Cov0.9970.9970.9970.9970.9970.9970.993
Hotelling1.0001.0001.0001.0001.0001.0001.000

Runtime. Our procedure consists of two main stages, witness estimation using the training data and inference using the testing data. Table R4 reports the estimation and inference times for MMD, ED, KFDA, PWD, and our proposed Max-Cov procedure. The PWD contains 9 witnesses, whereas Max-Cov combines 12 witnesses, allowing us to examine how runtime changes as the number of witnesses increases. The experiments are conducted on a server equipped with an Intel Xeon Platinum 8488C CPU and a single NVIDIA GeForce RTX 4090 GPU.

The computational cost of estimation depends on the witness construction method. For MMD, ED, and KFDA, the witnesses admit explicit formulas, whereas PWD requires numerical optimization. Consequently, the runtime of our method is dominated by PWD witness estimation. Despite including three additional witnesses, our method has a total runtime comparable to the PWD baseline. The inference stage involves computing the test statistic and its critical value, which is generally much less computationally demanding than witness estimation.

Table R4: Runtime comparison (seconds) under Model A with r=1r=1 over 20 runs, with standard deviations in parentheses.

MMDEDKFDAPWDMax-Cov
Estimation0.01720.01100.068044.809544.9058
(0.0015)(0.0009)(0.0034)(1.8832)(1.8866)
Inference0.00110.00050.00050.00400.0055
(0.0021)(0.0001)(0.0001)(0.0005)(0.0005)
Total0.01830.01150.068544.813444.9112
(0.0024)(0.0009)(0.0034)(1.8835)(1.8869)
Reviewer_U2r1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes to combine multiple two-sample testing procedures by aggregating learned witness functions. Its aggregated test statistic, Max-Cov, whitens the vector of mean witness differences by an estimated covariance matrix and then takes the maximum absolute coordinate. They establish asymptotic Type-I error control for the proposed test and show that it is consistent whenever at least one witness carries sufficiently strong signal.

Critical strengths
  • Combining witnesses trained by different procedures is a useful idea as it harnesses power from them when combined carefully
  • The theoretical analysis is rigorous and the paper is easy to follow
  • The experimental gains are strong (at least against the evaluated baselines)
Critical weaknesses

The paper could benefit from better positioning their work with respect to the prior literature. I also strongly advise revising the introduction and list of contributions.

  • The already cited work of Hu and Lin [1] uses the same covariance-whitened absolute maximum and independent-Gaussian critical value, applied to multiple PWD witnesses. The manuscript extends this to heterogeneous witnesses, together with a more general normalized fourth-moment formulation. However, such strong overlap should be made very clear in the introduction itself.

  • The work also overstates its contributions regarding "their key observation" that most of these tests can be viewed under the IPM framework. This observation is ubiquitous in the prior literature [2, 3]. In a similar vein, Bansal et al (2026) [4] derive robustness bounds for imperfect density-ratio estimates of the Hu and Lei's Conformal C2ST (RT in the current manuscript). It is also uncited.

[1] Hu, X., & Lin, Z. (2026). Neural Wasserstein Two-Sample Tests. arXiv preprint arXiv:2601.21732.

[2] Liu, F., Xu, W., Lu, J., Zhang, G., Gretton, A. & Sutherland, D.J.. (2020). Learning Deep Kernels for Non-Parametric Two-Sample Tests. <i>Proceedings of the 37th International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 119:6316-6326 Available from https://proceedings.mlr.press/v119/liu20m.html.

[3] Kübler, J.M., Jitkrittum, W., Schölkopf, B. & Muandet, K.. (2022). A Witness Two-Sample Test . <i>Proceedings of The 25th International Conference on Artificial Intelligence and Statistics</i>, in <i>Proceedings of Machine Learning Research</i> 151:1403-1419 Available from https://proceedings.mlr.press/v151/kubler22a.html.

[4] Bansal, V., Chen, T. & Scott, J.G.. (2026). Conformal C2ST: Turning weak classifiers into strong two-sample tests. <i>Proceedings of the 43rd International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 306:6325-6352 Available from https://proceedings.mlr.press/v306/bansal26a.html.

Questions for authors
  • Which witness representations are new relative to Liu et al., Kübler et al., and the ED/MMD literature? How novel is the aggregation rule and theory relative to the Hu and Lin? Please align the Introduction and contribution claims with those distinctions.
  • On the experiments: Appendix B notes that MMD and ED witnesses are normalized, KFDA is left on its fitted scale, and spectral normalization is used for networks. However, symmetric covariance whitening followed by an infinity norm is sensitive to rescaling and it matters for aggregating heterogeneous witnesses whose scales differ. Could the authors provide justification via a sensitivity analysis?
  • Can the authors also show experimental comparison results for RT and C2ST?
5 comments
AuthorsOpenReview ↗

Response to Reviewer U2r1 (1)

We thank the reviewer for appreciating the value of our approach, the rigor of our theoretical analysis and the strong experimental gains. Below we address the weaknesses (W) and questions (Q) raised by the reviewer.

W1. Connection to Hu and Lin [1]

We agree that the original manuscript did not sufficiently emphasize the connection to Hu and Lin [1]. This work develops neural Wasserstein two-sample tests and aggregates multiple PWD witnesses across hyperparameter configurations using a covariance-whitened absolute maximum statistic and independent-Gaussian critical values. Our Max-Cov statistic builds on this aggregation principle, and we do not claim novelty for the aggregation rule itself.

Our main methodological extension is to move beyond aggregating witnesses from a single discrepancy family. While Hu and Lin [1] aggregate PWD witnesses to adapt to unknown sparsity levels and projection dimensions, our framework combines heterogeneous witnesses derived from different testing procedures, including MMD, ED, KFDA, PWD, RT and ff-divergences, to address test selection under unknown discrepancy structures. We also investigate two alternative aggregation statistics, Max-Var and Hotelling, within this general framework.

We have revised the Introduction to explicitly acknowledge the methodological connection to Hu and Lin [1] and clarify that our contribution lies in extending covariance-aware witness aggregation to heterogeneous testing procedures.

W2. Witness-based perspective and additional related work

We acknowledge that witness functions and their connections to two-sample testing are well established. In particular, MMD and ED admit known witness representations, Liu et al. [2] develop deep-kernel two-sample tests, and Kübler et al. [3] construct kernel-based witness tests. We do not claim novelty for these existing representations.

Our framework takes a broader witness-based perspective: rather than restricting attention to discrepancies arising from the IPM framework, we only require witness functions that exhibit differences in expectations under PP and QQ. While some existing procedures naturally admit such representations, we further illustrate how discriminative witnesses can be constructed from other testing principles, including rank-based tests (RT) and ff-divergences (see Appendix E). This enables us to incorporate witnesses originating from different discrepancy measures and optimization objectives into a common aggregation framework.

By representing these witnesses through a vector of mean differences, our method aggregates complementary evidence while accounting for dependence among witnesses. The main contribution is therefore not the individual witness representations, but a general adaptive testing framework that leverages heterogeneous witnesses to achieve robust power across diverse alternatives. We have revised the Introduction and contribution statements to clarify this distinction and more accurately position our work relative to existing literature.

We also thank the reviewer for bringing the relevant work Bansal et al. [4] to our attention. We have cited this work in the Introduction.

AuthorsOpenReview ↗

Response to Reviewer U2r1 (2)

Q1. Novelty of witness representations, aggregation, and theory

We thank the reviewer for requesting a clearer clarification of our contributions. We distinguish them at three levels.

Witness representations. We do not claim novelty for existing witness constructions, including those associated with MMD, ED and kernel-based witness tests [3]. Instead, we provide a unified formulation that accommodates witnesses from heterogeneous testing principles. In addition to standard discrepancy-based witnesses, we illustrate constructions based on rank-based tests and ff-divergences (Appendix E).

Aggregation methodology. As clarified in the response to W1, the covariance-whitened maximum aggregation rule was already developed in Hu and Lin [1]. Our methodological contribution is to extend its application from multiple PWD witnesses to heterogeneous witnesses originating from different testing procedures. We also consider Max-Var and Hotelling statistics as alternative aggregation strategies within the same framework.

Theoretical analysis. We extend the theoretical analysis of [1] in three aspects. First, we establish the asymptotic null distribution and a Berry--Esseen type bound under weaker normalized fourth-moment conditions, without restricting the witness learning to PWD. Second, we derive a generic consistency guarantee whenever at least one constituent witness has sufficiently strong signal relative to its estimation errors in Theorem 2, and specialize this result to the witnesses considered in our experiments in Appendix G of the revised version (Appendix D of original submission). Third, we show that even when the positive-definiteness assumption is violated, the pseudoinverse-based statistics TnT_n and Tn′T_n^{\prime} achieve asymptotically conservative Type-I error control, see Appendix C. Together, these results extend both the applicability and theoretical understanding of covariance-whitened witness aggregation.

We have revised the Introduction, contribution statements and related-work discussion to make these distinctions explicit and more accurately position our work relative to the existing literature.

Q2. Sensitivity to witness scaling

Thanks for the comment. Max-Var and Hotelling are invariant to nonzero rescaling of individual witnesses under their standard formulations, whereas Max-Cov is generally not scale-invariant. To assess its practical sensitivity, we conduct additional experiments replacing the normalized MMD and ED witnesses with their unnormalized counterparts. As shown in Table R1, the empirical Type-I error rates and power remain largely unchanged, indicating that Max-Cov is empirically robust to witness scaling. These results are included in Appendix D.5 of the revised manuscript.

Table R1: Empirical rejection rates with and without witness normalization.

ModelUnnormalizedNormalized
Null
Gaussian0.0670.067
Gaussian mixture0.0670.067
Alternative
A, r=1r=10.9700.970
A, r=30r=301.0001.000
B, τ=0.1\tau=0.10.9430.943
B, τ=0.7\tau=0.71.0001.000
C, τ=0.1\tau=0.11.0001.000
C, τ=0.7\tau=0.70.2140.217
D, β=0.2\beta=0.20.6730.670
D, β=0.5\beta=0.50.9970.997
E, β=0.2\beta=0.20.8970.893
E, β=0.5\beta=0.51.0001.000
AuthorsOpenReview ↗

Response to Reviewer U2r1 (3)

Q3. Comparison with RT and C2ST

We provide additional comparisons with C2ST [5] and RT with neural networks [6] in Table R2. Both methods exhibit limited power across most of the considered settings. These results are included in Appendix D.2 of the revised manuscript.

Table R2: Empirical rejection rates of different methods.

ModelC2STRT-NNMax-CovMax-VarHotelling
Null
Gaussian0.0200.0400.0670.0470.070
Gaussian mixture0.0470.0470.0670.0570.060
Alternative
A, r=1r=10.0970.0600.9700.6370.987
A, r=30r=300.0630.0601.0000.9871.000
B, τ=0.1\tau=0.10.0430.0470.9430.9530.957
B, τ=0.7\tau=0.70.0570.0471.0000.9431.000
C, τ=0.1\tau=0.10.2200.5571.0001.0001.000
C, τ=0.7\tau=0.70.3000.3470.2170.5570.353
D, β=0.2\beta=0.20.1930.0430.6700.5100.803
D, β=0.5\beta=0.50.7630.0430.9970.9701.000
E, β=0.2\beta=0.20.0600.0670.8930.2770.937
E, β=0.5\beta=0.50.1100.0631.0000.4631.000

[1] Hu and Lin, Neural Wasserstein Two-Sample Tests. arXiv:2601.21732.

[2] Liu et al., Learning Deep Kernels for Non-Parametric Two-Sample Tests. ICML 2020.

[3] Kübler et al., A witness two sample test. AISTATS 2022.

[4] Bansal et al., Conformal C2ST: Turning weak classifiers into strong two-sample tests. ICML 2026.

[5] Lopez-Paz and Oquab, Revisiting classifier two-sample tests. ICLR 2017.

[6] Hu and Lei, A two-sample conditional distribution test using conformal prediction and weighted rank sum. JASA 2024.

Reviewer_r1YzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes aggregating over learned witness functions, and a test statistic which whitened the empirical witness mean difference for the learned witnesses. The resultant test is calibrated and consistent whenever an individual constituent test is consistent.

Critical strengths
  1. The general motivation of so-called "structures of distributional discrepancy" is sound and relevant. It is important conceptually for practitioners to understand the correct test to use in high-dimensional setting. A general recipe to combine hypothesis tests with diverse properties, if executed properly and compared extensively with existing methods, could be a useful machinery in practice.
Critical weaknesses
  1. The paper did not engage sufficiently with prior literature on aggregation in testing. The claim that prior work only addressed aggregation of MMD tests is not true, as shown by (Ribero et al. Regularized f-Divergence Kernel Tests) which aggregated f-divergence tests.

  2. Importantly, no comparison is made, empirically or theoretically, with neural network-based two-sample testing, which naturally exhibit adaptivity to variations in distribution and are designed with high dimensional data in mind.

Questions for authors

The first question concerns comparison with related work.

  1. Comparison with work using spectral regularization to improve the separation radius and power of MMD based hypothesis test. The prominent example in this line of work is the spectrally regularized MMD test (Hagrass, Srimperumbudur and Li), which also aims to improve two-sample testing by leveraging covariance information discarded by the MMD, and directly addresses the challenge mentioned in the introduction, namely that difference may be weak but spread across many coordinates? This line of work employs a whitening step which is also very similar to the whitening procedure used in the paper, and connects directly with KFDA but it is not cited or discussed.
  2. Relatedly, the claim that prior work only considered aggregation of MMD tests is not true. Aggregation is a general principle to improve the power of hypothesis testing that has been applied broadly. See for instance Ribero et al. Regularized f-Divergence Kernel Tests for aggregating tests based on f-divergences, and application to diverse fields such as machine unlearning and differential privacy.
  3. There is no empirical comparison with MMDAgg or other publicly available aggregation method. Therefore the empirical simulation cannot be deemed to be complete.
  4. Working with the witness function itself is a natural idea. I'm less familiar with the testing literature, but in gradient flows it is very very common to estimate or learn the witness function itself. A recent example is Tian et al. Sobolev Regularized MMD Gradient Flow. Yet the only relevant citations address failings of p-value combination, and does not provide any discussion of prior efforts to learn witness functions or combine learned witness functions.
  5. Crucially, there is no comparison with "deep" two-sample tests, for instance work by Kirchler et al. Two-sample Testing Using Deep Learning and Liu et al. Learning Deep Kernels for Non-Parametric Two-Sample Tests. These tests are explicitly designed to address the problem being studied here, which is to scale to high dimensional data and adapt to features in the dataset in order to improve power. These methods are well cited and subsequent work has built on them. The adaptivity of neural networks to low dimensional features is not only folklore in practice but also well established in ML theory, e.g. Suzuki and Nitanda Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov space. Neither the literature review nor the empirical evaluation cannot be deemed complete without a comparison against deep two-sample tests.
  6. High dimensional two-sample testing is in fact a deep and active area of research, a quick search brought up Yan and Zhang Kernel Two-Sample Tests in High Dimension: Interplay Between Moment Discrepancy and Dimension-and-Sample Orders, Gao and Shao Two Sample Testing in High Dimension via Maximum Mean Discrepancy and a few others classical and recent work. These work rigorously established guarantees under theoretical high dimensional asymptotics. The aforementioned deep neural net based work provided practical method with strong empirical performance. It is regrettable that in this work the word high-dimensional appeared to be used in a very loose way, which is backed up only by weak empirical simulations. Moreover, it is not clear to me at all whether the assumptions made in the theoretical analysis, such as Assumption 2, would continue to hold in truly high dimensional setting.
  7. The requirement that V^{-1/2}Sigma V^{-1/2} is dominated below in Lowener order by a constant multiple of identity matrix caused significant confusion for me. Is this an implicit requirement that the constituent witness functions cannot be collinear in some sense? If so, this appears rather strong. Can the authors address how the method would occur if I unwittingly include two identical witness functions in the procedure, or an additional learned witness function nearly lies in the span of the existing ones? I would like to see an empirical or theoretical response to this question.
10 comments
AuthorsOpenReview ↗

Response to Reviewer r1Yz (1)

We thank the reviewer for recognizing the motivation of our work. Below we address the weaknesses (W) and questions (Q) raised by the reviewer.

Q1. Comparison with spectrally regularized MMD test

We thank the reviewer for pointing out this relevant work. Spectrally regularized MMD (srMMD) [1] incorporates covariance information through spectral regularization to improve the detection power and separation rates of MMD-based tests. We agree that this approach is related to KFDA and shares similarities with the whitening procedure used in our framework.

However, the two approaches address distinct but complementary objectives. While srMMD aims to enhance the sensitivity of a kernel-based test to certain alternatives, our work focuses on the broader challenge of test selection under unknown discrepancy structures. Rather than improving an individual test, our framework aggregates heterogeneous witnesses derived from different testing procedures, aiming to achieve robust power across diverse alternatives.

The role of covariance information also differs. srMMD applies spectral regularization to the kernel covariance operator to improve the detection of distributional discrepancies, whereas our method exploits the joint covariance structure of heterogeneous witness mean differences to construct an aggregated test with an asymptotically pivotal null distribution. Thus, despite their shared use of covariance information, the two methods differ in both their constructions and primary objectives.

We have also conducted additional comparisons with srMMD. As shown in Table R1, srMMD achieves high power under some alternatives, such as Model A with r=30r=30, Model B with τ=0.7\tau=0.7 and Model C with τ=0.7\tau=0.7, but exhibits substantially lower power in other settings. In contrast, our method generally achieves more robust performance across diverse alternatives. Full comparison results are provided in Appendix D.2 of the revised manuscript. We have also cited this work in the Introduction and provided further discussion in Appendix A.

Table R1: Empirical rejection rates of different methods.

ModelsrMMDff-AggMMD-DDMMDMax-CovMax-VarHotelling
Null
Gaussian0.0070.0170.0770.0730.0670.0470.070
Gaussian mixture0.0230.0530.0370.0430.0670.0570.060
Alternative
A, r=1r=10.3130.2600.3030.0430.9700.6370.987
A, r=30r=301.0001.0001.0000.0571.0000.9871.000
B, τ=0.1\tau=0.10.0500.1170.0800.0570.9430.9530.957
B, τ=0.7\tau=0.71.0001.0000.9930.0571.0000.9431.000
C, τ=0.1\tau=0.10.5270.9270.4800.1471.0001.0001.000
C, τ=0.7\tau=0.70.4770.7300.3900.2070.2170.5570.353
D, β=0.2\beta=0.20.0470.6830.0900.1000.6700.5100.803
D, β=0.5\beta=0.50.4270.9970.6700.4570.9970.9701.000
E, β=0.2\beta=0.20.1230.1570.0670.0730.8930.2770.937
E, β=0.5\beta=0.50.8900.5830.1400.1501.0000.4631.000
AuthorsOpenReview ↗

Response to Reviewer r1Yz (2)

W1/Q2. Clarification and expanded discussion of related aggregation methods

We would like to clarify a possible misunderstanding: our original submission did not claim that prior aggregation methods were restricted to MMD tests. In fact, we explicitly discussed the aggregation of heterogeneous tests through their pp-values in the Introduction. We also reviewed MMDAgg [1] and MMD-FUSE [2] in Appendix A as representative aggregation methods closely related to our setting. Empirical comparisons were provided with conventional pp-value aggregation methods in Section 5 and with MMDAgg and MMD-FUSE in the Appendix.

We agree that the discussion of existing aggregation approaches could be more comprehensive. We thank the reviewer for bringing regularized ff-divergence kernel tests [3] to our attention. This work develops kernel-based estimators and permutation tests for ff-divergences, together with an aggregation procedure ff-Agg that combines tests across different ff-divergences.

Our framework differs in two key aspects. First, its scope extends beyond specific discrepancy families. By representing heterogeneous procedures through witness functions whose expectations differ under PP and QQ, our framework can incorporate witnesses derived from MMD, ED, KFDA, PWD, RT and even ff-divergences. In particular, for a convex function ϕ\phi and density ratio r=dP/dQr=dP/dQ, the optimal variational witness gϕ=ϕ′(r)g_\phi=\phi'(r) satisfies EP[gϕ]−EQ[gϕ]>0\mathbb E_P[g_\phi]-\mathbb E_Q[g_\phi]>0 whenever P≠QP\neq Q. A proof and representative examples are provided in Appendix E. Second, the aggregation mechanisms differ. While ff-Agg combines tests through permutation-calibrated multiple-testing correction, our method directly aggregates heterogeneous witness mean differences using their joint covariance structure, yielding an asymptotically pivotal test without resampling under the stated regularity conditions.

We have cited these aggregation-based methods in the Introduction and provided further discussion in Appendix A.

Q3. Comparison with MMDAgg and other aggregation methods

Empirical comparisons with MMDAgg and MMD-FUSE were already provided in Appendix C.1 of the original submission (Appendix D.1 of the revised version). The results demonstrate the competitive and robust performance of our approach across diverse alternatives. In Appendix D.1 of the revised manuscript, we have added experimental comparisons with ff-Agg. As shown in Table R1, ff-Agg achieves high power in some settings but remains sensitive to the structure of the alternative, further motivating the aggregation of complementary testing procedures.

Q4. Connections to witness learning and related work

We agree that witness functions have been widely used to characterize distributional discrepancies, not only in two-sample testing but also in other applications, such as distribution matching and gradient flows. For instance, Tian et al. [5] employ Sobolev-regularized witnesses to guide MMD gradient flows.

Our contribution does not lie in learning individual witness functions, but in developing a unified framework for aggregating complementary witnesses derived from heterogeneous testing procedures. In our original submission, we cited related work [4] employing witness-based techniques in the Introduction and included empirical comparisons with MMD-witness and KFDA-witness tests. Rather than optimizing a particular witness, our framework leverages multiple witness mean differences to construct an adaptive test, aiming to achieve robust power across diverse alternatives without prior knowledge of the underlying discrepancy structure.

Importantly, our framework also provides a broader perspective on witness construction. Beyond conventional discrepancy-based witnesses, we illustrate how discriminative witnesses can be constructed from other testing principles, including ranking and variational formulations of ff-divergences, see Section 3.1 and Appendix E. This allows witnesses obtained from different discrepancy measures and learning objectives to be incorporated into a common aggregation framework.

We have cited the relevant work [5] in the Introduction and expanded the discussion in Appendix A of the revised manuscript, clarifying both the connections to existing witness learning approaches and the distinct objectives and contributions of our framework.

AuthorsOpenReview ↗

Response to Reviewer r1Yz (3)

W2/Q5. Comparison with deep two-sample tests

We thank the reviewer for highlighting these important works. We agree that neural networks have been widely used in two-sample testing to learn informative representations and improve power in high-dimensional settings. For example, Kirchler et al. [6] propose two-sample tests based on learned neural network representations, while Liu et al. [7] learn deep kernels by optimizing a criterion related to test power. The theoretical adaptivity of neural networks to underlying function structures has also been studied extensively, including Suzuki and Nitanda [8]. We have cited these works in the Introduction and expanded the discussion to cover deep two-sample tests and neural network adaptivity in Appendix A of the revised manuscript.

Our framework addresses a complementary aspect of adaptivity. While deep two-sample tests exploit adaptivity through neural representation or kernel learning, our approach focuses on adaptivity across heterogeneous testing procedures by aggregating complementary witnesses. We have added empirical comparisons with deep two-sample tests, including DMMD [6] and MMD-D [7], in Appendix D.2. As shown in Table R1, their performance varies considerably across different alternative structures, highlighting the importance of adaptivity across heterogeneous testing procedures.

Moreover, these deep two-sample tests can be incorporated into our framework. In Appendix E.2, we demonstrate how MMD-D [7] and DMMD [6] admit witness representations that can be incorporated into our aggregation framework. Thus, representation learning and heterogeneous witness aggregation are complementary rather than competing approaches, and our framework can potentially benefit from advances in deep two-sample testing.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Further Clarifications (Part 1)

1. Moment conditions in Lemma C.1.

We thank the reviewer for this careful observation. Our original argument implicitly treated the fitting sample as fixed and considered only the testing sample sizes tending to infinity. Under this conditional asymptotic setting, the learned witnesses and their distributions remain fixed, so finite second moments suffice for the law of large numbers and multivariate central limit theorem used in Steps 2 and 3.

We agree, however, that this argument does not directly apply when the fitting and testing sample sizes increase simultaneously, since the learned witnesses may vary with the sample size.

To address this issue, we have revised Lemma C.1 and its proof under the same asymptotic framework as the main text. Specifically, we now impose the normalized fourth-moment conditions in Assumption 2, with Ln/n→0L_n/n\to0, together with a uniform lower bound on the smallest nonzero eigenvalue of the witness correlation matrix.

Under these conditions, we establish covariance consistency and stability of the pseudoinverse-based whitening in Step 2. In Step 3, the fourth-moment conditions ensure the Lindeberg condition, allowing us to apply the multivariate Lindeberg–Feller theorem for triangular arrays and obtain the required Gaussian approximation, even when the learned witnesses vary with nn.

The revised proof retains the original conclusion of asymptotically conservative Type-I error control under singular covariance matrices.

We appreciate the reviewer for identifying this issue, which has helped us strengthen the theoretical justification of Lemma C.1.

2. Splitting ratio.

We thank the reviewer for raising this concern. We agree that selecting the splitting ratio with the smallest p-value does not generally preserve Type-I error control, even if each candidate test is individually valid. Our previous discussion did not sufficiently clarify this limitation.

First, we emphasize that our proposed method uses a prespecified splitting ratio of 0.5 throughout all experiments, a commonly adopted choice in sample-splitting-based testing [1, 2]. Our additional sensitivity experiments in Appendix D.8 show that this choice provides a favorable empirical balance between witness estimation and statistical inference. We therefore recommend 0.5 as the default choice in practice, without requiring data-driven tuning.

Second, the adaptive tuning strategy discussed in Q4 was intended only as a possible heuristic extension, rather than a component of our proposed method. A related strategy has been considered in high-dimensional MANOVA for tuning the partial standardization parameter [3], where candidate parameters are screened based on empirical Type-I error estimates before selecting the one yielding the smallest p-value. Nevertheless, we acknowledge that such screening does not itself guarantee Type-I error control after data-dependent selection. Establishing a valid calibration procedure while retaining the potential power benefits of adaptive tuning requires additional theoretical and empirical investigation.

We have therefore revised Appendix F.4 to clarify that this heuristic is not accompanied by a Type-I error guarantee and is not recommended as part of our proposed procedure. Developing a theoretically justified adaptive splitting strategy is beyond the scope of the present work, which focuses on dependence-aware aggregation of heterogeneous witnesses.

AuthorsOpenReview ↗

Summary of Contributions and Rebuttal Updates

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all reviewers for their time and constructive feedback. As the discussion phase comes to an end, we would like to summarize the recognized strengths of our work, its main contributions, and the substantial additions and revisions made during rebuttal.

1. Strengths recognized by the reviewers

We appreciate the positive assessments of several aspects of our work:

  • Relevant and meaningful research motivation (r1Yz). Reviewer r1Yz recognized that the motivation concerning different structures of distributional discrepancies is sound and relevant, and highlighted the potential practical value of a general approach for combining tests with diverse properties.
  • Useful and general witness-based framework (U2r1, XGBQ). Reviewer U2r1 recognized the usefulness of combining witnesses learned through different procedures, while Reviewer XGBQ highlighted the unified witness perspective for combining complementary procedures and accounting for their dependence.
  • Theoretical contributions (U2r1, XGBQ). Reviewer U2r1 acknowledged the rigor of our theoretical analysis. Reviewer XGBQ specifically recognized the quantitative Gaussian approximation bound in Theorem 1, which relaxes the exponential-moment requirement of Hu and Lin (2026) in certain regimes.
  • Strong empirical performance (U2r1). Reviewer U2r1 highlighted the strong experimental gains against the evaluated baselines.

2. Main contributions of our work

Our work addresses an important challenge in two-sample testing: selecting an appropriate test when the underlying discrepancy structure is unknown. Rather than developing a test tailored to a particular discrepancy family, we propose a general framework for adaptively aggregating heterogeneous learned witnesses.

Specifically, our main contributions include:

  • A general witness-based framework beyond IPMs. We provide a unified perspective for incorporating witnesses derived from heterogeneous two-sample procedures, including MMD, ED, KFDA, and PWD. Importantly, our framework is not restricted to conventional integral probability metric (IPM)-based methods. We further demonstrate how informative witnesses can be constructed from other testing principles, including rank-based tests and ff-divergences, extending the applicability of witness-based aggregation.
  • Dependence-aware adaptive aggregation. We construct an adaptiv test that aggregates complementary witness statistics from heterogeneous procedures, extending beyond a single class of discrepancies while explicitly accounting for dependence among the witnesses.
  • Theoretical guarantees. We establish asymptotic Type-I error control, quantitative Gaussian approximation bounds, and consistency guarantees under suitable regularity conditions.
  • Robust empirical performance. Extensive experiments demonstrate competitive and robust power across diverse alternatives, where individual testing procedures can exhibit substantially different sensitivities.

3. Substantial additions and clarifications during rebuttal

We have carefully addressed the reviewers' concerns and strengthened the manuscript through extended theoretical analysis, additional experiments, and further clarifications.

In particular, we have:

  • Extended the theoretical analysis to establish asymptotically conservative Type-I error control under singular witness correlation matrices.
  • Expanded comparisons with existing aggregation methods and other two-sample tests.
  • Conducted additional experiments examining redundant and collinear witnesses, witness scaling, pseudoinverse truncation thresholds, sample splitting ratios, and computational costs.
  • Further clarified the generality and novelty of our framework, including its applicability beyond conventional IPM-based methods.
  • Clarified the theoretical scope and assumptions, particularly the interpretation of our guarantees in high-dimensional settings and the conditions imposed on learned witnesses.

4. Final remarks

We believe that these additional theoretical results, empirical evidence, and clarifications have substantially strengthened the manuscript and addressed important concerns raised during the review process.

We respectfully ask the Area Chair and reviewers to reconsider their overall assessment of our work in light of its original contributions and the additional results and clarifications provided during rebuttal.

Thank you again for your time and careful consideration.

Best regards,

Authors

AuthorsOpenReview ↗

Further Clarifications (Part 2)

3. Methodological and theoretical advances over [4].

We thank the reviewer for requesting further clarification. We agree that the distinction between methodological generalization and new theoretical arguments deserves greater emphasis.

Methodological contribution: adaptivity across heterogeneous testing principles. [4] aggregate neural Wasserstein witnesses across projection dimensions and sparsity constraints, achieving adaptivity within the PWD family. In contrast, our work addresses a broader problem: adaptively combining testing procedures based on fundamentally different discrepancy measures and learning objectives. We develop a unified witness-based framework accommodating MMD, ED, KFDA, and PWD, and further illustrate its applicability beyond conventional IPMs through witness constructions derived from rank-based tests and ff-divergences. This enables cross-method adaptivity rather than parameter adaptivity within a single discrepancy family.

The quantitative Gaussian approximation under weaker moment conditions. The Berry--Esseen-type analysis in [4] relies on exponential-moment conditions, and its corresponding error bounds cannot be directly applied under our normalized fourth-moment assumption. In particular, two errors require separate control: (i) the Gaussian approximation error for the population-whitened witness mean differences, and (ii) the studentization error arising from replacing the population covariance matrix with its sample estimate.

In Appendix H, we control the first error using a multivariate Berry--Esseen inequality together with the normalized fourth-moment bounds. For the second error, we establish covariance perturbation bounds and control the resulting random deviation ∣Tn−Tn0∣|T_n-T_n^0| through a randomized concentration inequality. Combining these arguments yields the quantitative Gaussian approximation bound O(Ln/n)O(\sqrt{L_n/n}) under Ln/n→0L_n/n\to0, without requiring the exponential-moment conditions of [4]. Thus, the new analysis concerns establishing the approximation under weaker assumptions rather than introducing a fundamentally different Gaussian approximation technique.

General and method-specific power guarantees. [4] derive detection conditions by controlling the estimation and approximation errors associated with learned PWD witnesses, including those arising from projection optimization and neural network approximation. These PWD-specific bounds do not directly apply to other witness constructions, whose estimation errors have different sources. Specifically, MMD requires controlling the estimation of kernel mean embeddings, ED requires controlling empirical distance-based witnesses, and KFDA additionally involves perturbation analysis of regularized covariance operators. We therefore establish a generic consistency guarantee in Theorem 2 and derive procedure-specific bounds in Appendix G, with detailed proofs in Appendix J. In particular, we derive bounds for MMD and ED in Section J.1 and for KFDA in Lemma J.3, while directly using the existing PWD bound from [4] for the PWD component. These analyses yield detection conditions for heterogeneous witnesses that cannot be obtained directly from the PWD-specific results.

Broader Type-I error analysis. We acknowledge that the fundamental strategy of sample splitting, covariance whitening, and Gaussian approximation is related to [1]. Our contribution here is not a fundamentally different Gaussian approximation technique, but establishing the null guarantees for general learned witnesses under weaker normalized fourth-moment conditions, rather than the exponential-moment conditions in [4]. Furthermore, the revised Lemma C.1 extends asymptotically conservative Type-I error control to singular witness correlation matrices, which are particularly relevant when aggregating redundant or strongly dependent witnesses.

Thus, our contribution is not simply replacing PWD witnesses with other witnesses in an existing test. It is a general framework for cross-method aggregation, supported by a witness-agnostic consistency theorem, method-specific detection guarantees, and broader null-distribution analysis.

[1] Kübler et al., A witness two sample test. AISTATS 2022.

[2] Hu and Lei, A two-sample conditional distribution test using conformal prediction and weighted rank sum. JASA 2024.

[3] Lin et al., High-Dimensional MANOVA Via Bootstrapping and Its Application to Functional and Sparse Count Data. JASA 2023.

[4] Hu and Lin, Neural Wasserstein Two-Sample Tests. arXiv:2601.21732.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.