- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper combines MMD, ED, KFDA, and PWD through learned witness functions, using covariance adjustment to account for dependence among witness statistics evaluated on the same testing sample. It establishes asymptotic Type-I error control and consistency when at least one witness carries sufficiently strong signal. Numerical experiments demonstrate competitive or superior power across diverse distributional shifts while maintaining Type-I error control.
Critical strengths
The unified witness perspective provides a useful framework for combining complementary procedures, including MMD, ED, KFDA, and PWD, while accounting for dependence among their statistics. Theorem 1 provides a quantitative Gaussian approximation bound under stated conditions. This relaxes the exponential-moment requirement used for the corresponding bound in Hu & Lin (2026) in certain regimes.
Critical weaknesses
-
The methodological novelty over Hu & Lin (2026) requires further clarification. While the relaxed moment conditions provide a useful theoretical contribution, Max-Cov has the same form as their Equation (10). The main methodological addition appears to be combining witnesses from different procedures, and the significance of this extension needs stronger justification.
-
The theoretical guarantees hold for a fixed number of witnesses and require a uniformly nondegenerate correlation matrix. It remains unclear whether they extend to larger collections that contain highly correlated or redundant witnesses. Moreover, the pseudoinverse used in implementation does not carry these guarantees over to settings where population nondegeneracy fails.
Questions for authors
-
The benefit of combining witnesses from different procedures needs clearer justification. In similar regimes, Hu & Lin (2026) already aggregate PWD witnesses generated under different parameter configurations using the same form of statistic. Could a collection containing only PWD witnesses, matched in size to the proposed collection, achieve comparable power? What gains can be attributed specifically to including witnesses from other procedures? The comparison should use the same number of witnesses. The current PWD baseline uses nine witnesses, while the proposed aggregation seems to use twelve, so the gains may partly come from using more witnesses.
-
The construction and selection of the witness collection needs clearer guidance. Why were MMD, ED, KFDA, and PWD chosen? A paragraph or two explaining this choice may help. Furthermore, what properties must a new witness have to be included, and how can Assumption 2 be verified for the combined collection? Could the selection or aggregation of candidate witnesses be optimized for test power in broader regimes?
-
The implementation requires further assessment of numerical stability and computational cost. How are eigenvalues close to zero handled when computing the pseudoinverse? How sensitive are Type-I error and power to this choice? Could the authors also report how runtime changes as the number of witnesses increases?
-
The sample-splitting strategy in Appendix B.2 needs clearer justification and further assessment of its effect on test performance. Why was the sample splitting ratio set to 0.5, and how sensitive are Type-I error and power to this ratio and the random split? Could this ratio be optimized to improve test power while maintaining Type-I error control?
-
The code for reproducing the experiments does not appear to be available. Could the authors provide anonymized code during the rebuttal period?
5 comments
Response to Reviewer XGBQ (2)
Q1. Benefits of combining witnesses from different procedures
Thank you for the comment. As suggested, we provide comparisons with PWD baseline containing 12 witnesses, obtained by including an additional sparsity parameter to match the size of our proposed collection. As shown in Table R2, the PWD baseline still exhibits substantial power loss under several alternatives, particularly Model A with and Model B with . In contrast, our method achieves more robust power across diverse alternatives, demonstrating the benefit of aggregating witnesses from heterogeneous procedures rather than simply increasing the number of witnesses. These additional experiments are included in Appendix D.6 of the revised manuscript.
Table R2: Empirical rejection rates of different methods.
| Model | PWD (12) | PWD (9) | Max-Cov | Max-Var | Hotelling |
|---|---|---|---|---|---|
| Null | |||||
| Gaussian | 0.063 | 0.063 | 0.067 | 0.047 | 0.070 |
| Gaussian mixture | 0.060 | 0.073 | 0.067 | 0.057 | 0.060 |
| Alternative | |||||
| A, | 0.933 | 0.980 | 0.970 | 0.637 | 0.987 |
| A, | 0.383 | 0.383 | 1.000 | 0.987 | 1.000 |
| B, | 0.950 | 0.950 | 0.943 | 0.953 | 0.957 |
| B, | 0.193 | 0.130 | 1.000 | 0.943 | 1.000 |
| C, | 0.997 | 1.000 | 1.000 | 1.000 | 1.000 |
| C, | 0.270 | 0.250 | 0.217 | 0.557 | 0.353 |
| D, | 0.677 | 0.700 | 0.670 | 0.510 | 0.803 |
| D, | 0.993 | 0.997 | 0.997 | 0.970 | 1.000 |
| E, | 0.837 | 0.833 | 0.893 | 0.277 | 0.937 |
| E, | 0.990 | 1.000 | 1.000 | 0.463 | 1.000 |
Q2. Construction and selection of the witness collection
We provide further clarification on the choice of witnesses, the requirements for incorporating new witnesses, the assessment of Assumption 2, and potential strategies for optimizing witness selection. We have also added the relevant discussion to the revised manuscript, see Section 5 and Appendix F.3.
Choice of witnesses. We selected MMD, ED, KFDA, and PWD to represent complementary approaches to two-sample testing. MMD captures discrepancies through kernel mean embeddings, ED through pairwise distances, KFDA through covariance-aware discrimination in an RKHS, and PWD through learned Wasserstein projections with different structural constraints. These methods emphasize different aspects of distributional differences and may exhibit complementary power across alternatives. Our goal is not to identify an optimal fixed collection, but to demonstrate that heterogeneous witnesses can be combined within a unified adaptive testing framework.
Requirements for new witnesses. Our framework is highly flexible in incorporating new witnesses, provided they are learned independently of the testing sample. For the Gaussian calibration, the moment and nondegeneracy conditions in Assumption 2 ensure asymptotic validity. In practice, the empirical correlation matrix and standardized fourth moments provide useful diagnostics, although they cannot rigorously verify the population assumptions (see our response below on assessment of Assumption 2). Moreover, our extended theoretical analysis establishes asymptotic Type-I error control for the pseudoinverse-based Max-Cov statistic, even when the population covariance matrix is singular. Alternatively, permutation calibration provides finite-sample Type-I error control under the null exchangeability assumption, without requiring moment or nondegeneracy conditions. For consistency, at least one constituent witness must provide sufficiently strong discriminative signal, as characterized in Theorem 2.
Assessment of Assumption 2. In practice, the empirical correlation matrix and standardized fourth moments provide useful diagnostics, although they cannot rigorously verify the population assumptions. If the estimated correlation matrix is nearly singular, one possible remedy is backward elimination, as suggested in [1]. Specifically, let be the sample correlation after excluding the -th witness. We remove the witness indexed by and continue this process recursively until the retained collection is sufficiently well conditioned.
Potential optimization of witness selection. The witness collection could be further optimized using an independent validation sample to select candidates based on their estimated discriminative strength and dependence structure. Such selection may improve power by retaining informative and complementary witnesses. Importantly, selection must be performed independently of the final testing sample to preserve the validity of our existing calibration. Developing a systematic power-optimal selection strategy is an interesting extension beyond the scope of the present work.
Response to Reviewer XGBQ (3)
Q3. Numerical stability and runtime
We conduct additional experiments to assess the numerical stability of the pseudoinverse computation in Appendix D.7 and examine how runtime changes as the number of witnesses increases in Appendix D.9.
Numerical stability. We thank the reviewer for raising this concern. In our implementation, we compute the pseudoinverse using PyTorch's torch.linalg.pinv() with its default tolerance, which treats eigenvalues below a numerical threshold as zero. To assess numerical stability, we conduct additional experiments with different truncation thresholds: 1e-17, 2.66e-15 (default), 1e-13, 1e-10, 1e-8, 1e-6 and 1e-4. As shown in Table R3, the empirical Type-I error rates and power remain stable for thresholds up to 1e-10. Larger thresholds generally lead to more conservative Type-I error rates and reduced power in some settings. Overall, the results suggest that the proposed procedures are reasonably robust to the truncation threshold, provided it is reasonably small.
Table R3: Sensitivity analysis of empirical rejection rates to the truncation thresholds.
| Model | Method | 1e-17 | 2.66e-15 | 1e-13 | 1e-10 | 1e-8 | 1e-6 | 1e-4 |
|---|---|---|---|---|---|---|---|---|
| Null | ||||||||
| Gaussian | Max-Cov | 0.067 | 0.067 | 0.067 | 0.067 | 0.057 | 0.053 | 0.047 |
| Hotelling | 0.070 | 0.070 | 0.070 | 0.070 | 0.057 | 0.040 | 0.027 | |
| Gaussian mixture | Max-Cov | 0.067 | 0.067 | 0.067 | 0.067 | 0.053 | 0.050 | 0.037 |
| Hotelling | 0.060 | 0.060 | 0.060 | 0.060 | 0.047 | 0.030 | 0.027 | |
| Alternative | ||||||||
| A, | Max-Cov | 0.970 | 0.970 | 0.970 | 0.970 | 0.970 | 0.970 | 0.970 |
| Hotelling | 0.987 | 0.987 | 0.987 | 0.987 | 0.987 | 0.980 | 0.980 | |
| A, | Max-Cov | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| Hotelling | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | |
| D, | Max-Cov | 0.670 | 0.670 | 0.670 | 0.670 | 0.663 | 0.657 | 0.583 |
| Hotelling | 0.803 | 0.803 | 0.803 | 0.803 | 0.787 | 0.723 | 0.670 | |
| D, | Max-Cov | 0.997 | 0.997 | 0.997 | 0.997 | 0.997 | 0.997 | 0.993 |
| Hotelling | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
Runtime. Our procedure consists of two main stages, witness estimation using the training data and inference using the testing data. Table R4 reports the estimation and inference times for MMD, ED, KFDA, PWD, and our proposed Max-Cov procedure. The PWD contains 9 witnesses, whereas Max-Cov combines 12 witnesses, allowing us to examine how runtime changes as the number of witnesses increases. The experiments are conducted on a server equipped with an Intel Xeon Platinum 8488C CPU and a single NVIDIA GeForce RTX 4090 GPU.
The computational cost of estimation depends on the witness construction method. For MMD, ED, and KFDA, the witnesses admit explicit formulas, whereas PWD requires numerical optimization. Consequently, the runtime of our method is dominated by PWD witness estimation. Despite including three additional witnesses, our method has a total runtime comparable to the PWD baseline. The inference stage involves computing the test statistic and its critical value, which is generally much less computationally demanding than witness estimation.
Table R4: Runtime comparison (seconds) under Model A with over 20 runs, with standard deviations in parentheses.
| MMD | ED | KFDA | PWD | Max-Cov | |
|---|---|---|---|---|---|
| Estimation | 0.0172 | 0.0110 | 0.0680 | 44.8095 | 44.9058 |
| (0.0015) | (0.0009) | (0.0034) | (1.8832) | (1.8866) | |
| Inference | 0.0011 | 0.0005 | 0.0005 | 0.0040 | 0.0055 |
| (0.0021) | (0.0001) | (0.0001) | (0.0005) | (0.0005) | |
| Total | 0.0183 | 0.0115 | 0.0685 | 44.8134 | 44.9112 |
| (0.0024) | (0.0009) | (0.0034) | (1.8835) | (1.8869) |
Response to Reviewer XGBQ (1)
We thank the reviewer for the appreciation of the value of our unified witness-based framework and theoretical contributions. Below we address the weaknesses (W) and questions (Q) raised by the reviewer.
W1. Novelty relative to Hu and Lin [1]
Hu and Lin [1] develop neural Wasserstein two-sample tests and aggregate PWD witnesses across hyperparameter configurations to adapt to unknown sparsity levels and projection dimensions. In contrast, our framework addresses the broader challenge of test selection when the underlying discrepancy structure is unknown. We adopt a general witness-based perspective that accommodates witnesses derived from different discrepancy measures and testing principles, including constructions beyond the conventional IPM framework. By aggregating complementary witnesses from heterogeneous procedures, including MMD, ED, KFDA, and PWD, our method extends adaptivity beyond a single discrepancy family, aiming to achieve robust power across diverse alternatives. While our Max-Cov statistic builds on the covariance-whitened maximum aggregation of [1], we also investigate two alternative aggregation statistics, Max-Var and Hotelling, within the same general framework.
We extend the theoretical analysis of [1] in three aspects. First, we establish the asymptotic null distribution and a Berry--Esseen type bound under weaker normalized fourth-moment conditions, without restricting the witness learning to PWD. Second, we derive a generic consistency guarantee whenever at least one constituent witness has sufficiently strong signal relative to its estimation errors in Theorem 2, and specialize this result to the witnesses considered in our experiments in Appendix G of the revised version (Appendix D of original submission). Third, we show that even when the positive-definiteness assumption is violated, the pseudoinverse-based statistics Tn and Tn′ achieve asymptotically conservative Type-I error control, see Appendix C. Together, these results extend both the applicability and theoretical understanding of covariance-whitened witness aggregation.
We have revised the Introduction to make these distinctions explicit and more accurately position our work relative to the existing literature.
W2. Theoretical guarantees with correlated or redundant witnesses
The nondegeneracy of the correlation matrix is a standard regularity condition for whitening, which is used to establish the asymptotically pivotal null distribution. However, it is not necessary for conservative Type-I error control. In Appendix C of the revised manuscript, we extend our theoretical analysis to show that, the pseudoinverse-based statistics Tn and Tn′ yield asymptotically conservative Type-I error control even when the population covariance matrix is singular. Alternatively, permutation calibration provides finite-sample Type-I error control without requiring nondegenerate correlation matrix.
We further provide numerical experiments to examine the effects of redundant collinear witnesses in Appendix D.4. Specifically, we introduce an exact duplicate of the MMD witness into the aggregation procedure. As shown in Table R1, the empirical Type-I error rates and power remain largely unchanged across the considered settings, suggesting that the inclusion of an identical witness does not substantially affect the performance of our procedure.
Table R1: Comparison of empirical rejection rates with and without a duplicated witness.