acceptodds
Under review as a conference paper at ICLR 2027

Zero-Flow Two-Sample Tests

Abstract

Motivated by the success of modern flow-based generative models in modeling complex data, we study two-sample testing through the lens of flow-based methods. We propose the Zero-Flow Two-Sample Test (ZF2ST), built on the zero-flow criterion, which characterizes distributional equality through a time-reversal antisymmetry of a learnable velocity field. We extend this criterion and further develop the Zero-Flow Discrepancy, an identifying discrepancy that controls the Wasserstein distance, and derive a variational representation in terms of a witness function. This representation naturally leads to a witness-based test whose power is governed by the signal-to-noise ratio (SNR), allowing direct power maximization for witness learning. ZF2ST learns the witness on one data split and performs testing on held-out samples, thereby maintaining Type-I error control and admitting a simple asymptotic null distribution. Experimentally, ZF2ST performs competitively across various synthetic and real-world benchmarks, while showing particularly strong performance in distinguishing image distributions from different sources.

Primary area:
generative models
Keywords:
Flow matching, Two-sample test, Statistical discrepancy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zVU6OpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper develops a two-sample test from a time-reversal symmetry of flow-matching probability flux. It extends the zero-flow characterization to distributions with finite first moments and establishes a Wasserstein bound for the resulting Zero-Flow Discrepancy (ZFD). A variational representation motivates learning a vector-valued witness on a training split and evaluating its directional alignment on independent held-out pairs. Witnesses are trained either by regression or by maximizing a regularized signal-to-noise ratio. Swap antisymmetry permits finite-sample sign-flip calibration, and branch/time averaging reduces auxiliary variance. Experiments cover synthetic distributions, HIGGS, natural-image distribution shifts, and real-versus-generated CIFAR-10 images. The reported image results are promising, particularly for ZF-SNR, although important parts of the experimental protocol are not yet reproducible from the supplied package.

Critical strengths
  1. A coherent connection between the population characterization and a valid testing procedure. Theorem 3.1 and Proposition 3.1 give a useful characterization and quantitative Wasserstein control without requiring densities. More importantly for the implemented test, sample splitting and score antisymmetry provide finite-sample null calibration without requiring accurate estimation of the entire residual field (Appendix A.9). Reusing precomputed scores for sign flips makes this statistically principled construction computationally attractive.

  2. Promising empirical power with a small image witness. The reported gains on ImageNet versus ImageNetV2 and RF-generated CIFAR-10 are substantial. Table 1b also includes several generator families and reports a meaningful exception: MMD-FUSE and MMDagg outperform ZF-SNR on MeanFlow. The variance-reduction studies and backbone ablation provide useful supporting evidence. These results suggest a practical advantage worth investigating, although the backbone experiment does not fully isolate the contribution of architecture, objective, and optimization.

Critical weaknesses
  1. The validation protocol and total data budget are unresolved. Appendix C.4, lines 1392–1393, states that a validation set as large as the training set was used to select hyperparameters and regularization. However, inspection of all seven main experiment scripts finds 18 calls to the ZF trainers and no call supplying val_set; the trainers' validation branches only log diagnostics. This does not establish that tuning never occurred elsewhere, but the supplied code does not reproduce it. Please specify the search space, selection criterion, tuning frequency, and source of validation samples, including whether their cost is included in nn and matched across methods. This is especially important for CIFAR-10.1: the reported 1,000 training and 1,021 testing images already exhaust its 2,021 images. A further disjoint validation set cannot come from that same collection. Clarifying this is necessary to assess both the sample-efficiency comparison and independence of the final test from tuning.

  2. The central generated-image experiments lack necessary sample-generation artifacts. The supplied sampler_genmodel.py loads four precomputed tensors for ScoreSDE, RF, MeanFlow, and StyleGAN-XL. None is present in the supplied data/ directory, and the README does not explain how to obtain or construct them. The RF Euler experiments similarly expect per-NFE tensors. Appendix C.2 provides upstream repositories, checkpoints, and some sampling information, but this does not reproduce the actual tensors, including their conversion to the required uint8 representation. The provided data/fid.py consumes existing tensors rather than generating them. Please supply the tensors or an executable generation pipeline with exact configurations, seeds, preprocessing/quantization, and artifact identifiers. Until then, Table 1b and the RF experiments cannot be independently regenerated from this package.

  3. Some implementation and evaluation details require reconciliation before interpreting the reported precision. Section 4.3, Eq. (18), and Algorithm 1 specify i.i.d. uniform time samples, whereas one_step_snr_loss and the default draw_test_times sample one time from each of KK equal strata. This still preserves the mean alignment and swap antisymmetry, but the displayed i.i.d. variance equality is not the exact variance of the implementation. Please describe stratification and give the corresponding expression. Separately, the CIFAR-10.1 alternative sampler reuses the entire fixed held-out QQ subset within each training run while resampling PP. Consequently, these repeats are not fresh draws from both populations as the protocol suggests. A conditional finite-corpus evaluation is legitimate; please state that estimand and clarify what the reported standard errors include. Neither observation by itself establishes invalid Type-I calibration.

Questions for authors
  • Scope of consistency. Propositions 4.3 and A.1 explicitly require a fixed witness with positive population alignment. These are useful conditional results, but definiteness of ZFD does not establish that neural training yields such a witness for every alternative. Please preserve this distinction in the interpretation of the method. Failure to reject is not evidence that ZFD or Wasserstein distance is small.
  • Moment assumptions. The regression identity in Lemma 3.1/Eq. (9) should explicitly state the second-moment condition used in Appendix A.4. The first-moment population characterization should be distinguished from the integrability assumptions needed for regression and asymptotic testing.
6 comments
AuthorsOpenReview ↗

Clarifications on Weakness: Validation, Reproducibility and Implementation

We sincerely thank the reviewer for the careful assessment of our paper, particularly for the effort devoted to examining our implementation details. We appreciate the constructive comments and address them below.

Weakness 1: Validation Protocol and Total Data Budget

Please specify the search space, selection criterion, tuning frequency, and source of validation samples, including whether their cost is included in nn and matched across methods

We thank the reviewer for raising this important concern. We acknowledge that the tuning procedure was not clearly described in Appendix C.4, which we have now clarified in the revision.

  • Search space: Learning rate 5×10−45\times10^{-4} to 5×10−35\times10^{-3}; λ\lambda from 10−410^{-4} to 10−110^{-1}.
  • Selection criterion: Positive and stabilized validation SNR, used as a practical indicator of witness learning quality and training stability.
  • Tuning freq: Once per experimental setting at a fixed sample size; no dataset-specific or trial-specific retuning during formal evaluations.
  • Validation samples:
    • Synthetic: Independently generated samples with different random seeds; no overlap with formal evaluations.
    • HIGGS: n=m=1500n=m=1500, split into 750 training and 750 validation samples per distribution under the alternative hypothesis.
    • Image datasets: ImageNet and ImageNetV2 only, n=m=1500n=m=1500, with the same train/val split as for HIGGS. Hyperparameters then applied to all image experiments without dataset-specific retuning.

For HIGGS and ImageNet vs. ImageNetV2, preliminary tuning samples may reappear in subsequent randomized train/test splits. These validation samples were not included in the reported nn. All formal evaluations used fixed hyperparameters after tuning.

Since our experiments largely follow [1], we adopted its published baseline hyperparameters without retuning. For the additional ImageNet and real-versus-generated CIFAR-10 experiments, we used the CIFAR-10 vs. CIFAR-10.1 settings from [1]. Thus, tuning budgets were not explicitly matched: ZF2ST was tuned once, whereas the MMD-D and C2ST were not retuned.

Weakness 2: The Lack of Sample-generation Artifacts

Please supply the tensors or an executable generation

We thank the reviewer for pointing out the missing artifacts. We have updated the supplementary materials. Please refer to the last part of readme.md for the tensors used in the experiments. The original PNG files can be reconstructed using data/fid.py.

The reviewer can reproduce our results using the provided tensors or independently generate images from the corresponding official model repos and convert them using data/compress.py. We expect the resulting testing power to be close to our reported values.

Weakness 3: Implementation and Evaluation Details Requiring Reconciliation

We thank the reviewer for carefully examining our implementation and evaluation protocol. We address both concerns below.

Stratified Time Sampling

but the displayed i.i.d. variance equality is not the exact variance of the implementation

Eq. (18) (now Eq. (17) after revision) considers i.i.d. uniform time sampling, whereas our implementation uses stratified sampling for further variance reduction. We partition [0,1/2][0,1/2] into KK equal strata I1,…,IKI_1,\ldots,I_K and independently draw tk∼Unif(Ik)t_k\sim\mathrm{Unif}(I_k). Denoting the resulting score by sh,Kstra\mathfrak{s}_{\mathbf{h},K}^{\mathrm{stra}}, in contrast to the i.i.d. score sh,K\mathfrak{s}_{\mathbf{h},K}, we obtain the exact variance decomposition

=\sigma^2_{X,Y}+\frac1K\mathbb E_{X,Y}[\sigma^2_{t\mid X,Y}]-\underbrace{\frac1{K^2}\mathbb E_{X,Y} \left[\sum_{k=1}^K(\mu_k-\bar\mu)^2\right]}_{\text{Between-stratum variance reduction}}\leq \sigma_K^2(\mathbf{h}),$$ where $\mu_k:=\mathbb E_{t\sim\mathrm{Unif}(I_k)}[\bar{\mathfrak{s}}_{\mathbf{h}}(t)\mid X,Y]$ and $\bar\mu:=K^{-1}\sum_{k=1}^K\mu_k$, with the remaining notation defined as in the revised Eq. (17). Thus, the i.i.d. variance expression serves as an upper bound for the variance of the stratified estimator. We have clarified this in a new remark in Appendix B.1. ### CIFAR-10.1 Estimand and Standard Errors > please state that estimand and clarify what the reported standard errors include. Our CIFAR-10 vs. CIFAR-10.1 evaluation follows a protocol similar to [1]. The **estimand** is the rejection probability conditional on the available CIFAR-10.1 corpus, averaged over randomized data splits, witness training, and repeated testing. The reported mean $\pm$ SE is computed across the 10 trial-level rejection rates, reflecting variability across training runs and repeated testing, but not uncertainty from newly collected CIFAR-10.1 samples. We have clarified this in the revised Appendix C.1.
AuthorsOpenReview ↗

Clarification of the Questions

Question 1: Scope of Consistency

but definiteness of ZFD does not establish that neural training yields such a witness for every alternative

We agree with the reviewer’s observation. To address this concern, we have added high-probability bounds for ZF-Reg and ZF-SNR in the revised Appendix E. Under the stated assumptions, both learned witnesses achieve positive alignment with probability at least 1−δ1-\delta whenever

{\rm ZFD}(p,q) &> \varepsilon_{\rm app}^{\rm reg} +\varepsilon_{\rm stat}^{\rm reg} +\varepsilon_{\rm opt}^{\rm reg},\\\\ {\rm SNR}_\lambda^* &> \varepsilon_{\rm app}^{\rm snr} +\varepsilon_{\rm stat}^{\rm snr} +\varepsilon_{\rm opt}^{\rm snr}. \end{aligned}$$ Here, ${\rm ZFD}(p,q)$ and ${\rm SNR}_\lambda^*$ denote the respective population signals for ZF-Reg and ZF-SNR. The errors account for approximation, finite-sample estimation, and optimization, respectively. Detailed bounds and proofs are provided in Appendix E.1. As the statistical errors vanish as $n_{\rm tr}\to\infty$, positive alignment holds with probability tending to one when the population signals exceed the approximation and optimization errors by fixed positive margins. Combined with Proposition 4.3, this provides sufficient conditions for conditional testing consistency, under the required non-degeneracy condition. These conditions are **sufficient, not necessary**. As the reviewer noted, failure to reject may reflect insufficient witness alignment rather than closeness between distributions. We have clarified this point in the revision. ## Question 2: Moment assumptions We thank the reviewer for pointing out this distinction. We have explicitly added the finite second-moment assumption $\mathbb E\\|Y-X\\|^2 < \infty$ to Lemma 3.1 in the revised version. **References (Primarily Cited in [Clarifications on Weakness](https://openreview.net/forum?id=E3VOWpZeF6&noteId=C1BjjjwZKH))** [1] Learning Deep Kernels for Non-Parametric Two-Sample Tests. Liu et al. ICML2020.

Thanks, I have read the comments.

Reviewer_JZKPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a flow-based two-sample testing method. The authors establish a necessary and sufficient condition for distributional equality under general conditions. For estimation, it uses a variational representation, then learns a witness function and then estimates its associated inner product unbiasedly on held-out data. For inference, it uses a sign-flip permutation test leveraging symmetry and discusses variance reduction through averaging auxiliary randomness. Empirical studies demonstrate competitive performance.

Critical strengths
  1. Equivalence between zero flow and distributional equality. Theorem 3.1 extends the previous result to weaker assumptions: under finite first moments, the proposed discrepancy vanishes if and only if the two distributions are equal.

  2. Effective use of symmetry. The symmetry of the construction provides a connection between the population discrepancy and the permutation test. Under the null hypothesis, exchanging the endpoints of a pair, with the corresponding transformation of the auxiliary branch variable, preserves the joint distribution while reversing the test statistic. This gives a finite-sample valid test conditional on the independently learned witness hh.

Critical weaknesses
  1. Guarantees for the learned witness and resulting power. The variational formulation motivates a held-out linear alignment statistic that avoids the positive quadratic bias associated with directly squaring the estimated gg. However, it is unclear how accurately the optimal hh can be learned. Poor alignment between hh and gg can reduce power.

  2. Pairing XX and YY may limit data efficiency, particularly with unequal sample sizes. The inference procedure uses one-to-one pairs. However, the two samples may have unequal sizes, and how to handle this deserves some discussion. In addition, the pairs may be formed randomly, and reducing the variability introduced by this pairing could potentially improve power.

Questions for authors

My questions concern the weaknesses discussed above. I am open to raise the score if the questions are adequately addressed.

  1. Can the authors comment on how well the learned witness approximates the optimal witness? This may affect the power in practice.

  2. Can averaging over additional pairings reduce variance? How to accommodate unequal sample sizes?

6 comments
AuthorsOpenReview ↗

Clarifications on Weakness 1 and Question 1

We sincerely thank the reviewer for the thoughtful assessment of our work and the insightful questions. We address each question below.

Weakness 1:Guarantees for the Learned Witness and Resulting Power

Poor alignment between hh and gg can reduce power.

We agree that imperfect witness learning can reduce power. We address this concern together with the related question below.

Question 1: Can the authors comment on how well the learned witness approximates the optimal witness?

We quantify the quality of the learned witness by establishing high-probability bounds for both ZF-Reg and ZF-SNR in the Appendix E of the revised version. However, accurate recovery of the optimal witness is not necessary for test consistency, as we clarify at the end.

For ZF-Reg, Eq. (33) in the proof of Theorem E.1 directly yields the following high-probability L2L^2 recovery bound:

\leq \varepsilon_{\rm app}^{\rm reg} +\varepsilon_{\rm stat}^{\rm reg} +\varepsilon_{\rm opt}^{\rm reg}.$$ Thus, $\widehat{\mathbf h}_{\rm reg}$ approximates optimal $\mathbf g$ up to function-class approximation, finite-sample statistical, and optimization errors, with high probability. For **ZF-SNR**, due to scale invariance stated in Proposition 4.1, our bound controls the SNR optimality gap rather than the $L^2$ recovery error. Theorem E.2 (Eq. (40)) establishes: $${\rm SNR}_\lambda^* -{\rm SNR}_\lambda(\widehat{\mathbf h}_{\rm snr}) \leq \varepsilon_{\rm app}^{\rm snr} +\varepsilon_{\rm stat}^{\rm snr} +\varepsilon_{\rm opt}^{\rm snr}.$$ Thus, $\widehat{\mathbf h}_{\rm snr}$ approaches the unrestricted optimal SNR direction \$\\mathbf h\_{\\rm snr}^* \\propto \\big(\\Sigma+\\lambda I\\big)^{-1} \\mathbf g\$, $\Sigma:=\mathbb E[(Y-X)(Y-X)^\top|\widetilde Z_t]$, up to the same three sources of error, with high probability. Detailed bounds and proofs are provided in Appendix E. As noted above, test consistency only requires positive population alignment, $\tau(\widehat{\mathbf h})>0$, and $0<\sigma^2(\widehat{\mathbf h})<\infty$ (Proposition 4.3). Sufficient conditions for positive alignment are established in Appendix E of the revised manuscript, as summarized in our response to reviewer zVU6 (Question 1) [[link]](https://openreview.net/forum?id=E3VOWpZeF6&noteId=2vzf0klUTD).
AuthorsOpenReview ↗

Clarifications on Weakness 2 and Question 2

Weakness 2: Data Efficiency of One-to-One Pairing and Unequal Sample Sizes

the two samples may have unequal sizes

reducing the variability introduced by this pairing could potentially improve power

We thank the reviewer for these suggestions. We have added a discussion of unequal sample sizes and full cross pairing in the Appendix B.5 of revised PDF, and address both questions below.

Question 2.1: Can averaging over additional pairings reduce variance?

Yes. Averaging over additional pairings can reduce variance, and averaging over all possible cross-sample pairings leads to full-cross pairing. This eliminates the additional variability due to random pairing and yields a U-statistic estimator with no greater variance. However, it requires O(nte2)O(n_{\rm te}^2) witness evaluations, compared with O(nte)O(n_{\rm te}) for one-to-one pairing, analogous to quadratic- versus linear-time MMD [1]. Moreover, fast wild-bootstrap calibration does not directly apply to full-cross pairing, for which we need to resort to pooled-label permutations.

We further conducted an ablation on Blobs (n=m=200n=m=200), using the same trained witnesses and testing samples. Results are reported as mean ±\pm SE over 10 training trials, each with 100 testing repetitions.

MethodOne-to-one power (%)Full-cross power (%)One-to-one time (ms/test)Full-cross time (ms/test)
ZF-Reg18.0 ± 2.921.5 ± 4.11.43 ± 0.0414.00 ± 0.31
ZF-SNR61.7 ± 5.869.9 ± 5.51.48 ± 0.0414.19 ± 0.32

Testing time includes witness evaluation and calibration. Full-cross pairing achieves higher empirical power, but at approximately 10 times the testing cost. This computational overhead may be even more substantial for image data, where witness evaluations are more expensive. We therefore retain one-to-one pairing as the default.

Question 2.2: How to accommodate unequal sample sizes?

Equal sample sizes are not required for ZF2ST. In training stage, the dynamic pairing (Appendix B.2) naturally accommodates unequal sample sizes. For n≠mn\neq m with n=ntr+nten=n_{\rm tr}+ n_{\rm te} and m=mtr+mtem=m_{\rm tr}+ m_{\rm te}, we choose nte=mten_{\rm te}=m_{\rm te} and use all remaining ntr≠mtrn_{\rm tr} \neq m_{\rm tr} for training, without discarding data. Alternatively, full-cross pairing directly accommodates unequal testing sizes at quadratic computational cost.

We further evaluated Blobs with n=200n=200 and m=250m=250, including MMD baselines. Results follow the same evaluation protocol stated above.

MethodTrain (ntr/mtr)(n_{\rm tr}/m_{\rm tr})Test (nte/mte)(n_{\rm te}/m_{\rm te})Power (%)Time (ms/test)
ZF-Reg (One-to-one)100/150100/10019.7 ± 2.81.04 ± 0.05
ZF-Reg (Full-cross)100/125100/12522.2 ± 4.112.93 ± 0.31
ZF-SNR (One-to-one)100/150100/10070.1 ± 6.31.10 ± 0.08
ZF-SNR (Full-cross)100/125100/12578.1 ± 5.212.09 ± 0.39
MMD-FUSE—200/25061.9 ± 1.03.15 ± 0.08
MMDagg—200/25056.8 ± 1.611.19 ± 0.10

Both ZF-SNR variants achieve higher empirical power than the MMD baselines, while one-to-one pairing is substantially faster.

Overall, full-cross pairing is a viable extension, particularly for highly unequal sample sizes, while one-to-one pairing remains our default for computational efficiency.

Reference

[1] A Kernel Two-Sample Test. Gretton et al. JMLR2012.

I thank the authors for the detailed reply!

For the reply of Question 2.1: Can averaging over additional pairings reduce variance?

  • Can author expand "Moreover, fast wild-bootstrap calibration does not directly apply to full-cross pairing, for which we need to resort to pooled-label permutations"?

  • As for the computation of full-cross pairing, can that be paralleled?

  • For full-cross pairing, one can sample from possible pairs in comparison to enumerate all. That can save computation.

For the reply of Question 2.2: How to accommodate unequal sample sizes?

Thanks for the explanation. I feel n = 200 and m = 250 are not too far away from n = m. What about n = 200 and m = 2000? (This may happen when some data are easier to collect). Will the method still use <200 out of m = 2000 for testing?

Reviewer_fX64OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces zero-flow two-sample tests (ZF2ST), which formulates a two-sample test based on flow-based transport maps between then two distributions. It leverages the zero-flow criterion, which builds the test upon the time-reversal property of the transport map under H0H_0 (p=q). It is the evaluated on different benchmarks, from CIFAR10 to imagenet, where it distinguishes generated from real samples and re-created dataset version from the original.

Critical strengths

Hypothesis tests enlarge the toolbox of the machine learning and are a promising step towards more rigor in both research and application of machine learning models. Similarly, flow models have shown a great promise to scale to high-dimensional problems, especially in images. Therefore, combining both promises to scale the hypothesis test to high-dimensions, enabling more applications.

The paper is well structured and readable, explaining the mathematical concepts sufficiently.

Critical weaknesses

There are a few problems with the paper, the first one is the design of the test. While the theoretical section is well argued, it assumes a "perfect" flow model and is only considering the perfect flow. But since the flow model is trained, its transport map is never really the idea transport map. A proper discussion of this discrepancy appears to be missing both from a theoretical perspective (what happens with the test?) and from a practical perspective (when is the flow good enough to "trust the test?"). Especially for the test to be useful in practice, this question should be sufficiently answered and its consequences explored sufficiently.

Secondly, I wonder whether the experimental evaluation is really trustworthy. The test shows a high power when separating the dataset CIFAR-10 vs CIFAR-10.1, but the distributions should be very close as the test-set generation process is closely followed. The hypothesis test is not so straightforward as traditional methods though, especially because the flow map is trained. An experiment where H_0 is true and another where H_0 is false, especially in high dimensions and with real datasets, and a discussion of practical aspects that considers the influence of training and the asymptotic normality of proposition 4.2 is crucial.

A final, more practical limitation, is the assumption of equal population size throughout the paper. This assumption seems quite unrealistic.

The writing is sometimes hard to follow, as it is focused on a lot of desiring the mathematical steps, but less on the consequences. What do we want to prove now? Why? What are the consequences of the statements. More context would really benefit the reader from understanding what we are currently proving.

Questions for authors
  1. Can the assumption of equal population size be lifted?
  2. What is the practical effect of the A0 assumption? Is it a restrictive assumption (even if it is lifted later)
  3. In Equation 11, the expectation is wrt to what?
  4. In proposition 4.2, what is the definition of τ(h)\tau(h)? Is this later assumed to be 00? Why?
  5. How sensitive is the test wrt to the hyper-parameter (K,λ)(K, \lambda) in the experiments
  6. How sensitive is the test wrt a badly trained flow-map, and what are the consequences?
  7. How can a practitioner know when the flow if trained sufficiently?
  8. Why is CIFAR-10 and CIFAR-10.1 so well separated?
  9. What are the empirical results for a setting where H0 is true (especially also for real data, like Imagenet or CIFAR-10).
8 comments
AuthorsOpenReview ↗

Clarification on Weakness 1: The Assumption of a “Perfect” Flow Model

We sincerely thank the reviewer for the careful assessment and constructive suggestions. Our main goal is to develop a statistically valid two-sample test based on ZFD. To this end, we first establish ZFD as a statistical discrepancy under finite first moments, together with a quantitative Wasserstein bound, and then construct a two-sample test using learned vector-field witnesses. We have also revised the presentation of our theoretical results. We clarify the main concerns below.

Weakness 1: The Assumption of a “Perfect” Flow Model

it assumes a "perfect" flow model and is only considering the perfect flow.

We appreciate the reviewer's concern about the gap between the learned and population flows. In fact, the validity of ZF2ST, ie, the type-I error control, does not require a perfectly learned flow.

This robustness to imperfectly trained flow follows from Proposition 4.1, which motivates testing the linear alignment τ(h)\tau(\mathbf h) between an arbitrary flow h\mathbf h and the population flow g\mathbf g. Specifically, Eq. (11) (now Eq. (10) in the revised version) gives

=\underbrace{\mathbb E[\langle\mathbf h(\widetilde Z_t,t),Y-X\rangle]}_{\text{No need to estimate }\mathbf g}$$ By the tower property (Appendix A.5), this alignment can be estimated directly from held-out samples. Under $H_0:p=q$, we have $\mathbf g=0$ by Theorem 3.1, hence $\tau(\mathbf h)=0$ regardless of flow quality. Through sample splitting, the learned flow $\widehat{\mathbf h}$ is independent of the test data, so this property holds for our trained $\widehat{\mathbf h}$. Furthermore, our wild-bootstrap calibration guarantees **finite-sample Type-I error control** for such imperfect flow. We also evaluate ZF2ST using imperfectly trained flows, as shown in the table below. Notably, imperfect training is not unique to ZF2ST. [1,3] similarly rely on learned models that may be imperfectly trained. > its transport map is never really the idea transport map. It is worth clarifying that ZF2ST **does not require explicitly constructing a transport map or solving an ODE for generation**. Instead, the learned vector field is used directly to detect distributional differences. > what happens with the test? Imperfect flow may reduce testing power, but does not compromise Type-I error control. To illustrate the **effect of imperfect flow model**, we evaluated ZF2ST on Blobs ($n=200$) with varying training steps. Results are reported as mean $\pm$ SE over 10 training trials. | Training steps | ZF-Reg Type-I (%) | ZF-SNR Type-I (%) | ZF-Reg Power (%) | ZF-SNR Power (%) | |---:|---:|---:|---:|---:| | 1 | 5.1 ± 0.8 | 5.0 ± 0.9 | 4.5 ± 0.6 | 5.5 ± 0.8 | | 1,000 | 5.1 ± 0.8 | 5.3 ± 0.9 | 11.2 ± 2.0 | 32.5 ± 5.9 | | 2,000 | 4.9 ± 0.9 | 6.2 ± 0.9 | 18.0 ± 2.9 | 61.7 ± 5.8 | | 4,000 | 6.0 ± 0.7 | 5.8 ± 0.9 | 27.6 ± 3.1 | 74.0 ± 5.0 | | 10,000 | 5.4 ± 0.5 | 6.1 ± 0.7 | 34.1 ± 3.2 | 74.2 ± 4.7 | **Type-I error remains close to $\alpha=0.05$ even after a single training step**, while testing power generally improves with additional training. We also provide quantitative guarantees on learned witness quality in our response to reviewer JZKP (Question 1) [[link]](https://openreview.net/forum?id=E3VOWpZeF6&noteId=ASepGB1Djm). > when is the flow good enough to "trust the test?" There are two aspects to the trustworthiness of the test. First, **test validity**: Type-I error control does not require a perfectly trained flow as discussed at beginning. Second, **testing power**: Proposition 4.3 guarantees that the power approaches 1 as $n_{\rm te}\to\infty$, provided the learned flow $\mathbf h$ has positive alignment $\tau(\mathbf h)>0$ and finite, nonzero variance. We further establish sufficient conditions for positive alignment in Appendix E of the revised manuscript. **References** [1] Learning Deep Kernels for Non-Parametric Two-Sample Tests. Liu et al. ICML2020. [2] MMD-FUSE: Learning and Combining Kernels for Two-Sample Testing Without Data Splitting. Biggs et al. NeurIPS2023. [3] Revisiting Classifier Two-Sample Tests. Lopez-Paz and Oquab. ICLR2017
AuthorsOpenReview ↗

Clarifications on Remaining Weaknesses: Trustworthiness of Evaluations and Equal Sample Size Assumption

Weakness 2: Trustworthiness of the Experimental Evaluation

but the distributions should be very close as the test-set generation process is closely followed.

We agree that CIFAR-10.1 was collected using a protocol closely following CIFAR-10. However, similar collection procedures do not guarantee identical distributions, and even subtle distributional shifts can be detected with high statistical power. Indeed, established MMD-based tests [1, 2] also achieve substantial power on this benchmark, indicating that the observed separation is not unique to ZF2ST.

An experiment where H_0 is true and another where H_0 is false...

... a discussion of practical aspects that considers the influence of training and the asymptotic normality of proposition 4.2 is crucial.

We agree that experiments under both H0H_0 and H1H_1 are important. Experiments where H0H_0 is true are reported in Appendix C.5 (Type-I error), while experiments where H0H_0 is false are presented in Section 5 (Power experiments), including high-dimensional real-world datasets. The influence of training is further addressed in our response to Weakness 1 [link].

Regarding the asymptotic normality in Proposition 4.2, it is primarily used to characterize testing power in Proposition 4.3. In practice, we use wild-bootstrap calibration (Section 4.3, Stage-II), which guarantees finite-sample Type-I error control (Appendix A.9) without relying on asymptotic normality.

Weakness 3: Unrealistic Equal Sample Size Assumption

more practical limitation, is the assumption of equal population size

We thank the reviewer for raising this concern. Equal sample sizes are not a fundamental requirement of ZF2ST, but are assumed for simplicity, as similarly considered in Section 3 of [3]. We have clarified how unequal sample sizes can be accommodated in the revised Appendix B.5. We also conducted an experiment with unequal sample sizes, as reported in our response to reviewer JZKP (Question 2.2, Table 2) [link], along with further details.

AuthorsOpenReview ↗

Clarifications of the Questions

Q1: Can the assumption of equal population size be lifted?

Yes. As discussed above.

Q2: What is the practical effect of the A0 assumption? Is it a restrictive assumption (even if it is lifted later)

Assumption (A0) holds for many common distributions, such as Gaussian, Laplace, and Gamma distributions, but excludes others, such as uniform distributions. However, it is not required by ZF2ST.

Q3: In Equation 11 (now 10), the expectation is wrt to what?

The expectation is taken wrt X∼pX\sim p, Y∼qY\sim q, t∼Unif⁡[0,1/2]t\sim\operatorname{Unif}[0,1/2], and B∼Ber⁡(1/2)B\sim\operatorname{Ber}(1/2), as sh(X,Y,t,B)\mathfrak{s}_{\mathbf h}(X,Y,t,B) is a function of all four random variables.

Q4: In proposition 4.2, what is the definition of τ(h)\tau(h)? Is this later assumed to be 0? Why?

As discussed in Weakness 1, τ(h)\tau(\mathbf h) is the linear alignment between h\mathbf h and g\mathbf g. Indeed,it is not assumed to be zero. It equals zero under H0H_0, since g=0\mathbf g=0 by Theorem 3.1.

Q5: How sensitive is the test wrt to the hyper-parameter (K,λ)(K,\lambda) in the experiments

KK controls the number of time samples used for variance reduction, with diminishing gains as KK increases (Appendix D.1, Figure 7). The parameter λ\lambda stabilizes the ZF-SNR training objective. We further evaluated sensitivity to λ\lambda over a broad range, reporting power (%) as mean ±\pm SE over 10 training trials.

Datasetλ=10−4\lambda=10^{-4}10−310^{-3}10−210^{-2}10−110^{-1}10010^{0}10110^{1}10210^210310^310410^4
Blobs (n=200n=200)61.3 ± 5.761.7 ± 5.861.7 ± 5.456.0 ± 4.544.4 ± 3.331.7 ± 3.630.7 ± 3.930.2 ± 3.829.8 ± 3.8
ImageNet vs. ImageNetV2 (n=2000n=2000)63.5 ± 6.563.5 ± 6.563.5 ± 6.563.2 ± 6.562.3 ± 6.661.2 ± 6.666.3 ± 5.460.6 ± 4.446.7 ± 4.6

Power is stable for λ∈[10−4,10−2]\lambda\in[10^{-4},10^{-2}] on both datasets.

Q6: How sensitive is the test wrt a badly trained flow-map, and what are the consequences?

As demonstrated in Weakness 1, insufficient flow training mainly reduces testing power, while Type-I error remains close to the prescribed level.

Q7: How can a practitioner know when the flow if trained sufficiently?

Rather than focusing on flow estimation accuracy, we primarily aim to achieve high testing power. In practice, we monitor whether the validation signal-to-noise ratio (SNR) becomes positive and stabilizes as a practical indicator of witness quality. This follows the same principle as [1], which also optimizes an estimated SNR as a surrogate for testing power.

Q8: Why is CIFAR-10 and CIFAR-10.1 so well separated?

We addressed this in Weakness 2 [link]

Q9: What are the empirical results for a setting where H0 is true (especially also for real data, like Imagenet or CIFAR-10).

These experiments are reported in Appendix C.5 (Figure 5 and Table 4).

References

[1] Learning Deep Kernels for Non-Parametric Two-Sample Tests. Liu et al. ICML2020.

[2] MMD-FUSE: Learning and Combining Kernels for Two-Sample Testing Without Data Splitting. Biggs et al. NeurIPS2023.

[3] Revisiting Classifier Two-Sample Tests. Lopez-Paz and Oquab. ICLR2017

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.