- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies sequential two-sample testing with a critic learned only from past observations and evaluated on a fresh pair . Each score is the critic's mean difference plus two zero-mean perturbations: a Rademacher-signed pair distance and an independent Rademacher jitter. Under the null, the scores are conditionally symmetric martingale differences.
The paper has three main contributions:
-
Variance stabilization for a regularized dictionary. Adaptive critic learning can leave a random predictable-variance scale, so studentization alone does not guarantee a Gaussian limit (counterexample in C.4). For a fixed finite Lipschitz dictionary with entropic (softmax) regularization and temperature , the critic's null variance vanishes while its drift converges to the largest dictionary contrast. This yields a studentized Gaussian test under a -moment null, including point-mass nulls (Prop. 3.2).
-
Anytime-valid inference. A closed-form half-normal mixture e-process on the same scores gives anytime-valid inference without moment assumptions (Thm. 3.3).
-
Drift convergence for general critics. Exact or approximate empirical Kantorovich–Rubinstein (KR) critics, and restricted critic classes, have population drift converging to or to the restricted discrepancy (Thms. 3.4–3.5).
The validation includes:
-
an 8-D dictionary simulation, covering variance concentration and a comparison of half-normal vs. online Newton step (ONS) betting on identical scores;
-
a temperature and perturbation sensitivity study;
-
a 1-D exact-KR comparison against permutation and energy tests;
-
a stratified analysis of Palmer penguin morphology.
Critical strengths
-
It identifies and formalizes a genuine calibration pitfall for learned critics. Past-only learning preserves null centering but can make the variance scale random. The counterexample in C.4 shows that conditional symmetry plus observed studentization cannot replace a deterministic variance-scale condition. This matters because learned-witness and martingale two-sample tests are increasingly common, and the clarification applies beyond Wasserstein critics.
-
It gives a concrete, rigorously analyzed construction that satisfies this condition. For the regularized dictionary, bounds (11)–(12) make the trade-off transparent: the temperature suppresses the null critic variance at the training-noise scale, while costing at most in drift. Point-mass nulls are handled through external jitter, and measurability issues are treated carefully (e.g., Lemma D.2). I found no errors in the proofs I checked (Prop. 3.1, Prop. 3.2, Thms. C.1, C.2, 3.3). This technical care is the main reason the paper is not a clear rejection.
Critical weaknesses
-
The main Gaussian guarantee covers a much narrower object than the title and abstract suggest, and its power cost is not quantified.
-
Narrow scope. Gaussian calibration is established only for a fixed finite dictionary, which in the experiments is . Its target, , is neither nor consistent against general alternatives.
-
Unverified condition for Wasserstein critics. For actual Wasserstein critics (full KR, restricted, or neural), condition (20) is assumed rather than verified. The 1-D study reports a predictable-variance coefficient of variation of about 0.35 at , so users who want a Wasserstein critic in practice obtain only the e-process.
-
Likely loss of local power. Heuristically, for a contrast the critic is approximately , so the drift is about . Summed over n pairs against fluctuations, this suggests a detection boundary near rather than . Theorem C.5 could address this question but is not applied to the proposed test, so the price paid for calibration remains unclear.
-
-
The novelty relative to prior work is not clearly delineated, and the empirical evidence is too limited to establish practical value.
-
Overlap with prior work. The past-witness/fresh-pair design with studentized Gaussian calibration already appears in the martingale kernel test (Chatterjee & Ramdas, 2026). Perturbation of degenerate studentized statistics follows Corradi et al. (2024). The half-normal mixture of under conditional symmetry is a classical self-normalized construction (de la Peña et al., 2004).
-
Limited experimental scope. All experiments are low-dimensional: , for KR, and a 4-D application with per group. There are no standard ML benchmarks and no comparisons to the witness test, cross-MMD, deep-kernel MMD, C2ST, or the martingale kernel test. Neural critics are discussed but never run.
-
Weak comparative results. Where comparisons exist, the method often loses. Fixed-KR is below permutation under location, mixture, and contamination alternatives at small n. In the penguin application it gives , versus 0.002 for both permutation baselines. In addition, the half-normal rule's advantage over ONS coincides with a higher null crossing rate (e.g., 0.030 vs. 0.016 under t5).
-
Questions for authors
-
(Weakness 1) Beyond the finite dictionary, under what verifiable conditions does the learned KR critic satisfy condition (20), and what local-power cost does the proposed regularization impose?
-
(Weakness 2) What is the key technical advance over existing martingale two-sample tests, and what evidence supports a practical advantage over established learned-witness baselines?
7 comments
Response to Reviewer m5A2 (2/5)
W2. We credit past-witness/fresh-pair Gaussian testing to Chatterjee and Ramdas (2026), perturbation for degenerate studentized statistics to Corradi et al. (2024), and the conditional-symmetry exponential and one-sided normal-mixture constructions to de la Peña et al. (2004) and Howard et al. (2021). The contribution specific to this paper is the entropy-regularized one-Lipschitz learner's simultaneous null-variance and fixed-alternative drift bounds, together with KR population-drift recovery without fitted-function convergence. The new local-power result makes the associated regularization cost explicit.
The added comparisons include Hotelling, a held-out empirical linear witness, linear and RBF martingale MMD, and RBF cross-MMD. The held-out linear comparator uses the training mean difference on the fixed feature vector; it is a simple sample-split witness. The numerical comparisons cover these fixed-feature and RBF procedures. Deep-kernel MMD, C2ST, neural critics and standard ML datasets remain outside the evaluated benchmark set.
In the original one-dimensional study, the full-KR normal-reference frequency is 0.302 under scale change versus 0.268 for permutation and 0.184 for energy, whereas under location change it is 0.684 versus 0.768 and 0.766. The former rows illustrate a response to scale changes; their Gaussian calibration requires in probability for a deterministic , where is the accumulated predictable variance. For sequential detection, an independent paired rerun (2000 null and 1000 alternative streams) raises location crossing at to 1.000 for half-normal and 0.999 for ONS, from 0.862 and 0.778 with noise; the associated noise-free normal-null rates are 0.0105 and 0.0010. With perturbations, the rerun gives 0.0190/0.0115 under the null and 0.862/0.778 under location change; the submitted batch gives 0.030/0.016 and 0.882/0.790. Half-normal has higher null crossing as well as higher detection, with both null rates below 0.05; the power comparison is at the common nominal threshold rather than equal attained null crossing. The practical evidence is therefore most useful for understanding the variance/detection tradeoff and noise-free monitoring, with the comparator results in Q2 giving the corresponding fixed-horizon cost.
For the original one-dimensional comparison, the normal-reference rejection frequency of Fixed-KR is 0.642 under the mixture-weight 0.7 alternative, compared with 0.702 for permutation and 0.720 for energy. Under 0.05 contamination, the corresponding values are 0.274, 0.320 and 0.206. Permutation has higher rejection frequency than Fixed-KR in both of these settings, while Fixed-KR exceeds energy under contamination. The Fixed-KR frequencies use a normal reference whose validity depends on predictable-variance stabilization; the permutation columns use randomization calibration. These comparisons are in revised Section 4.3 (Table 4), Appendix B.3 (Table 9) and Appendix H (Tables 11–12).
Response to Reviewer m5A2 (3/5)
Q1. A sufficient condition for broader anchored one-Lipschitz critics is convergence in , up to a predictable sign, to a deterministic anchored one-Lipschitz function , with having a finite distance moment. The common Lipschitz envelope bounds their norms, and
gives variance convergence in when the displayed function discrepancy tends to zero in probability. Averaging then yields in probability for fixed perturbations. This gives an explicit sufficient condition on the learned critics.
An exact verifiable KR case is a null supported on two points with masses , , and separation . For any measurable exact empirical optimizer, deterministic training sizes , fixed , , and the distance stabilizer,
Whenever the empirical proportions differ, exact optimality forces . The tie probability is at most , so its averaged contribution vanishes. The bounded scores and positive variance limit give with , and hence Gaussian calibration.
The three-point law shows the additional issue for richer KR fits. With slopes selected by the empirical cumulative-count signs, : it equals for equal signs and for opposite signs. With bounded update gaps, each sign pattern persists over a substantial final fraction of the stream with probability bounded away from zero. Consequently retains separated ranges and fails deterministic variance stabilization. To see the separation, let be the two cumulative-count differences at . Its increment covariance is . For , both events and have positive limiting probability by the two-dimensional central limit theorem. Independently, the future increments stay within in each coordinate with probability at least . On the resulting events, the respective signs persist throughout the final three quarters, giving and . Since the distance second moment equals one, equals twice this average plus , which has two separated ranges of positive probability. The continuous-KR experiments retain their conditional Gaussian interpretation, while their symmetry-based monitoring remains valid.
For the regularized dictionary, the local-power result gives the boundary and noncentrality displayed in W1. The drift and studentization calculation in W1 handles the overlapping prefixes directly; the result also gives consistency above the test-specific boundary. Revised Appendix I gives the two-point case (Proposition I.1) and the three-point example, and Appendix G.1 the local-power result.
Q2. The technical advance is an explicit one-Lipschitz learning construction with both null-variance control and fixed-alternative drift recovery, plus the separation of Gaussian calibration from conditional-symmetry monitoring. The new local-power theorem identifies the price of that construction.
The fixed-horizon comparison below uses the common input . HT is Hotelling; LM is linear martingale MMD; HW is the held-out empirical linear witness; RM is the RBF martingale MMD test of Chatterjee and Ramdas (2026), and RC is RBF cross-MMD. HT/LM use within-pair randomization, HW uses independent evaluation data, and the RBF methods use their native normal reference on a richer nonlinear class. The table uses fresh streams, independent of the submitted experiment: 2000 per null and 1000 per alternative.
| Setting | Default | HT | LM | HW | RM | RC | |
|---|---|---|---|---|---|---|---|
| Normal null | 1000 | 0.0535 | 0.0520 | 0.0555 | 0.0395 | 0.0440 | 0.0520 |
| Standardized null | 1000 | 0.0445 | 0.0490 | 0.0495 | 0.0540 | 0.0360 | 0.0470 |
| Location shift 0.4 | 250 | 0.366 | 0.784 | 0.750 | 0.733 | 0.687 | 0.712 |
| Location shift 0.4 | 1000 | 0.970 | 1.000 | 0.999 | 1.000 | 0.999 | 1.000 |
| Scale factor 1.3 | 250 | 0.794 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| Scale factor 1.3 | 1000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
Response to Reviewer m5A2 (1/5)
Thank you for your time and valuable comments.
W1. We distinguish three inferential targets. The regularized dictionary has Gaussian calibration and fixed-alternative consistency for DH=maxj∣(P−Q)hj∣>0. Approximate full/restricted KR optimization recovers the corresponding population drift. Conditional symmetry supports anytime-valid monitoring for arbitrary predictable critics. For example, P=(δ−1+δ1)/2 and Q=δ−2/4+δ0/2+δ2/4 have identical x and ∣x∣ means but W1=1, explicitly identifying the finite dictionary's restricted target.
The local-power result is as follows. With bounded update gaps and ϑi=cri−β, 0<β<1/2, the learner's separation rate is n−1/4−β/2. For independent row-wise pairs, fixed anchored one-Lipschitz features, a fixed finite zero-critic initialization, fixed τ≥0, ρ>0, uniform local-array (2+δ) distance moments and κn=EPn⊗Qnd(X,Y)2→κ0,
n1/4+β/2Δn→b⟹Tn⇒N(Kc(1+β)σ0∣b∣2,1),σ02=τ2κ0+ρ2.
The default gives n−3/8. Every single deterministic temperature schedule shared across the rows and satisfying ri<i, ri→∞, ϑi→0 and riϑi2→∞ has limiting rejection α at root-n feature contrasts under these local-array and fixed-perturbation assumptions. In the additional local-alternative experiment, rejection at shift 3n−3/8 is 0.331, 0.354, 0.330 for n=250,1000,4000, versus 0.094, 0.090, 0.065 at shift 3n−1/2.
For the continuous KR and application projection fits, Gaussian calibration is conditional on predictable-variance stabilization. We therefore describe these measurements as normal-reference rejection frequencies. The exact KR cases in Q1 supply a concrete verification and a concrete obstruction to treating this condition as automatic.
Put pβ=1/4+β/2. The local-power calculation uses the signed-softmax map gj(z)=sinh(zj)/∑kcosh(zk). Let ξni denote the fresh-pair score and Gn,i−1 the past observations and signs. Put Dn=∣Δn∣2, B0=supn∑jVarPnhj+VarQnhj<∞, and let εni denote the training error. Then E∣εni∣2≤B0/ri, ∣g(z)∣≤min(1,∣z∣) and ∣g(z)−z/K∣≤4∣z∣2. For the conditional score mean mni=Δn⊤g((Δn+εni)/ϑi), the triangle inequality gives
\mathbb E\left|\frac{\sum_i m_{ni}}{\sqrt n} -\frac{D_n^2}{K\sqrt n}\sum_i\vartheta_i^{-1}\right| &\le\frac{D_n\sqrt{B_0}}{K\sqrt n}\sum_i\frac{1}{\sqrt{r_i}\vartheta_i}\\\\ &\quad+\frac{4D_n}{\sqrt n}\sum_i\frac{D_n^2+B_0/r_i}{\vartheta_i^2}. \end{aligned}$$ This bound applies directly to the overlapping prefixes. At $D_n=O(n^{-p_\beta})$, its three contributions have orders $n^{\beta/2-1/4}$, $n^{\beta/2-1/4}$ and $n^{3\beta/2-3/4}$, all tending to zero. Since $\sum_i\vartheta_i^{-1}\sim n^{1+\beta}/\\{c(1+\beta)\\}$, the normalized drift tends to $\\|b\\|^2/\\{Kc(1+\beta)\\}$. For the critic coefficients $a_{ni}=g((\boldsymbol\Delta_n+\varepsilon_{ni})/\vartheta_i)$, $$\frac1n\sum_i\mathbb E\\|a_{ni}\\|^2 \le\frac1n\sum_i\min\left\\{1,\frac{D_n^2+B_0/r_i}{\vartheta_i^2}\right\\}=o(1).$$ Thus the average conditional variance of the critic component vanishes, and the independent signs give $n^{-1}\sum_i\mathbb E[(\xi_{ni}-m_{ni})^2\mid\mathcal G_{n,i-1}]\to\sigma_0^2$. The uniform $(2+\delta)$ moment bound gives conditional Lindeberg. For $s=1+\min(\delta,2)/2$, the martingale moment bound for centered squared increments is $O(n^{1-s})$; together with $n^{-1}\sum_i m_{ni}^2\le D_n^2\to0$, this yields $n^{-1}\sum_i\xi_{ni}^2\to\sigma_0^2$. The martingale central limit theorem and the drift limit give the displayed noncentral normal law. Revised Appendix G.1 (Proposition G.1) gives this result, and Appendix I the KR variance examples.