- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes SeqGoF, a nonparametric, anytime-valid sequential goodness-of-fit test for a composite null where the data distribution is an (unknown) convex mixture of K reference distributions. The method constructs a kernel-based witness by projecting the empirical kernel mean embedding onto the convex hull of the reference embeddings in an RKHS, centers the resulting score by its worst-case expectation over the null, and then performs predictable betting to produce an e-process. The authors prove supermartingale validity (level-α control under arbitrary stopping), almost-sure power one with a positive rate against any fixed alternative outside the convex hull (for characteristic kernels), and first-order stopping-time asymptotics; they also present synthetic experiments and a real-world application to hard-drive SMART telemetry.
Critical strengths
-
The SeqGoF testing procedure introduced in this paper provides a principled and elegant approach to handling mixture heterogeneity in sequential goodness-of-fit. By leveraging the convex hull of kernel mean embeddings, it reduces composite-null calibration to a tractable finite-dimensional operation.
-
The construction of a predictable, data-adaptive witness via projection onto the convex hull combined with worst-case component-wise centering is novel in the sequential testing-by-betting literature.
-
The approach extends kernel-based sequential GoF beyond a single reference distribution to mixture composite nulls in a way that remains computationally manageable.
Critical weaknesses
-
In the introduction, the authors suggest that the paper is motivated by changing of proportions of mixture of distributions under the null hypothesis. However, in Section 2, the null assumes i.i.d. sampling from a single (but unknown) mixture within the convex hull. Thus, time-varying mixture weights under the null are not covered (despite motivating language about composition shifts), and it is not clear whether the procedure could reject the null hypothesis with only composition drift.
-
Experiments section is not sufficient. In particular, no baseline is considered. The authors should consider compare with other methods to demonstrate the effectiveness of theirs proposed method in the heterogeneity settings they considered in their paper. Also, there is no sensitivity about kernel choice and bandwidth.
-
There are multiple hyperparameters in this paper. However, analysis (either theoretically or practically) and guidance on choosing ε, λmax, K, and reference components (especially in the real-data example) is limited.
Questions for authors
-
Could you clarify the intended null with respect to time variation? The theory assumes i.i.d. from a fixed mixture in the hull; do you consider slowly drifting mixture weights to be a change (to be detected) or part of the null? If the latter, can the framework be adapted to handle time-varying weights while preserving e-validity?
-
Could you compare other baselines in experiments section?
-
How sensitive is performance to the kernel and its bandwidth? Would aggregating multiple kernels improve robustness/power, and can the betting framework accommodate such aggregation without breaking validity?
-
In the real-data study, how were K and the component definitions chosen (tertiles of age)? Did you try alternative partitions or learned components (e.g., clustering) and, if so, how did this affect both level and delay?
3 comments
Thank you for addressing my concerns, espically adding the new Section 4 and comparison to the baseline. I will raise my score to 3 since I believe the paper is stronger now. However, could you clarify the following points further?
-
In figure 1 and 2, it seems to me that the dashed line is for proposed method and the real line is for the baseline? If this is correct, could you make this clearer in the legend? Currently it is not very obvious. Also, the presentation for the figures could be better. For example, in Figure 1, Type I error, I cannot tell which one is the proposed method and which one is for baseline in Scenario 2; in Figure 2, the two methods are indistinguishable.
-
In figure 2, the two methods are almost indistinguishable in the figure. Does this suggest that the proposed method's main advantage is for Gaussian reference distributions?
-
The new theoretical results in Section 4 is great. Could you also provide corresponding experiments for the results?
We thank you for your comments that have led to a significantly improved version of the paper. We respond to your remaining comments below.
-
We have revised Figures 1 and 2 to improve their readability. Figure 1 is now represented by Figures 5 and 6 on page 35 while Figure 2 is now Figures 7 and 8 on page 36. In the revised figures, SeqGoF is represented by solid blue lines and ZL24 by dashed orange lines, with distinct plotting symbols for the different scenarios. Additionally, and to respect the page limit, in Section 5 we now report the numerical results at selected sample sizes in Tables 1 and 2, allowing direct comparisons even when the curves overlap.
-
We have revised Figure 2, now presented as Figures 7 and 8 (page 36), to improve the distinction between the methods and added Table 2 in Section 5 to facilitate direct numerical comparisons.
Table 2 and Figures 7 and 8 demonstrate that SeqGoF performs well beyond the Gaussian setting. In particular, its empirical-reference implementation maintains accurate type-I error control while exhibiting increasing power against non-Gaussian alternatives outside the composite null. Although ZL24 appears to achieve greater sample efficiency in Figure 8 (and Table 2), this apparent advantage must be interpreted cautiously because ZL24 exhibits substantial type-I error inflation under the composite null (Figure 7 and Table 2). In fact in Figure 7, the type I error curves for ZL24 under scenarios 1 and 3 exceed 0.1 for all values of n and are, therefore, not shown. These results demonstrate that SeqGoF provides reliable testing performance for non-Gaussian reference distributions, with valid inference under the composite null.
- We thank you for this suggestion. We have added two numerical experiments (Experiments 3 and 4) in Appendix G to investigate the theoretical results established in Section 4. Experiment 3 considers Gaussian reference distributions, while Experiment 4 considers non-Gaussian (Gamma–Exponential) reference distributions. Each experiment includes two settings.
In Setting 1, we examine type-I error control under time-varying mixing weights, considering both gradually evolving weights and abrupt changes. The results (Figures 9 and 11) demonstrate that SeqGoF maintains type-I error close to the nominal level, whereas ZL24 exhibits substantial error inflation.
In Setting 2, we investigate power under change-point alternatives, where the observations initially follow a time-varying distribution within the composite null before transitioning to a fixed alternative. The results (Figures 10 and 12) show that SeqGoF gains power following the change point, with detection performance depending on the change-point location and the strength of the alternative. In contrast, ZL24 frequently rejects well before the true change point, reflecting its inability to accommodate changes in the null mixing weights.
Together, these experiments provide empirical support for the theoretical results in Section 4, illustrating the robustness of SeqGoF to time-varying mixing weights and its ability to detect subsequent departures from the composite null.
Response to reviewer xwLr
Thank you for you thoughtful questions, below are our responses.
Q1 & W1. Thank you for raising this important point. In our framework, changes in the mixing proportions are considered part of the null as long as the underlying reference components remain unchanged. Motivated by this comment, we have extended our theoretical results to accommodate time-varying mixing proportions (Section 4 and Appendix B). Specifically, we establish that SeqGoF retains its anytime-valid type-I error control even when the mixing weights vary arbitrarily over time and depend on past observations (Theorem 4). Thus, the extended framework accommodates not only slowly drifting but also abrupt changes in the mixing proportions under the null, without requiring any modification to the original betting procedure. Furthermore, we show that the asymptotic wealth growth, power-one guarantee, and first-order stopping-time asymptotics continue to hold when the process follows a time-varying mixture null up to an unknown finite change point and subsequently generates i.i.d. observations from a fixed alternative outside the composite null (Appendix B.2).
Q2 & W2. Thank you for this comment. In the revised manuscript, we include the sequential goodness-of-fit test of Zhou and Liu (2024) as an empirical baseline in both simulation experiments (Figures 1 and 2). Since their method is designed for a simple null, we implement it by instantiating our composite null at specific choices of mixing weights (see Appendix C.3 for details). Our numerical results show that Zhou and Liu (2024) controls the type-I error when the specified mixture coincides with the data-generating distribution but exhibits substantial error inflation when the mixing proportions differ (left panels of Figures 1 and 2). In contrast, SeqGoF maintains empirical type-I error at or near the nominal level across all the scenarios considered. Moreover, SeqGoF demonstrates greater sample efficiency in the Gaussian experiment (right panel of Figure 1). Although Zhou and Liu (2024) achieves higher sample efficiency in the non-Gaussian experiment (Figure 2), this apparent advantage should be interpreted with caution given its lack of type-I error control under the composite null (left panel of Figure 2). These comparisons highlight the importance of explicitly accounting for unknown mixing proportions in sequential goodness-of-fit testing.
We have also expanded our discussion of the related work by Martínez-Taboada and Ramdas (2025), highlighting the methodological and computational differences between their approach and SeqGoF in the context of composite mixture nulls (Appendix C.2).
Q3. In the revised manuscript (Appendix D, Figure 4), we investigate the sensitivity of SeqGoF to three kernels (Gaussian, Laplace, and IMQ) and four bandwidths. While empirical type-I error control is robust across specifications, power exhibits varying degrees of sensitivity. Aggregating the wealth processes across all twelve kernel–bandwidth combinations substantially mitigates this sensitivity and achieves power comparable to the best-performing individual specifications. Importantly, such aggregation preserves anytime-valid type-I error control by the closure of nonnegative supermartingales under convex combinations. We have incorporated these empirical findings in Appendix D of the revised manuscript.
Q4. In the revised manuscript (page 5), we clarify the roles of ε and λmax and conduct a sensitivity analysis over nine configurations, considering ε∈{10−2,10−6,10−8} and λmax∈{20,5000,10000} (Appendix E, Figure 5). The empirical type-I error and power curves are nearly indistinguishable across these configurations, suggesting limited sensitivity to these hyperparameters over the range considered. We have also expanded the SMART data analysis to investigate the choice of K and the construction of reference components. We discuss the details in our response to your next comment. These additional analyses provide practical guidance on constructing reference components using both domain knowledge and data-driven approaches.
W3. In the original analysis, the component distributions were constructed by partitioning healthy drives according to age tertiles (SMART 9). In the revised manuscript (Appendix F.1), we examine alternative age-based partitions with K=2,3,4, as well as a data-driven clustering approach that selects K=2 by maximizing the mean silhouette coefficient. None of the four specifications triggers a false alarm over 5000 held-out healthy observations. Detection delays are comparable across the age-based partitions and somewhat longer under the clustering-based specification. Please see Table 2 in Appendix F.1.