- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies prequential binary classification with drift, heterogeneous features, and missing observations under a use-and-discard protocol. OnlineNet combines a slow pathway using mask-aware cumulative normalization with a fast pathway receiving current values and their availability mask. A bounded, non-attenuating multiplicative gate lets the fast representation modulate the slow representation, while training-time logit adjustment uses class counts from previously revealed labels. Experiments on over 20 datasets show that the proposed OnlineNet outperforms several baseline methods for online learning.
Critical strengths
-
The combination of incomplete observations, evolving distributions, and prohibited instance replay is practically relevant. The prequential protocol and distinction between data retention and formal privacy are clearly stated.
-
The implementation is compact and the method is clearly presented. Eqs (4)-(5) give an explicit gate range, and Alg. 1 updates feature statistics and class counts only after prediction and the supervised update.
Critical weaknesses
-
This work is largely an incremental design based on existing methods, and the novelty relative to existing no-buffer online learners needs sharper positioning. Section 3's test-then-train, single-instance update protocol is already compatible with most evaluated baselines, and Table 5 explicitly marks HBP, MODL, HI2, and several classical methods as not retaining prior instances. Therefore, the contribution is principally a particular architecture for this setting rather than the no-retention setting itself. Welford normalization and Menon et al.'s logit adjustment are established ingredients, leaving the gate as the main technical addition. Please explain its distinction from standard conditional feature modulation and residual gating, and isolate the benefit over a compact masked-input MLP with the same causal normalization and prior-adjusted loss. This would make the architectural contribution more convincing.
-
The experiments are not sufficient, because this paper targets a very challenging and practical setting of learning from wild and non-stationary data. However, the experiments are conducted on synthetic datasets with self-crafted feature missing. Section 5.1 removes features independently with a fixed probability and ensures at least one feature remains observed; all tasks are binary, with at most 50,000 instances and majority fractions below 77%. This does not test correlated sensor failures, missingness linked to values or labels, changing availability mechanisms, feature birth/death, severe evolving class imbalance, or co-evolution of features and labels. Experiments on more large-scale real-world datasets with millions of instances, self-evolving distributions, and severe class imbalance should be conducted to evaluate the effectiveness of the proposed architecture. For example, in financial fraud detection (a binary classification task), the fraudulent patterns are continually evolving, the fraudulent rate keeps changing over time, feature missing is common, and the class imbalance problem is a major concern. Such a realistic setting should be considered in this paper, since this paper studies streaming learning with concept drift, missing values, and heterogeneous features. Moreover, the classification task considered is binary, and there are many scenarios with multiple classes. It is better if the authors can provide these experimental results.
-
The contribution of each component in the OnlineNet should be further evaluated. As shown in Tab.13, in many cases, removing certain components can lead to performance improvement. Moreover, the contribution of the proposed modules is not significant, and the current ablations do not fully isolate the proposed fast-slow mechanism from normalization and model capacity. Tab. 3 (average results) shows the largest balanced-accuracy drop when removing causal normalization, from 77.422 to 73.427, whereas removing the fast pathway gives 76.772. The first result establishes that scaling matters, but it does not establish that two complementary pathways are necessary.
-
Cumulative context and prior estimates can become stale under the very drift the method targets. Appendix C.1 uses all-history Welford means and variances, and Eq. (7) estimates class frequencies from all previously revealed labels. After a long stable period, a new regime has diminishing influence on these statistics; a formerly rare class can remain strongly adjusted even when its current frequency changes. The cumulative scores and sliding-window plots in this paper do not specifically quantify this failure mode.
Questions for authors
see weaknesses
6 comments
W1.3: OnlineNet vs. masked-input MLP variant
Thank you for your comment. We compared OnlineNet with a compact masked-input MLP variant, referred to as OnlineMLP-OPA, using the same causally normalized inputs, availability masks, and OPA prior-adjusted loss. The results are reported below. Each cell shows the OnlineMLP-OPA score, followed by the difference (OnlineMLP-OPA − OnlineNet) in percentage points.
| Dataset | Accuracy | AUROC | AUPRC | Balanced Accuracy |
|---|---|---|---|---|
| wdbc | 78.38 (−4.50) | 86.49 (−2.80) | 81.07 (−3.23) | 74.34 (−6.37) |
| spam | 86.72 (−4.27) | 93.68 (−2.36) | 88.95 (−3.56) | 83.12 (−6.26) |
| diabetes_f | 64.58 (−2.43) | 60.43 (−3.71) | 43.75 (−5.62) | 51.76 (−4.20) |
| spambase | 77.53 (−2.99) | 83.79 (−3.18) | 77.29 (−3.89) | 75.37 (−3.66) |
| LUdata | 63.12 (−3.23) | 70.82 (−3.22) | 70.78 (−3.39) | 63.12 (−3.18) |
W2.1: Datasets and Experimental Setup
Thank you for your comment. We would like to provide the following clarifications.
(a) Dataset Coverage and Experimental Breadth. We clarify that our evaluation is not limited to synthetic datasets. Our original experiments cover 27 datasets, comprising 15 real-world datasets spanning 11 application domains and 12 synthetic datasets generated using MOA. Among the real-world datasets, four are time-series datasets, namely LUdata, spam, weather, and elec. The datasets collectively cover numerical, categorical, mixed-type, and binary-valued features. Across all datasets, the number of instances ranges from 569 to 50,000, and the number of features ranges from 3 to 499. The majority-class percentages range from 50.32% to 76.19%. The synthetic datasets further incorporate controlled abrupt and gradual concept drifts with predefined transition positions. The 27 datasets evaluated at three observation probabilities yield 81 dataset–availability conditions. This enables systematic evaluation across different data scales, feature types, and feature availability levels.
(b) Feature Availability Protocol. We adopt a controlled feature-masking protocol, following prior online learning studies [1–4], to systematically evaluate OnlineNet's robustness under different degrees of feature missingness. For each instance, each feature is independently sampled as observed with probability . Specifically, we consider three feature observation probabilities, $p \in \{0.25, 0.50, 0.75\} $, corresponding to nominal missing rates of 75%, 50%, and 25%, respectively. If all features are sampled as unavailable, one feature is randomly retained. This protocol enables controlled and reproducible comparisons across different feature availability levels.
(c) Scope of Research. Our work focuses on strict online learning for wild streaming data, particularly the challenges of concept drift, varying feature availability, and heterogeneous features. Accordingly, our evaluation examines different levels of feature missingness and both abrupt and gradual concept drifts across real-world and synthetic datasets. The scenarios raised by the reviewer, including correlated sensor failures, missingness mechanisms dependent on feature values or labels, dynamically changing availability mechanisms, feature birth and death, severe and evolving class imbalance, and the co-evolution of features and labels, involve additional problem dimensions and assumptions that extend beyond the specific scope of wild streaming data investigated in this work.
[1] Yi He et al., Online Learning in Variable Feature Spaces with Mixed Data, ICDM, 2021.
[2] Rohit Agarwal et al., Aux-Drop: Handling Haphazard Inputs in Online Learning Using Auxiliary Dropouts, TMLR, 2023.
[3] Antonios Valkanas et al., MODL: Multilearner Online Deep Learning, AISTATS, 2025.
[4] Yulin Wang et al., Online Learning for Wild Streaming Data with Delayed Feedback, SIGKDD, 2026.
W2.2: Extreme Class Imbalance
Thank you for your comment. As detailed in our response to W2.1, we have made efforts to ensure broad experimental coverage across challenging streaming conditions. We provide the following further clarifications and additional experimental results.
(a) Scope of This Work. We agree that evaluating our approach on larger-scale financial fraud detection datasets is valuable. However, developing methods specifically targeting extreme class imbalance is beyond the scope of this work. We focus on strict online learning for wild streaming data, aiming to balance adaptation and knowledge retention under intermittent feature missingness and non-stationarity without retaining historical instances. Although OPA accounts for class-frequency effects during online learning, it is not intended as a dedicated method for extreme class imbalance. This problem is commonly studied as online class imbalance learning [1–3]. Integrating specialized imbalance-aware learning strategies into the strict no-retention setting is therefore an important direction for future research.
(b) Additional Experiments. To provide further empirical evidence, we conduct additional experiments on BAF Variant IV, a synthetic bank account fraud benchmark based on anonymized real-world data. It contains 1,000,000 instances, including 11,030 fraudulent cases (1.103%), and 30 input features. The records span eight months and are processed chronologically, with an approximately 5.89% native feature missingness rate. We evaluate two feature availability settings, =0.5 and =1.0, where the former randomly masks 50% of originally observed feature values, whereas the latter preserves the original missingness. The comparative results are reported below, with BA denoting Balanced Accuracy.
| Method | =0.5 AUROC | =0.5 BA | =1 AUROC | =1 BA |
|---|---|---|---|---|
| OnlineNet | 68.37 | 50.12 | 78.60 | 51.39 |
| MODL | 50.00 | 50.00 | 50.00 | 50.00 |
| Aux-Drop | 49.58 | 50.00 | 48.60 | 50.00 |
| OLIFL | 50.01 | 49.99 | 50.02 | 49.98 |
| OLVF | 51.38 | 50.09 | 40.45 | 50.04 |
| ORV$^3$V | 45.86 | 50.00 | 45.04 | 50.00 |
These results demonstrate OnlineNet's scalability to million-instance streams and its stronger ranking discrimination. They also highlight the remaining challenges in minority-class classification.
[1] Aristotelis Chrysakis et al., Online Continual Learning from Imbalanced Data, 2020, ICML.
[2] Dianlong You et al., Online Learning From Incomplete and Imbalanced Data Streams, 2023, IEEE TKDE.
[3] Jiahang Tu et al., Adaptive Learning in Imbalanced Data Streams With Unpredictable Feature Evolution, 2025, IEEE TKDE.
W2.3: Multi-class Classification
Thank you for your comment. We provide the following clarifications and additional experimental evaluations.
(a) Fairness Considerations. Some baseline methods were originally designed specifically for binary classification. Although these methods can, in principle, be extended to multi-class classification through a One-vs-Rest (OvR) strategy, doing so would require adapting their underlying online learning and prediction mechanisms. Since the original papers do not specify how these methods should be adapted to multi-class settings, we limited our original evaluation to binary classification to avoid introducing additional design choices that could compromise the fairness of comparisons.
(b) Additional Experiments. We conduct additional comparative experiments on Asfault, a five-class streaming dataset. For fair comparison, we select only neural network-based baselines from the same model family as OnlineNet. The following table reports the average performance at different feature availability rates.
| Model | Accuracy | Balanced Accuracy | AUROC | AUPRC |
|---|---|---|---|---|
| HBP_multiclass | 61.16 | 31.57 | 71.85 | 32.10 |
| HI²_multiclass | 81.76 | 61.19 | 93.34 | 69.01 |
| Aux-Drop_multiclass | 55.58 | 23.83 | 72.36 | 29.13 |
| OnlineNet_multiclass | 84.33 | 75.72 | 96.15 | 78.76 |
W1.1: Positioning and Contributions
Thank you for your comment. We provide the following clarifications.
(a) Relationship to Existing No-Buffer Online Learners. As discussed in Appendix A of the original paper, privacy protection and data governance motivate the formulation of constraints governing the use and retention of streaming data. However, existing paradigms lack a unified specification of these requirements. Standard online learning emphasizes sequential prediction and incremental model updates but does not prescribe whether historical instances should be retained. Buffer-free learning excludes replay buffers but does not constrain historical-sample access or batch size. For example, in online time-series forecasting, historical samples may be repeatedly accessed as temporal contexts, whereas online continual learning may employ mini-batch processing and multiple training epochs. Therefore, we characterize Strict Online Learning through explicit constraints on instance usage, update frequency, and historical data retention.
(b) Test-Then-Train, No-Retention, and Single-Instance Updates. We clarify that the prequential test-then-train procedure is a standard evaluation protocol in online learning. Single-instance updates and the non-retention of prior instances adopted by some of the evaluated baselines are algorithmic choices for incremental adaptation. In particular, discarding prior instances avoids repeated reliance on potentially outdated observations as concepts evolve. We emphasize that we have never claimed that these practices themselves constitute a novel learning setting or a methodological contribution.
(c) Contributions. Our contribution extends beyond a particular architectural design. We systematically investigate the learning challenges posed by wild streaming data under the strict online learning constraints described above. As discussed in the Introduction, this setting introduces a distinct stability–plasticity dilemma under uncertain single-instance evidence. Each update relies on potentially incomplete observations, while predictive relationships evolve over time. Without access to historical instances for replay, potentially disruptive updates cannot be reassessed or corrected by revisiting those observations. Moreover, evolving class frequencies can introduce bias. These coupled challenges constitute the central learning problem investigated in this work.
W1.2: Components and Gate
Thank you for your comment. We provide the following clarifications and additional experiments. We will incorporate the corresponding changes into the revised paper and highlight them in blue.
Components
We do not claim Welford normalization or logit adjustment to be novel contributions. We adapt these components to our setting. For Welford normalization, we develop a mask-aware causal normalization scheme based on Welford's algorithm to accommodate feature missingness. For logit adjustment, since the global class prior is unavailable in streaming data, we estimate class priors prequentially from previously revealed labels and use these estimates to adjust logits in the training objective.
We emphasize that the primary technical contribution of this work is OnlineNet, a lightweight neural Fast–Slow framework. Although relatively simple, its architecture reflects a deliberate design choice for learning from wild streaming data, where computational efficiency and generalizability are important. OnlineNet has fewer than 0.5% as many parameters as a state-of-the-art model while achieving strong predictive performance across diverse datasets and domains.
BAM vs. Standard Conditional Feature Modulation and Residual Gating
(a) Difference. The key distinction lies in how BAM integrates instantaneous evidence into the slow representation. Standard conditional feature modulation typically applies feature-wise scaling and shifting, whereas residual gating controls the contribution of a separately computed residual representation. In contrast, BAM directly applies multiplicative, bounded, and non-attenuating modulation to the slow representation. Weak responses from the fast pathway keep the gate near its multiplicative identity, while stronger responses allow amplification. By preventing attenuation, BAM protects accumulated contextual information and keeps the slow representation as the basis of the fused representation, while allowing the fast pathway to regulate its feature responses.
(b) Additional Experiments. We compare BAM with FiLM-style modulation and Gated Residual Fusion, which represent standard conditional feature modulation and residual gating, respectively. All other model components and training settings remain unchanged. The averaged results are shown below.