acceptodds
Under review as a conference paper at ICLR 2027

Fair Ensemble Learning for Imbalanced Data Streams

Abstract

Learning from data streams requires online updates and stable predictive performance. However, class imbalance, sensitive-group imbalance, and instance-level difficulty imbalance are widespread in real-world data streams, where they reshape online training signals and may induce or amplify unfair predictions. Their distinct fairness effects and the limitations of existing fairness-aware methods under multi-source imbalance remain insufficiently understood. We analyze 204 controlled stationary synthetic streams using a unified profile-based prequential protocol. The results reveal distinct effects of the three imbalance sources on online fairness and highlight the challenges of maintaining fairness under their combined influence. Existing fairness interventions also exhibit source-dependent limitations and can incur substantial accuracy costs. Guided by these findings, we propose instance reWeighting for fair Ensemble Learning under imbaLanced data streams (WELL). WELL jointly reweights the incoming instance according to label-sensitive cells and moderates within-cell difficulty, with theoretical conditions for fairness improvement and bounds on accuracy loss. Across five real-world streams and 19 synthetic settings, WELL outperforms state-of-the-art fairness-aware streaming methods in the overall accuracy-fairness trade-off.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
fairness, ethics, data stream

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TF73OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper separates three imbalance sources in fair online classification: class imbalance (CI), sensitive-group imbalance (SI) and instance-difficulty imbalance (IDI). A study over 204 stationary synthetic streams and 15 learners finds that CI redistributes disparities, SI degrades fairness more broadly, imbalance amplifies label disparity, and rare/outlier-dominated streams degrade sharply. Six fairness-aware baselines do not transfer across sources (e.g. CFSMOTE drops from ~0.98 to 0.57 BA as outliers rise). The proposed WELL reweights each instance by group-imbalance, distributional-association and decision-disparity terms, moderated by a kNN difficulty score, and spreads the weight across an ensemble. Evaluation covers 19 synthetic settings and five real streams.

Critical strengths
  1. The paper shows that class imbalance, group imbalance and instance difficulty affect fairness in different ways, and that existing fair methods do not work equally well across them. For example, rebalancing groups does not help with hard or unusual instances. This changes how the field should think about “imbalance”, and it is the main reason the paper has value.

  2. A large, well-designed controlled study uses 204 synthetic streams, 15 learners and 6 fairness-aware methods, varying each type of imbalance separately. This makes the findings above reliable and useful to other researchers, even apart from the proposed method.

Critical weaknesses
  1. WELL is best only by average rank on one combined score (GM4), with no actual values, variance or statistical tests in the main text, and the combined score is never changed to check the ranking. WELL is only 4th in accuracy, behind three of its own simplified versions. So the claim that it "outperforms the state of the art" is not supported. This is the main reason for my negative score.

  2. The paper is about data streams, where changes in the data over time (drift) are the defining problem, but all streams are stationary. The method is therefore not tested in the setting it was designed for.

Questions for authors
  1. Per-setting BA, Kappa, SPD, DI with dispersion for WELL, FABBOO and CFSMOTE; how many differences exceed seed variation?
  2. Does WELL stay first under another aggregation (normalized mean, Pareto dominance count)?
  3. Does Theorem 1 prove or assume parity-correcting shifts? Any condition that guarantees them?
  4. Results for one gradual and one abrupt drift setting.
  5. Is there an online diagnostic telling when difficulty moderation helps?
  6. Behaviour under delayed labels or a missing sensitive attribute?
  7. Why does a positive Delta_amp show amplification rather than a different operating point?
  8. Were the coefficients selected on the evaluation streams? Please report a held-out selection.
  9. Add the missing fairness references and position WELL against in-processing methods.
9 comments
AuthorsOpenReview ↗

Response to Reviewer TF73 (part 1/7)

We thank the reviewer for recognizing the value of our controlled study of CI, SI, and IDI and for the feedback that helps us better substantiate WELL’s effectiveness. We have supplemented the revised manuscript with more complete empirical evidence, including standard deviations across repeated runs, alternative aggregation and Pareto analyses, drift validation, and an online diagnostic, together with further methodological clarifications. These additions help present WELL’s overall advantages and adaptability to drift more comprehensively. These revisions are highlighted in red.

Q1. Statistical Results [W1]

Appendix Tables 15–17 of the submitted manuscript have reported per-setting averages of BA, Kappa, SPD, and DI for WELL and all baseline methods, including FABBOO and CFSMOTE, together with statistical analysis using Wilcoxon signed-rank tests against WELL. We have added standard deviations over the 10 runs to the corresponding synthetic and real-world comparison tables in Appendix I. In the revised manuscript, these comparison tables are numbered Tables 21–23. Section 8’s “Experiment Setup” now explicitly directs readers to these mean ± SD results and Wilcoxon comparisons.

(1) Complete results.

Appendix Tables 21–23 now report the average BA, Kappa, SPD, DI, and GM4 over 10 independent runs for each setting, covering WELL and all baseline methods. Specifically, Table 21 covers synthetic settings S0–S8, Table 22 covers S9–S18, and Table 23 covers five real-world datasets. The revised tables now present BA, Kappa, SPD, DI, and GM4 as mean ± SD, with means and SDs, respectively.

(2) Statistical tests.

The symbols +, =, and − in Appendix Tables 21–23 indicate that the baseline performs significantly better than WELL, shows no statistically significant difference, or performs significantly worse than WELL, respectively. The statistical analysis is a two-sided Wilcoxon signed-rank test with a threshold of p<0.05.

The examples below additionally report standard deviations. Combining the numerical results in (1) with these statistical tests, we have updated the corresponding tables in Appendix I to report mean ± SD alongside the paired Wilcoxon significance markers, with the captions specifying that + indicates the baseline performs significantly better than WELL, = indicates no statistically significant difference between the baseline and WELL, and − indicates the baseline performs significantly worse than WELL.

SettingMethodBA↑Kappa↑SPD↓DI↓
AdultWELL0.7696±0.00340.4381±0.00750.0037±0.00090.0144±0.0031
AdultFABBOO0.7499±0.0019(−)0.3868±0.0014(−)0.0087±0.0037(−)0.0242±0.0019(−)
AdultCFSMOTE0.7918±0.0027(+)0.5279±0.0028(+)0.2106±0.0039(−)0.5539±0.0081(−)
KDDWELL0.7703±0.00240.2984±0.00320.0044±0.00110.0276±0.0070
KDDFABBOO0.7554±0.0030(−)0.3485±0.0026(+)0.0048±0.0021(=)0.0284±0.0035(=)
KDDCFSMOTE0.8227±0.0028(+)0.3539±0.0123(+)0.0895±0.0016(−)0.4150±0.0220(−)

To address how many differences exceed seed variation, we summarize the paired Wilcoxon comparisons with FABBOO and CFSMOTE. Each comparison uses the 10 paired runs for one metric in one setting. Across 24 settings, four metrics, and two baselines, this yields 192 comparisons (24×4×2). Of these 192 comparisons, 186 detect statistically significant differences between WELL and the corresponding baseline.

AuthorsOpenReview ↗

Response to Reviewer TF73 (part 2/7)

Q2. Alternative Aggregations [W1]

Following the reviewer’s suggestions, we additionally use Min–Max mean (MM4) and Z-score mean (ZM4) to combine the four metrics, covering accuracy (BA and Kappa) and fairness (SPD and DI). We also perform Pareto dominance analysis (PDA4) to count how often each method dominates others, is dominated, or lies on the Pareto frontier. The new results show that WELL remains first under both normalized means and achieves the highest dominance count, further supporting its overall accuracy–fairness advantages.

In Section 8 of the revised manuscript, we have introduced MM4, ZM4, and PDA4 in “Experiment Setup” and added their results to “Additional validation”. We have also updated Figure 6’s left panel to compare GM4, MM4, and ZM4 ranks. In Appendix I.1, we have added “Alternative Aggregations and Pareto Analysis”, which specifies the normalization formulas and shared reference pool, the 240 setting–run ranking protocol, Friedman/Nemenyi tests, Pareto definitions, and the complete seven-method comparison table.

(1) WELL ranks first under both normalized means.

WELL still ranks first among the seven methods under GM4, MM4, and ZM4, demonstrating its superior overall accuracy–fairness performance across different aggregation schemes. MM4 and ZM4 respectively average the four Min–Max-normalized and Z-score-standardized losses per run. To establish a unified and appropriate scale, we determine shared normalization parameters for each metric using all settings, methods, and runs.

(2) WELL remains on the Pareto frontier and is never dominated across all settings.

WELL’s Pareto dominance results further support its overall accuracy–fairness advantage: it records the highest dominance count (PDA4-A: 458), is never dominated (PDA4-B: 0), and remains on the Pareto frontier in every evaluated run (PDA4-F: 100%). We compare the seven methods pairwise across all evaluated settings. A method dominates another if it is no worse on all four metrics and strictly better on at least one.

Results cover 24 settings (19 synthetic and five real), with 10 runs each. PDA4-A and PDA4-B count dominance and dominated relations across settings, runs, and opponents. PDA4-F is the percentage of runs on the frontier. The table reports average ranks for MM4 and ZM4. Bold marks the best values.

MethodMM4 rank↓ZM4 rank↓PDA4-A↑PDA4-B↓PDA4-F↑
WELL1.101.094580100.00%
FABBOO1.901.912763187.08%
CFSMOTE3.103.19129697.50%
CFDS5.025.25728569.17%
FOMOS5.435.321033010.83%
FAHT5.545.751520350.42%
FairPCA5.915.50030524.17%


Q3. Conditions for Parity Correction

Theorem 1 takes parity-correcting shifts as a premise and establishes fairness improvement and accuracy-loss bounds under its stated conditions. Proposition 2 in Appendix B.2 (p. 19, Eqs. 26–27) derives sufficient conditions for these shifts from WELL’s actual weighting rule. The matched rule retains the same group-imbalance and difficulty terms while removing the two fairness corrections. In the weighted population-risk analysis, positive empirical cell probabilities, nonnegative correction coefficients and multipliers, and historical positive-label and positive-prediction rates ordered consistently with the matched response imply that the higher-positive-rate group’s effective threshold cannot decrease and the lower-positive-rate group’s cannot increase. These are the required parity-correcting directions. Together with the theorem’s no-crossing condition and a nonzero probability of changed predictions, they yield strict improvements in SPD and DI.

We have revised the paragraph following Theorem 1 in Section 7 to explicitly reference Proposition 2’s sufficient conditions for the assumed parity-correcting shifts. We have also added two explanatory paragraphs in Appendix B.2, connecting WELL’s log-weight terms to the threshold directions and distinguishing the directional conditions from no crossing, nonzero switching mass, and realization of weighted population minimizers. These paragraphs also explain the role of the learner-disagreement bounds in transferring population improvements to implemented online responses.

AuthorsOpenReview ↗

Response to Reviewer TF73 (part 3/7)

Q4. Drift Validation [W2]

Our study focuses on understanding the fairness effects of CI, SI, and IDI and developing WELL to improve the accuracy–fairness trade-off in such imbalanced stationary streams. Although drift is not the primary focus of our study, we recognize the reviewer’s point that evaluating drift scenarios is important for assessing the practical applicability of streaming methods. We have therefore expanded our evaluation to include abrupt and gradual drift when CI, SI, and IDI coexist, providing a more comprehensive assessment of WELL’s accuracy–fairness performance under changing distributions. Across the six evaluated drift settings, WELL achieves the best average ranks under GM4, MM4, and ZM4.

We have added a drift-evaluation summary to Section 8’s “Additional validation” paragraph, with a reference to the new “Drift Validation” subsection in Appendix I.1. This subsection details the six abrupt and gradual drift configurations on S16–S18 and the evaluation protocol, and includes two tables: full-stream GM4 means ± standard deviations with Wilcoxon comparisons, and aggregate GM4/MM4/ZM4 ranks with PDA4 results.

(1) Better effectiveness in Drift scenarios.

WELL achieves the best overall accuracy–fairness performance than all compared baselines on average across the drift scenarios, supporting its adaptability when CI, SI, and IDI coexist. We introduce drift in P(Y∣S)P(Y\mid S), a conditional distribution commonly studied in fairness research, by changing label SPD from 0 to 0.15. We evaluate S16–S18, where CI, SI, and IDI coexist, under both abrupt and gradual drift configurations.

The table reports full-stream GM4 means ± standard deviations across 10 runs per setting (200k samples per run). A/G denote abrupt/gradual drift. For the six baselines with matching runs, +, =, and − indicate significantly better performance, no statistically significant difference, and significantly worse performance relative to WELL, respectively, using two-sided Wilcoxon signed-rank tests with a significance threshold of p<0.05.

MethodS16-AS16-GS17-AS17-GS18-AS18-G
WELL (ours)0.0460±0.00450.0474±0.00490.0239±0.00750.0343±0.01910.0673±0.00380.0713±0.0058
FABBOO0.0902±0.0092(−)0.0602±0.0077(−)0.1161±0.0268(−)0.0633±0.0276(−)0.1083±0.0034(−)0.1128±0.0182(−)
CFSMOTE0.1764±0.0024(−)0.1577±0.0030(−)0.1889±0.0025(−)0.1697±0.0043(−)0.1962±0.0089(−)0.1765±0.0078(−)
FAHT0.1874±0.0113(−)0.1638±0.0126(−)0.2105±0.0072(−)0.1805±0.0115(−)0.1984±0.0518(−)0.1400±0.0485(−)
CFDS0.1836±0.0042(−)0.1593±0.0050(−)0.2147±0.0061(−)0.1889±0.0066(−)0.2309±0.0189(−)0.2035±0.0170(−)
FOMOS0.1521±0.0043(−)0.1330±0.0041(−)0.1809±0.0357(−)0.1742±0.0322(−)0.1565±0.0323(−)0.1349±0.0312(−)
FairPCA0.1326±0.0075(−)0.1083±0.0146(−)0.0761±0.0176(−)0.0717±0.0118(−)0.0770±0.0117(−)0.0723±0.0158(−)

(2) Overall evaluation and Pareto results.

Following the reviewer’s suggestions in Q2, we evaluate overall accuracy–fairness performance using four approaches: GM4, MM4, ZM4, and Pareto dominance analysis (PDA4). In general, WELL ranks first under GM4, MM4, and ZM4, with average ranks of 1.0, 1.2, and 1.2, respectively. WELL also achieves the largest dominance count and lies on the Pareto frontier in all six configurations.

MethodGM4 rank↓MM4 rank↓ZM4 rank↓PDA4-A↑PDA4-B↓PDA4-F↑
WELL (ours)1.01.21.2120100.0%
FABBOO2.51.81.890100.0%
CFSMOTE5.03.33.50266.7%
FAHT6.24.34.50716.7%
CFDS6.74.75.02366.7%
FOMOS4.25.85.30916.7%
FairPCA2.56.86.70266.7%
Reviewer_4QMKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a new method to improve upon tradeoffs of accuracy and fairness related metrics in an online stream learning setup. They do so after empirical investigation of classical, imabalanced, and fairness aware stream learners, identifying a gap in more explicitly accounting for different difficulties arising from different imbalances (CI/SI/IDI).

Existing methods typically do not address all three types of imbalance simultaneously, esp the instance difficulty.

Critical strengths
  • High quality technical formulation and notation. This is noteworthy because it focuses the claims within the paper on a narrower, but much clearer defined problem setup (e.g. through stream stationarity).
  • Thorough experimental setup with relevant and sufficiently many existing methods, ablations. Relevant because it far exceeds many of the experimental setups in related work.
  • Very relevant related work.
Critical weaknesses
  • The method suffers from a highly complex interplay between an ensemble of (in the main part not closer described) base learners, with WELL affecting only a weight factor that then get processed by an incompletely described diversification operator, to then be consumed by base-learner specific update functions. This limits the interpretability of model components considerably, even though the arguments that explain the methods design read as fairly reasonable.
  • There appears to be a (moderate) risk of circularity in the method design, stream feature identification, and performance attribution: Stream generators that are designed to isolate borderline/rare instance effects likely provide a good test environment for methods that explicitly account for such effects, and additionally allow for cleaner attribution of the methods parameters onto such features. As the real-world stream results look equally good, I consider this a minor concern.
  • Instance difficulty is assessed based on the local same-class neighbourhood. This may be problematic for minority classes, esp protected minority classes where scarcity is even worse, potentially resulting in unstable difficulty estimates. A more detailed discussion and analysis of this aspect should be considered.
  • The accuracy–fairness trade-off is taken for granted in this work. However, prior work for the static setting questions whether this trade-off is inherent. What is the case in the streaming setting? Is the trade-off theoretically or empirically unavoidable, or does it depend on the data distribution, type of imbalance, fairness criterion, and learning setup?
Questions for authors
  • Some more clarification regarding \Xi, the diversification operator, and the base learners would be welcome.
  • Some notation seems to be rarely reused and the paper may benefit from removal of them without sacrificing mathematical exactness
  • Clarify the trade-off question (see weaknesses)
  • In the related work, please explain how the proposed method compares to each and every of them? Esp. for ensemlbe methods which are the closest in terms of learning models
6 comments
AuthorsOpenReview ↗

Response to Reviewer 4QMK (part 1/3)

We thank the reviewer for the detailed feedback and positive assessment of our work. We have uploaded the revised manuscript, which more clearly explains WELL’s diversification mechanism and base learners, simplifies auxiliary notation, and clarifies the scope of the observed accuracy–fairness conflicts. It also provides detailed comparisons with related methods, strengthens the discussion of borderline/rare instances’ practical relevance, and explains the contribution of group imbalance correction under sample scarcity. We address each point below. These revisions are highlighted in green.

Q1. Component description [W1]

We thank the reviewer for the comments, which have helped us clarify WELL’s components in the revised manuscript. Starting from the weight ww computed by WELL for the current instance, the diversification operator sequentially adds zero-mean Gaussian noise to obtain the training weight wkw_k for each base learner kk. Here ϵt\boldsymbol{\epsilon}_t, denoted by XiX_i in the submitted manuscript, represents the collection of Gaussian perturbations used in this process. These perturbations make the same instance contribute differently to each learner’s update, promoting ensemble diversity through randomized training, following the principle of online bagging [1–3]. WELL determines the instance’s initial training influence, while these perturbations introduce training differences across learners. We use Hoeffding Adaptive Trees, widely used incremental stream learners [4,5], as the base learners, consistent with a baseline in our comparison.

We have revised Algorithm 1, the “Ensemble of base learners” paragraph in Section 7, and the “WELL Implementation Details” appendix to explain the Gaussian perturbations, additive diversification mechanism, and HAT base learners.

[1] N. C. Oza and S. J. Russell, “Online bagging and boosting,” in Proceedings of the Eighth International Workshop on Artificial Intelligence and Statistics, 2001, pp. 229–236.

[2] D. Wood, T. Mu, A. M. Webb, H. W. J. Reeve, M. Luján, and G. Brown, “A unified theory of diversity in ensemble learning,” Journal of Machine Learning Research, vol. 24, no. 359, pp. 1–49, 2023.

[3] J. A. Soloff, R. F. Barber, and R. Willett, “Bagging provides assumption-free stability,” Journal of Machine Learning Research, vol. 25, no. 131, pp. 1–35, 2024.

[4] J. Montiel, J. Read, A. Bifet, and T. Abdessalem, “Scikit-Multiflow: A multi-output streaming framework,” Journal of Machine Learning Research, vol. 19, no. 72, pp. 1–5, 2018.

[5] A. Lourenço, J. Rodrigo, J. Gama, and G. Marreiros, “DFDT: Dynamic fast decision tree for IoT data stream mining on edge devices,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 29, 2026, pp. 24053–24060.

Q2. Notation simplification

We thank the reviewer for the suggestion. We have reviewed and simplified the rarely reused auxiliary notation throughout the manuscript while preserving mathematical exactness. These revisions are highlighted in green.

Q3. Accuracy–fairness trade-off [W4]

We thank the reviewer for the comments, which have helped us clarify the impact of data distribution, imbalance type, and fairness criterion on the accuracy–fairness trade-off in the revised manuscript. Our experiments, presented in Figure 30 of Appendix F.7, empirically show that conflicts are very common, but their prevalence does not imply a universally unavoidable trade-off in streaming. Within our evaluated learning setup, their strength depends on the data distribution, imbalance type, and fairness criterion. Figure 30 in Appendix F.7 provides the following evidence.

(1) Data distribution. Under class imbalance, Figure 30(a) and (b) represent different data distributions. Accuracy and most fairness metrics exhibit negative relationships in both settings, but the strength of these relationships varies across distributions.

(2) Imbalance type. Figure 30(a) and (c) compare class imbalance and sensitive group imbalance under fair stream data. Conflicts between accuracy and most fairness metrics remain common, but their strength differs between the two imbalance types.

(3) Fairness criterion. In Figure 30(a), (b), and (c), different fairness metrics exhibit different correlations with accuracy. This shows that the strength of the observed accuracy–fairness conflict also depends on the fairness criterion used.

We have revised Finding 6 in Section 6 and expanded the discussion of Figure 30 in Appendix F.7 to explain how the observed accuracy–fairness conflicts vary with the data distribution, imbalance type, and fairness criterion within our evaluated learning setup.

AuthorsOpenReview ↗

Response to Reviewer 4QMK (part 2/3)

Q4. Comparison with related methods

We thank the reviewer for the comments. We have expanded the “Fair Streaming Learning” paragraph in Section 2 and provided a detailed comparison in Appendix E.5, covering the six baselines’ mechanisms, use of ensembles, and handling of CI, SI, and IDI, with particular attention to FABBOO and CFSMOTE.

WELL’s key innovation is to combine group representation correction with instance difficulty moderation within a unified online weighting framework, jointly addressing CI, SI, and IDI. Among the closest ensemble methods, FABBOO adjusts training weights according to class proportions and adapts decision thresholds using prediction-fairness feedback. Its imbalance correction primarily targets CI, without dedicated mechanisms for SI or IDI. CFSMOTE also uses an ensemble and addresses CI and SI through subgroup oversampling, but does not explicitly account for IDI. Compared with these methods, WELL addresses subgroup scarcity while distinguishing instances within each subgroup by their local difficulty.

The remaining baselines intervene at different levels: FAHT incorporates fairness into tree splitting; FOMOS uses multi-objective optimization of feature weights; FairPCA improves representation fairness through feature projection; and CFDS monitors group representation within data blocks. These methods do not jointly address CI, SI, and IDI.

We have included this comparison table and the accompanying discussion in Appendix E.5.

MethodEnsembleCI handlingSI handlingIDI handling
FABBOOYesYesNoNo
CFSMOTEYesYesYesNo
FAHTNoNoNoNo
FOMOSNoNoNoNo
FairPCANoNoNoNo
CFDSNoNoYesNo
WELL (ours)YesYesYesYes


W2. Widespread occurrence of borderline/rare instances in real-world data

Thank you for your comment, which has helped us better demonstrate the practical relevance of our study in the revised manuscript. We have expanded the discussion in Section 8 to highlight borderline and rare instances in all five real-world streams and WELL’s effectiveness in settings without borderline or rare instances. We have also added the scope and ranking procedure of this additional analysis to Appendix I.2.

Borderline and rare instances occur across diverse real-world imbalanced datasets and pose substantial learning challenges [1,2]. Related recent studies examine borderline oversampling and difficulty-aware imbalanced learning [3,4]. We therefore systematically investigate their effects on fairness, using the findings to explain performance degradation and guide WELL’s design for real-world streams.

WELL explicitly accounts for these difficulty patterns, and results on both synthetic and real-world streams demonstrate its advantages in balancing accuracy and fairness. Notably, our additional analysis shows that even in settings without borderline or rare instances, WELL still achieves the best overall performance among the compared state-of-the-art methods.

We have added this analysis and the corresponding ranking table to Appendix I.2 of the revised manuscript to demonstrate WELL’s overall accuracy–fairness advantage across scenarios with and without borderline or rare instances.

The table below summarizes overall accuracy–fairness performance using average GM4 ranks for scenarios with and without borderline or rare instances. WELL ranks first in both groups.

Scenario groupWELL (ours)FABBOOCFSMOTECFDSFAHTFairPCAFOMOS
With borderline or rare1.471.934.275.876.273.404.80
Without borderline or rare1.563.003.224.444.565.335.89

[1] K. Napierała and J. Stefanowski, “Types of minority class examples and their influence on learning classifiers from imbalanced data,” Journal of Intelligent Information Systems, vol. 46, no. 3, pp. 563–597, 2016.

[2] J. A. Sáez, B. Krawczyk, and M. Woźniak, “Analyzing the oversampling of different classes and types of examples in multi-class imbalanced datasets,” Pattern Recognition, vol. 57, pp. 164–178, 2016.

[3] H. Sun, J. Li, and X. Zhu, “A novel expandable borderline SMOTE over-sampling method for class imbalance problem,” IEEE Transactions on Knowledge and Data Engineering, vol. 37, no. 5, pp. 2183–2199, 2025.

[4] M. Son, I. Koo, J. Park, and C. Kim, “Difficulty-aware balancing margin loss for long-tailed recognition,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 19, 2025, pp. 20522–20530.

AuthorsOpenReview ↗

Response to Reviewer 4QMK (part 3/3)

W3. Difficulty estimation under scarcity

Thank you for your feedback, which has helped us better present WELL’s effectiveness under sample scarcity. Our explanation below draws on the ablation results already reported in Table 19 of Appendix I.4 in the submitted manuscript, corresponding to Table 25 in the revision.

In the revised manuscript, we have expanded the existing paragraph “WELL components play complementary roles” in Section 8 and the discussion in Appendix I.4 to explain how group imbalance correction can help offset difficulty-based downweighting under sample scarcity.

Difficulty estimates can be unstable when local samples are scarce. By assigning higher training weights to minority classes and groups, WELL’s group imbalance term can help offset potential downweighting caused by unreliable local-support estimates. The table below compares WELL (right) with the variant removing only the group imbalance term (left), with difficulty moderation retained in both. In S16–S18, where CI, SI, and IDI coexist, WELL achieves significantly lower GM4 in all three settings, including S17 and S18, where both CI and SI are severe. These results support the contribution of group imbalance correction to maintaining overall performance when difficulty moderation is used under sample scarcity.

Values are means over 10 runs. +/−/= denote better/worse/no detected difference relative to WELL using Wilcoxon tests (p < 0.05).

SettingBA↑Kappa↑SPD↓DI↓GM4↓
S16.7124−/.7170.4267+/.4210.0069−/.0024.0463−/.0155.0861−/.0501
S17.6920−/.7053.4626+/.4380.0165−/.0063.1603−/.0577.1462−/.0890
S18.6282−/.6502.3315=/.3284.0111−/.0017.1601−/.0262.1459−/.0567
Reviewer_UTviOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines how class imbalance, sensitive-group imbalance, and instance-level difficulty affect fairness in online learning. It proposes WELL, which reweights incoming observations using cell frequencies, historical predictions, and local label consistency, and evaluates it on synthetic and real-world streams.

Critical strengths
  • Varying the imbalance sources separately helps readers distinguish their effects on fairness.
  • The paper clearly presents the empirical observations that inspired WELL, making the motivation behind the approach easy to follow.
Critical weaknesses
  • I found the transition from Section 4’s evaluation protocol to the experiments in Sections 5–6 abrupt. Before the notation and Proposition 1, it would help to explain why intermediate fairness matters, and whether the goal is to be fair at every point in the stream or on average. It also wasn’t clear to me how these experiments lead to WELL. I’d suggest shortening the motivating analysis to leave more room for the method and its results.
  • I’m concerned about the cost of updating the ensemble and searching the buffer for every incoming observation. Appendix I.1.1 gives a complexity analysis, and Table 12 reports 10.87 minutes for WELL versus 3.13 minutes for FABBOO on 200,000 instances—roughly 3.5 times as long in this benchmark. The main text should discuss whether this cost is acceptable for the streaming applications the authors have in mind.
Questions for authors
  1. Is the practical objective low average disparity over time, or avoiding substantial disparity at any point?
  2. How does per-instance latency change with feature dimension and buffer size on the hardware used?
3 comments
AuthorsOpenReview ↗

Response to Reviewer UTvi (part 1/2)

We thank the reviewer for recognising the value of our CI/SI/IDI analysis, the clarity of our presentation, and the motivation for WELL, and for the constructive comments and questions. We have revised the manuscript to address all of the reviewer’s comments by clarifying the fairness objective, streamlining the motivating analysis, and reporting additional latency measurements and their practical implications. These revisions are highlighted in blue.

Q1. Objective

Thank you for the opportunity to clarify our practical objective. Our primary objective is low average disparity over time, reflecting fairness throughout the stream rather than only at its endpoint. However, a low average can mask periods of substantial disparity, during which decisions may disproportionately disadvantage a group. WELL therefore also aims to reduce large disparities during the stream through per-instance corrective feedback based on observed group disparities.

In the revised manuscript, we have added a new paragraph titled “Motivation for Intermediate Fairness Evaluation” at the start of Section 4 to explain the need to evaluate fairness throughout the stream and clarify our objective of low average disparity over time.

Q2. Millisecond-scale per-instance latency

To answer this question, we conducted additional experiments. Together with the results in our submission, these show that WELL achieves the best average ranks in fairness and overall accuracy–fairness performance among the seven evaluated methods (Figure 6) with low inference latency in our additional experiments: below 1 ms at 2–20 features and below 1.8 ms at 100 features on the AMD EPYC 9654 CPU.

Specifically, in the additional experiments, we tested feature dimensions d∈2,5,10,20,100d \in \\{2,5,10,20,100\\} and buffer sizes B∈1000,2000,4000,6000B \in \\{1000,2000,4000,6000\\}. Both inference and update latency increase with dd. Larger buffers mainly increase training (update) latency. Inference does not query the buffer, and its latency shows no consistent increase with BB. All experiments used the same AMD EPYC 9654 CPU. Each algorithm was run on a single designated CPU core with one computational thread.

In the revised manuscript, we have added Table 13 (also pasted below) to Appendix I.1.1 to report mean inference and update latencies with their standard deviations, together with a new paragraph “Per-instance latency” describing the timing settings. We evaluated each configuration over five runs of 200,000 instances. Each cell reports mean ± standard deviation across five runs, in ms per instance. We have also added a new paragraph “Efficiency and applicability” to Section 8 to summarize these findings.

(a) Inference latency per instance (ms)

d\Bd \backslash B1000200040006000
20.193±0.020.193 \pm 0.020.173±0.000.173 \pm 0.000.216±0.040.216 \pm 0.040.192±0.030.192 \pm 0.03
50.276±0.040.276 \pm 0.040.270±0.050.270 \pm 0.050.280±0.040.280 \pm 0.040.291±0.040.291 \pm 0.04
100.354±0.050.354 \pm 0.050.359±0.060.359 \pm 0.060.316±0.030.316 \pm 0.030.341±0.070.341 \pm 0.07
200.458±0.040.458 \pm 0.040.474±0.030.474 \pm 0.030.516±0.030.516 \pm 0.030.592±0.080.592 \pm 0.08
1001.631±0.501.631 \pm 0.501.482±0.271.482 \pm 0.271.714±0.201.714 \pm 0.201.633±0.281.633 \pm 0.28

(b) Model update latency per instance (ms)

d\Bd \backslash B1000200040006000
22.048±0.172.048 \pm 0.172.012±0.052.012 \pm 0.052.656±0.392.656 \pm 0.392.711±0.272.711 \pm 0.27
52.976±0.282.976 \pm 0.283.157±0.443.157 \pm 0.443.599±0.383.599 \pm 0.384.028±0.424.028 \pm 0.42
103.714±0.313.714 \pm 0.313.911±0.463.911 \pm 0.464.015±0.334.015 \pm 0.334.647±0.614.647 \pm 0.61
204.815±0.154.815 \pm 0.155.291±0.255.291 \pm 0.255.941±0.295.941 \pm 0.296.966±0.686.966 \pm 0.68
10018.543±4.8518.543 \pm 4.8521.031±2.8021.031 \pm 2.8026.919±4.1926.919 \pm 4.1928.643±0.9728.643 \pm 0.97
AuthorsOpenReview ↗

Response to Reviewer UTvi (part 2/2)

W1. Evaluation protocol and motivation

Thank you for your feedback, which has helped us improve the manuscript’s organization.

(1) Why intermediate fairness matters & Goal. Online predictions inform decisions as instances arrive, and these decisions immediately affect individual opportunities [1]. Endpoint-only evaluation may conceal earlier group disparities, motivating us to consider fairness throughout learning. Thus, our primary evaluation goal is low average disparity over time, with the fairness profile retaining disparity changes over valid prequential prefixes. At the algorithm level, WELL’s decision-disparity term also aims to avoid substantial disparity at any point in the stream by strengthening corrective feedback in response to larger observed group disparities and adjusting instance weights in subsequent model updates.

(2) From Empirical Findings to WELL’s Design. Section 4 provides the common evaluation protocol for the empirical analyses in Sections 5–6. The motivating empirical analysis in Section 5 identifies the distinct fairness effects of CI, SI, and IDI, highlighting the need to consider both group-level imbalance and instance-level difficulty in WELL’s design. RQ1 in Section 6 further reveals the limitations of existing methods under different imbalance conditions and the accuracy costs that can accompany improved fairness. Figure 5 summarizes how these empirical findings motivate the design of WELL’s four modules.

(3) Manuscript revisions. Following the reviewer’s suggestions, we have incorporated the clarifications in (1) and (2) into the manuscript and made further revisions to improve its organization and provide more space for the method and its results. Specifically:

  • We have added “Motivation for Intermediate Fairness Evaluation” at the start of Section 4, immediately before “Profile-based Fairness Evaluation,” to explain the motivation and average-disparity evaluation goal.
  • We have linked Section 5’s “Experiment Setup” to Section 4’s fairness-profile protocol and clarified in Section 6’s opening that RQ1 uses the same protocol.
  • In Section 7, we have revised “From empirical findings to WELL” accompanying Figure 5 to connect the findings to cell-level correction and within-cell difficulty moderation.
  • We have also shortened the setup and Findings 1–2 and 4 in Section 5, retaining all four findings and all figures while moving detailed metric-wise and severity comparisons to the corresponding “Detailed evidence” subsections of Appendix F.
  • Under “Instance weight” in Section 7, we have explained how the exponential factor adjusts influence across cells while the difficulty factor moderates contributions within cells.
  • Under “WELL components play complementary roles” in Section 8, we have added an interpretation of the fairness terms’ role in the accuracy–fairness trade-off.

[1] A. Dal Pozzolo, G. Boracchi, O. Caelen, C. Alippi, and G. Bontempi, “Credit card fraud detection: A realistic modeling and a novel learning strategy,” IEEE Transactions on Neural Networks and Learning Systems, vol. 29, no. 8, pp. 3784–3797, 2018.

W2. Significant fairness and overall gain with negligible per-instance extra inference/update time

Thank you for your feedback, which has helped us clarify WELL’s efficiency in the revised manuscript. Compared with FABBOO, WELL achieves average relative improvements of 2.42% in accuracy and 62.63% in fairness, alongside an 18.18% improvement in overall performance over advantaged scenarios. WELL’s inference latency remains competitive at 0.224 ms per instance, compared with 0.152 ms for FABBOO. The additional computational cost mainly comes from model updates: 3.349 ms per instance for WELL versus 1.545 ms for FABBOO, about 2.2× as long but only 1.804 ms extra.

Following the reviewer’s suggestion, we have expanded the runtime discussion under “Additional validation” in Section 8 into a separate paragraph titled “Efficiency and applicability”. This paragraph summarizes the inference–update comparison above. We have also documented the supplementary timing settings and distinguished full-run measurements from runtime estimates in Appendix I.1.1.

AuthorsOpenReview ↗

Kindly ask for feedback from Reviewer UTvi

Dear Reviewer UTvi,

Following your valuable comments, we have carefully revised the manuscript and clarified the points you raised in our responses. With the discussion period approaching its conclusion, we would greatly appreciate it if you could take a moment to reconsider your assessment of the paper. If you find that your main concerns have been satisfactorily resolved, we would very much appreciate it if you could consider adjusting your score accordingly.

We are thankful for your careful review and for the thoughtful remarks you have provided.

Best regards,
Paper 27903 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.