- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper separates three imbalance sources in fair online classification: class imbalance (CI), sensitive-group imbalance (SI) and instance-difficulty imbalance (IDI). A study over 204 stationary synthetic streams and 15 learners finds that CI redistributes disparities, SI degrades fairness more broadly, imbalance amplifies label disparity, and rare/outlier-dominated streams degrade sharply. Six fairness-aware baselines do not transfer across sources (e.g. CFSMOTE drops from ~0.98 to 0.57 BA as outliers rise). The proposed WELL reweights each instance by group-imbalance, distributional-association and decision-disparity terms, moderated by a kNN difficulty score, and spreads the weight across an ensemble. Evaluation covers 19 synthetic settings and five real streams.
Critical strengths
-
The paper shows that class imbalance, group imbalance and instance difficulty affect fairness in different ways, and that existing fair methods do not work equally well across them. For example, rebalancing groups does not help with hard or unusual instances. This changes how the field should think about “imbalance”, and it is the main reason the paper has value.
-
A large, well-designed controlled study uses 204 synthetic streams, 15 learners and 6 fairness-aware methods, varying each type of imbalance separately. This makes the findings above reliable and useful to other researchers, even apart from the proposed method.
Critical weaknesses
-
WELL is best only by average rank on one combined score (GM4), with no actual values, variance or statistical tests in the main text, and the combined score is never changed to check the ranking. WELL is only 4th in accuracy, behind three of its own simplified versions. So the claim that it "outperforms the state of the art" is not supported. This is the main reason for my negative score.
-
The paper is about data streams, where changes in the data over time (drift) are the defining problem, but all streams are stationary. The method is therefore not tested in the setting it was designed for.
Questions for authors
- Per-setting BA, Kappa, SPD, DI with dispersion for WELL, FABBOO and CFSMOTE; how many differences exceed seed variation?
- Does WELL stay first under another aggregation (normalized mean, Pareto dominance count)?
- Does Theorem 1 prove or assume parity-correcting shifts? Any condition that guarantees them?
- Results for one gradual and one abrupt drift setting.
- Is there an online diagnostic telling when difficulty moderation helps?
- Behaviour under delayed labels or a missing sensitive attribute?
- Why does a positive Delta_amp show amplification rather than a different operating point?
- Were the coefficients selected on the evaluation streams? Please report a held-out selection.
- Add the missing fairness references and position WELL against in-processing methods.
9 comments
Response to Reviewer TF73 (part 2/7)
Q2. Alternative Aggregations [W1]
Following the reviewer’s suggestions, we additionally use Min–Max mean (MM4) and Z-score mean (ZM4) to combine the four metrics, covering accuracy (BA and Kappa) and fairness (SPD and DI). We also perform Pareto dominance analysis (PDA4) to count how often each method dominates others, is dominated, or lies on the Pareto frontier. The new results show that WELL remains first under both normalized means and achieves the highest dominance count, further supporting its overall accuracy–fairness advantages.
In Section 8 of the revised manuscript, we have introduced MM4, ZM4, and PDA4 in “Experiment Setup” and added their results to “Additional validation”. We have also updated Figure 6’s left panel to compare GM4, MM4, and ZM4 ranks. In Appendix I.1, we have added “Alternative Aggregations and Pareto Analysis”, which specifies the normalization formulas and shared reference pool, the 240 setting–run ranking protocol, Friedman/Nemenyi tests, Pareto definitions, and the complete seven-method comparison table.
(1) WELL ranks first under both normalized means.
WELL still ranks first among the seven methods under GM4, MM4, and ZM4, demonstrating its superior overall accuracy–fairness performance across different aggregation schemes. MM4 and ZM4 respectively average the four Min–Max-normalized and Z-score-standardized losses per run. To establish a unified and appropriate scale, we determine shared normalization parameters for each metric using all settings, methods, and runs.
(2) WELL remains on the Pareto frontier and is never dominated across all settings.
WELL’s Pareto dominance results further support its overall accuracy–fairness advantage: it records the highest dominance count (PDA4-A: 458), is never dominated (PDA4-B: 0), and remains on the Pareto frontier in every evaluated run (PDA4-F: 100%). We compare the seven methods pairwise across all evaluated settings. A method dominates another if it is no worse on all four metrics and strictly better on at least one.
Results cover 24 settings (19 synthetic and five real), with 10 runs each. PDA4-A and PDA4-B count dominance and dominated relations across settings, runs, and opponents. PDA4-F is the percentage of runs on the frontier. The table reports average ranks for MM4 and ZM4. Bold marks the best values.
| Method | MM4 rank↓ | ZM4 rank↓ | PDA4-A↑ | PDA4-B↓ | PDA4-F↑ |
|---|---|---|---|---|---|
| WELL | 1.10 | 1.09 | 458 | 0 | 100.00% |
| FABBOO | 1.90 | 1.91 | 276 | 31 | 87.08% |
| CFSMOTE | 3.10 | 3.19 | 129 | 6 | 97.50% |
| CFDS | 5.02 | 5.25 | 72 | 85 | 69.17% |
| FOMOS | 5.43 | 5.32 | 10 | 330 | 10.83% |
| FAHT | 5.54 | 5.75 | 15 | 203 | 50.42% |
| FairPCA | 5.91 | 5.50 | 0 | 305 | 24.17% |
Q3. Conditions for Parity Correction
Theorem 1 takes parity-correcting shifts as a premise and establishes fairness improvement and accuracy-loss bounds under its stated conditions. Proposition 2 in Appendix B.2 (p. 19, Eqs. 26–27) derives sufficient conditions for these shifts from WELL’s actual weighting rule. The matched rule retains the same group-imbalance and difficulty terms while removing the two fairness corrections. In the weighted population-risk analysis, positive empirical cell probabilities, nonnegative correction coefficients and multipliers, and historical positive-label and positive-prediction rates ordered consistently with the matched response imply that the higher-positive-rate group’s effective threshold cannot decrease and the lower-positive-rate group’s cannot increase. These are the required parity-correcting directions. Together with the theorem’s no-crossing condition and a nonzero probability of changed predictions, they yield strict improvements in SPD and DI.
We have revised the paragraph following Theorem 1 in Section 7 to explicitly reference Proposition 2’s sufficient conditions for the assumed parity-correcting shifts. We have also added two explanatory paragraphs in Appendix B.2, connecting WELL’s log-weight terms to the threshold directions and distinguishing the directional conditions from no crossing, nonzero switching mass, and realization of weighted population minimizers. These paragraphs also explain the role of the learner-disagreement bounds in transferring population improvements to implemented online responses.
Response to Reviewer TF73 (part 3/7)
Q4. Drift Validation [W2]
Our study focuses on understanding the fairness effects of CI, SI, and IDI and developing WELL to improve the accuracy–fairness trade-off in such imbalanced stationary streams. Although drift is not the primary focus of our study, we recognize the reviewer’s point that evaluating drift scenarios is important for assessing the practical applicability of streaming methods. We have therefore expanded our evaluation to include abrupt and gradual drift when CI, SI, and IDI coexist, providing a more comprehensive assessment of WELL’s accuracy–fairness performance under changing distributions. Across the six evaluated drift settings, WELL achieves the best average ranks under GM4, MM4, and ZM4.
We have added a drift-evaluation summary to Section 8’s “Additional validation” paragraph, with a reference to the new “Drift Validation” subsection in Appendix I.1. This subsection details the six abrupt and gradual drift configurations on S16–S18 and the evaluation protocol, and includes two tables: full-stream GM4 means ± standard deviations with Wilcoxon comparisons, and aggregate GM4/MM4/ZM4 ranks with PDA4 results.
(1) Better effectiveness in Drift scenarios.
WELL achieves the best overall accuracy–fairness performance than all compared baselines on average across the drift scenarios, supporting its adaptability when CI, SI, and IDI coexist. We introduce drift in , a conditional distribution commonly studied in fairness research, by changing label SPD from 0 to 0.15. We evaluate S16–S18, where CI, SI, and IDI coexist, under both abrupt and gradual drift configurations.
The table reports full-stream GM4 means ± standard deviations across 10 runs per setting (200k samples per run). A/G denote abrupt/gradual drift. For the six baselines with matching runs, +, =, and − indicate significantly better performance, no statistically significant difference, and significantly worse performance relative to WELL, respectively, using two-sided Wilcoxon signed-rank tests with a significance threshold of p<0.05.
| Method | S16-A | S16-G | S17-A | S17-G | S18-A | S18-G |
|---|---|---|---|---|---|---|
| WELL (ours) | 0.0460±0.0045 | 0.0474±0.0049 | 0.0239±0.0075 | 0.0343±0.0191 | 0.0673±0.0038 | 0.0713±0.0058 |
| FABBOO | 0.0902±0.0092(−) | 0.0602±0.0077(−) | 0.1161±0.0268(−) | 0.0633±0.0276(−) | 0.1083±0.0034(−) | 0.1128±0.0182(−) |
| CFSMOTE | 0.1764±0.0024(−) | 0.1577±0.0030(−) | 0.1889±0.0025(−) | 0.1697±0.0043(−) | 0.1962±0.0089(−) | 0.1765±0.0078(−) |
| FAHT | 0.1874±0.0113(−) | 0.1638±0.0126(−) | 0.2105±0.0072(−) | 0.1805±0.0115(−) | 0.1984±0.0518(−) | 0.1400±0.0485(−) |
| CFDS | 0.1836±0.0042(−) | 0.1593±0.0050(−) | 0.2147±0.0061(−) | 0.1889±0.0066(−) | 0.2309±0.0189(−) | 0.2035±0.0170(−) |
| FOMOS | 0.1521±0.0043(−) | 0.1330±0.0041(−) | 0.1809±0.0357(−) | 0.1742±0.0322(−) | 0.1565±0.0323(−) | 0.1349±0.0312(−) |
| FairPCA | 0.1326±0.0075(−) | 0.1083±0.0146(−) | 0.0761±0.0176(−) | 0.0717±0.0118(−) | 0.0770±0.0117(−) | 0.0723±0.0158(−) |
(2) Overall evaluation and Pareto results.
Following the reviewer’s suggestions in Q2, we evaluate overall accuracy–fairness performance using four approaches: GM4, MM4, ZM4, and Pareto dominance analysis (PDA4). In general, WELL ranks first under GM4, MM4, and ZM4, with average ranks of 1.0, 1.2, and 1.2, respectively. WELL also achieves the largest dominance count and lies on the Pareto frontier in all six configurations.
| Method | GM4 rank↓ | MM4 rank↓ | ZM4 rank↓ | PDA4-A↑ | PDA4-B↓ | PDA4-F↑ |
|---|---|---|---|---|---|---|
| WELL (ours) | 1.0 | 1.2 | 1.2 | 12 | 0 | 100.0% |
| FABBOO | 2.5 | 1.8 | 1.8 | 9 | 0 | 100.0% |
| CFSMOTE | 5.0 | 3.3 | 3.5 | 0 | 2 | 66.7% |
| FAHT | 6.2 | 4.3 | 4.5 | 0 | 7 | 16.7% |
| CFDS | 6.7 | 4.7 | 5.0 | 2 | 3 | 66.7% |
| FOMOS | 4.2 | 5.8 | 5.3 | 0 | 9 | 16.7% |
| FairPCA | 2.5 | 6.8 | 6.7 | 0 | 2 | 66.7% |
Response to Reviewer TF73 (part 1/7)
We thank the reviewer for recognizing the value of our controlled study of CI, SI, and IDI and for the feedback that helps us better substantiate WELL’s effectiveness. We have supplemented the revised manuscript with more complete empirical evidence, including standard deviations across repeated runs, alternative aggregation and Pareto analyses, drift validation, and an online diagnostic, together with further methodological clarifications. These additions help present WELL’s overall advantages and adaptability to drift more comprehensively. These revisions are highlighted in red.
Q1. Statistical Results [W1]
Appendix Tables 15–17 of the submitted manuscript have reported per-setting averages of BA, Kappa, SPD, and DI for WELL and all baseline methods, including FABBOO and CFSMOTE, together with statistical analysis using Wilcoxon signed-rank tests against WELL. We have added standard deviations over the 10 runs to the corresponding synthetic and real-world comparison tables in Appendix I. In the revised manuscript, these comparison tables are numbered Tables 21–23. Section 8’s “Experiment Setup” now explicitly directs readers to these mean ± SD results and Wilcoxon comparisons.
(1) Complete results.
Appendix Tables 21–23 now report the average BA, Kappa, SPD, DI, and GM4 over 10 independent runs for each setting, covering WELL and all baseline methods. Specifically, Table 21 covers synthetic settings S0–S8, Table 22 covers S9–S18, and Table 23 covers five real-world datasets. The revised tables now present BA, Kappa, SPD, DI, and GM4 as mean ± SD, with means and SDs, respectively.
(2) Statistical tests.
The symbols +, =, and − in Appendix Tables 21–23 indicate that the baseline performs significantly better than WELL, shows no statistically significant difference, or performs significantly worse than WELL, respectively. The statistical analysis is a two-sided Wilcoxon signed-rank test with a threshold of p<0.05.
The examples below additionally report standard deviations. Combining the numerical results in (1) with these statistical tests, we have updated the corresponding tables in Appendix I to report mean ± SD alongside the paired Wilcoxon significance markers, with the captions specifying that + indicates the baseline performs significantly better than WELL, = indicates no statistically significant difference between the baseline and WELL, and − indicates the baseline performs significantly worse than WELL.
To address how many differences exceed seed variation, we summarize the paired Wilcoxon comparisons with FABBOO and CFSMOTE. Each comparison uses the 10 paired runs for one metric in one setting. Across 24 settings, four metrics, and two baselines, this yields 192 comparisons (24×4×2). Of these 192 comparisons, 186 detect statistically significant differences between WELL and the corresponding baseline.