- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Resampling approaches for mitigating class-wise performance disparities predominantly focus on reallocating training data according to class frequency, however, this setting ignores that classes also differ in learning difficulty. Some following works validate that class hardness provide additional basis for allocating training resources across classes, but these evidence are still mostly under the frequency-imbalanced setting, which can not disentangle the contribution from correcting the representation deficit and hardness-guided resources allocation. This work is therefore motivated to test the hardness-based allocation under the frequency-balanced setting, which attempts to disentangle the above two parts. Two complementary case studies illustrate that hardness-guided resampling strategy could reduce class-wise recall gap among different classes, which shows that frequency imbalance is not the only factor that results in class-wise performance disparity.
Critical strengths
- The problem itself is meaningful, this paper highlights that the inherent learning hardness of a class itself can also lead to issues of class-wise fairness beyond class frequency.
- The comparison among three oversampling strategies (rEDM, aEDM, and hEDM) supports the hardness of oversampled data determines its utility for improving class-wise equity.
Critical weaknesses
- The main investigation in this paper is only two case studies under the frequency-balanced setting, which is too limited to convince the readers. More importantly, this paper is positioned as expanding the hardness-base resampling effectiveness evidence from frequency-imbalanced dataset to frequency-balanced one. As a phenomenon investigation paper, I can not find any additional finding or conclusion from it given similar existing work [1], a comprehensive novelty clarification is needed during the rebuttal.
- In line 74, the author claimed that "This creates an important ambiguity: when ... Testing hardness-based allocation in a frequency-balanced setting addresses this ambiguity". However, it is hard to understand why testing under a frequency-balanced setting could addresses the ambiguity. The hardness-based reallocation on training dataset also changes the number of samples of the hard class, which may result in the observed improvement of class-wise equity.
[1] "Reducing Class Bias In Data-Balanced Datasets Through Hardness-Based Resampling", Pawel Pukowski, Venet Osmani. https://arxiv.org/abs/2504.07031
Questions for authors
- There are two repeated sentences in line 87-89.
- Is it reasonable to use recall gap as a representative metric for class-wise equity? Does the reduction in the recall gap come at the cost of increasing the gap in other class-wise equity metrics?
- The whole section 3 and 4 are centered on the two case studies, is there any deep analysis to help understand why hardness-based reallocation could be effective for reducing class-wise recall gap.
- How to define the "hardness" and estimate it?