- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates a data-pruning federated learning framework designed to address the challenge of noisy labels. The authors propose a Loss Signature Score (LSS) to characterize the learning difficulty of samples based on their consistent performance, calculated by combining the average loss within a sliding window with the loss gain. By integrating the LSS into a dynamic federated pruning framework, the approach enables adaptive sample selection in scenarios involving heterogeneous data. Experimental results on several datasets show that LSS outperforms existing methods in terms of accuracy.
Critical strengths
-
Overall, the paper is well-written with a clear presentation.
-
Investigating data pruning in federated noisy label learning is a significant problem and direction.
-
Extensive experiments have been conducted to evaluate the performance of the proposed framework.
Critical weaknesses
-
The paper primarily presents overall accuracy results, but it fails to provide sufficient evidence that it can effectively distinguish between noisy samples and hard clean samples. It would be beneficial to include metrics such as the actual noise rate within the retained set (before and after pruning), as well as the precision, recall, or AUROC for the identification of noisy samples.
-
While positioned as federated learning with noisy labels, this work lacks comparisons with methods that directly address this problem.
-
In Table 3, the mean-only baseline also performs very strongly in the CIFAR-100N setting; the overall effectiveness of the entire module therefore requires further clarification.
-
Enforcing category quotas based on observation labels may result in the retention of incorrect labels, and could even increase the noise rate within the retained set.
-
The time window and refresh interval are core parameters; corresponding sensitivity analyses should be conducted.
-
The synthetic noise consists of only 10% and 20% symmetric noise; such noise levels are too low to be sufficiently challenging or practically meaningful.
-
The model definition allows for different noise rates for each client, but the synthetic experiments primarily employ a uniform noise rate.
Questions for authors
Refers to the weakness
2 comments
I still maintain that, for a study on noisy-label FL, comparing with relevant FNLL baselines is fundamental. Without these comparisons, the work is difficult to find convincing.
Although some human-annotated or naturally occurring datasets contain a larger proportion of noise, the maximum level is only around 40%. In contrast, these FNLL baselines typically evaluated performance against synthetic noise at all levels, from low to high, reaching 70% or 80%. Consequently, the noise ratios currently used in synthetic settings are still not solid enough.
These concerns are not difficult to clarify, and baselines like FedCorr have open-source code. Running experiments like CIFAR-10/100 doesn't take much time. Therefore, I will still retain my existing negative score.
Dear Reviewer,
Please take a moment to review the authors' rebuttal and provide your feedback on whether your concerns have been adequately addressed. Thank you for your time and effort.