- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper aims to address the limitations of the existing detection-and-correction paradigm in federated learning environments with noisy samples. In particular, this paper highlights the issue of delayed detection, arguing that even after noisy clients are identified, their harmful historical influence can still lead to performance degradation. To address this issue, this paper proposes FedHSR, which consists of two key components. First, Historical Replay repairs the historical updates of noisy clients. Second, Historical Response Calibration compensates for the model shift introduced by the historical repair process.
Critical strengths
- The paper is generally well-organized, with clear notation and a smooth, logical flow of ideas. However, there are a few aspects that could be improved. First, although Figure 3 serves as an overview of the proposed framework, it provides limited insight into how Historical Replay and Historical Response Calibration operate. In contrast, Algorithm 1 in the Appendix provides a clearer and more intuitive explanation of the Historical Self-Repair process. Second, Figure 3 contains three instances of , but it is unclear whether they all refer to the same model. It seems that they represent different models, and if so, their distinctions should be explicitly clarified. Finally, Section 3.2 appears to assume that noise begins at round , whereas Section 3.1 denotes the noise onset round as . This inconsistency may cause confusion and should be clarified to ensure consistent notation throughout the paper.
- This paper provides theoretical guarantees for estimating the aggregated responses of clean clients and the resulting trajectory repair, strengthening the theoretical foundation of the proposed approach.
- The idea of repairing the harmful historical influence of noisy clients after their detection appears to be novel. If this approach has not been explored in prior work, this paper could offer a new perspective and a promising research direction for addressing noisy clients in federated learning.
Critical weaknesses
- Initially, FedHSR appeared to be an independent method. However, after reading the entire paper, it seems to be highly dependent on the underlying backbone methods. In particular, the experimental results suggest that FedHSR is applied as a plug-in module to existing methods (e.g., FedNoRo, FedCorr, and FedRG) that already achieve strong performance on specific datasets or under certain experimental settings. In this regard, it does not seem fair to present FedHSR as outperforming these existing methods. Instead, it would be more appropriate to report the results as "Existing Method + Ours" (e.g., "FedRG + Ours") to clearly indicate the contribution of FedHSR. Even taking this into account, the performance improvements achieved by FedHSR appear to be rather marginal.
- Once noisy clients are detected, if a repair process is necessary, why not simply restart the training process from scratch, using all available data from clean clients and leveraging noisy samples from noisy clients through semi-supervised learning, without Historical Replay and Historical Response Calibration? In other words, rather than reverting to a previous training trajectory for repair, it seems reasonable to expect that restarting federated learning from scratch with the corrected samples could achieve comparable or even better performance. Therefore, the necessity of Historical Replay and Historical Response Calibration is not sufficiently clear, and further justification for these components is needed.
- The experimental evaluation is somewhat limited in terms of the datasets considered. It would be beneficial to conduct additional experiments on mini-WebVision, as in [1], or the ISIC dataset, as in [2], to further validate the effectiveness of the proposed method. Moreover, since the method presented in [1] appears to achieve strong performance under similar experimental settings, it would be important to include this method as an additional baseline for a more comprehensive performance comparison.
[1] Kim, Sangmook, et al. "FedRN: Exploiting k-reliable neighbors towards robust federated learning." Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 2022.
[2] Wu, Nannan, et al. "FedNoRo: Towards noise-robust federated learning by addressing class imbalance and label noise heterogeneity." arXiv preprint arXiv:2305.05230 (2023).
Questions for authors
- What are the computational costs of the compared methods, including FedHSR? In particular, does FedHSR incur additional computational overhead during the repair process compared to the backbone methods? It would be helpful to provide a comparison of computational costs across all methods.
- The experimental setting used in Table 3 does not seem to be clearly specified. Which dataset and experimental configuration were used for these results? I assume that the experiments were conducted on CIFAR-10 with ( = 1.0). However, if this is the case, the backbone performance reported in Table 3 does not match the corresponding results of FedNoRo. Could the authors clarify this discrepancy?
- As mentioned in the Weaknesses section, what would be the performance if federated learning were restarted from scratch using the corrected samples, without Historical Replay and Historical Response Calibration? Such a comparison would help clarify the necessity of these two components.
- What would be the performance under two additional settings: (1) when all training samples are clean (i.e., the performance upper bound) and (2) when all noisy samples are removed before training? These results would provide useful reference points for assessing the effectiveness of FedHSR.
6 comments
Response to Reviewer jLeS
W3. Additional datasets and method.
Thank you for suggesting mini-WebVision, ISIC, and FedRN. We evaluated both datasets under federated protocol in table 1 (Dirichlet β=1) in our paper.
Results on ISIC2019.
| Method | Accuracy (%) |
|---|---|
| FedAvg | 22.99 |
| DivideMix | 19.02 |
| FedCorr | 17.14 |
| FedNoRo | 25.20 |
| FedFixer | 20.58 |
| FedDiv | 19.61 |
| FedELC | 25.24 |
| FedClean | 17.00 |
| FedRG | 18.89 |
| FedHSR(Noro) (ours) | 26.26 |
mini-WebVision.
| Method | Accuracy (%) |
|---|---|
| FedAvg | 29.68 |
| DivideMix | 17.00 |
| FedRN | 23.28 |
| FedCorr | 33.24 |
| FedNoRo | 32.56 |
| FedFixer | 30.00 |
| FedDiv | 24.32 |
| FedELC | 26.60 |
| FedClean | 13.92 |
| FedRG | 22.48 |
| FedHSR(corr) (ours) | 35.84 |
From this, we can observe that our method still perform best on the new dataset. Note that FedRN does not outperform FedAvg on the mini-WebVision dataset. This may be due to differences in our dataset protocol. However, we ensure that our protocol is consistent with the one described in our paper and aligns with the standard protocol schemes commonly used in existing methods.
FedRN
We also evaluated FedRN under the federated protocol in table 1 in our paper.
| Dataset | Dirichlet β | Label noise | FedRN (%) | FedHSR(ours) (%) |
|---|---|---|---|---|
| CIFAR-10 | 1.0 | Symmetric | 56.24 | 76.97 |
| CIFAR-10 | 1.0 | Pair | 50.15 | 77.84 |
| CIFAR-10 | 1.0 | Mixed | 54.06 | 77.83 |
| CIFAR-10 | 0.5 | Symmetric | 46.47 | 72.65 |
| CIFAR-10 | 0.5 | Pair | 46.82 | 71.76 |
| CIFAR-10 | 0.5 | Mixed | 44.75 | 75.20 |
| CIFAR-100 | 1.0 | Symmetric | 35.30 | 54.27 |
| CIFAR-100 | 1.0 | Pair | 33.90 | 55.44 |
| CIFAR-100 | 1.0 | Mixed | 37.72 | 54.59 |
| CIFAR-100 | 0.5 | Symmetric | 32.26 | 46.31 |
| CIFAR-100 | 0.5 | Pair | 30.21 | 48.93 |
| CIFAR-100 | 0.5 | Mixed | 32.99 | 49.62 |
From this, we can observe that the performance of FedRN is not good, and there remains a gap compared to the ours result presented in our Table 1.
Q2. Discrepancy settings in Table 3.
Thank you for pointing out this discrepancy. We did not clearly document the experimental settings in Table 3. Table 1 reports the real FedNoRo reference results and Table 3 use the cifar-10 with β=1 with wrong results. We will revise Table 3 and its description in our manuscript.
Overall
Thank you again for reading each part of our paper so carefully. We hope the additional experiments address your concerns about the range of datasets and that our new ablations clarify the contributions of Historical Replay and Historical Response Calibration. Your suggestions on the presentation will help us make the revised manuscript clearer. If our response addresses your concerns, we would be grateful if you would consider raising your rating.
Thank you for providing the additional experimental results and clarifications.
However, I remain concerned that the contribution of FedHSR is relatively limited based on these results.
First, although FedHSR reduces historical repair time by 34.2% compared to restarting from scratch, I do not consider this reduction sufficiently substantial to justify the proposed framework, particularly given the additional complexity it introduces. Moreover, the 1.35 percentage-point accuracy gap between FedHSR (60.05%) and restarting from scratch (61.40%) is not negligible.
Second, since FedHSR operates as a plug-in framework relying on existing detectors, its practical application requires selecting an appropriate detector that performs well under a given setting. Identifying and evaluating such detectors may introduce additional computational and experimental costs, which do not appear to be reflected in the reported efficiency comparison.
Therefore, I am not yet convinced that FedHSR provides a sufficiently compelling performance-efficiency trade-off to establish a strong contribution over existing approaches. I would appreciate further justification of whether the computational savings meaningfully outweigh the accuracy degradation and additional detector-selection costs in practical federated learning settings.
Response to Reviewer jLeS
We sincerely thank you for carefully reading our paper and acknowledging its ideas; your suggestions have been of great help in refining our manuscript. Below is our response.
S1. Inconsistency in Figure 3 and Section 3.
We are very grateful for your careful review of every part of the paper, we will revise then in the manuscript.
W1. Relation with existing detectors.
Thank you for your careful reading of our paper. The core contribution of FedHSR is a repair framework which can draw on any existing detectors. Therefore, it is indeed more appropriate to describe it as "existing detector + our approach." We will address this in the manuscript.
In addition, we agree that repair quality also depends on the detector. However, we believe FedHSR has two advantages in this setting.
First is flexibility. FedHSR allows us to choose a detector suited to each setting, such as FedNoRo for client-level detection and FedRG for sample-level detection. This flexibility allows the same historical repair framework to extend across datasets, noise settings, and detection granularities.
Second is effectiveness. We conduct oracle experiments with the corrected detector in our paper, where FedHSR improves over existing methods by 2.32 percentage points. This demonstrates that our repair method can still achieve improvements without the impact from detector quality.
W2, Q1, Q3 and Q4. Contribution on effectiveness and computational costs.
Thank you for your suggestion. We would like to restate our contributions here. We identify that the delayed detection in existing methods will impact federated training, therefore, we propose FedHSR to repair this historical training trajectory.
Based on your review, we use two variants to describe our adventage:
a. no-repair, which ignores delayed detection issues and continue training, is highly efficient but preform poorly.
b. restarted from scratch, which retrains the whole trajectory, is high performance but poor efficiency.
Therefore, our approach represents a trade-off: it achieves results comparable to restarted from scratch methods, while reducing computational cost by the response calibration.
To demonstrate this conclusion, we conducted a comprehensive set of experiments that covered every scenario from your reviews, and also included computational costs you asked about.
a. Clean Fedavg (Q4). Training Fedavg framework with a clean dataset which did not add any noise samples.
b. Filtered Fedavg (Q4). Training Fedavg framework with a filtered dataset which delated noise samples.
c. Noisy Fedavg. Training Fedavg framework with a noisy dataset which add noise samples.
d. No-repair (W2,Q3). Training Fedavg framework with a noisy dataset and fix noise samples after detection.
e. Restarted from scratch (W2,Q3). Training Fedavg framework with a noisy dataset and retrain the whole trajectory after detection.
g. Restarted from scratch with FixMatch (W2). Training Fedavg framework with a noisy dataset, and using all available data from clean clients and leveraging noisy samples from noisy clients through semi-supervised learning.
f. Our FedHSR. Training Fedavg framework with a noisy dataset and repair the training trajectory.
We evaluated both datasets on oracle detector on CIFAR-10 and inject label noise into 30 of 100 clients, with a label corruption rate of 60%–80% within those clients.
It can be observed that, under oracle detection conditions, retraining (Restarted from scratch) can yield results comparable to ideal setting using a clean dataset (Clean Fedavg). However, simply removing noisy samples (Filtered Fedavg) does not produce good results. Furthermore, merely correcting samples without fixing historical trajectories performs ineffective (No-repair), while employing a semi-supervised training approach (Restarted from scratch with FixMatch) in such cases also perform worse results. In contrast, our method (Our FedHSR) achieves results behind only and nearly those from retraining (Restarted from scratch), while using 34.2% less time for historical repair.