acceptodds
Under review as a conference paper at ICLR 2027

Historical Self-Repair for Federated Learning with Delayed Detection of Noisy Labels

Abstract

Federated learning with noisy labels aims to train robust global models in federated settings where clients may hold noisy labels. Most existing methods follow a detection-and-correction paradigm: they identify noisy clients or samples during training and correct the detected noise through label correction, filtering, or robust optimization. However, this paradigm faces the problem of delayed detection, where some noisy samples are detected after multiple communication rounds. Although these samples can be corrected after being detected, their updates have already affected the global model during the earlier training step, leaving harmful historical influence on subsequent optimization. To address this problem, we propose FedHSR, a historical self-repair framework for robust model training. FedHSR maintains the global training trajectory and repairs the affected historical trajectory through two modules. Historical Replay reconstructs the historical updates of noisy clients based on the detected data, while Historical Response Calibration estimates the responses of clean clients to the model shift induced by historical repair without requiring them to repeat their historical training. We further establish theoretical bounds for both response estimation and trajectory repair. Experiments show that FedHSR effectively repairs the affected trajectory and consistently improves the performance of existing methods. Code is available at https://anonymous.4open.science/r/ICLR-59567.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
federated learning, historical self-repair

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jLeSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to address the limitations of the existing detection-and-correction paradigm in federated learning environments with noisy samples. In particular, this paper highlights the issue of delayed detection, arguing that even after noisy clients are identified, their harmful historical influence can still lead to performance degradation. To address this issue, this paper proposes FedHSR, which consists of two key components. First, Historical Replay repairs the historical updates of noisy clients. Second, Historical Response Calibration compensates for the model shift introduced by the historical repair process.

Critical strengths
  1. The paper is generally well-organized, with clear notation and a smooth, logical flow of ideas. However, there are a few aspects that could be improved. First, although Figure 3 serves as an overview of the proposed framework, it provides limited insight into how Historical Replay and Historical Response Calibration operate. In contrast, Algorithm 1 in the Appendix provides a clearer and more intuitive explanation of the Historical Self-Repair process. Second, Figure 3 contains three instances of wt+1w_{t+1}, but it is unclear whether they all refer to the same model. It seems that they represent different models, and if so, their distinctions should be explicitly clarified. Finally, Section 3.2 appears to assume that noise begins at round tt, whereas Section 3.1 denotes the noise onset round as t0t_0. This inconsistency may cause confusion and should be clarified to ensure consistent notation throughout the paper.
  2. This paper provides theoretical guarantees for estimating the aggregated responses of clean clients and the resulting trajectory repair, strengthening the theoretical foundation of the proposed approach.
  3. The idea of repairing the harmful historical influence of noisy clients after their detection appears to be novel. If this approach has not been explored in prior work, this paper could offer a new perspective and a promising research direction for addressing noisy clients in federated learning.
Critical weaknesses
  1. Initially, FedHSR appeared to be an independent method. However, after reading the entire paper, it seems to be highly dependent on the underlying backbone methods. In particular, the experimental results suggest that FedHSR is applied as a plug-in module to existing methods (e.g., FedNoRo, FedCorr, and FedRG) that already achieve strong performance on specific datasets or under certain experimental settings. In this regard, it does not seem fair to present FedHSR as outperforming these existing methods. Instead, it would be more appropriate to report the results as "Existing Method + Ours" (e.g., "FedRG + Ours") to clearly indicate the contribution of FedHSR. Even taking this into account, the performance improvements achieved by FedHSR appear to be rather marginal.
  2. Once noisy clients are detected, if a repair process is necessary, why not simply restart the training process from scratch, using all available data from clean clients and leveraging noisy samples from noisy clients through semi-supervised learning, without Historical Replay and Historical Response Calibration? In other words, rather than reverting to a previous training trajectory for repair, it seems reasonable to expect that restarting federated learning from scratch with the corrected samples could achieve comparable or even better performance. Therefore, the necessity of Historical Replay and Historical Response Calibration is not sufficiently clear, and further justification for these components is needed.
  3. The experimental evaluation is somewhat limited in terms of the datasets considered. It would be beneficial to conduct additional experiments on mini-WebVision, as in [1], or the ISIC dataset, as in [2], to further validate the effectiveness of the proposed method. Moreover, since the method presented in [1] appears to achieve strong performance under similar experimental settings, it would be important to include this method as an additional baseline for a more comprehensive performance comparison.

[1] Kim, Sangmook, et al. "FedRN: Exploiting k-reliable neighbors towards robust federated learning." Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 2022.

[2] Wu, Nannan, et al. "FedNoRo: Towards noise-robust federated learning by addressing class imbalance and label noise heterogeneity." arXiv preprint arXiv:2305.05230 (2023).

Questions for authors
  1. What are the computational costs of the compared methods, including FedHSR? In particular, does FedHSR incur additional computational overhead during the repair process compared to the backbone methods? It would be helpful to provide a comparison of computational costs across all methods.
  2. The experimental setting used in Table 3 does not seem to be clearly specified. Which dataset and experimental configuration were used for these results? I assume that the experiments were conducted on CIFAR-10 with (β\beta = 1.0). However, if this is the case, the backbone performance reported in Table 3 does not match the corresponding results of FedNoRo. Could the authors clarify this discrepancy?
  3. As mentioned in the Weaknesses section, what would be the performance if federated learning were restarted from scratch using the corrected samples, without Historical Replay and Historical Response Calibration? Such a comparison would help clarify the necessity of these two components.
  4. What would be the performance under two additional settings: (1) when all training samples are clean (i.e., the performance upper bound) and (2) when all noisy samples are removed before training? These results would provide useful reference points for assessing the effectiveness of FedHSR.
6 comments
AuthorsOpenReview ↗

Response to Reviewer jLeS

We sincerely thank you for carefully reading our paper and acknowledging its ideas; your suggestions have been of great help in refining our manuscript. Below is our response.

S1. Inconsistency in Figure 3 and Section 3.

We are very grateful for your careful review of every part of the paper, we will revise then in the manuscript.

W1. Relation with existing detectors.

Thank you for your careful reading of our paper. The core contribution of FedHSR is a repair framework which can draw on any existing detectors. Therefore, it is indeed more appropriate to describe it as "existing detector + our approach." We will address this in the manuscript.

In addition, we agree that repair quality also depends on the detector. However, we believe FedHSR has two advantages in this setting.

  • First is flexibility. FedHSR allows us to choose a detector suited to each setting, such as FedNoRo for client-level detection and FedRG for sample-level detection. This flexibility allows the same historical repair framework to extend across datasets, noise settings, and detection granularities.

  • Second is effectiveness. We conduct oracle experiments with the corrected detector in our paper, where FedHSR improves over existing methods by 2.32 percentage points. This demonstrates that our repair method can still achieve improvements without the impact from detector quality.

W2, Q1, Q3 and Q4. Contribution on effectiveness and computational costs.

Thank you for your suggestion. We would like to restate our contributions here. We identify that the delayed detection in existing methods will impact federated training, therefore, we propose FedHSR to repair this historical training trajectory.

Based on your review, we use two variants to describe our adventage:

  • a. no-repair, which ignores delayed detection issues and continue training, is highly efficient but preform poorly.

  • b. restarted from scratch, which retrains the whole trajectory, is high performance but poor efficiency.

Therefore, our approach represents a trade-off: it achieves results comparable to restarted from scratch methods, while reducing computational cost by the response calibration.

To demonstrate this conclusion, we conducted a comprehensive set of experiments that covered every scenario from your reviews, and also included computational costs you asked about.

  • a. Clean Fedavg (Q4). Training Fedavg framework with a clean dataset which did not add any noise samples.

  • b. Filtered Fedavg (Q4). Training Fedavg framework with a filtered dataset which delated noise samples.

  • c. Noisy Fedavg. Training Fedavg framework with a noisy dataset which add noise samples.

  • d. No-repair (W2,Q3). Training Fedavg framework with a noisy dataset and fix noise samples after detection.

  • e. Restarted from scratch (W2,Q3). Training Fedavg framework with a noisy dataset and retrain the whole trajectory after detection.

  • g. Restarted from scratch with FixMatch (W2). Training Fedavg framework with a noisy dataset, and using all available data from clean clients and leveraging noisy samples from noisy clients through semi-supervised learning.

  • f. Our FedHSR. Training Fedavg framework with a noisy dataset and repair the training trajectory.

We evaluated both datasets on oracle detector on CIFAR-10 and inject label noise into 30 of 100 clients, with a label corruption rate of 60%–80% within those clients.

Methodtest accuracy (%)Regular training (s)Historical repair (s)
Clean Fedavg61.40815.90.0
Filtered Fedavg56.78647.10.0
Noisy Fedavg53.35813.00.0
No-repair59.17818.10.0
Restarted from scratch61.40817.5487.0
Restarted from scratch with FixMatch57.89827.2499.1
Our FedHSR60.05822.8320.5

It can be observed that, under oracle detection conditions, retraining (Restarted from scratch) can yield results comparable to ideal setting using a clean dataset (Clean Fedavg). However, simply removing noisy samples (Filtered Fedavg) does not produce good results. Furthermore, merely correcting samples without fixing historical trajectories performs ineffective (No-repair), while employing a semi-supervised training approach (Restarted from scratch with FixMatch) in such cases also perform worse results. In contrast, our method (Our FedHSR) achieves results behind only and nearly those from retraining (Restarted from scratch), while using 34.2% less time for historical repair.

AuthorsOpenReview ↗

Response to Reviewer jLeS

W3. Additional datasets and method.

Thank you for suggesting mini-WebVision, ISIC, and FedRN. We evaluated both datasets under federated protocol in table 1 (Dirichlet β=1) in our paper.

Results on ISIC2019.

MethodAccuracy (%)
FedAvg22.99
DivideMix19.02
FedCorr17.14
FedNoRo25.20
FedFixer20.58
FedDiv19.61
FedELC25.24
FedClean17.00
FedRG18.89
FedHSR(Noro) (ours)26.26

mini-WebVision.

MethodAccuracy (%)
FedAvg29.68
DivideMix17.00
FedRN23.28
FedCorr33.24
FedNoRo32.56
FedFixer30.00
FedDiv24.32
FedELC26.60
FedClean13.92
FedRG22.48
FedHSR(corr) (ours)35.84

From this, we can observe that our method still perform best on the new dataset. Note that FedRN does not outperform FedAvg on the mini-WebVision dataset. This may be due to differences in our dataset protocol. However, we ensure that our protocol is consistent with the one described in our paper and aligns with the standard protocol schemes commonly used in existing methods.

FedRN

We also evaluated FedRN under the federated protocol in table 1 in our paper.

DatasetDirichlet βLabel noiseFedRN (%)FedHSR(ours) (%)
CIFAR-101.0Symmetric56.2476.97
CIFAR-101.0Pair50.1577.84
CIFAR-101.0Mixed54.0677.83
CIFAR-100.5Symmetric46.4772.65
CIFAR-100.5Pair46.8271.76
CIFAR-100.5Mixed44.7575.20
CIFAR-1001.0Symmetric35.3054.27
CIFAR-1001.0Pair33.9055.44
CIFAR-1001.0Mixed37.7254.59
CIFAR-1000.5Symmetric32.2646.31
CIFAR-1000.5Pair30.2148.93
CIFAR-1000.5Mixed32.9949.62

From this, we can observe that the performance of FedRN is not good, and there remains a gap compared to the ours result presented in our Table 1.

Q2. Discrepancy settings in Table 3.

Thank you for pointing out this discrepancy. We did not clearly document the experimental settings in Table 3. Table 1 reports the real FedNoRo reference results and Table 3 use the cifar-10 with β=1 with wrong results. We will revise Table 3 and its description in our manuscript.

Overall

Thank you again for reading each part of our paper so carefully. We hope the additional experiments address your concerns about the range of datasets and that our new ablations clarify the contributions of Historical Replay and Historical Response Calibration. Your suggestions on the presentation will help us make the revised manuscript clearer. If our response addresses your concerns, we would be grateful if you would consider raising your rating.

Thank you for providing the additional experimental results and clarifications.

However, I remain concerned that the contribution of FedHSR is relatively limited based on these results.

First, although FedHSR reduces historical repair time by 34.2% compared to restarting from scratch, I do not consider this reduction sufficiently substantial to justify the proposed framework, particularly given the additional complexity it introduces. Moreover, the 1.35 percentage-point accuracy gap between FedHSR (60.05%) and restarting from scratch (61.40%) is not negligible.

Second, since FedHSR operates as a plug-in framework relying on existing detectors, its practical application requires selecting an appropriate detector that performs well under a given setting. Identifying and evaluating such detectors may introduce additional computational and experimental costs, which do not appear to be reflected in the reported efficiency comparison.

Therefore, I am not yet convinced that FedHSR provides a sufficiently compelling performance-efficiency trade-off to establish a strong contribution over existing approaches. I would appreciate further justification of whether the computational savings meaningfully outweigh the accuracy degradation and additional detector-selection costs in practical federated learning settings.

Reviewer_rWbdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This submission focus on the federated learning with imperfect labels. Existing FLNL methods intend to identify and rectify those wrong samples. However, this submission argue that the noisy label detection is delayed and these wrong samples already contributed to the weight updating. Reverting these damage takes a longer training. To this end, this submission proposes FedHSR with adopt historical replay recompute revert the adversary effect of the noisy labels. The results on CIFAR-10/100 show that the proposed FedHSR improves the label-noise robustness.

Critical strengths
  • Interesting motivation and method design: this submission proposes a method to deal with the indirect historical adversary effect of noisy labels in federating learning. It may inspire the community to introducing unlearning techniques to develop better FLNL methods.
  • The proposed methods is plug-and-play: the historical replay recompute is orthogonal to other FLNL methods and can further improve the label-noise robustness.
  • Comprehensive improvement: the reported performance improvement is consistent and the repairing speed is improved 1.27–1.75× over full historical retraining.
Critical weaknesses
  • Limited evaluations: Lack of experimental results on the Clothing1M which contains the large-scale real noisy labels;
  • Limited literature review: No concept comparison between FedRecover[1], FedNLU[2], and FedGR[3]
  • The effectiveness of this response diagnosis method is less impressive. A relative error of 1 corresponds to the predicted response is zero; yet, in the provided illustrations, the error for FedHSR frequently exceeds 1. The mere alignment of scalar signs is insufficient to demonstrate an accurate recovery of vector responses.
  • The delayed report requires verification: There is a discrepancy between the improvements in the endpoint metrics shown in Table 6 and the average gains indicated in Figure 5.

[1] FedRecover: Recovering from Poisoning Attacks in Federated Learning using Historical Information

[2] FedNLU: Robust federated learning with noisy label unlearning

[3] Learning Locally, Revising Globally: Global Reviser for Federated Learning with Noisy Labels

Questions for authors

Please refer to the weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer rWbd

We sincerely thank you for your careful reading and review comments, particularly your observations on our experimental results, which have helped us develop a more robust piece of work. Below is our response.

W1. Evaluation on large-scale datasets

Thank you for suggesting large-scale datasets. We evaluated Clothing1M under federated protocol in table 1 (Dirichlet β=1) in our paper.

MethodAccuracy (%)
FedAvg67.26
DivideMix49.82
FedCorr63.80
FedNoRo67.05
FedFixer68.56
FedDiv57.00
FedELC64.34
FedRG62.42
FedHSR68.67

As can be seen, our method also performs the best on large-scale datasets.

W2. Comparison with FedRecover, FedNLU, and FedGR

Thank you for suggesting these related papers. We provide the following clarifications.

  • FedRecover removes all noisy clients, however, these client samples still contain useful information to the server model, removing these samples directly can reduce training performance. In contrast, our method mitigates the impact of these noisy samples, thereby yielding better results.
  • FedNLU mitigate the impact of such noisy samples through unlearning, this technique cannot fully mitigate the impact of noisy information in the server model. In contrast, our method employs a retraining strategy that only incorporates the corrected samples into the model, leading to further performance improvements.
  • FedGR follows a "detect-and-repair" paradigm but overlooks the issue of "delayed detection" that we have identified. In contrast, our method repairs the history trajectory to achieve better performance.

We will incorporate these distinctions into the revised manuscript.

We also conducted experiments to compare these methods. For FedNLU and FedGR, we performed the comparison under federated protocol in table 1 (Dirichlet β=1) in our paper. It is worth noting that the code for these two methods has not been made public, we can only attempt to reproduce them based on our understanding of the paper. For FedRecover, since it assumes that the noise is directly observable, which is different from our settings in table 1, we employed an oracle federated protocol with correct noisy samples in our paper. We used the same CIFAR-10 oracle setting as in our response to Reviewer jLeS, with 30 noisy clients out of 100, and 60%–80% label corruption within those clients.

The results are as follows:

FedNLU and FedGR

DatasetDirichlet βNoiseFedNLUFedGRFedHSR (ours)
CIFAR-101.0Sym62.2835.1076.97
CIFAR-101.0Pair60.6032.4277.84
CIFAR-101.0Mixed57.8033.6777.83
CIFAR-100.5Sym51.8830.6972.65
CIFAR-100.5Pair46.8730.4271.76
CIFAR-100.5Mixed42.4328.7775.20
CIFAR-1001.0Sym31.2612.7254.27
CIFAR-1001.0Pair30.7613.4055.44
CIFAR-1001.0Mixed34.668.9254.59
CIFAR-1000.5Sym26.557.5546.31
CIFAR-1000.5Pair29.887.3048.93
CIFAR-1000.5Mixed26.957.5549.62

FedRecover

MethodAccuracy (%)
FedAvg53.35
FedRecover56.51
FedHSR60.05

As we have shown, FedNLU is unable to completely unlearn noisy content, while FedGR fails to repair the noisy history, consequently, both of them perform worse than our method. Furthermore, under oracle setting with known noisy clients, FedRecover fails to fully leverage the useful information contained within those noisy clients, resulting in worse performance than that of our FedHSR.

W3. Relative error and vector response diagnostics

Thank you for pointing this out. We acknowledge that our figure obscured the relative response error between the "repaired" and "unrepaired" versions. This occurred because the responses were generated using an excessively large step size, which introduced significant fluctuations and led to worse results than those of the unrepaired method. Furthermore, we acknowledge our metric lacked a comparison in terms of angles.

To clarify these, we will provide a detailed breakdown of the corrected results.

We reran the oracle experiment. We report the mean relative response error and track whether the error for each epoch exceeds 1. Additionally, we add an angular comparison to analyze discrepancies in direction.

MethodMean relative response error ↓Instances with error < 1Mean update angle ↓
No response1.0000/8453.88°
FedHSR0.92184/8450.04°

It is worth noting that, due to the high dimensionality of the latent space, the angle cannot be adjusted to near zero. However, our method allows both the angle and distance to be closer to the ground-truth trajectory across all epochs.

AuthorsOpenReview ↗

Response to Reviewer rWbd

W4. Table 6 and Figure 5

Thank you for pointing this out. In Figure 5, we present the average gain over 40 rounds, which is why the results differ, we will clarify this in the manuscript.

Overall

Thank you again for your thoughtful review. We hope the additional experiments address your concerns about the breadth of our datasets and baseline comparisons, and also clarify how well FedHSR estimates the response vector during historical repair. Your careful reading of the experiments also helped us clarify the experimental settings in the manuscript. If these additions address your concerns, we would be grateful if you would consider raising your rating.

Thanks for your response. I am curious that the result about the FedNLU and FedGR. The reported results is completely different from their originally reported performance even their setups are more diffcult, i.g., β=0.3\beta = 0.3. And the improvement on Clothing1m is marginal. If possible, please explain and open-source the exp code repos.

Thank you for taking the time to review our experiments carefully. We address your two concerns below.

First, on Clothing1M, the results we initially reported used FedNoRo as the detector. Our method improved accuracy from 67.05% to 68.67%. Your concern about the small gain may come from comparing this result with FedFixer’s stronger baseline of 68.56%. Due to time constraints, we had not yet completed the run with FedFixer. That run is now complete, and adding our method to FedFixer improves accuracy from 68.56% to 69.58%. We hope these gains address your concern that the improvement is marginal.

Second, regarding the lower results for FedNLU and FedGR, neither paper provides public code, so we implemented both methods based on their descriptions. We have shared our code in the anonymous repository https://anonymous.4open.science/r/ICLR-59567. If you spot any issues in our implementations, we would appreciate your feedback and can promptly make corrections and rerun the comparisons.

If our response addresses your concerns about the experimental results, we kindly ask that you reconsider your score. Thank you again for your time and feedback.

Reviewer_o8CKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper addresses the model repair issue in federated learning when noisy labels are delayed in being discovered. Since incorrect labels have already affected the previous training, simply correcting the labels and continuing the training is insufficient to eliminate the accumulated historical impact. Therefore, the authors propose FedHSR, which utilizes the historical global model trajectory to replay the local training of the affected clients after correcting the labels, and calibrates the approximate updates of the remaining clients on the repair trajectory through historical responses, thereby reducing the cost of full re-training. The paper further provides theoretical analysis and experimental evaluation to discuss the repair effect and computational efficiency of this method.

Critical strengths

1.the problem is of practical significance. This paper focuses on the impact that noise labels have had on the model before they were discovered, shifting the research focus from the identification and correction of current labels to the restoration of the historical training process. This setting takes into account the reality that noise detection often has a delay, and also provides a valuable perspective for evaluating label correction methods. 2. the method design reflects consideration of the cost of restoration. The authors replay the affected clients and use historical information to approximate the updates of the remaining clients, avoiding the requirement for all clients to re-participate in the complete training for each restoration. The scalar response estimation at the aggregation level is relatively simple, and it also reduces the need for storing the historical updates of each client individually, providing a worthy exploration option for the trade-off between restoration effect, computational cost, and storage cost.

Critical weaknesses

1: The differences are insufficient compared to history-based federated recovery methods. FedRecover has advanced model recovery through delayed detection: malicious clients may have already polluted the global model before sufficient evidence accumulates to identify them. It also leverages stored historical information to reconstruct the recovered model and estimate client updates, thereby avoiding full retraining on clients. FedHSR shares the same motivation and recovery structure. Its distinguishable differences lie in corrected client replay, scalar response estimation at the aggregation level, and different trade-offs between storage and recomputation. While these differences may be valuable, the paper does not demonstrate their advantages over the closest prior methods. To fully validate its algorithmic contribution, it is necessary to combine the same corrected client replay mechanism with a FedRecover-style update estimator and conduct comparative evaluations based on metrics such as storage, communication, computation, and recovery error. 2: The theoretical analysis does not adequately cover the actual federated training process. When deriving error bounds for response approximation, the paper directly treats clients' local updates as single-step gradient updates computed at the current global model. However, in practice, experiments employ multi-step local training, where clients upload accumulated model changes along their local optimization trajectory. The resulting response depends not only on the gradient at the starting point but also on the number of local steps, learning rate, and intermediate iteration positions. For training methods involving round-dependent objectives or persistent client states, updates may further depend on historical states, going beyond the assumptions made in the current analysis. Therefore, even if the underlying loss function satisfies the required smoothness and strong convexity conditions, existing error bounds cannot be directly applied to real-world multi-step local updates. This gap weakens the theoretical support for the proposed algorithm. We recommend that the authors clearly specify under which training settings their conclusions hold and provide additional analysis on multi-step local updates, explaining how the number of local steps and state dependencies affect response estimation errors and the stability of the correction process.

Questions for authors
  1. the theoretical analysis treats client-side local updates as single-step gradient updates, whereas the experiments employ multi-step local training. Could the authors clarify the scope of applicability of existing theoretical results and specify what additional assumptions are required when transitioning from single-step gradients to multi-step local updates? How do changes in the number of local training steps and learning rate affect the response approximation error?

  2. for training methods involving round-dependent objectives or persistent client states, how does the repair process handle these objectives and states? Is the relevant historical state synchronized and reconstructed during replay? If only the model trajectory is corrected while preserving the original states, would this affect the consistency between the repaired result and a full retraining with corrected labels?

  3. to what extent can response estimates on historical trajectories be applied to the repaired training trajectory? When noise is detected late, affects many clients, or data heterogeneity is high, the two trajectories may diverge significantly. Can the authors conduct targeted experiments to demonstrate the reliability of response approximation under such conditions, and whether approximation errors accumulate over replay rounds?

3 comments
AuthorsOpenReview ↗

Response to Reviewer o8CK

We sincerely thank you for reading our paper carefully and recognizing the practical importance of repairing the effects of delayed label detection. Below is our response.

W1. Comparison with FedRecover

Thank you for suggesting FedRecover. We provide the following clarifications.

The key difference is that FedRecover addresses model poisoning by malicious clients, while our FedHSR addresses the historical effects of label noise after delayed detection. These settings lead to two adventages of our FedHSR.

  • First, since the poisoned clients are malicious, FedRecover removes these clients. In our FedHSR, clients with noisy labels still hold useful information. FedHSR retains these clients and uses their correctly labeled data, which can improve model performance.
  • Second, since the poisoned clients are sparse and distinct, FedRecover stores every client gradients in every epoch from the original training, while our FedHSR considers settings where detection can take many rounds. As Figure 1 shows, some detectors still detect mislabeled samples after 100 rounds. Storing every client’s update over such a long period can be costly. Even with 100 clients training a ResNet-18, the history can take tens of gigabytes. This storage cost can become prohibitive in practice. In our FedHSR, as we emphasize in the manuscript, we only store the history of the server model, which significantly reduces resource consumption.

We also conducted experiments to illustrate this advantage.

We report the CIFAR-10 results with three methods shared the same first 60 training rounds, with 30 of 100 clients affected by 60%–80% pairwise label corruption.

  • FedHSR our methods.
  • FedRecover delate noisy clients and repair trajectory with its Estimator.
  • our Historical Replay + FedRecover Estimator repair noisy clients by our Historical Replay and repair trajectory with FedRecover Estimator.
MethodPeak test accuracy (%) ↑Regular training (s) ↓Historical repair (s) ↓Total time (s) ↓Historical storage (GB) ↓
FedRecover56.511,372.1713.92,717.436.50
our Historical Replay + FedRecover Estimator60.51853.4911.02,190.139.78
our FedHSR60.05815.0323.01,563.56.64

As we introduced, FedRecover performs poorly in terms of effectiveness, time cost and memory cost. While its specialized "Estimator" component can indeed yield little better results, it comes at the cost of extremely high memory usage (6x than ours). Furthermore, as the number of training rounds increases (such as 500 or 1,000 rounds), the memory consumption of FedRecover grows linearly. In contrast, our method only stores the server model for each round, resulting in a much slower rate of growth in resource requirements compared to FedRecover.

W2 and Q1. Theory

Thank you for pointing out the difference between our theory and experiments. We agree that the single-step assumption in Proposition 1 differs from the multi-step mini-batch training used in practice. This simplification follows prior work in federated learning. For example, FedPM limits its convergence analysis to one local update and uses multiple local steps in its experiments. Ishii et al., AAAI 2026

However, we have also updated the bound in our Proposition 1, incorporating a discussion regarding mini-batches. For a single full-gradient update with the same total step size, Proposition 1 gives the range [−KηLt,−Kημt][-K\eta L_t,-K\eta\mu_t] for ata_t. We assume that mini-batch and full gradients share the same norm bound in the local repair region. Accounting the mini-batch sampling gives the following range for ata_t under multi-step local SGD.

\le a_t \le -K\eta\mu_t+\frac{2D}{\|s_t\|_2}, \qquad D=2K\eta G+\frac{LG\eta^2K(K-1)}{2}.$$ The true response depends on updates from two starting models, so its change is at most $2D$. Together with the change in $a_t$, this gives the following bound. $$\lVert R_t-\widehat R_t\rVert_2 \le K\eta(L_t-\mu_t)\lVert\delta_{t-1}\rVert_2+\alpha_t, \qquad \alpha_t=2D\left(1+\frac{\lVert\delta_{t-1}\rVert_2}{\lVert s_t\rVert_2}\right).$$ We will clarify the single-step assumption and add this derivation to the manuscript. The bound shows how the number of local steps and the learning rate affect the response estimate. We also tested these changes in CIFAR-10. The table reports the mean across the four rounds. | Local training setting | Replay Only relative error | FedHSR relative error | |---|---:|---:| | Half the local steps | 1.000 | **0.915** | | Half the learning rate | 1.000 | **0.892** | In both settings, FedHSR had a lower relative response error than Replay Only at all four rounds. This indicates that these changes will not result in a failure response.
AuthorsOpenReview ↗

Response to Reviewer o8CK

Q2. Client states during replay

Thank you for pointing out how client states carried across rounds can affect replay. In FedRG, as we repair the global model trajectory, we also recompute the label transition matrix, class prototypes, and confidence statistics of the affected clients at each round. This prevents the detected wrong labels from continuing to affect training through these states. This also avoids differences between replay and full retraining with corrected labels that would arise from reusing the affected clients' original states.

Q3. Response error over replay rounds

Thank you for asking how response errors change over replay rounds. We ran a CIFAR-10 oracle experiment. The trajectory error is the distance to the trajectory obtained by retraining with correct labels.

Training roundFedHSR relative response errorReplay Only normalized trajectory errorFedHSR normalized trajectory error
410.9230.4180.387
430.9020.5800.511
450.9080.6510.556
470.9210.6960.589
490.9260.7430.626
510.9320.7850.660
530.9350.8270.691
540.9370.8470.706

We acknowledge that the error grew as replay progressed. However, FedHSR always had a lower relative response error than Replay Only (1.000) in training steps. This means that repairing a long step is indeed more difficult, but we can still achieve improvements. We will add these results to the revised manuscript.

Overall

Thank you again for your thoughtful review. We hope the additional comparison with FedRecover and the replay experiments address your concerns about the cost and accuracy of historical repair. Your questions will help us clarify the scope of our theory and explain how replay handles client states in the revised manuscript. If our response addresses your concerns, we would be grateful if you would consider raising your rating.

Thanks for the author's rebuttal response. I maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.