acceptodds
Under review as a conference paper at ICLR 2027

Federated Data Pruning via Activation Stability

Abstract

Federated Learning (FL) keeps data on-device, but pays for it in compute: clients spend most of every round re-processing data that have already stopped contributing to learning. Existing FL data-pruning methods rely on static importance scores, server-side validation, or aggregated centroids: assumptions that break under privacy constraints and non-IID heterogeneity. In this work, we ask whether a purely local, training-aware data redundancy signal exists, and answer in the affirmative. We introduce activation stability-based data pruning at the batch level for FL, a lightweight, architecture-agnostic plug-in that permanently removes batches whose mean hidden-activation variance saturates across iterations, a signal already produced by the local forward pass, requiring no extra communication, no server data, and no architectural change. We show that activation stability and client drift evolve on decoupled time scales, explaining why local pruning remains reliable under heterogeneity. Across CIFAR-10/100, SVHN, and ImageNet-1K with ResNet-18/50 and CvT, our framework discards up to 45% of training batches at almost no accuracy cost, saving up to 45% of the GPU node-hours.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federated Learning, Data Pruning, Activation Stability, Efficient Distributed Deep Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QyoGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adapts activation-stability-based batch pruning to federated learning. Each client tracks the mean standard deviation of hidden activations for fixed batches and permanently removes batches whose scores change by less than a scheduled threshold. To handle intermittent participation, the method tightens the threshold for moderately stale clients and resets activation histories after longer absences. Experiments cover four image datasets, several architectures, heterogeneous partitions, and partial participation, reporting accuracy, cumulative data savings, and training-time savings. The contribution is primarily empirical; no convergence or pruning-safety guarantee is provided.

Critical strengths

1.The method is lightweight and locally implementable, without additional communication or server-held validation data. Evaluation extends to ImageNet and multiple architectures, and the reported timing measurements include activation-tracking overhead.

2.The staleness mechanism addresses a concrete issue introduced by intermittent client participation. Its ablation shows meaningful improvements over retaining old histories without correction, supporting the usefulness of this federated adaptation.

Critical weaknesses

1.The principal explanatory claim is insufficiently supported, limiting the contribution beyond B-PAS. The central activation statistic and permanent batch-pruning rule are inherited from prior work. Stability of a mean standard deviation does not establish that representations have converged or that the corresponding data no longer contribute useful gradients. The drift comparisons and nonsignificant statistical tests do not establish independence, equivalence across heterogeneity levels, or safety after subsequent aggregation. Direct evidence about the learning utility of pruned batches is particularly important because deleted batches are never reconsidered.

2.The comparative evaluation does not adequately isolate the benefit of the proposed criterion and federated adaptation. The FedCS comparison uses published results rather than a controlled rerun, making the large accuracy differences difficult to attribute to pruning quality. Partial-participation and client-scaling experiments lack corresponding full-data baselines. Moreover, input-space similarity reaches 85.14% versus 85.48% for activation stability when given the same pruning schedule. Together, these results leave unclear how much benefit comes from the activation criterion, the pruning schedule, or differences in training protocols.

Questions for authors
  1. Does the score predict reduced loss-gradient contribution or reduced future learning utility for individual batches? Is there evidence that batches selected for removal remain unimportant after later global aggregations?

2.How is DSI calculated, including early termination, and how is the published FedCS pruning fraction converted into the same cumulative data-usage budget?

3.What are the full-data results under the same partial-participation and client-scaling settings? How does the method compare with directly applying B-PAS locally and with a simple predetermined late-training pruning schedule at matched computational budgets?

4.What constitutes an independent observation in the heterogeneity tests, and how are temporal and within-client correlations handled?

4 comments
AuthorsOpenReview ↗

Response to Reviewer QyoG (1/4)

We thank the reviewer for the precise and constructive review, and for recognizing that the method is lightweight and locally implementable, that the evaluation covers ImageNet and multiple architectures with overhead included, and that the staleness mechanism yields meaningful gains. We agree that the contribution is primarily empirical, and below we state clearly what our evidence establishes. All changes are reflected in the revised manuscript.


W1 / Q1. Does activation stability indicate reduced learning utility, and do pruned batches stay unimportant?

The per-batch criterion and its link to learning utility come from prior work. Our pruning criterion is inherited from B-PAS (Alam et al., 2026), which studied the relationship between activation stability and the learning utility of individual batches in the centralized setting:

  • The per-batch change in mean activation standard deviation is large early in training, decreases monotonically and then saturates, indicating diminishing learning utility (B-PAS, Fig. 5) (Alam et al., 2026).
  • A random baseline that prunes the same number of batches consistently degrades accuracy, so stability identifies non-informative batches rather than simply reducing the amount of data.
  • Pruning favors neither easy nor hard examples and preserves class balance (B-PAS, App. A.6) (Alam et al., 2026).

We build on this established criterion. Our paper therefore asks a different question: whether the criterion remains valid under federated, heterogeneous, partially participating training, and what is needed for it to work there.

Evidence in the federated setting.

  • (i) Controlled selection. Our random baseline drops exactly as many batches as our method, on every client, at every iteration, so both follow the same pruning schedule and learning-rate trajectory. It still loses 0.9–4.9 points (Table 6, e.g. 82.46% vs. 85.48% on Non-IID CIFAR-10 and 63.25% vs. 68.16% on Non-IID CIFAR-100). The batches selected by stability are therefore safer to remove than arbitrary ones, as in B-PAS.
  • (ii) Timing. Most pruning happens late: in the main setting, the first batch is pruned around round 80 and the median batch at round 161 of 187, after most learning has taken place. This limits the cost of permanent removal.
  • (iii) No systematic side effects. Class proportions are preserved per client (mean Jensen–Shannon divergence 0.055) and globally.
  • (iv) Robustness. The criterion is insensitive to which layers are tracked (11 configurations, 0.68-point spread).

Safety after subsequent aggregation. This concern is specific to FL, and it is what our staleness mechanism addresses. A client's reference scores are always taken at its last participation:

  • A client returning after a short absence receives a stricter threshold.
  • A client absent for more than kk rounds discards its history and rebuilds it under the current global model before pruning again.

Without this mechanism, i.e., B-PAS applied locally, accuracy drops by 3.7 and 5.8 points at 75% and 50% participation, which shows that comparisons across distant global models do lead to unsafe decisions.

We agree that stability of the mean activation standard deviation does not by itself constitute a per-batch guarantee on gradient magnitude, and we do not claim one. Re-admitting pruned batches is a natural extension, which we leave to future work.

AuthorsOpenReview ↗

Response to Reviewer QyoG (2/4)

W1 (Contd.)

On the drift comparisons and tests. We agree with the reviewer. We do not claim that activation stability and client drift are independent, nor that non-significant tests establish equivalence across heterogeneity levels. The analysis in Section 4 is descriptive, and its purpose is narrower: to check whether the heterogeneity that drives client drift also visibly affects the local signal our pruning rule uses.

  • The two signals behave differently (Fig. 3). Activation stability changes sharply only in the first few rounds. Afterwards only small residual fluctuations remain, which settle within a narrow band (±0.001) at every heterogeneity level. Client drift varies much more, and its trajectory depends on heterogeneity. At α=0.5\alpha=0.5 and 1.01.0 it rises over many rounds before leveling off. At α=0.1\alpha=0.1 it is non-monotonic and fluctuates noticeably more. Both signals eventually settle, but their magnitude and their dependence on heterogeneity differ.
  • Stability shows no detectable dependence on α\alpha (Fig. 4). Its trajectories overlap across α∈{0.1,0.5,1.0}\alpha\in\{0.1,0.5,1.0\}. The ANOVA and Kruskal–Wallis results (p=0.93p=0.93, 0.400.40) are reported as descriptive. As the appendix now states, they are not a formal test of equivalence, because consecutive rounds of one run are correlated (please see Q4).
  • What we infer, and what we do not. The pruning rule uses only each batch's own stability score. That score shows no detectable dependence on α\alpha, while drift clearly does. Global disagreement therefore does not visibly propagate into the local pruning signal. We treat this as consistent with local pruning remaining usable under heterogeneity, not as proof that it does.

W2. Isolating the benefit of the criterion, the schedule and the federated adaptation

Criterion vs. schedule. We agree with the reviewer's reading of the input-space similarity result, and we now discuss it in the main text (Section 5, "Random pruning vs. ours"):

Selection rule, same pruning schedule (Non-IID CIFAR-10)Accuracy
Random82.46%
Input-space similarity85.14%
Activation stability (ours)85.48%

Once the schedule is fixed, a reasonable redundancy score recovers most of the benefit, because each batch is a random mix of the client's samples. What activation stability provides that a static score cannot is the schedule itself: a training-aware decision of when and how much to prune, made locally, during training and without communication. A static similarity score has no notion of training progress; in this experiment, it inherits the schedule from our method.

The federated adaptation vs. B-PAS. Applying B-PAS directly on each client loses 3.7 points at 75% participation (83.26% vs. 86.95%) and 5.8 points at 50% (78.34% vs. 84.16%). The appendix ablation is now labelled "B-PAS (local)" and discussed in the main text.

FedCS. We agree this is not a controlled comparison: no implementation is available, and we use its published numbers. The revised Section 5 and the Table 2 caption now state this explicitly, describe the comparison as indicative, and point to the controlled baselines above, which were all run in our own pipeline under the same schedule.

Full-data baselines. We have added full-data training at every participation rate (Table 4) and for every client-scaling setting (Appendix, Table "Client scale"). The partial-participation and client-scaling results are now directly comparable with full data in the same setting (please see Q3).

AuthorsOpenReview ↗

Response to Reviewer QyoG (3/4)

Q2. How is DSI computed, and how was the FedCS pruning fraction converted?

Section 5 now gives the definition explicitly, following B-PAS: DSI=1−∑t=1Ts∑i=1n∣Si(t)∣T∑i=1nmi,\mathrm{DSI}=1-\frac{\sum_{t=1}^{T_s}\sum_{i=1}^{n}|\mathcal S_i^{(t)}|}{T\sum_{i=1}^{n} m_i},

  • ∣Si(t)∣|\mathcal S_i^{(t)}| is the number of active batches client ii trains on at iteration tt.
  • mim_i is the number of batches of client ii before training.
  • TT is the scheduled number of iterations, and Ts≤TT_s\le T is the iteration at which training stops.

Early termination occurs only when every batch on every client has been pruned (Section 3), and the remaining iterations then count as zero data. Section 5 includes a worked example.

For FedCS, we interpret its reported pruning rates (pf,pl)(p^f,p^l) as the overall data fraction removed, and match the closest DSI. This is an approximation: FedCS prunes once, early in training, so its fraction applies to almost all rounds, whereas our savings accumulate over training. The two budgets are similar but not identical, which the Table 2 caption now states.


Q3. Full-data results under partial participation and client scaling; direct B-PAS; a predetermined schedule

Full-data results under partial participation. We thank the reviewer for this suggestion. We have added full-data training at every participation rate to Table 4. Together with the direct application of B-PAS, all methods can now be compared at the same participation rate (Non-IID CIFAR-10, α=0.1\alpha=0.1):

ParticipationFull dataOurs (k=5k=5)Ours (k=20k=20)B-PAS (local)
100%85.71%85.48%85.48%85.48%
75%82.69%86.95%82.08%83.26%
50%77.13%84.16%80.83%78.34%

At full participation no client is ever stale, so kk has no effect and our method coincides with B-PAS. Under partial participation:

  • Full-data training degrades as participation drops (85.71 → 82.69 → 77.13%), because fewer client updates are averaged in each round.
  • Our method with k=5k=5 clearly exceeds full data: by 4.3 points at 75% participation and 7.0 points at 50%, while still removing data (DSI 19.2% and 12.3%).
  • Directly applying B-PAS locally gains much less (+0.6 and +1.2 points over full data), and stays 3.7 and 5.8 points below our method. The gap grows as participation becomes sparser and references become staler. This is the failure mode our staleness mechanism is designed to correct.
  • A large staleness threshold (k=20k=20) is close to full data at 75% (−0.6 points) and above it at 50% (+3.7). It remains well below k=5k=5, confirming that tying pruning decisions to the current global model matters.

Full-data results under client scaling. We have also added full-data training to the client-scaling experiments (Appendix, Table "Client scale"):

ClientsParticipationFull dataPruning
1050%77.13%84.16%
2025%59.39%60.97%
5010%49.45%54.63%

Both methods lose accuracy as the same data is split across more clients under a fixed 200-round budget. Pruning exceeds full-data training in all three settings (+7.0, +1.6 and +5.2 points) while removing 10–14% of the data.

Why pruning can exceed full data under sparse participation. We attribute this to variance reduction. Subsampling clients raises the variance of the aggregated update, and pruning removes saturated batches that contribute noise but little signal, which helps most when fewer client updates are averaged per round. This is consistent with B-PAS (Alam et al., 2026), which likewise observed that activation-stability-based pruning can occasionally improve accuracy over full-data training. We have added this explanation to Section 5 and the appendix.

A predetermined late-training schedule at matched compute. We did not run this baseline, and we acknowledge it as a limitation. The closest control in the paper is the random baseline (Table 6). It drops exactly as many batches as our method on every client at every iteration, so it follows our schedule and data budget exactly, but it ignores the per-batch signal. It loses 0.9–4.9 points. Together with the input-space similarity result (please see W2), this indicates that both the schedule and the selection matter.

AuthorsOpenReview ↗

Response to Reviewer QyoG (4/4)

Q4. What constitutes an independent observation in the heterogeneity tests?

The reviewer is right. In our tests each observation is the mean stability over the probe batches at one communication round of a single training run per α\alpha. Consecutive rounds are temporally correlated, and observations within a run are not independent. We have revised the paper accordingly:

  • Appendix: now states the unit of observation, both null and alternative hypotheses, and that the resulting pp-values (ANOVA 0.930.93, Kruskal–Wallis 0.400.40) are descriptive rather than a formal test of equivalence.
  • Section 4: now calls them "descriptive statistical tests", and the appendix states that they are not a formal test of equivalence.

A seed-level analysis with an equivalence test would be the appropriate formal test, and we consider it a useful addition for future work.

We thank the reviewer again for these precise comments, which have made our claims more accurate and the paper more rigorous. We hope the clarifications and additions address the concerns, and we would be grateful if the reviewer would consider raising the score.

Reviewer_ar2HOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to address data pruning issue in federated learning. Specifically, the authors partition each client’s local dataset into static batches and compute the per-layer standard deviation of activations in each communication round. The mean of these standard deviations across all layers serves as the indicator for pruning decisions. If the difference between two consecutive indicators falls below a heuristically chosen threshold, the corresponding batches are permanently pruned from training. An exponentially increased mechanism is designed to ensure a balanced pruning strategy. Four datasets are used to evaluate the proposed algorithm.

Critical strengths

1.The motivation is clear. This paper aims to design local data-pruning strategies that reduce data redundancy in federated learning.

2.The proposed algorithm depends only on local information and therefore does not increase communication overhead.

3.Taking the staleness into account, which is a common issue in federated learning.

Critical weaknesses

1.The whole algorithm is heuristically designed. Using activation stability as the indicator for data pruning is the core idea of this paper, which has not been theoretically verified. Moreover, the pruning strategy is only based on heuristically determined threshold.

2.As B-PAS proposed in Alam et al. (2026) has already used the activation stability to prune data, the paper’s novelty is limited to designing heuristic methods to decide the pruning threshold.

3.The batches are predetermined on each client, so information from combinations of samples across different batches is lost during training, which compromises training performance.

4.The pruning strategy is sensitive to the learning rate, but the paper does not evaluate this hyper-parameter. Moreover, in Figure 1, pruning occurs after 100 communication rounds. If learning rate decay is used in training, this behavior is probably caused by the smaller learning rate.

5.All claims are empirically evaluated on Dirichlet partition, which only incurs limited heterogeneity on non-iid settings.

6.The experimental results contain anomalies. For instance, in Table 1, the performance with data pruning is higher than that of training on the full data.

Questions for authors

1.This paper is hard to read. The authors should carefully proofread and revise the manuscript.

2.The NodeHour saving is affected by the early stop strategy. Reporting the accuracy and the NodeHour of full-data training with early stop would be more objective comparison.

3.The performance of training without batch partition should be reported to avoid the bias introduced by partition.

4.The experiments only use 10 clients partitioning, a more complicated experimental settings would be more convincing.

5.Table 5 seems to compare different algorithms on an IID partition, which understates the performance of FedProx and FedNova.

6.The comparison with FedCS is unfair because its performance is cited from the original paper, and FedAvg with full data outperforms FedCS by more than 20%.

7.The null-hypothesis and alternative hypothesis are not reported in the paper.

5 comments
AuthorsOpenReview ↗

Response to Reviewer ar2H (1/3)

We thank the reviewer for the careful reading and for recognizing the clear motivation, the absence of extra communication, and the handling of staleness. We address every weakness and question below. All changes are reflected in the revised manuscript.


W1. The algorithm is heuristically designed

We agree that our method, like most data-pruning methods (e.g., forgetting scores, EL2N, InfoBatch, B-PAS), is empirically motivated rather than derived from a convergence guarantee, and we do not claim otherwise. We do, however, provide controlled evidence that the signal is informative in FL, beyond what a heuristic threshold alone would give:

  • Selection matters. A random baseline drops exactly as many batches as our method, on every client, at every iteration, so both follow the same schedule and reach the same DSI. It still loses 0.9–4.9 points (Table 6, e.g. 82.46% vs. 85.48% on Non-IID CIFAR-10 and 63.25% vs. 68.16% on Non-IID CIFAR-100).
  • The FL adaptation matters. Applying the centralized rule (B-PAS) directly on each client loses 3.7 and 5.8 points at 75% and 50% participation. Our staleness mechanism recovers this loss (Table 4 and the appendix ablation).
  • No harmful side effects. Pruning preserves per-client and global class proportions (mean Jensen–Shannon divergence 0.055), and the score is robust to which layers are tracked (11 configurations, 0.68-point spread).
  • Consistent behaviour under heterogeneity. Stability trajectories overlap across α∈{0.1,0.5,1.0}\alpha\in\{0.1,0.5,1.0\} (Fig. 4). Client drift, by contrast, varies much more and its trajectory depends strongly on heterogeneity (Fig. 3).

The threshold is a two-parameter schedule, and it can be calibrated cheaply on a 10% data subset (Figure 5, Appendix "Fast δ calibration"). We list data-driven, adaptive thresholds as future work in the conclusion.


W2. Novelty limited to designing the pruning threshold

A small clarification: the exponential threshold schedule is inherited from B-PAS, and Section 3 now states this explicitly, so it is not our contribution. Our contributions are:

  1. The federated adaptation. Per-client stability histories keyed to each client's last participation, and the staleness-aware mechanism. Without it, B-PAS applied locally loses 3.7–5.8 points under partial participation.
  2. FL-specific analyses that do not arise in centralized training.
    • Activation stability and client drift behave very differently. Stability changes sharply only in the first few rounds, after which only small residual fluctuations remain, settling within a narrow band (±0.001) at every heterogeneity level. Drift varies much more, and its trajectory is strongly shaped by heterogeneity: it is non-monotonic and fluctuates noticeably more under α=0.1\alpha=0.1.
    • There is no detectable dependence of stability on heterogeneity.
    • Pruning stays class-balanced under Dirichlet partitions.
    • Layer aggregation does not mask learning in deeper layers.
    • With identical schedules, FL prunes more conservatively than centralized training (Appendix, "Comparison of centralized and federated pruning").
  3. A practical result in a largely unexplored area. Data pruning in FL has received little attention, and existing methods need static scores, server validation data, or aggregated statistics. Our method is on-device and communication-free, and it removes up to 45% of batches. On ImageNet with ResNet-50 it saves up to 45.3% of GPU node-hours, and on Non-IID ImageNet it reaches 67.40% vs. 65.84% for full data with 45% DSI.

W3 / Q3. Fixed batches lose cross-batch sample combinations

Fixing batches is required for the signal to be well defined. The stability score is a property of a batch and is compared across iterations. If batches were regenerated every epoch, a change in σˉ\bar\sigma would mainly reflect which samples were drawn rather than whether the representation has converged, and no batch identity would persist to be pruned. We follow the protocol of B-PAS (Alam et al., 2026): batches are fixed once at initialization, samples are shuffled within each batch every epoch, and standard augmentation is applied.

This protocol does not compromise training:

  • Every sample is still trained on in every epoch until its batch is pruned.
  • Re-running Non-IID CIFAR-10 with three independent random batch assignments changes accuracy by only ±0.54 points (max spread 1.1; Appendix, "Sensitivity to batch assignment"), so results do not depend on a particular grouping of samples.
  • Accuracy with pruning stays close to full-data training across all datasets and architectures (Table 3).
AuthorsOpenReview ↗

Response to Reviewer ar2H (2/3)

W4. Sensitivity to the learning rate; pruning after ~100 rounds

LR ablation. We do evaluate the learning rate (It was already there in the Appendix, Table "Training ImageNet-1K with ResNet-50 under different learning rates"). With LR 0.2, pruning is more aggressive (DSI 63%) and accuracy drops to 72.22%; with LR 0.1, DSI is 42% and accuracy 73.32%. As expected, larger steps make activation statistics noisier and trigger earlier pruning. The main text now references this ablation.

On the figure. We believe the reviewer may be referring to Figure 2 rather than Figure 1. Figure 1 is the framework diagram and contains no pruning curve. Figure 2 shows the number of remaining batches over training on ImageNet-1K for nine (δs,δe)(\delta_s,\delta_e) schedules.

Why and when pruning starts. The onset is governed mainly by the threshold schedule. The exponential schedule starts small (δs\delta_s) and grows toward δe\delta_e. Early in training, activation statistics change quickly, so few batches satisfy ∣Δσˉ∣≤δ(t)|\Delta\bar\sigma|\le\delta^{(t)}. As representations stabilize and the threshold grows, more batches meet the condition. The timing therefore depends strongly on (δs,δe)(\delta_s,\delta_e):

  • In Figure 2, aggressive schedules remove all batches by iteration 2.
  • Conservative schedules start pruning much later.
  • Intermediate schedules start in between.

This behaviour was analysed comprehensively in B-PAS, which studied 45 threshold settings on CIFAR-10. Our defaults were selected the same way: from the nine-schedule sweep on ImageNet (Figure 2, Table 1) and the ten-configuration sweeps on CIFAR-10 IID/Non-IID (Appendix). We deliberately chose schedules that prune conservatively while representations are still changing quickly.

Is it just learning-rate decay? We agree that the step size is part of the training dynamics that drive stabilization. However, two controls show that our results are not explained by "late in training" alone:

  1. The random baseline prunes the same number of batches in the same rounds, and therefore under the same learning-rate schedule, yet it is consistently worse (0.9–4.9 points).
  2. In the new multi-epoch experiments (Appendix "Multiple local epochs"), training uses a constant learning rate with no decay at all. Activations still stabilize, and pruning starts in round 5 (E=2E=2) and round 1 (E=5E=5).

W5. Only Dirichlet partitions

The Dirichlet partition with α=0.1\alpha=0.1 is a standard high-heterogeneity setting in FL; it is also the setting used by FedCS. We vary heterogeneity widely:

  • α∈{0.1,0.5,1.0}\alpha\in\{0.1,0.5,1.0\} in the heterogeneity analysis (Section 4).
  • An extremely heterogeneous α=0.05\alpha=0.05 (Appendix), where pruning reaches 77.18% vs. 76.94% for full data while removing 22.2% of the data.
  • Partial participation and up to 50 clients, which add system heterogeneity on top of data heterogeneity.

We agree that pathological label-skew (shard) partitions would be a useful further test, and we note them as future work.


W6. Pruning above full-data accuracy

We respectfully note that this is not an anomaly. It is well established in the data-pruning literature that removing redundant data does not necessarily hurt accuracy and can match or slightly exceed full-data training. In particular, B-PAS (Alam et al., 2026) reports slightly higher accuracy with pruning: 78.43% on ImageNet-1K while saving 47% of the data, and 95.60% on CIFAR-10 with 25% DSI.

In our results, the gains are modest and not universal. On CIFAR-10/100 and SVHN, pruning is at or slightly below full data (Table 3), and we do not claim a systematic improvement. We have revised Section 5 to be precise about the cause. The full-data ImageNet model does not overfit (its validation accuracy is still rising at round 281), so we no longer attribute the gain to regularization. A plausible FL-specific factor is reduced client drift: pruning shrinks each client's local dataset as training proceeds, and so reduces local steps per round. The new multi-epoch experiments support this:

Local epochs EEFull dataPruning
185.71%85.48%
283.68%86.29%
580.63%82.42%

As clients take more local steps on skewed data, full-data accuracy falls (85.71% → 83.68% → 80.63%), which is the classic signature of client drift (Li et al., 2020; Karimireddy et al., 2020). Pruning is above full data exactly in these drift-prone settings, and slightly below it at E=1E=1, where drift is smallest.

AuthorsOpenReview ↗

Response to Reviewer ar2H (3/3)

Questions

Q1. Readability. We have carefully revised the manuscript, and it is now better organized and easier to read:

  • All citations use the parenthetical form.
  • Tables sit next to the text that discusses them, with the best results highlighted.
  • The Figure 1 caption is short.
  • Section 3 is condensed so that the pruning rule appears once (Eq. 1, including the staleness factor).
  • DSI is defined by an explicit formula.
  • Overstated claims have been made more precise.

Q2. Node-hour savings and early stopping. Our early termination is not a separate training trick that full-data training could also use. Training stops only when every batch on every client has been pruned (Section 3), which full-data training cannot trigger. Validation-based early stopping would not reduce the full-data cost either: on ImageNet, its validation accuracy is still improving at round 281 of 300. Schedules that never empty the dataset also save node-hours without early termination (Table 1: 12.6%, 19.2% and 25.1% for the three schedules without an early-stop round). Following B-PAS, Section 5 now defines DSI=1−∑t=1Ts∑i∣Si(t)∣\/(T∑imi)\mathrm{DSI}=1-\sum_{t=1}^{T_s}\sum_i|\mathcal S_i^{(t)}|\/(T\sum_i m_i) with a worked example, so iterations after termination count as zero data. The Table 1 caption explains the "Early Stop Round" column.

Q3. Training without batch partition. Please see W3: fixed batches are needed for the stability score to be defined, and the results are insensitive to the particular batching (±0.54 points).

Q4. More complicated settings. Beyond the 10-client setting, the paper (mostly the appendix) includes:

  • 20 clients at 25% participation and 50 clients at 10% participation.
  • Partial participation at 75% and 50% (Table 4).
  • Extreme heterogeneity (α=0.05\alpha=0.05).
  • Multiple local epochs (E=2E=2 and E=5E=5), each with a matched full-data baseline: pruning reaches 86.29% vs. 83.68% with 45.7% fewer GPU-hours at E=2E=2, and 82.42% vs. 80.63% with 69.6% fewer GPU-hours at E=5E=5.
  • Further analyses: different optimizers (SGD, Adam, AdaGrad, RMSProp), batch normalization, layer selection, class balance, the input-space similarity baseline, and tracking overhead (1.2% of wall-clock time).

Q5. FedProx/FedNova on IID data. The reviewer is right that this comparison uses an IID partition, and the caption now says so. Its purpose is to show that our plug-in works unchanged with different aggregation rules, not to rank those algorithms. Our heterogeneous experiments (α∈{0.05,0.1}\alpha\in\{0.05,0.1\}, partial participation, more clients, multiple local epochs) use FedAvg.

Q6. Comparison with FedCS. We agree that this is not a controlled comparison, and the revised Section 5 and the Table 2 caption now say so explicitly. No public implementation of FedCS is available, so we compare against its published numbers in the same setup (dataset, architecture, α\alpha). Part of the absolute gap may reflect differences between training pipelines, and the matched DSI values are approximate because FedCS prunes once, early, whereas our savings accumulate later. We therefore present FedCS as an indicative comparison. Our controlled evidence comes from baselines run in our own pipeline under the same pruning schedule:

Baseline (Non-IID CIFAR-10)Accuracy
Random selection82.46%
Input-space similarity85.14%
B-PAS applied locally, 75% participation83.26% (ours: 86.95%)
B-PAS applied locally, 50% participation78.34% (ours: 84.16%)
Ours, full participation85.48%

Q7. Null and alternative hypotheses. We now state both hypotheses in the appendix:

  • ANOVA: H0H_0: equal mean stability across α\alpha, against H1H_1: at least one mean differs.
  • Kruskal–Wallis: H0H_0: identical distributions, against H1H_1: at least one group is stochastically larger or smaller.

We also state the unit of observation (the mean stability over probe batches at one communication round). Because consecutive rounds are correlated, we describe the resulting pp-values (0.930.93 and 0.400.40) as descriptive evidence of no detectable difference, not as a formal test of equivalence.

We thank the reviewer again for the detailed comments, which have helped us make the paper clearer and more rigorous. We hope these clarifications and additions address the concerns, and we would be grateful if the reviewer would consider raising the score.

Reviewer_P5V5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to address the issue of data redundancy in federated learning (FL), where clients continue to repeatedly processing batches during the later stages of training. This paper adapts the centralized B-PAS method to the FL setting, proposing a batch-level pruning strategy based on activation stability. During local forward passes, this paper calculates the mean standard deviation of activations across selected hidden layers for each batch. If the change between two consecutive participation rounds falls below an exponentially increasing threshold, δ(t)\delta(t), the batch is permanently pruned. To account for the intermittent participation of FL, this paper introduces a "staleness" mechanism. If a client is absent for more than kk rounds, its historical data tracking is discarded, and a baseline is reconstructed by performing one full round without pruning. This process is entirely local, requiring no additional communication, server-side data, or architectural modifications, with an overhead of approximately 1.2%. Experiments on CIFAR-10/100, SVHN, and ImageNet-1K demonstrate that up to 45% of batches can be pruned with negligible impact on accuracy, resulting in savings of up to 45% in GPU node-hours.

Critical strengths
  1. The paper addresses a practical problem in Federated Learning (FL). Most existing research on efficiency focuses on model pruning and communication compression, while data-side redundancy has been largely overlooked.

  2. The proposed method is lightweight and deployment-friendly. It derives from existing activations during the forward pass and does not necessitate a server-side validation set or aggregated statistics. Furthermore, it requires no modifications to the model architecture and is plug-and-play with existing FL optimizers.

Critical weaknesses
  1. My major concern with this paper lies in its writing and presentation. Specifically:

1.1 Citation formatting is incorrect throughout the manuscript. Almost all citations use the \citet style (e.g., "computer vision Chen et al. (2023), medical imaging Koutsoubis et al. (2024)", "FedAvg McMahan et al. (2017)", "CvT Wu et al. (2021)"); these should instead be enclosed in parentheses using \citep.

1.2 Tables are misaligned with the corresponding text. Although wraptable is used for all tables, they frequently do not align with the relevant discussion in the main text. For instance, Table 2 (comparing FedCS) appears next to the "Performance analysis on ImageNet-1K" section, which actually discusses Table 1. Similarly, the section "Comparison with SOTA method" (near line 388) analyzes Table 2, yet Table 3 is displayed alongside it; likewise, the section "Architecture and dataset robustness" analyzes Table 3, but is placed next to Table 4.

1.3 Results in the tables lack visual emphasis. The best-performing results in Tables 1, 2, 3, 5, and 6 are not highlighted (e.g., via bold text). Given that Table 1 contains 9 configurations and 6 columns of numerical data, it is difficult to discern the "balanced point" recommended by the authors without such emphasis.

1.4 The caption for Figure 1 is excessively long. It essentially restates the methodology described in Section 3—including full definitions of symbols and the formal equations for pruning conditions—occupying nearly a third of a page. A caption should merely describe the components of the figure; methodological details should be reserved for the main text.

1.5 Section 3 contains significant redundancy. The pruning mechanism is described at least four times: in the Figure 1 caption, the introductory paragraph of Section 3, the "Pruning" subsection, and the subsequent formal definition.

1.6 The layout is overly crowded; the authors are advised to revise the formatting.

  1. The proposed method is based on B-PAS and adapted for a Federated Learning (FL) setting; the only addition is a heuristic "staleness" factor. Consequently, the method novelty is limited.

  2. The assumptions underlying the experiments are overly strong. For instance, the experiments involve only 10 clients, with a single local epoch per round, and batches are fixed at initialization rather than being shuffled. Although the authors claim that this approach "naturally extends to multiple epochs," this has not been verified. Furthermore, fixing the batches is, in itself, a strong assumption. Additionally, δ\delta is extremely sensitive and task-dependent; the values ​​used are [10−7,10−3][10^{-7}, 10^{-3}] for CIFAR and [5×10−5,5×10−4][5 \times 10^{-5}, 5 \times 10^{-4}] for ImageNet.

Questions for authors

Please see weakness.

5 comments
AuthorsOpenReview ↗

Response to Reviewer P5V5 (1/3)

We sincerely thank the reviewer for the careful reading and constructive feedback, and for recognizing that data-side redundancy is an overlooked problem in FL and that our method is lightweight, deployment-friendly and plug-and-play. We address each point below. All changes are reflected in the revised manuscript.


W1. Writing, presentation and formatting

We agree, and we have revised the manuscript throughout:

  • (1.1) Citations. All citations now use the parenthetical form (\citep).
  • (1.2) Table placement. Every table now sits next to the paragraph that discusses it.
  • (1.3) Visual emphasis. The best results are bolded. In Table 1, the default schedule (5×10−5,5×10−4)(5\times10^{-5},5\times10^{-4}) is shaded, and the text explains why it is the default: it gives the highest data savings among schedules within 0.6 points of full-data accuracy. When node-hours matter most, (10−5,5×10−4)(10^{-5},5\times10^{-4}) is the better choice.
  • (1.4) Figure 1 caption. Reduced to three sentences that describe only the figure.
  • (1.5) Redundancy in Section 3. The pruning rule is now stated once, as Eq. (1), ∣σˉ(t)−σˉ(tprev)∣≤fi δ(t)|\bar\sigma^{(t)}-\bar\sigma^{(t_{prev})}|\le f_i\,\delta^{(t)}, which already includes the staleness factor (fi=1f_i=1 without staleness). All other places refer back to Eq. (1), and the Section 3 introduction is condensed.
  • (1.6) Layout. These changes free substantial space, which we used to reduce crowding. Section 5 also now defines DSI by an explicit formula.
AuthorsOpenReview ↗

Response to Reviewer P5V5 (2/3)

W2. Novelty

Data pruning in FL is largely unexplored. Efficiency work in FL has focused almost entirely on model pruning and communication compression. The few existing FL data-pruning methods rely on static importance scores computed before training (FedCS), server-held validation gradients (gradient-alignment coresets), or aggregated statistics. These assumptions conflict with FL's privacy model and with evolving, heterogeneous training. To our knowledge, ours is the first method that prunes data on-device, during training, from a signal the forward pass already computes, with no extra communication and no server data.

This is not a direct extension of B-PAS. We credit B-PAS for the per-batch stability score and the threshold schedule (Section 3). B-PAS, however, was designed for centralized, IID, fully-attended training, and moving it to FL raises problems it does not address. Our results show these problems matter:

  1. Naive transfer fails under partial participation. Applying B-PAS directly on each client, with a persistent history, loses 3.7 points at 75% participation (83.26% vs. 86.95%) and 5.8 points at 50% (78.34% vs. 84.16%) on Non-IID CIFAR-10. The cause is that a returning client compares its fresh scores against references computed under an older global model. Our per-client histories, keyed to each client's last participation, and the staleness mechanism recover this loss (Table 4 and the appendix staleness ablation, labelled "B-PAS (local)").
  2. The signal behaves differently in FL. With identical (δs,δe)(\delta_s,\delta_e) schedules, FL prunes noticeably more conservatively than centralized B-PAS, because heterogeneous clients and round-based aggregation slow activation stabilization (Appendix, "Comparison of centralized and federated pruning"). The centralized behaviour therefore cannot simply be assumed.
  3. New FL-specific analyses. These address questions that do not arise in centralized training:
    • Stability vs. client drift (Fig. 3). Activation stability saturates and stays within a narrow band (±0.001), while client drift is strongly shaped by heterogeneity and fluctuates far more under α=0.1\alpha=0.1. The two signals capture different things: drift reflects global disagreement across clients, stability reflects local representational convergence. This is consistent with a purely local pruning decision remaining usable under non-IID data.
    • Consistency across heterogeneity (Fig. 4). Stability trajectories for α∈{0.1,0.5,1.0}\alpha\in\{0.1,0.5,1.0\} overlap closely and saturate at nearly the same rounds. Descriptive ANOVA (p=0.93p=0.93) and Kruskal–Wallis (p=0.40p=0.40) tests detect no difference. At even stronger heterogeneity (α=0.05\alpha=0.05), pruning matches full data (77.18% vs. 76.94%) while removing 22.2% of the data.
    • Class balance under permanent removal. Per-client class proportions are preserved (mean Jensen–Shannon divergence 0.055), as is the global class distribution.
    • Layer aggregation. Across 11 tracked-layer configurations, accuracy varies by only 0.68 points, so averaging across layers does not mask learning in deeper layers.

On the staleness factor being "heuristic." The rule is simple by design, so that it adds no communication and only one hyperparameter (kk). Its form follows directly from the failure mode above: a stale reference makes a batch look stable when it is not. Moderately stale clients therefore receive a stricter threshold (fif_i decays linearly from 1 to 0.5), and severely stale clients (ci>kc_i>k) discard their history and rebuild it before pruning again. Each component is validated empirically:

  • Removing the mechanism costs 3.7–5.8 points.
  • A small kk (5) clearly outperforms a large kk (20): 86.95% vs. 82.08% at 75% participation. A large kk rarely triggers the stricter threshold or the reset, which matches the mechanism's intent.

The results are significant. The framework transfers across four datasets, three backbones (including a transformer) and IID/Non-IID partitions:

  • ImageNet-1K / ResNet-50: it removes 42% of training batches within 0.6 points of full-data accuracy (73.32% vs. 73.92%), and saves up to 45.3% of GPU node-hours.
  • Non-IID ImageNet-1K: with 45% of the data removed, it reaches 67.40% against 65.84% for full data.
  • Partial participation: it exceeds full-data training at the same participation rate by 4.3 points at 75% and 7.0 points at 50% (86.95% vs. 82.69%, 84.16% vs. 77.13%).
  • Plug-in: it works with FedAvg, FedProx and FedNova without modification.

Because the signal comes from the forward pass, tracking costs only 1.2% of wall-clock time. These savings therefore translate directly into lower client compute and energy, which matters most in FL.

AuthorsOpenReview ↗

Response to Reviewer P5V5 (3/3)

W3. Experimental assumptions

Number of clients. Besides the 10-client setting, the appendix ("Client scale") (Table 20) reports 20 clients at 25% participation and 50 clients at 10% participation, and Table 4 reports 75% and 50% participation. Pruning still removes 10–14% of the data at 20 and 50 clients. Under sparse participation, clients often return after more than kk rounds; the staleness reset then makes pruning more conservative rather than acting on outdated scores, which is the intended behaviour.

Multiple local epochs. We agree this needed verification and have added it (new Appendix "Multiple local epochs") (Appendix A.6 in the revised manuscript). On Non-IID CIFAR-10 with an aggressive δ = [10−5,10−3][10^{-5},10^{-3}], the stability check runs after every local epoch. Within a round, it compares consecutive local epochs; across rounds, it compares against the client's last epoch of its previous round, when the global model has changed through aggregation. Each setting is compared with a full-data run from the same split and initialization:

Local epochs EEFull dataPruningDSIGPU-hours (full → pruning)
283.68%86.29%53.7%2.55 → 1.39 (−45.7%)
580.63%82.42%74.5%6.11 → 1.85 (−69.6%)

The method therefore extends directly to multi-epoch training. At E=5E=5, the high DSI is largely due to early termination: every batch has been pruned by round 110, so the remaining scheduled rounds use no data. The gain over full data is consistent with client drift. Full-data accuracy falls as EE grows, because clients take more local steps on skewed data, and pruning reduces local steps late in training. We report this as a plausible explanation, not a systematic effect.

Fixed batches at initialization. This design is needed for the signal to be meaningful. The stability score σˉB\bar\sigma_{B} is a property of a batch, and the pruning rule compares it across iterations. If batches were regenerated every epoch, a change in σˉ\bar\sigma would mostly reflect which samples were drawn, not whether the representation has converged, and no batch identity would persist to be pruned. We follow B-PAS (Alam et al., 2026), which uses the same protocol: batches are fixed once at initialization, and samples are shuffled within each batch every epoch, with standard data augmentation. This protocol does not hurt or bias training:

  • Every sample is still trained on in every epoch until its batch is pruned.
  • Re-running Non-IID CIFAR-10 with three independent random batch assignments changes accuracy by only ±0.54 points (max spread 1.1), so results do not depend on a particular batching (Appendix, "Sensitivity to batch assignment").
  • Because batches are random mixes of each client's samples, pruning removes data class-agnostically (mean JSD 0.055).
  • Accuracy with pruning stays close to full-data training across all datasets (Table 3).

Sensitivity of δ. We agree that δ matters; it controls the accuracy–savings trade-off. Two clarifications and one practical point:

  • The values are schedule endpoints, not tuning ranges. [10−7,10−3][10^{-7},10^{-3}] (CIFAR) and [5×10−5,5×10−4][5\times10^{-5},5\times10^{-4}] (ImageNet) are the start and end values of an exponential schedule, not intervals within which δ must be tuned (now stated in Sections 3 and 5).
  • The behaviour is smooth. Across nine schedules spanning two orders of magnitude on ImageNet (Table 1), accuracy varies gradually between 72.5% and 75.1%, with no sharp cliff. The different scales for CIFAR and ImageNet reflect different activation magnitudes, and one schedule transfers across CIFAR-10, CIFAR-100, and SVHN.
  • δ can be calibrated quickly. As shown in Figure 5 (Appendix, "Fast δ calibration"), running the threshold sweep on only 10% of the ImageNet training data preserves the relative ordering of all schedules in both data savings and accuracy. Larger thresholds consistently prune more, smaller ones prune less, and the trends closely match full-data runs. A suitable schedule can therefore be chosen from short partial runs and then used for full training. This is consistent with B-PAS, where the 10%-subset calibration on ImageNet took about 5.3k s versus 46.4k s for full runs, roughly 9× faster. Tuning δ thus costs about as much as tuning a standard hyperparameter such as the learning rate.

We thank the reviewer again for these suggestions, which have substantially improved the clarity and completeness of the paper. We hope our responses address the concerns and would be grateful if the reviewer would consider updating the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.