- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper adapts activation-stability-based batch pruning to federated learning. Each client tracks the mean standard deviation of hidden activations for fixed batches and permanently removes batches whose scores change by less than a scheduled threshold. To handle intermittent participation, the method tightens the threshold for moderately stale clients and resets activation histories after longer absences. Experiments cover four image datasets, several architectures, heterogeneous partitions, and partial participation, reporting accuracy, cumulative data savings, and training-time savings. The contribution is primarily empirical; no convergence or pruning-safety guarantee is provided.
Critical strengths
1.The method is lightweight and locally implementable, without additional communication or server-held validation data. Evaluation extends to ImageNet and multiple architectures, and the reported timing measurements include activation-tracking overhead.
2.The staleness mechanism addresses a concrete issue introduced by intermittent client participation. Its ablation shows meaningful improvements over retaining old histories without correction, supporting the usefulness of this federated adaptation.
Critical weaknesses
1.The principal explanatory claim is insufficiently supported, limiting the contribution beyond B-PAS. The central activation statistic and permanent batch-pruning rule are inherited from prior work. Stability of a mean standard deviation does not establish that representations have converged or that the corresponding data no longer contribute useful gradients. The drift comparisons and nonsignificant statistical tests do not establish independence, equivalence across heterogeneity levels, or safety after subsequent aggregation. Direct evidence about the learning utility of pruned batches is particularly important because deleted batches are never reconsidered.
2.The comparative evaluation does not adequately isolate the benefit of the proposed criterion and federated adaptation. The FedCS comparison uses published results rather than a controlled rerun, making the large accuracy differences difficult to attribute to pruning quality. Partial-participation and client-scaling experiments lack corresponding full-data baselines. Moreover, input-space similarity reaches 85.14% versus 85.48% for activation stability when given the same pruning schedule. Together, these results leave unclear how much benefit comes from the activation criterion, the pruning schedule, or differences in training protocols.
Questions for authors
- Does the score predict reduced loss-gradient contribution or reduced future learning utility for individual batches? Is there evidence that batches selected for removal remain unimportant after later global aggregations?
2.How is DSI calculated, including early termination, and how is the published FedCS pruning fraction converted into the same cumulative data-usage budget?
3.What are the full-data results under the same partial-participation and client-scaling settings? How does the method compare with directly applying B-PAS locally and with a simple predetermined late-training pruning schedule at matched computational budgets?
4.What constitutes an independent observation in the heterogeneity tests, and how are temporal and within-client correlations handled?
4 comments
Response to Reviewer QyoG (2/4)
W1 (Contd.)
On the drift comparisons and tests. We agree with the reviewer. We do not claim that activation stability and client drift are independent, nor that non-significant tests establish equivalence across heterogeneity levels. The analysis in Section 4 is descriptive, and its purpose is narrower: to check whether the heterogeneity that drives client drift also visibly affects the local signal our pruning rule uses.
- The two signals behave differently (Fig. 3). Activation stability changes sharply only in the first few rounds. Afterwards only small residual fluctuations remain, which settle within a narrow band (±0.001) at every heterogeneity level. Client drift varies much more, and its trajectory depends on heterogeneity. At and it rises over many rounds before leveling off. At it is non-monotonic and fluctuates noticeably more. Both signals eventually settle, but their magnitude and their dependence on heterogeneity differ.
- Stability shows no detectable dependence on (Fig. 4). Its trajectories overlap across . The ANOVA and Kruskal–Wallis results (, ) are reported as descriptive. As the appendix now states, they are not a formal test of equivalence, because consecutive rounds of one run are correlated (please see Q4).
- What we infer, and what we do not. The pruning rule uses only each batch's own stability score. That score shows no detectable dependence on , while drift clearly does. Global disagreement therefore does not visibly propagate into the local pruning signal. We treat this as consistent with local pruning remaining usable under heterogeneity, not as proof that it does.
W2. Isolating the benefit of the criterion, the schedule and the federated adaptation
Criterion vs. schedule. We agree with the reviewer's reading of the input-space similarity result, and we now discuss it in the main text (Section 5, "Random pruning vs. ours"):
| Selection rule, same pruning schedule (Non-IID CIFAR-10) | Accuracy |
|---|---|
| Random | 82.46% |
| Input-space similarity | 85.14% |
| Activation stability (ours) | 85.48% |
Once the schedule is fixed, a reasonable redundancy score recovers most of the benefit, because each batch is a random mix of the client's samples. What activation stability provides that a static score cannot is the schedule itself: a training-aware decision of when and how much to prune, made locally, during training and without communication. A static similarity score has no notion of training progress; in this experiment, it inherits the schedule from our method.
The federated adaptation vs. B-PAS. Applying B-PAS directly on each client loses 3.7 points at 75% participation (83.26% vs. 86.95%) and 5.8 points at 50% (78.34% vs. 84.16%). The appendix ablation is now labelled "B-PAS (local)" and discussed in the main text.
FedCS. We agree this is not a controlled comparison: no implementation is available, and we use its published numbers. The revised Section 5 and the Table 2 caption now state this explicitly, describe the comparison as indicative, and point to the controlled baselines above, which were all run in our own pipeline under the same schedule.
Full-data baselines. We have added full-data training at every participation rate (Table 4) and for every client-scaling setting (Appendix, Table "Client scale"). The partial-participation and client-scaling results are now directly comparable with full data in the same setting (please see Q3).
Response to Reviewer QyoG (3/4)
Q2. How is DSI computed, and how was the FedCS pruning fraction converted?
Section 5 now gives the definition explicitly, following B-PAS:
- is the number of active batches client trains on at iteration .
- is the number of batches of client before training.
- is the scheduled number of iterations, and is the iteration at which training stops.
Early termination occurs only when every batch on every client has been pruned (Section 3), and the remaining iterations then count as zero data. Section 5 includes a worked example.
For FedCS, we interpret its reported pruning rates as the overall data fraction removed, and match the closest DSI. This is an approximation: FedCS prunes once, early in training, so its fraction applies to almost all rounds, whereas our savings accumulate over training. The two budgets are similar but not identical, which the Table 2 caption now states.
Q3. Full-data results under partial participation and client scaling; direct B-PAS; a predetermined schedule
Full-data results under partial participation. We thank the reviewer for this suggestion. We have added full-data training at every participation rate to Table 4. Together with the direct application of B-PAS, all methods can now be compared at the same participation rate (Non-IID CIFAR-10, ):
| Participation | Full data | Ours () | Ours () | B-PAS (local) |
|---|---|---|---|---|
| 100% | 85.71% | 85.48% | 85.48% | 85.48% |
| 75% | 82.69% | 86.95% | 82.08% | 83.26% |
| 50% | 77.13% | 84.16% | 80.83% | 78.34% |
At full participation no client is ever stale, so has no effect and our method coincides with B-PAS. Under partial participation:
- Full-data training degrades as participation drops (85.71 → 82.69 → 77.13%), because fewer client updates are averaged in each round.
- Our method with clearly exceeds full data: by 4.3 points at 75% participation and 7.0 points at 50%, while still removing data (DSI 19.2% and 12.3%).
- Directly applying B-PAS locally gains much less (+0.6 and +1.2 points over full data), and stays 3.7 and 5.8 points below our method. The gap grows as participation becomes sparser and references become staler. This is the failure mode our staleness mechanism is designed to correct.
- A large staleness threshold () is close to full data at 75% (−0.6 points) and above it at 50% (+3.7). It remains well below , confirming that tying pruning decisions to the current global model matters.
Full-data results under client scaling. We have also added full-data training to the client-scaling experiments (Appendix, Table "Client scale"):
| Clients | Participation | Full data | Pruning |
|---|---|---|---|
| 10 | 50% | 77.13% | 84.16% |
| 20 | 25% | 59.39% | 60.97% |
| 50 | 10% | 49.45% | 54.63% |
Both methods lose accuracy as the same data is split across more clients under a fixed 200-round budget. Pruning exceeds full-data training in all three settings (+7.0, +1.6 and +5.2 points) while removing 10–14% of the data.
Why pruning can exceed full data under sparse participation. We attribute this to variance reduction. Subsampling clients raises the variance of the aggregated update, and pruning removes saturated batches that contribute noise but little signal, which helps most when fewer client updates are averaged per round. This is consistent with B-PAS (Alam et al., 2026), which likewise observed that activation-stability-based pruning can occasionally improve accuracy over full-data training. We have added this explanation to Section 5 and the appendix.
A predetermined late-training schedule at matched compute. We did not run this baseline, and we acknowledge it as a limitation. The closest control in the paper is the random baseline (Table 6). It drops exactly as many batches as our method on every client at every iteration, so it follows our schedule and data budget exactly, but it ignores the per-batch signal. It loses 0.9–4.9 points. Together with the input-space similarity result (please see W2), this indicates that both the schedule and the selection matter.
Response to Reviewer QyoG (4/4)
Q4. What constitutes an independent observation in the heterogeneity tests?
The reviewer is right. In our tests each observation is the mean stability over the probe batches at one communication round of a single training run per . Consecutive rounds are temporally correlated, and observations within a run are not independent. We have revised the paper accordingly:
- Appendix: now states the unit of observation, both null and alternative hypotheses, and that the resulting -values (ANOVA , Kruskal–Wallis ) are descriptive rather than a formal test of equivalence.
- Section 4: now calls them "descriptive statistical tests", and the appendix states that they are not a formal test of equivalence.
A seed-level analysis with an equivalence test would be the appropriate formal test, and we consider it a useful addition for future work.
We thank the reviewer again for these precise comments, which have made our claims more accurate and the paper more rigorous. We hope the clarifications and additions address the concerns, and we would be grateful if the reviewer would consider raising the score.
Response to Reviewer QyoG (1/4)
We thank the reviewer for the precise and constructive review, and for recognizing that the method is lightweight and locally implementable, that the evaluation covers ImageNet and multiple architectures with overhead included, and that the staleness mechanism yields meaningful gains. We agree that the contribution is primarily empirical, and below we state clearly what our evidence establishes. All changes are reflected in the revised manuscript.
W1 / Q1. Does activation stability indicate reduced learning utility, and do pruned batches stay unimportant?
The per-batch criterion and its link to learning utility come from prior work. Our pruning criterion is inherited from B-PAS (Alam et al., 2026), which studied the relationship between activation stability and the learning utility of individual batches in the centralized setting:
We build on this established criterion. Our paper therefore asks a different question: whether the criterion remains valid under federated, heterogeneous, partially participating training, and what is needed for it to work there.
Evidence in the federated setting.
Safety after subsequent aggregation. This concern is specific to FL, and it is what our staleness mechanism addresses. A client's reference scores are always taken at its last participation:
Without this mechanism, i.e., B-PAS applied locally, accuracy drops by 3.7 and 5.8 points at 75% and 50% participation, which shows that comparisons across distant global models do lead to unsafe decisions.
We agree that stability of the mean activation standard deviation does not by itself constitute a per-batch guarantee on gradient magnitude, and we do not claim one. Re-admitting pruned batches is a natural extension, which we leave to future work.