- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper addresses federated class-incremental learning with a trusted server, synchronized stage transitions, client-specific task orders, and spatial label skew. Clients cache a server-owned conditional generator for local replay and upload classifier snapshots, BN buffers, real-data coverage masks, and update-deviation factors. At each task boundary the server selects class-eligible cached experts by real-data coverage, weights their supervision by a Jensen-Shannon disagreement score with a singleton fallback combining predictive confidence, BN consistency, and update deviation, consolidates the generator with class, distillation, and BN-moment losses, and fine-tunes the classifier with reliability-weighted clean replay plus a BN-gated FGSM term. Validation covers CIFAR-10, CIFAR-100, and 64x64 ImageNet, five tasks and twenty clients against fifteen baselines, with five runs. The headline is CIFAR-100 +0.88 accuracy and -1.70 forgetting against DGR-FCL, with smaller ImageNet margins of 0.30 and 0.70. The paper discloses that its ablations remove only whole losses or branches, and the release covers the pipeline but its canonical config differs from the appendix.
Critical strengths
The paper is unusually candid about the limits of its own evidence. It states that its ablations do not isolate the named mechanisms, that its standard deviations establish no paired significance, that its budgets are unequal, and that its proxies carry no performance guarantee. This is consequential because it lets the reader separate what is established from what is hoped, and it is the reason I treat the paper as an honest report rather than an overclaim.
The pipeline is complete and the formal content that exists is correct. The release covers the consolidation server, reliability scorer, gate, clients, and metrics, with tests; the Jensen-Shannon identity, the KL decomposition, and the FGSM algebra are correct; and the coverage-mask formalization is clean. This is consequential because it makes the work reproducible at the pipeline level and gives future work a usable base even if the current claims do not hold.
Critical weaknesses
The headline accuracy claim is not established, and the named mechanisms are never priced. The ImageNet margin is 0.30 points with standard deviations of 0.25 and 0.30 (Welch t about 1.7), yet it is headlined in the abstract; the three mechanisms in the title are never individually ablated, as the paper concedes; and the method falls from 58.4 to 32.9 to 24.2 across 5, 10, and 20 tasks with no baselines at the longer horizons. This is consequential because the paper's contribution is the claim that these mechanisms improve federated CIL, and the evidence cannot attribute even the thin gains.
The release contradicts the paper. The canonical config disagrees with Appendix E.4 on roughly fourteen values, so which setting produced Table 1 is unknowable; most of the zip is a sweep grid for a baseline that appears in no table; and there are no per-table configs or result dumps. This is consequential because reproducibility is the one claim the artifact is supposed to support, and it fails at the headline table.
Questions for authors
Clarification requests (these could change my recommendation):
-
Which hyperparameter set produced Table 1, Appendix E.4 or fedrua_paper.yaml? Please reconcile them and ship per-table configs.
-
Can you add the four missing ablations: uniform reliability weights, singleton fallback removed, gate always open or always closed, and eligibility versus random selection?
-
Why are there no baselines at 10 and 20 tasks, where FedRUA falls to 33 and 24 percent? Please add the strongest three or scope the claim to five tasks.
-
With paired per-seed tests, does the ImageNet accuracy margin survive? And what do the five seeds re-randomize?
-
How does FedRUA differ from FedCVAE (NeurIPS 2023), which pre-empts the server-side data-free generator skeleton? I treat FedGTG as awareness only, since it is arXiv-only.
-
What is the per-round byte payload (classifier, generator, BN buffers, masks) versus FedCIL and TARGET?
Suggestions for future work (not needed for my assessment):
-
Add a standard benchmark such as CORe50 or DomainNet and a standard round budget with convergence curves.
-
Add a memory-matched real-exemplar replay baseline and equalize replay budgets.
-
Discuss the privacy implications of uploading classifier snapshots and BN buffers, which are the inputs of feature inversion.
2 comments
Although the authors have addressed my concerns I'm afraid the fundamental contributions presented in the manuscript fall short of the prestigious AI conference such as ICLR. Such fundamental novelty weaknesses cannot be remedied via more experiments or more results since the substrate of the paper is essentially unimpressive.
I thank the authors for their extra efforts and I believe their work has a good chance of getting accepted at 2nd tier conferences. Best wishes.
Thank you for recognizing the problem and suggesting further tests of tranfer and access to information.
Q1: Result provenance and configuration reconciliation.
A: Thank you for your remark. The available run records do not establish which settings produced Table 1. Appendix E.4 documents the manuscript protocol. The released YAML shares its client batch size, learning-rate schedule.
Table R2. Key differences between Appendix E.4 and the released CIFAR-100 configuration and code.
g_lr.We will clarify these differences in the revised manuscript and add a release index linking each traceable table row to its run IDs, resolved configurations, and per-seed results. The index will flag rows with missing records.
Q2: Mechanism-specific controlled ablations.
A: Following your suggestion, we conducted additional experiments with mechanish-specific controlled ablations. We will update results and discuss these in our revised manuscript.
Table R3. CIFAR-100 accuracy for FedRUA ablations of reliability weighting, the singleton fallback, FGSM gating, and teacher selection under IID and Dirichlet (α = 0.1) client allocations.
Q3: Ten-task and twenty-task comparison.
A: Following your suggestion, we added standalone 10- and 20-task CIFAR-100 comparisons with TARGET, FedCIL, and GLFC. We will include Table R3 in the revised manuscript and limit claims against the full Table 1 baseline set, including DGR-FCL and CAN, to the five-task setting.
Table R4. Final average accuracy (%) on CIFAR-100 with standalone 10- and 20-task schedules. Values are mean ± SD.
FedRUA’s mean accuracy exceeded TARGET’s, the strongest baseline in Table R4, by 2.68 percentage points at 10 tasks and 1.28 points at 20 tasks.
Q4: Per-seed evidence and performance claims.
A: Table 1 reports ImageNet-1K accuracy of (44.30 \pm 0.25%) for FedRUA and (44.00 \pm 0.30%) for DGR-FCL over five runs (different seeds). We will report the 0.30 percentage-point gap as a difference in means and note the need for paired validation on more complex datasets in the revised manuscript.
Q5: Relationship to FedCVAE.
A: FedCVAE-KD distills client-trained decoders into a server decoder for data-free one-shot FL. The closer continual-learning precedent is MFCL, which trains a generator from the global classifier at task boundaries and shares it for local replay. We will also cite the FedGTG for its server-trained image and feature generators. FedRUA selects teachers by real-data class coverage, weights their guidance, and applies a BN gate to the additional FGSM loss during server classifier fine-tuning.
Q6: Communication and resource accounting.
A: Please refer to Q7 in our response to Reviewer qjzy.