acceptodds
Under review as a conference paper at ICLR 2027

Retention is Leakage: Forgetting Does Not Confer Privacy in Federated Continual Learning

Abstract

Federated continual learning is often presented as private by design. Clients keep their raw data, and the data of an early task is assumed to fade from the system as the model forgets it. We test this assumption against an adversary who observes every broadcast rather than only the final model. The assumption fails in two ways. First, forgetting does not remove leakage. On CIFAR-100 and ImageNet-R, six further tasks of plain federated averaging erase about three quarters of the above-chance accuracy on an early task. The transcript still holds 87% and 97% of the above-chance membership leakage about that task. Second, retention is leakage. Every anti-forgetting mechanism we test preserves most of the accuracy and keeps or amplifies leakage. Exemplar replay raises above-chance membership leakage by a factor of 2.6 to 4.7 over six tasks. The factor grows with the share of client data that the buffer stores. A method that retains exact second-moment statistics exposes members with a true positive rate of at least 0.92 at a 1% false positive rate. Secure aggregation does not change this. We explain both failures with a lifelong accounting framework. It names the unit of privacy, records which releases read each datum, and separates retention operators that read old data again from operators that do not. Only the second kind admits a privacy cost that is independent of the length of the stream. We build a retention operator of that kind, which retains the release rather than the data. Under example-level privacy at it reaches 81% accuracy on CIFAR-100, against 88% without privacy, and a likelihood-ratio audit finds no leakage above chance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Data Privacy, Federated Continual Learning, Retention, Federated Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_87FHOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether forgetting in federated continual learning reduces privacy leakage. It argues that declining accuracy on an old task does not erase information already present in the training transcript, while mechanisms that repeatedly use old data to retain accuracy may create further leakage. The paper formalizes this distinction through an artifact ledger and an influence horizon, audits membership leakage from different views of the training process, and proposes a method that privatizes each task’s statistics once and reuses the resulting releases. Its experiments report persistent transcript leakage despite forgetting, increased leakage under several retention methods, and a more favorable privacy–accuracy trade-off for the proposed method.

Critical strengths

Analyzing how forgetting relates to privacy leakage offers a novel and valuable perspective.

Critical weaknesses

The writing is so poor that the paper is difficult to follow. The motivation, problem formulation, and proposed framework are not explained in a coherent sequence. The paper introduces a release channel, four adversarial views, an artifact ledger, touched sets, privacy units, and an influence horizon in rapid succession, without using proper language to show how these concepts relate to one another. The propositions and theorems are also presented without almost any explaination. As a result, substantial effort is required simply to identify the paper’s main argument.

The novelty and contributions are likewise unclear. The paper lists a vocabulary, a membership-leakage audit, a privacy-accounting criterion, and a retention operator as separate contributions, but does not clearly distinguish the new technical elements in each from established tools such as differential privacy composition, membership inference audits, and noisy statistical summaries. In particular, I do not consider introducing “a vocabulary that makes the question answerable” to be a substantive novel contribution on its own. The authors need to identify precisely which results and design choices are original, and explain how they advance beyond the existing techniques they build upon.

Questions for authors

See weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 87FH

We thank the reviewer. We agree that the presentation made the argument hard to follow, and we have rewritten it. The revised PDF uses the extra page for the changes below.

W1. Coherence of the formulation, and explanation of the results

The paper makes one argument in three steps.

  • A federated adversary sees every broadcast, not only the final model. So the object to analyse is the transcript of all broadcasts.

  • Whether a lifelong privacy guarantee exists depends on one number: how many releases read each datum as tasks keep arriving.
    - If it stays bounded (contractive), the guarantee is fixed for any number of tasks (Theorem 2).

    - If it grows (accumulating), additive Gaussian noise cannot give any finite guarantee (Proposition 3).
    
  • Replay keeps reading old data, so it is accumulating, and our audit shows that its leakage grows. Retaining releases instead of data is contractive. Our operator does this and keeps high accuracy.

Section 3 now opens with this roadmap. Each concept answers one question in the chain, and the concepts appear in that order, illustrated with one running example (a client that keeps a replay buffer):

  • release channel: what leaves the clients;
  • views: who sees which part;
  • artifact ledger and touched set: which data each release read;
  • unit: whose data the guarantee protects;
  • influence horizon: how many releases read one unit's data. This is the quantity Theorem 2 and Proposition 3 use.

Before each theorem and proposition, we now add a short paragraph that says what it states, why it holds, and where the paper uses it.

W2: Novelty, and what is new beyond existing tools

We acknowledge the vocabulary is not a contribution on its own. We remove it from the list and restate the contributions below. For each one, we separate what we build on from what is new.

  • A measurement and a finding.

    • What we build on: the likelihood-ratio membership test (Carlini et al., S&P 2022).
    • What is new: to our knowledge, the first audit of membership leakage across tasks on the federated transcript, with views that model secure aggregation. Prior federated attacks study one round or one model. The centralised forgetting result (Jagielski et al., ICLR 2023) studies one model.
    • The finding: it reverses that result. In federation, forgetting does not remove Leakage, and every retention mechanism we tested keeps or increases it. New rebuttal results strengthen this. Methods that retain statistics leak through those statistics already at the target's task. On CUB-200, M2's broadcast statistics alone reach a TPR of 0.995 at 1% FPR.
  • A characterization.

    - What we build on: standard composition (R\'enyi differential privacy). Composition bounds the cost of a known number of releases.
    - What is new: we identify the influence horizon of the retention operator as the quantity that decides whether that number stays bounded as tasks arrive. A bounded horizon is sufficient for a guarantee that does not grow with the number of tasks (Theorem~2). For additive Gaussian mechanisms it is also necessary (Proposition~3). We also give an account that measures the horizon from the releases alone. Prior lifelong private learning (Lai et al., 2022) treats a single centralised learner, without clients or units.
    
  • An operator.

    • What we build on: noisy sufficient statistics, as in analytic learning (e.g., AFCL).
    • what is new: a retention rule that releases each task once and carries forward only releases. We prove it contractive, so its Guarantee is fixed for any number of tasks while it still prevents forgetting.
    • New comparison: in the rebuttal we compare it with DP-SGD and DP-FedAvg at matched privacy. All three were tuned on the same wide grid with the same selection procedure on public data. The grid was widened after a sensitivity check on the test split, which we disclose. At the example level our operator leads in all 9 cells (3 datasets ×\times 3 values of ϵ0\epsilon_0); for example, 0.828 against 0.611 on CIFAR-100 at ϵ0=4\epsilon_0 = 4.

I thank the authors for their response and their efforts during the rebuttal. However, I do not see substantial improvement in the presentation. The introduction, for example, still lacks a coherent structure that clearly establishes the motivation, the limitations of existing methods, and the contributions of the work. The writing remains below the standard expected of a research paper, making it difficult to assess the work’s contributions. Nevertheless, in recognition of the authors’ efforts during the rebuttal, I am raising my score to 2.

AuthorsOpenReview ↗

New Paper Uploaded

Thank you for your feedback and for increasing your score. It appears your comment was submitted right before our revised PDF was uploaded to the system.

We have now uploaded the updated manuscript with all revisions highlighted in blue. In particular, we substantially rewrote and restructured the Introduction (Section 1) to clearly establish the motivation, the limitations of existing methods, and our three core contributions (explicitly separating what we build on from what is new), and we added a roadmap, running example, and intuitive explanations before every theorem and proposition in Section 3.

If you have the time, we would greatly appreciate it if you could review the newly uploaded PDF and reconsider your score based on these presentation improvements.

Reviewer_mjT5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper dives deep into privacy in federated continual learning, and reveals that in the transcript adversary case (the adversary observes every broadcast): (1) forgetting previous tasks doesn’t remove the leakage; (2) any method with a retention mechanism keeps the leakage the same or increases it. The paper further shows that only sending summary statistics and adding noise to the aggregation can keep up accuracy without sacrificing privacy.

Critical strengths

The finding is useful and interesting. The claim that forgetting does not guarantee leakage removal is well supported. The experiments also cover quite different aspects of the paper, especially the controlled experiments on retention vs leakage and secure aggregation vs leakage.

Critical weaknesses

The main weakness of the paper is that the experiments are not thoroughly discussed and explained, especially the results that do not hold or do not support the claims made. There are important results that need more in-depth discussion and are directly related to the main thesis of the paper, but are left out. The main one is the M2 and M3 results in the part where the paper analyzes how mechanisms that keep retention increase leakage. The results show that these two, while not storing old samples and reading no old data, increase leakage significantly. This contradicts their claim about contractive operators, which is important in the paper, and thus needs thorough discussion instead of just “did not isolate the cause of this growth.”

Another weakness is about the units in the paper. The paper mainly focuses on example-level privacy and its results hold for that, but for other units it lacks clear discussion and claims. The proposed operator seems to only work at the example level. The authors say “every claim in this paper carries its unit,” so it is expected to see how the case differs for other units. It is also important to discuss the units in FCL and what the expected unit to protect usually is in this domain.

Questions for authors

Other than the concerns I listed in the weakness section, I have some questions:

  • Your proposed operator needs a public reference dataset, and each dataset you studied needs a corresponding in-distribution one. Do you have any insight into how well your operator can generalize to an OOD reference dataset? Since the focus is privacy, requiring an in-distribution reference set counters the contribution.
  • How are the five target records chosen, just at random?

Overall I like the findings, but there are critical weaknesses, so I rate it a 2. I would be happy to raise the rating if my concerns and questions are addressed during the rebuttal.

2 comments
AuthorsOpenReview ↗

Response to Reviewer mjT5

We thank the reviewer for the constructive feedback of our work. We now respond to the concerns raised.

W1: M2 and M3 raise leakage without reading old data. Does this contradict contractivity?

We acknowledge the concern raised even though we maintain there is no contradiction. However, we acknowledge we need to do an analysis on this claim. Contractivity bounds what any attack can learn from noised releases. It says nothing about the score of one fixed attack. The precise argument is post-processing. After the target's task, every broadcast of M2 or M3 is a function of earlier broadcasts and of data independent of the target. By the data-processing inequality, the best possible attack on the transcript therefore gains nothing after the target's task (e=0e = 0). The growth we reported must come from our attack, not from new information.

Our audit scored only the logit margin (family F1F_1) for M2 and M3. At the target's task, however, both methods also release statistics. M2 broadcasts per-class Gaussian means and variances (family F6F_6). M3 releases a task covariance and the subspace computed from it (family F5F_5).

We fixed a prediction in advance: the release alone at e=0e = 0 leaks at least as much as the margin attack at e=6e = 6. We tested it on the paper's own shadow federations. Rerunning the margin attack reproduces our reported numbers exactly.

TPR at 1% FPR, CIFAR-100 / CUB-200 / ImageNet-R:

  • M2 broadcast statistics, e=0e = 0: 0.244 / 0.995 / 0.647. Margin, e=6e = 6: 0.042/0.329/0.235.

  • M3 broadcast subspace, e=0e = 0: 0.050/0.181/0.102. Margin, e=6e = 6: 0.038/0.090 /0.090.

  • M3 task covariance (server view): 0.999/1.000/1.000.

The prediction holds in all 12 cells. For the M3 subspace on CIFAR-100 and ImageNet-R the intervals overlap, so we claim ``at least as much'', not more. The release scores are constant in ee, as a one-time release should be. So the information was in the transcript at e=0e = 0. The margin attack read it only later, through the classifier trained on top of these releases. This also shows that the margin attack understated the leakage of M2 and M3.

In the revision we make three changes: (1) we replace ``we did not isolate the cause'' with this analysis; (2) we state the post-processing argument as a proposition; (3) we add the release-attack rows to Table 2.

W2. Units of privacy

We agree that client-level privacy is the standard unit in federated learning (McMahan et al., ICLR 2018). Our theory covers all units. Corollary 7 gives:

  • a fixed guarantee for one client in one task (U2U_2);
  • Wϵ0W\epsilon _0 for one client over a window of WW tasks (U3U_3);
  • no finite guarantee from any additive Gaussian mechanism for a client that contributes forever (U4U_4). This is a property of the problem, not of our operator.

In practice, client-level accuracy is limited by the number of clients, because the noise must hide a whole client's contribution. While checking this, we found an error in our 50/100-client arm. It reused a clip norm calibrated for 10 clients, so the noise could not shrink. With a clip norm matched to client size (CIFAR-100, U2U_2, ϵ0=4\epsilon_0 = 4), accuracy grows as follows:

Clients101005001,0002,000
Our operator0.0150.0950.6020.7190.782

A tuned DP-FedAvg baseline is better with 100 clients (0.132) and worse with 1{,}000 (0.620). Both methods were tuned with the same procedure on public data, on a grid widened after a sensitivity check on the test split. We disclose this in the revision.

Therefore, +++ client-level guarantees become practical in federations with hundreds of clients or more. With few clients, such as a few hospitals, the example-level guarantee is the realistic one. We add this discussion to Section 3.2, add the scaling results to Section 6.6, and correct the 50/100-client statement.

Q1. Out-of-distribution public data: The requirement is mild. With no projection (public data choose only the ridge λ\lambda), CIFAR-100 reaches 0.818 at ϵ0=4\epsilon_0 = 4, against 0.810 with in-distribution PCA. With PCA fitted on a different dataset, and every setting chosen there, it reaches 0.738. With a data-independent random projection, it reaches 0.760. The revised appendix reports all datasets.

Q2. How the five targets are chosen: In each client shard of each task, 5 targets are drawn uniformly at random without replacement, which gives 500 targets per run. The pool is fixed across the 1{,}024 shadows. In every shadow, the membership of every record, targets included, is drawn independently with probability 0.5.

While answering this, we noticed that the membership draw depends on the shadow index but not on the run seed. So shadow ii has the same membership pattern in all three seeds, while the streams, partitions and target pools differ. Point estimates are unaffected, and we state this in the appendix.

Thank you for the detailed response. I still think the presentation could be improved, but given the additional experiments, I am raising my overall rating to a weak accept.

Reviewer_ihiZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents the viewpoint that model forgetting is not equivalent to privacy security. Moreover, through numerous experiments, it has been proved that the traditional replay-based method will inevitably lead to privacy leakage. Based on this, the paper proposes a new strategy. It requires the client to calculate the second-order feature covariance matrix after dimensionality reduction and strict clipping locally, and then the server performs secure aggregation and mandatory Gaussian noise addition. Extensive experimental results and theoretical analysis have proved the effectiveness of the method.

Critical strengths
  1. The article presents a novel viewpoint: it is argued that model forgetting is not equivalent to privacy security. Moreover, it demonstrates that the current mainstream methods have a significant risk of privacy exposure mathematically.
  2. The paper effectively resolves the contradiction of "having to retain knowledge while discarding data" by utilizing and uploading the noisy second-order covariance matrices of the features.
Critical weaknesses
  1. The paper uses the matrix P trained on a public dataset to project the features. Will this lead to the matrix P and the performance of the entire model being overly dependent on the data distribution of the public dataset? If the projection matrix P is not used, how would the model perform? It is suggested that the authors provide ablation experiments or relevant explanations regarding the matrix P.
  2. Although this method performed well on the CIFAR-100 dataset, its performance on the CUB dataset deteriorated significantly under the same privacy protection conditions. This reveals the limitations of the model's generalization ability. Although the authors explained that this was due to the relatively small number of training samples per class in the CUB dataset, the performance of this method on the ImageNet-R dataset was still very poor. Does this indicate that this method has certain limitations?
  3. The core mechanism of this leakage prevention approach is essentially noise addition. If noise is added to the updated parameters of the current mainstream continual learning paradigm (such as "foundation model + prompt learning"), can it also achieve better anti-forgetting effects while ensuring information security? It is suggested that the author should supplement a comparative analysis of this method with the noise injection strategies of the current mainstream large model prompt paradigm in the discussion section.
Questions for authors

Please refer to the weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer ihiZ

We thank the reviewer for the constructive feedback of our work. We ran new experiments for each weakness. All settings were chosen on public data only, and the test split was used only for the final numbers.

W1: Dependence on the projection PP and on the public data

PP is fitted on a public split that no client holds, so it costs no privacy. To test the dependence of the method on it, we compared three alternatives (unit U1U_1, 3 seeds): (1) No projection: (P=IP = I; public data choose only the ridge λ\lambda). CIFAR-100: 0.564 / 0.751 / 0.818 / 0.843 at ϵ0=1/2/4/8\epsilon_0 = 1 / 2 / 4 / 8, against 0.596 / 0.690 / 0.810 / 0.818 with PCA. ImageNet-R at ϵ0=8\epsilon_0 = 8: 0.434 against 0.385; (2) PCA fitted on a different dataset, with every setting also chosen there CIFAR-100: 0.738 at ϵ0=4\epsilon_0 = 4; (3) A data-independent random projection, with its settings chosen on the other dataset. CIFAR-100: 0.760 at ϵ0=4\epsilon_0 = 4. CUB-200: 0.339 at ϵ0=8\epsilon_0 = 8, against 0.161 with PCA. So PP is optional. It helps mainly at small ϵ\epsilon, where fewer dimensions mean less noise. Without it, accuracy matches or exceeds PCA at ϵ0≥2\epsilon_0 \geq 2 on CIFAR-100. The full table for all datasets is in the revised appendix.

W2. Lower accuracy on CUB-200 and ImageNet-R

This is a real limitation of example-level noise, and the paper states it. Its main cause is the number of examples per class. The noise has a fixed size, while the statistics grow with the number of examples. We tested this inside one dataset by reducing CIFAR-100 to 25 examples per class, as in CUB-200. At ϵ0=4\epsilon_0 = 4, accuracy falls from 0.810 to 0.182, while the non-private solution barely changes (0.883 to 0.854). On ImageNet-R, part of the gap is the task itself: even the non-private solution reaches only 0.635. The limitation is shared by other private methods. Under the same guarantee and the same tuning procedure (see W3), DP-SGD reaches 0.045 on CUB-200 and 0.112 on ImageNet-R at ϵ0=8\epsilon_0 = 8. Our operator reaches 0.405 and 0.429.

W3. Noise injection in other paradigms, such as prompt learning

Our theory states when this works. A method that reads each example only during its own task is contractive (Theorem 2 with h=1h = 1). Calibrated noise on its updates then gives a guarantee that does not grow with the number of tasks. Noisy prompt tuning without replay belongs to this class. Methods that replay stored data are accumulating, and for additive Gaussian noise Proposition~3 rules out a finite lifelong guarantee for them.

Training prompts through the backbone was beyond our compute budget. Instead, we added the closest baselines we could run on the same frozen features:

  • DP-SGD (example level);

  • DP-FedAvg (client level; McMahan et al., ICLR 2018).

The ledger confirms that both read each datum only once. Both baselines and our operator were tuned on the same wide grid, with the same selection procedure on public data. We widened the grid after a sensitivity check on the test split showed that the original grid lacked the small learning rates these baselines need. Selection itself used public data only, and the revision reports this.

Results (final average accuracy, 3 seeds):

  • Example level: our operator leads in all 9 cells (3 datasets ×\times 3 values of ϵ0\epsilon_0). At ϵ0=4\epsilon_0 = 4: CIFAR-100 0.828 against 0.611, CUB-200 0.172 against 0.018, ImageNet-R 0.340 against 0.097. It also forgets less on CIFAR-100 (BWT −0.065-0.065 against −0.178-0.178).

  • Client level (CIFAR-100, ϵ0=4\epsilon_0 = 4): with 10 clients, both stay at chance. DP-FedAvg is better with 100 clients (0.132 against 0.095). Our operator is better with 500 or more clients (with 1{,}000: 0.719 against 0.620).

Thus, noise on updates does give a lifelong guarantee, as the theory predicts. At the same guarantee, retaining noisy statistics gives higher accuracy in most settings. We add this comparison to Section 6.6 and the appendix.

Reviewer_ihiZOpenReview ↗

Comment

Thank you for the detailed response and the additional experiments.

The rebuttal addresses my main concerns. On the projection matrix, the new ablation with no projection, a PCA fitted on a different dataset, and a data-independent random projection shows that the method does not rely on the public data used to fit P, and that P mainly helps under small privacy budgets. On the lower accuracy on CUB-200 and ImageNet-R, the controlled experiment that reduces CIFAR-100 to 25 examples per class confirms that the drop is caused by the number of examples per class, and the authors acknowledge this as a limitation of example-level noise. On noise injection in other paradigms, the theoretical discussion clarifies which methods admit a lifelong guarantee, and the new comparisons with DP-SGD and DP-FedAvg on the same frozen features support the advantage of retaining noisy statistics in most settings.

I note that the accuracy in the low-data regime remains limited, and that a direct comparison with noisy prompt tuning is not included due to the compute budget. After considering the response together with the comments of the other reviewers, I will keep my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.