acceptodds
Under review as a conference paper at ICLR 2027

Decoupled in Time, Aligned in Space: Geometric Alignment for Temporally Heterogeneous Federated Continual Learning

Abstract

Federated Continual Learning (FCL) extends federated learning to evolving local data streams, but existing methods largely assume comparable task progression across clients. In realistic edge environments, heterogeneous task workloads cause clients to advance through continual learning at different rates, producing persistent temporal misalignment across learning states. This misalignment biases global aggregation and progressively distorts previously accumulated knowledge. We formalize this problem as ***temporal heterogeneity*** and propose **TARDIS** (**T**emporal-**A**nchored **R**epresentation **A**lignment for **D**rift-**I**nhibiting **S**tabilization). TARDIS progressively constructs a global Equiangular Tight Frame (ETF) as continual knowledge expands, providing shared geometric anchors that maintain representational consistency across heterogeneous client states. Extensive experiments on four continual-learning benchmarks show that TARDIS consistently outperforms representative FCL baselines under both Task-IL and Class-IL settings, with particularly strong gains in global-class prediction and knowledge retention.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Federated Learning, Federated Continual Learning, Neural Collapse

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9FZnOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper consider a temporal heterogeneity setting in Federated Continual Learning (FCL), where clients have asynchronous task progress. The paper proposed TARDIS, an algorithm that leverage the neural collapse to achieve the Equiangular Tight Frame (ETF) state, and make them as memory for the FCL system.

Critical strengths
  1. The idea of shared ETF anchors and the utilization of neural collapse is some how interesting and worth reading from my perspective.
  2. The results seem to be strong in class incremental learning.
  3. The paper apply a comprehensive ablation studies to prove the paper's technicality.
Critical weaknesses
  1. Aggregation bias and knowledge drift are long-standing FL/FCL issues, and the paper does not directly show that heterogeneous workloads cause or worsen them. Observations I–II are asserted rather than measured, and there is no synchronous schedule with the same partition as a control.
  2. It seems that the temporal heterogeneity is close to the asynchronous FCL setting [R1], The authors should consider about this more carefully to make a better positioning to the current paper scope of works.
  3. The ETF seems to be closely related to the current series of gradient projections in Continual Learning, especially the series of works in FCL, e.g., FOT [R2]. This part need a careful discussion to prove the novelty of ETF contribution. Furthermore, the discussion of model inversion seems to be incremental and disconnected from the whole storyline of ETF geometry. It should be noted that the current literature of applying model inversion in FCL has been considered, e.g., LANDER [R3] and TARGER [R4].
  4. The method is class-incremental by design, since expansion requires new, disjoint classes, so the general “FCL” framing in the title overclaims.
  5. The current presentation of the paper requires significant improvement. For instance, the neural collapse and the ETF appear without motivation in the introduction. Several core objects are never defined, e.g., the feature extractor fθf_\theta, the auxiliary head, the global model and the forgetting metric.
  • [R1]: Asynchronous federated continual learning, CVPRW.
  • [R2]: Federated orthogonal training: Mitigating global catastrophic forgetting in continual federated learning, ICLR 2024.
  • [R3]: Text-Enhanced Data-free Approach for Federated Class-Incremental Learning, CVPR 2024.
  • [R4]: TARGET: Federated Class-Continual Learning via Exemplar-Free Distillation, ICCV 2023.
Questions for authors
  1. How the "Heterogeneous Workloads" are important? How it related to the model performance?
  2. As the task-stream misalignment, how it related to the different learning states and distinct task-specific knowledge?
  3. How to recover a global ETF from the learned representation space? do we just train it and overfit to converge?
  4. According to the experimental setup, how the temporal heterogeneity is made? it is not detailed clearly in the paper. How are task-IL and class-IL setting is made according to those dataset?
  5. Although the historical data is not saved. however, please report about the memory of the storage of ETF?
  6. As dirichlet alpha are proposed. how is the robustness according to different type of heterogeneity? For example , beside the class skew, how about the task-order mismatch [R5]?
  • [R5]: HERO: A Heterogeneity-Aware Benchmark Library for Federated Continual Learning, NeurIPS 2026.
8 comments
AuthorsOpenReview ↗

Response to Reviewer 9FZn (part 1)

Dear Reviewer 9FZn,

We would like to thank the reviewer for taking the time to review our paper and provide valuable feedback. We conduct extensive experiments and address your concerns point by point below. We also updated the manuscript with changes highlighted in blue.

Comment 1:

"Aggregation bias and knowledge drift are long-standing FL/FCL issues, and the paper does not directly show that heterogeneous workloads cause or worsen them. Observations I–II are asserted rather than measured, and there is no synchronous schedule with the same partition as a control."

"How the "Heterogeneous Workloads" are important? How it related to the model performance?"

Response:

We thank the reviewer for raising this important question. To examine the effect of temporal misalignment, we follow your suggestion and compare FedAvg under synchronized task progression (Sync) and workload-induced temporal heterogeneity (TH) at α=0.5\alpha=0.5. As shown below, TH consistently yields lower accuracy and higher forgetting across the four benchmarks. These results provide further evidence that heterogeneous task progression can exacerbate forgetting in FCL.

Table 1. Effect of temporal misalignment on FedAvg (α=0.5\alpha=0.5). Acc: accuracy (%, ↑); F: forgetting (%, ↓). The Δ row shows the relative accuracy decrease under TH (single run, seed 0).

BenchmarkScheduleTask-IL Acc ↑Task-IL F ↓Class-IL Acc ↑Class-IL F ↓
CIFAR-10Synchronized (Sync)84.408.7531.5124.57
Temporal Heterogeneity (TH)68.0233.5824.6784.42
Δ Accuracy↓ 19.41%–↓ 21.71%–
CIFAR-100Synchronized (Sync)64.1921.8820.6537.40
Temporal Heterogeneity (TH)40.5040.6312.2160.06
Δ Accuracy↓ 36.91%–↓ 40.87%–
EMNISTSynchronized (Sync)85.7711.9728.5451.91
Temporal Heterogeneity (TH)43.9853.6415.7791.62
Δ Accuracy↓ 48.72%–↓ 44.74%–
Tiny ImageNetSynchronized (Sync)27.3539.979.4540.65
Temporal Heterogeneity (TH)20.3348.867.8448.09
Δ Accuracy↓ 25.67%–↓ 17.04%–

We further extend this experiment by varying the Dirichlet concentration α\alpha to examine FedAvg under different levels of temporal heterogeneity. As shown in the below, stronger temporal heterogeneity generally reduces accuracy.

Table 2. Effect of workload heterogeneity on FedAvg. Smaller α\alpha induces greater imbalance in client task workloads. Acc: accuracy (%, ↑); F: forgetting (%, ↓). Single run, seed 0.

BenchmarkDirichlet αTask-IL Acc ↑Task-IL F ↓Class-IL Acc ↑Class-IL F ↓
CIFAR-100.773.4528.5923.7073.81
0.568.0233.5824.6784.42
0.371.7816.3321.0867.90
CIFAR-1000.740.3742.5213.7958.46
0.540.5040.6312.2160.06
0.336.8637.139.7054.58
EMNIST0.744.1552.5113.6194.85
0.543.9853.6415.7791.62
0.340.2364.5312.8194.19
TinyImageNet0.723.8147.499.5549.09
0.520.3348.867.8448.09
0.319.1545.537.8637.14
AuthorsOpenReview ↗

Response to Reviewer 9FZn (part 2)

Comment 2:

"It seems that the temporal heterogeneity is close to the asynchronous FCL setting [R1], The authors should consider about this more carefully to make a better positioning to the current paper scope of works."

Response:

We thank the reviewer for bringing up this important question for better positioning the scope of work and pointing us to "Asynchronous Federated Continual Learning (AFCL) [Shenaj et al., 2023]". Although both AFCL and our temporal heterogeneity (TH) setting involve asynchronous aggregation and clients training different tasks at the same time, the asynchrony arises from fundamentally different sources, and the two settings therefore pose different problems.

In AFCL, each client receives its tasks at times and in an order set by its own circumstances, for example when its data become available or when the device can participate. The task schedule itself differs across clients and is independent of the learning problem. FedSpace accordingly focuses on keeping the global model stable when client updates arrive out of step with one another.

In TH, all clients follow the same task sequence, but each device can only process a limited amount of data between two communication rounds, due to its compute budget and the cost of communicating with the server. A task therefore occupies a client for as many rounds as its local workload requires. Because local data workload differ, clients spend different numbers of rounds on the same task and gradually fall out of step, even though they start together. At any given round, the server therefore receives updates from clients at different stages of the task sequence, and the clients with the most data on a task are still sending updates for it after most others have moved on. The asynchrony in TH thus comes from how much each task asks of a resource-constrained client, instead of from when the task arrives.

In the revision, we cite AFCL and add a detailed discussion of related work on asynchronous federated continual learning in Appendix A.1.1. We further compare TARDIS with FedSpace (proposed by Shenaj et al. in AFCL), when applied under temporal heterogeneity across all benchmarks.

Table R1. Comparison with FedSpace (Shenaj et al., CVPRW 2023), the method proposed for asynchronous FCL, when applied under temporal heterogeneity. Mean ± std over three random seeds. Acc: accuracy (%, ↑); F: forgetting (%, ↓). Bold indicates the better accuracy.

BenchmarkMethodTask-IL Acc ↑Task-IL F ↓Class-IL Acc ↑Class-IL F ↓
Split-CIFAR-10FedSpace68.23 ± 0.957.98 ± 1.0029.35 ± 2.5031.67 ± 0.63
TARDIS93.79 ± 0.280.46 ± 0.2065.61 ± 0.5613.52 ± 0.87
Split-CIFAR-100FedSpace35.11 ± 1.5649.40 ± 1.4514.01 ± 1.1260.44 ± 2.19
TARDIS76.06 ± 0.164.00 ± 0.3038.83 ± 0.5410.91 ± 0.48
Split-EMNISTFedSpace54.20 ± 1.7434.39 ± 2.7620.01 ± 0.7757.74 ± 1.60
TARDIS97.96 ± 0.010.32 ± 0.0585.62 ± 0.635.61 ± 1.22
Split-TinyImageNetFedSpace15.76 ± 0.1556.63 ± 0.207.88 ± 0.1666.26 ± 0.33
TARDIS42.68 ± 0.706.41 ± 0.5518.76 ± 1.105.13 ± 1.24

As shown in Table above, TARDIS outperforms FedSpace on all four benchmarks under both protocols, with substantially lower forgetting. A per-task inspection shows that FedSpace learns the most recent task well (e.g., Task-IL is 98.6% on the last Split-EMNIST task, 67.40% on the last Split-TinyImageNet) but drops to chance level on all earlier tasks. This follows from its design assumptions. FedSpace preserves old classes through prototypes aggregated from the classes that clients are currently training, which is effective in AFCL, where random task orders keep every class in circulation. Under TH, however, all clients eventually move past a task, after which its classes are no longer covered by the global prototypes and their classifier receives no supervision. TARDIS does not depend on old classes remaining active: their representations are tied to fixed ETF anchors shared by all clients and preserved by feature distillation, so knowledge of a finished task is retained even after no client trains it.

AuthorsOpenReview ↗

Response to Reviewer 9FZn (part 3)

Comment 3:

"The ETF seems to be closely related to the current series of gradient projections in Continual Learning, especially the series of works in FCL, e.g., FOT [R2]. This part need a careful discussion to prove the novelty of ETF contribution. Furthermore, the discussion of model inversion seems to be incremental and disconnected from the whole storyline of ETF geometry. It should be noted that the current literature of applying model inversion in FCL has been considered, e.g., LANDER [R3] and TARGER [R4]."

Response:

We thank the reviewer for highlighting these relevant studies. We provide a careful discussion as below to clarify their relationship to TARDIS below. We also added this discussion to the revised manuscript in Appendix A.1.3.

ETF anchoring vs. gradient projection.

Gradient-projection methods such as GPM [1] and the FCL study FOT [2] protect old tasks by restricting how parameters may change: FOT extracts a per-layer subspace of old-task activations after each task and projects the aggregated update of the next task to be orthogonal to it. ETF anchoring instead fixes where class features should be: each class is assigned a fixed vertex shared by all clients, and features are aligned to it, without restricting the direction of parameter updates. They are differ in three respects. First, FOT protected subspace grows with every task and progressively reduces the space available for new tasks, whereas adding classes to the ETF only adds target directions and leaves the parameter space unconstrained. Second, the orthogonality in TARDIS lies in the class-target space, not in the parameter space. Third, which is most relevant to our setting, FOT relies on synchronized task boundaries. In this case, the subspace is built once all clients have finished a task, and the projection is applied to the aggregated update under the assumption that all clients train the same new task. While under temporal heterogeneity, however, some clients are still training task tt while others have moved on, so removing the task tt directions from the aggregated update would also discard the progress of the clients still learning task tt. ETF alignment acts per sample on each client’s own classes and needs no shared task boundary, because consistency across clients comes from the shared anchors themselves. The two mechanisms are complementary rather than competing, and combining them is a possible direction for future work.

Model inversion and its role in TARDIS.

We acknowledge that model inversion and data-free knowledge transfer have been investigated in LANDER [3] and TARGET [4]. We do not claim novelty in model inversion itself. Instead, TARDIS$_{DF}$ is an optional extension for data-sensitive or memory-constrained environments where storing historical exemplars is undesirable. Synthetic samples allow previously learned classes to continue participating in ETF alignment and representation distillation without retaining raw replay data. The core TARDIS framework remains independent of this extension, as demonstrated by our buffer-based experiments.

We will revise the manuscript to state this explicitly in the contributions and present the variant in a separate extension subsection, so that it is not read as a core part of the method. We will also expand the related-work discussion in Appendix A.1.3 to clarify these distinctions and better separate the central contribution of temporal geometric anchoring from the optional data-free implementation.

References

[1] Saha, Gobinda, Isha Garg, and Kaushik Roy. "Gradient projection memory for continual learning." arXiv preprint arXiv:2103.09762 (2021).

[2] Bakman, Yavuz Faruk, et al. "Federated orthogonal training: Mitigating global catastrophic forgetting in continual federated learning." International Conference on Learning Representations. Vol. 2024. 2024.

[3] Tran, Minh-Tuan, et al. "Text-enhanced data-free approach for federated class-incremental learning." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[4] Zhang, Jie, et al. "Target: Federated class-continual learning via exemplar-free distillation." 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023.

Reviewer_SWvQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper mainly addresses the temporal heterogeneity problem in Federated Continual Learning (FCL), which refers to clients progressing through tasks at different rates, leading to aggregation bias and global knowledge drift, where old knowledge is progressively forgotten. To tackle this issue, the authors proposed the TARDIS method. The core idea is to progressively construct a global Equiangular Tight Frame (ETF) as the knowledge space expands, and use the shared geometric anchors it provides to align the representations of different clients, thereby maintaining cross-client representational consistency. Through extensive experiments on four continual-learning benchmarks, the study concluded that under both Task-IL and Class-IL settings, TARDIS consistently outperformed representative baseline methods, with particularly significant advantages in global-class prediction and knowledge retention.

Critical strengths
  1. The solved problems have practical significance. This paper focuses on the temporal heterogeneity issue in Federated Continual Learning (FCL), pointing out that heterogeneous task workloads cause clients to progress at different rates, thereby leading to aggregation bias and global knowledge drift. This problem is widespread in real scenarios such as edge computing and mobile devices, and has clear research value and practical significance.
  2. This paper innovatively introduces the Equiangular Tight Frame (ETF) into Federated Continual Learning (FCL). By dynamically constructing a global ETF as a shared geometric anchor, cross-client representation alignment is achieved under temporal decoupling. This cross-domain transfer provides a new geometric perspective for maintaining representational consistency in FCL.
  3. The paper also proposes buffer-based TARDIS and a data-free variant, TARDIS_DF, covering both settings where original historical samples can and cannot be retained. The former usually has better knowledge retention, while the latter remains competitive without real replay data, making it more suitable for privacy-sensitive scenarios. The data-free version indicates that the core ETF geometric anchoring mechanism does not rely on real replay data, and this dual-version design expands the application scenarios of the method.
  4. The experimental design is rigorous, and the results are reliable. The experiments are conducted on four continual-learning benchmarks and repeated with three random seeds, reporting the mean and standard deviation. This enhances the credibility and stability of the results, indicating that the performance advantage of the proposed method is not accidental.
Critical weaknesses
  1. TARDIS_DF relies on inverting the frozen local model to train the generator to synthesize old-class samples. However, the frozen model itself may have already experienced representation drift or forgetting for the old classes. The generator inherits these deviations, and the synthesized samples are then used to align old-class representations to their ETF anchors, which may continuously reinforce the deviations in the wrong direction. Although the final experimental results indicate that "the ETF geometric anchoring mechanism does not rely on real replay data", this does not allow us to ignore the problems in the generator itself.
  2. The ablation study on temporal heterogeneity is only conducted on two benchmarks (Split-CIFAR100 and Split-EMNIST). It would be more convincing to include the remaining two benchmarks, especially Split-TinyImageNet, to further verify the robustness of TARDIS under varying levels of temporal heterogeneity.
Questions for authors
  1. The paper assumes disjoint task label spaces, meaning that old classes will not reappear in new tasks. However, in real-world continual learning scenarios, old classes may be revisited, refined, or corrected, such as learning new subcategories of “cats” or correcting previous incorrect annotations. You also mentioned this issue in Section 6. Could TARDIS be extended by allowing bounded anchor drift, performing periodic re-estimation, or introducing a controlled update mechanism, so as to support the correction of old classes without violating cross-client geometric consistency?
  2. The paper uses Equiangular Tight Frames as geometric anchors, but does not discuss the relationship between the number of classes and the feature dimension. The strict equiangular simplex frame requires that the number of classes does not exceed the feature dimension plus one; when the number of classes exceeds the dimension, the strict framework does not exist and an approximate or alternative structure must be adopted. Under the current experimental setup, this condition is exactly met, so the problem is not exposed. Do the authors consider this geometric feasibility premise? In the case where the number of classes exceeds the dimension, is TARDIS still effective? Could the authors please explain this?
4 comments
AuthorsOpenReview ↗

Response to Reviewer SWvQ (part 1)

Dear Reviewer SWvQ,

We sincerely thank you for carefully reading our paper and providing thoughtful feedback. We are encouraged by the positive comments. We conducted additional experiments and answer your questions point by point below. We also updated the manuscript with changes highlighted in blue.

Comment 1:

"However, the frozen model itself may have already experienced representation drift or forgetting for the old classes. The generator inherits these deviations, and the synthesized samples are then used to align old-class representations to their ETF anchors, which may continuously reinforce the deviations in the wrong direction. Although the final experimental results indicate that "the ETF geometric anchoring mechanism does not rely on real replay data", this does not allow us to ignore the problems in the generator itself."

Response:

We thank the reviewer for raising this point. We agree that the generator inverts a teacher that may itself have drifted, so the synthesized samples can inherit part of that drift. Two properties of TARDIS (DF) prevent this inherited drift from accumulating over tasks. First, the synthesized samples are aligned to ETF vertices that the server constructs from federated class means and keeps fixed. Since no client’s teacher determines these targets, a drifted sample is pulled back toward its anchor rather than further along its drift. Second, a new generator is trained from scratch at every task boundary, so errors are not passed from one generator to the next.

Table 1. Re-inverted (default) vs. frozen inversion generator in TARDIS (DF). Frozen replays each old task from the generator inverted right after that task was learned, kept fixed afterwards; Re-inverted re-inverts the generator from the current model at every task boundary. Single run (seed 0). Acc: accuracy (%, ↑); F: forgetting (%, ↓). Bold indicates the better accuracy.

BenchmarkGeneratorTask-IL Acc ↑Task-IL F ↓Class-IL Acc ↑Class-IL F ↓
Split-CIFAR-10Frozen91.433.1144.6235.25
Re-inverted92.661.7449.958.27
Split-CIFAR-100Frozen76.705.3333.5424.27
Re-inverted78.156.0233.4223.65
Split-EMNISTFrozen97.890.2571.6616.57
Re-inverted97.670.3073.1412.38
Split-TinyImageNetFrozen36.886.5711.8010.42
Re-inverted48.9211.1719.0215.73

To examine this directly, we compare the default generator with a frozen variant that never sees later teachers: the samples of each old task are drawn from the generator inverted right after that task was learned, which is then kept fixed, while generator training is otherwise identical. As shown in the above Table, re-inverting from the current model performs on par with or better than the frozen variant on all benchmarks, with clear gains on Split-CIFAR-10 and Split-TinyImageNet. Re-inverted samples stay consistent with the network being trained, and the fixed ETF anchors correct their inherited deviation, whereas frozen generators were inverted against an earlier network and gradually lose this consistency. We included this ablation in the appendix.

Comment 2:

"Could TARDIS be extended by allowing bounded anchor drift, performing periodic re-estimation, or introducing a controlled update mechanism, so as to support the correction of old classes without violating cross-client geometric consistency?"

Response:

Thank you for raising this insightful suggestion. We agree that revisiting or refining previously learned classes is an important extension beyond our current disjoint-task formulation. TARDIS could accommodate such changes through a controlled update of its shared geometric anchors. For recurring classes, the existing anchors could be retained while their representations are updated using new observations or corrected annotations. When an existing class is subdivided, the server could progressively expand the ETF to assign distinct anchors to new subclasses, while coordinating updates to existing class representations to maintain geometric consistency. Importantly, anchor updates must be coordinated globally rather than performed independently by clients, since unconstrained changes could disrupt the simplex ETF geometry and cross-client consistency. Although these extensions are beyond the scope of our current study, they provide a promising direction for supporting evolving class semantics in federated continual learning.

AuthorsOpenReview ↗

Response to Reviewer SWvQ (part 2)

Comment 3:

"It would be more convincing to include the remaining two benchmarks, especially Split-TinyImageNet, to further verify the robustness of TARDIS under varying levels of temporal heterogeneity."

Response:

We thank the reviewer for this suggestion. We have extended the temporal heterogeneity ablation to all four benchmarks, including Split-CIFAR10 and Split-TinyImageNet. The additional results are presented in Appendix A.2.3 (Table 5) of the revised manuscript.

Table 2. Robustness of TARDIS to the degree of temporal heterogeneity on all four benchmarks. A smaller Dirichlet concentration α yields more unequal local workloads and therefore stronger temporal heterogeneity. Mean ± std over three independent runs. Acc: accuracy (%, ↑); F: forgetting (%, ↓).

BenchmarkαTask-IL Acc ↑Task-IL F ↓Class-IL Acc ↑Class-IL F ↓
Split-CIFAR-100.794.92 ± 0.210.71 ± 0.1669.85 ± 0.209.99 ± 0.23
0.593.79 ± 0.280.46 ± 0.2065.61 ± 0.5613.52 ± 0.87
0.393.25 ± 0.120.43 ± 0.1061.80 ± 0.4912.05 ± 0.74
Split-CIFAR-1000.776.61 ± 0.613.23 ± 0.0638.45 ± 0.3111.92 ± 0.54
0.576.06 ± 0.164.00 ± 0.3038.83 ± 0.5410.91 ± 0.48
0.373.66 ± 0.484.90 ± 0.2337.71 ± 0.4210.84 ± 0.23
Split-EMNIST0.797.41 ± 0.160.94 ± 0.0784.70 ± 0.229.51 ± 0.40
0.597.96 ± 0.010.32 ± 0.0585.62 ± 0.635.61 ± 1.22
0.397.80 ± 0.150.35 ± 0.0284.84 ± 0.154.92 ± 0.17
Split-TinyImageNet0.742.91 ± 0.608.06 ± 0.2919.83 ± 0.188.57 ± 0.49
0.542.68 ± 0.706.41 ± 0.5518.76 ± 1.105.13 ± 1.24
0.340.68 ± 0.537.14 ± 0.6518.38 ± 0.358.96 ± 0.74

Comment 4:

"The paper uses Equiangular Tight Frames as geometric anchors, but does not discuss the relationship between the number of classes and the feature dimension. The strict equiangular simplex frame requires that the number of classes does not exceed the feature dimension plus one; when the number of classes exceeds the dimension, the strict framework does not exist and an approximate or alternative structure must be adopted. Under the current experimental setup, this condition is exactly met, so the problem is not exposed. Do the authors consider this geometric feasibility premise? In the case where the number of classes exceeds the dimension, is TARDIS still effective? Could the authors please explain this?"

Response:

We thank the reviewer for raising this important point. An exact simplex ETF with KK vertices requires K≤d+1K \leq d+1, while our construction using KK orthonormal basis directions assumes K≤dK \leq d. This condition is satisfied in all our experiments, where the ResNet-18 feature dimension is d=512d=512 and the largest benchmark, Split-TinyImageNet, contains K=200K=200 classes.

For settings exceeding this dimensionality constraint, two extensions are possible. First, a learnable projection head could map backbone features into a higher-dimensional space (d′≥Kd' \geq K), allowing the existing ETF construction to be retained. Second, the exact ETF could be replaced with approximately well-separated hyperspherical prototypes, as explored by Mettes et al. [R1]. This possibility is further motivated by generalized neural collapse, where Jiang et al. [R2] characterize optimal classifier geometry through a maximum one-vs-rest margin when the number of classes exceeds the feature dimension.

Both approaches could retain the central principle of TARDIS: globally coordinated geometric anchors that maintain representation consistency across clients. However, their effectiveness under progressive class expansion remains to be evaluated, particularly when new anchors must be introduced without disrupting previously learned representations. We will clarify the dimensionality requirement in the revised manuscript and discuss these extensions as future directions.

[R1] Mettes, Pascal, Elise Van der Pol, and Cees Snoek. "Hyperspherical Prototype Networks." Advances in Neural Information Processing Systems, 32, 2019.

[R2] Jiang, Jiachen, et al. "Generalized Neural Collapse for a Large Number of Classes." Proceedings of the 41st International Conference on Machine Learning (ICML), 2024.

Thank you for the thoughtful responses and additional experiments. The expanded temporal-heterogeneity ablation across all four benchmarks addresses my concern about the original benchmark coverage. The new comparison for TARDIS_DF and the accompanying explanation provide useful evidence regarding the generator design and the potential effect of teacher drift. The discussion of revisited or refined classes, as well as settings where the number of classes exceeds the feature dimension, also clarifies the current scope of the method and identifies these cases as directions for future work.

Overall, I find the response helpful and am generally satisfied with the authors’ clarifications and additions. The point-by-point responses and the newly provided analyses make the paper’s assumptions, scope, and methodological choices clearer. I appreciate the authors’ effort to address the concerns raised during the review process, and I look forward to verifying these changes in the revised manuscript.In addition, I believe the score previously given was reasonable, so I am inclined to keep the current score.

Dear Reviewer,

We sincerely appreciate your thoughtful feedback and constructive engagement throughout the review process. All additional experiments, analyses, and discussions have been added to the revised manuscript, with changes highlighted in blue.

Thank you again for taking the time to read our responses and help us improve the paper.

Best,

Authors

Reviewer_uazLOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Temporal-Anchored Representation Alignment for Drift-Inhibiting Stabilization (TARDIS). It aims to address aggregation interference and catastrophic forgetting in federated continual learning (FCL) when different clients have different task progression rates. TARDIS progressively constructs a shared Equiangular Tight Frame (ETF) to align class representations across clients. It also combines feature distillation, exemplar-based or data-free replay, and task-progress-aware aggregation to reduce forgetting and interference. Experiments on four datasets show that TARDIS generally achieves higher accuracy and lower forgetting.

Critical strengths
  1. The authors propose a simple and intuitive approach for FCL, which achieves consistent improvements on four benchmark datasets.
  2. The data-free replay variant also achieves competitive performance.
Critical weaknesses
  1. The paper assumes that different clients have the same task sequence but different progression rates due to unequal local data volumes. Shenaj et al. (2023) (AFCL/FedSpace) has studied heterogeneous task orders and timing in FCL, but this work is neither cited nor discussed.
  2. TARDIS combines several components, such as progressive ETF alignment, feature distillation, replay, and weighted aggregation. However, it remains unclear which components contribute most to the improvements and how it differs from ProNC (Wang et al., ICLR 2026).
Questions for authors
  1. What are the main technical differences between TARDIS and ProNC? A comparison with a federated ProNC baseline (local ProNC + FedAvg) would be helpful.
  2. Could the authors provide some ablation which can start from FedAvg, gradually adding the ETF classifier, alignment, distillation, replay, and composite aggregation, to clarify which components drive the performance gains? Even results on a single dataset would be helpful.
4 comments
AuthorsOpenReview ↗

Response to Reviewer uazL (part 1)

Dear Reviewer uazL,

We sincerely appreciate your recognition of the motivation and significance of our work. We followed your suggestions and conducted extensive experiments with additional ablations to rigorously evaluate TARDIS. We will answer your questions point by point below. We also updated the manuscript with changes highlighted in blue.

Comment 1:

"Shenaj et al. (2023) (AFCL/FedSpace) has studied heterogeneous task orders and timing in FCL, but this work is neither cited nor discussed."

Response:

We thank the reviewers for pointing us to “Asynchronous Federated Continual Learning (AFCL) [Shenaj et al., 2023]”. Although both AFCL and our temporal heterogeneity (TH) setting involve asynchronous aggregation and clients training different tasks at the same time, the asynchrony arises from fundamentally different sources, and the two settings therefore pose different problems.

In AFCL, each client receives its tasks at times and in an order set by its own circumstances, for example when its data become available or when the device can participate. The task schedule itself differs across clients and is independent of the learning problem. FedSpace accordingly focuses on keeping the global model stable when client updates arrive out of step with one another.

In TH, all clients follow the same task sequence, but each device can only process a limited amount of data between two communication rounds, due to its compute budget and the cost of communicating with the server. A task therefore occupies a client for as many rounds as its local workload requires. Because local data workload differ, clients spend different numbers of rounds on the same task and gradually fall out of step, even though they start together. At any given round, the server therefore receives updates from clients at different stages of the task sequence, and the clients with the most data on a task are still sending updates for it after most others have moved on. The asynchrony in TH thus comes from how much each task asks of a resource-constrained client, instead of from when the task arrives.

In the revision, we cite AFCL and add a detailed discussion of related work on asynchronous federated continual learning in Appendix A.1.1. We further compare TARDIS with FedSpace (proposed by Shenaj et al. in AFCL), when applied under temporal heterogeneity across all benchmarks.

Table 1. Comparison with FedSpace (Shenaj et al., CVPRW 2023) under temporal heterogeneity. Mean ± std over three random seeds. Acc: accuracy (%, ↑); F: forgetting (%, ↓). Bold indicates the best result for each metric.

BenchmarkMethodTask-IL Acc ↑Task-IL F ↓Class-IL Acc ↑Class-IL F ↓
Split-CIFAR-10FedSpace68.23 ± 0.957.98 ± 1.0029.35 ± 2.5031.67 ± 0.63
TARDIS93.79 ± 0.280.46 ± 0.2065.61 ± 0.5613.52 ± 0.87
Split-CIFAR-100FedSpace35.11 ± 1.5649.40 ± 1.4514.01 ± 1.1260.44 ± 2.19
TARDIS76.06 ± 0.164.00 ± 0.3038.83 ± 0.5410.91 ± 0.48
Split-EMNISTFedSpace54.20 ± 1.7434.39 ± 2.7620.01 ± 0.7757.74 ± 1.60
TARDIS97.96 ± 0.010.32 ± 0.0585.62 ± 0.635.61 ± 1.22
Split-TinyImageNetFedSpace15.76 ± 0.1556.63 ± 0.207.88 ± 0.1666.26 ± 0.33
TARDIS42.68 ± 0.706.41 ± 0.5518.76 ± 1.105.13 ± 1.24

As shown in Table above, TARDIS outperforms FedSpace on all four benchmarks under both protocols, with substantially lower forgetting. A per-task inspection shows that FedSpace learns the most recent task well (e.g., Task-IL is 98.6% on the last Split-EMNIST task, 67.40% on the last Split-TinyImageNet) but drops to chance level on all earlier tasks. This follows from its design assumptions. FedSpace preserves old classes through prototypes aggregated from the classes that clients are currently training, which is effective in AFCL, where random task orders keep every class in circulation. Under TH, however, all clients eventually move past a task, after which its classes are no longer covered by the global prototypes and their classifier receives no supervision. TARDIS does not depend on old classes remaining active: their representations are tied to fixed ETF anchors shared by all clients and preserved by feature distillation, so knowledge of a finished task is retained even after no client trains it.

AuthorsOpenReview ↗

Response to Reviewer uazL (part 2)

Comment 2 & 3:

"However, it remains unclear which components contribute most to the improvements. "

"Could the authors provide some ablation which can start from FedAvg, gradually adding components to clarify which components drive the performance gains? "

Response:

We thank the reviewer for this suggestion. We have added a component-wise ablation starting from FedAvg and progressively incorporating replay, feature distillation, ETF anchoring (classifier and alignment), and composite-weighted aggregation. The experiments cover all four benchmarks under both Task-IL and Class-IL protocols. The results show that ETF anchoring provides substantial Class-IL accuracy gains, while replay and distillation improve knowledge retention. Composite aggregation provides further improvements in Class-IL accuracy. We also reported this in the revised manuscript iAppendix A.2.4 (Table 6).

Table 2. Component ablation of TARDIS on four benchmarks. Components are added cumulatively to FedAvg: ① replay buffer; ② feature distillation; ③ ETF anchoring (ETF classifier and alignment); ④ composite aggregation. Each cell reports accuracy / forgetting (%). Bold indicates the best accuracy per column.

Method①②③④Split-CIFAR-10 Task-ILSplit-CIFAR-10 Class-ILSplit-CIFAR-100 Task-ILSplit-CIFAR-100 Class-ILSplit-EMNIST Task-ILSplit-EMNIST Class-ILSplit-Tiny Image Net Task-ILSplit-Tiny Image Net Class-IL
FedAvg✗✗✗✗82.18 / 11.2039.77 / 22.9560.98 / 28.6020.69 / 43.4082.50 / 17.4329.31 / 66.1425.42 / 47.499.90 / 49.97
+ Replay✓✗✗✗90.53 / 0.9154.88 / 7.8972.37 / 4.5232.81 / 16.3697.67 / 0.4362.21 / 19.4433.89 / 24.4215.17 / 20.07
+ Distillation✓✓✗✗94.94 / 0.4452.74 / 6.4472.64 / 3.9134.98 / 15.6497.56 / 0.4364.81 / 4.5934.45 / 14.9115.64 / 18.23
+ ETF anchoring✓✓✓✗94.66 / 0.1164.80 / 6.8575.37 / 4.4238.49 / 13.6797.96 / 0.2585.13 / 8.1738.82 / 10.5818.34 / 17.67
TARDIS✓✓✓✓94.51 / 0.2266.04 / 5.2676.19 / 4.0439.26 / 11.2397.94 / 0.2886.05 / 6.2142.19 / 10.6219.10 / 17.33
AuthorsOpenReview ↗

Response to Reviewer uazL (part 3)

Comment 4:

"What are the main technical differences between TARDIS and ProNC? A comparison with a federated ProNC baseline (local ProNC + FedAvg) would be helpful. "

Response:

We thank the reviewer for this important clarification question. TARDIS adopts ProNC’s progressive ETF construction as its client-side objective. However, these two methods address different problems. As discussed in Section 2.3, ProNC considers a single learner that builds and extends its frame at well-defined task boundaries. Under temporal heterogeneity, these assumptions break down. Clients reach task boundaries at different rounds, no single client observes sufficient data to estimate the base geometry reliably, and independently built frames are mutually inconsistent, so aggregating models aligned to different frames destroys the alignment on which ProNC relies.

TARDIS therefore replaces the per-learner frame with a single global ETF that serves as a common coordinate system across clients as well as across time. Specifically, (i) the server constructs the base ETF from aggregated client class statistics (ii) the server expands the frame when the first client reaches a new task and broadcasts the updated anchors to all clients, regardless of their current task (iii) temporally aware composite aggregation. We further provide a data-free variant of TARDIS for memory-constrained settings in which historical data cannot be retained.

Following the reviewer’s suggestion, we compare TARDIS with a federated ProNC baseline (Local ProNC + FedAvg, in which each client constructs its own ETF) on all four benchmarks. TARDIS achieves higher accuracy and lower forgetting across all four benchmarks as shown below.

Table R4. Comparison with federated ProNC under temporal heterogeneity. ProNC is applied locally with FedAvg aggregation. Mean ± std over three runs. Acc: accuracy (%, ↑); F: forgetting (%, ↓). Bold indicates higher accuracy or lower forgetting.

BenchmarkMethodTask-IL Acc ↑Task-IL F ↓Class-IL Acc ↑Class-IL F ↓
Split-CIFAR-10FedAvg + ProNC90.47 ± 0.613.66 ± 0.2027.11 ± 0.8117.39 ± 0.73
TARDIS93.79 ± 0.280.46 ± 0.2065.61 ± 0.5613.52 ± 0.87
Split-CIFAR-100FedAvg + ProNC68.46 ± 0.7414.43 ± 0.4722.64 ± 0.4836.99 ± 1.38
TARDIS76.06 ± 0.164.00 ± 0.3038.83 ± 0.5410.91 ± 0.48
Split-EMNISTFedAvg + ProNC95.94 ± 0.191.26 ± 0.1861.92 ± 0.4932.11 ± 0.25
TARDIS97.96 ± 0.010.32 ± 0.0585.62 ± 0.635.61 ± 1.22
Split-TinyImageNetFedAvg + ProNC31.40 ± 0.4630.00 ± 0.6611.41 ± 0.3037.30 ± 1.08
TARDIS42.68 ± 0.706.41 ± 0.5518.76 ± 1.105.13 ± 1.24
AuthorsOpenReview ↗

Follow-up to the Reviewer

Dear Reviewer,

Thank you very much for your thoughtful review and valuable feedback. Following your suggestions, we have conducted extensive experiments, added additional discussions, and uploaded the revised manuscript with all changes highlighted in blue. We are happy to address any further concerns.

If our revisions have adequately addressed your concerns, we would be grateful if you would consider updating your score.

Thank you again for your time and consideration.

Best,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Summary

Dear AC,

Thank you for your engagement so far in the review process. We have updated the manuscript with changes highlighted in blue. Below, we summarize the key concerns raised and our corresponding responses:

  • How does TARDIS differ from asynchronous FCL? (Reviewers uazL, 9FZn)

    We clarify the differences from asynchronous FCL and add FedSpace comparisons in Section 2 and Appendices A.1.1, A.2.1.

  • What distinguishes TARDIS from ProNC and gradient-projection methods? (Reviewer 9FZn)

    We discuss the differences from FOT and add a federated ProNC baseline in Appendices A.1.2, A.2.6.

  • How does each TARDIS component contribute to performance? (Reviewer 9FZn)

    We add component-wise ablations across four benchmarks in Appendix A.2.4.

  • How reliable is data-free replay, and how does it differ from existing inversion methods? (Reviewers SWvQ, 9FZn)

    We clarify the optional data-free extension and add teacher-drift experiments in Section 4 and Appendix A.1.2, A.2.2.

  • How does temporal heterogeneity affect model performance? (Reviewer 9FZn)

    We add synchronized-versus-heterogeneous FedAvg comparisons in Appendix A.2.5 and clarify the relationship between task misalignment and learning states in Sections 1, 3.

  • How robust is TARDIS to different degrees and types of heterogeneity? (Reviewers SWvQ, 9FZn)

    We extend the heterogeneity ablation to all four benchmarks in Appendix A.2.3 and discuss task-order mismatch in Section 6.

  • How is the global ETF constructed, and what are its feasibility and storage requirements? (Reviewers SWvQ, 9FZn)

    We clarify ETF recovery and expansion in Section 4, geometric feasibility in Appendix A.1.3, and storage and computational overhead in Appendix A.2.11.

  • Are the motivation, technical definitions, and experimental protocols sufficiently clear? (Reviewer 9FZn)

    We revise Sections 1-4 and expand Appendix A.3.1 to clarify the motivation, model definitions, temporal heterogeneity setup, Task-IL/Class-IL protocols, and evaluation metrics.

  • Can TARDIS handle task-order mismatch and revisited or overlapping classes? (Reviewers SWvQ, 9FZn)

    We clarify the current class-incremental assumptions and discuss these extensions in Sections 3, 6.

Once again, we are truly grateful for the reviewers' valuable feedback and are happy to address any further concerns or questions.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.