- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies continual federated fine-tuning of large language models under heterogeneous client data and resources. The authors propose DoMA, a geometry-aware aggregation framework designed to reduce cross-client update interference and accumulated model drift across adaptation stages. Specifically, DoMA separates update magnitude and direction, aggregates client directions using a Riemannian Fréchet mean on a product-sphere manifold, and introduces stage-adaptive drift control to balance knowledge acquisition and retention. Experiments across language understanding, commonsense reasoning, sequential task learning, and multimodal adaptation show that DoMA generally outperforms the evaluated baselines in task performance, plasticity, and forgetting.
Critical strengths
-
The paper addresses a meaningful challenge in continual federated fine-tuning by considering both cross-client update interference and accumulated drift across adaptation stages. The proposed framework combines geometry-aware aggregation with adaptive drift control, providing a coherent approach to balancing knowledge acquisition and retention under heterogeneous client data.
-
The experiments cover multiple model scales, task settings, and heterogeneous client configurations, including sequential task learning and multimodal adaptation. DoMA shows consistent improvements over the evaluated baselines in both adaptation performance and knowledge retention. The paper also includes controlled ablation studies to examine the contributions of manifold aggregation and drift control, providing useful empirical support for the proposed framework.
Critical weaknesses
-
DoMA performs client-side training using low-rank adaptation and aggregates client directional matrices through a Riemannian Fréchet mean. However, the resulting global direction may not remain representable as a low-rank update, even when individual client updates are low-rank. The paper does not clearly explain how the aggregated global direction is represented and distributed to clients while maintaining parameter efficiency. This raises concerns about the actual communication overhead, particularly when the full directional matrix needs to be broadcast.
-
The paper evaluates sequential task-category learning across different model scales and reports substantial performance improvements. However, Appendix B.1 shows that the Qwen3-8B experiments use only 24 training examples per client, with evaluation limited to at most 32 examples per task. With such small evaluation sets, even a few changed predictions could noticeably affect the reported accuracy. Although the authors report means and standard deviations over five task orders, varying task orders does not fully account for training randomness or evaluation uncertainty. Therefore, it remains unclear whether the reported improvements would generalize to larger-scale continual learning settings.
Questions for authors
-
Could the authors further validate the reported improvements in sequential task learning using larger training and evaluation sets?
-
Could the authors further explain how the global directional matrix obtained after manifold aggregation maintains a parameter-efficient representation and is transmitted to clients? If broadcasting the full directional matrix is required, could the authors provide a complete communication cost analysis covering both uplink and downlink traffic?
Thanks to authors’ rebuttal. I checked the rebuttal and other reviewers’ comments. I keep my original score.