acceptodds
Under review as a conference paper at ICLR 2027

DoMA: Geometry-Aware Aggregation for Stable Continual Federated Fine-Tuning on the Heterogeneous Edge

Abstract

Federated fine-tuning (FFT) adapts large models to decentralized, privacy-sensitive data through parameter-efficient updates. Existing methods predominantly consider heterogeneous client updates in Euclidean space, implicitly treating independently optimized directions as linearly compatible. Under non-IID and continually evolving data, however, these directions encode distinct client- and task-specific semantics; Euclidean operation can attenuate, cancel, or distort informative components. Repeated across adaptation stages, this cross-client directional mismatch accumulates into global trajectory drift, forcing a stability–plasticity trade-off between acquiring new knowledge and preserving earlier capabilities. To address this, we propose DoMA (Drift-Controlled On-Manifold Aggregation), which jointly addresses these spatial and temporal sources of interference. DoMA decouples update magnitude from direction, computes intrinsic Riemannian Fr\'echet means on a product-sphere manifold, and adaptively regulates inter-stage displacement. Extensive experiments across task-isolated language adaptation, sequential task-category learning, and multimodal continual transfer show that DoMA consistently improves both adaptation and retention, outperforming existing FFT methods by up to in task performance and in new-task plasticity, while lowering average forgetting by up to compared to benchmarks. Code is available at https://anonymous.4open.science/r/DoMA-453D/.

Primary area:
optimization
Keywords:
Federated Fine-Tuning, Riemannian Optimization, Catastrophic Forgetting, Geometric-aware Fine-Tuning, Continual Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_jY9nOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies continual federated fine-tuning of large language models under heterogeneous client data and resources. The authors propose DoMA, a geometry-aware aggregation framework designed to reduce cross-client update interference and accumulated model drift across adaptation stages. Specifically, DoMA separates update magnitude and direction, aggregates client directions using a Riemannian Fréchet mean on a product-sphere manifold, and introduces stage-adaptive drift control to balance knowledge acquisition and retention. Experiments across language understanding, commonsense reasoning, sequential task learning, and multimodal adaptation show that DoMA generally outperforms the evaluated baselines in task performance, plasticity, and forgetting.

Critical strengths
  1. The paper addresses a meaningful challenge in continual federated fine-tuning by considering both cross-client update interference and accumulated drift across adaptation stages. The proposed framework combines geometry-aware aggregation with adaptive drift control, providing a coherent approach to balancing knowledge acquisition and retention under heterogeneous client data.

  2. The experiments cover multiple model scales, task settings, and heterogeneous client configurations, including sequential task learning and multimodal adaptation. DoMA shows consistent improvements over the evaluated baselines in both adaptation performance and knowledge retention. The paper also includes controlled ablation studies to examine the contributions of manifold aggregation and drift control, providing useful empirical support for the proposed framework.

Critical weaknesses
  1. DoMA performs client-side training using low-rank adaptation and aggregates client directional matrices through a Riemannian Fréchet mean. However, the resulting global direction may not remain representable as a low-rank update, even when individual client updates are low-rank. The paper does not clearly explain how the aggregated global direction is represented and distributed to clients while maintaining parameter efficiency. This raises concerns about the actual communication overhead, particularly when the full directional matrix needs to be broadcast.

  2. The paper evaluates sequential task-category learning across different model scales and reports substantial performance improvements. However, Appendix B.1 shows that the Qwen3-8B experiments use only 24 training examples per client, with evaluation limited to at most 32 examples per task. With such small evaluation sets, even a few changed predictions could noticeably affect the reported accuracy. Although the authors report means and standard deviations over five task orders, varying task orders does not fully account for training randomness or evaluation uncertainty. Therefore, it remains unclear whether the reported improvements would generalize to larger-scale continual learning settings.

Questions for authors
  1. Could the authors further validate the reported improvements in sequential task learning using larger training and evaluation sets?

  2. Could the authors further explain how the global directional matrix obtained after manifold aggregation maintains a parameter-efficient representation and is transmitted to clients? If broadcasting the full directional matrix is required, could the authors provide a complete communication cost analysis covering both uplink and downlink traffic?

Reviewer_XqU4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper focuses on the problem of adapting large models to decentralized, privacy-sensitive scenarios using Federated fine-tuning (FFT). It introduces DoMA (Drift-Controlled On-Manifold Aggregation) to address cross-client directional mismatch accumulation causing global trajectory drift across adaptation stages. Extensive experiments on eight commonsense reasoning benchmarks and seven natural language understanding benchmarks demonstrate the superior performance of the proposed method compared with SOTA methods.

Critical strengths

(+) This paper is well prepared with clear theoretical support

(+) This paper conducts extensive experiments in multiple domains and demonstrates improvement with a large margin compared to SOTA methods

Critical weaknesses

(-) The presentation of this paper can be further improved, e.g., some “equation equation” typos can be found in line 249 and line 446.

Questions for authors

See weaknesses.

1 comment

Thanks to authors’ rebuttal. I checked the rebuttal and other reviewers’ comments. I keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.