DoMA: Geometry-Aware Aggregation for Stable Continual Federated Fine-Tuning on the Heterogeneous Edge

Federated fine-tuning (FFT) adapts large models to decentralized, privacy-sensitive data through parameter-efficient updates. Existing methods predominantly consider heterogeneous client updates in Euclidean space, implicitly treating independently optimized directions as linearly compatible. Under non-IID and continually evolving data, however, these directions encode distinct client- and task-specific semantics; Euclidean operation can attenuate, cancel, or distort informative components. Repeated across adaptation stages, this cross-client directional mismatch accumulates into global trajectory drift, forcing a stability–plasticity trade-off between acquiring new knowledge and preserving earlier capabilities. To address this, we propose DoMA (Drift-Controlled On-Manifold Aggregation), which jointly addresses these spatial and temporal sources of interference. DoMA decouples update magnitude from direction, computes intrinsic Riemannian Fr\'echet means on a product-sphere manifold, and adaptively regulates inter-stage displacement. Extensive experiments across task-isolated language adaptation, sequential task-category learning, and multimodal continual transfer show that DoMA consistently improves both adaptation and retention, outperforming existing FFT methods by up to $8.72\%$ in task performance and $8.38\%$ in new-task plasticity, while lowering average forgetting by up to $5.99\%$ compared to benchmarks. Code is available at https://anonymous.4open.science/r/DoMA-453D/.