Fed3DLoRA: Cross-Layer Tensor Consensus for Heterogeneous Federated Fine-Tuning

Federated fine-tuning with low-rank adaptation (LoRA) has to accommodate clients whose resources support adapters of different ranks. Existing methods combine such adapters one weight matrix at a time, although every layer of a transformer reads from the same residual stream, which couples the updates of a projection across depth. Fed3DLoRA exploits this cross-layer structure: the server stacks the updates along depth into a tensor, builds a low-dimensional coordinate system shared across layers and clients, and expresses every client’s update, whatever its rank, as a compact core in it. This makes a client-specific, direction-dependent consensus tractable: cores are combined under metrics derived from the updates themselves, while everything outside the shared space is kept as the plain mean before the rank-constrained write-back. Aggregation runs entirely on the server and leaves client computation and per-round communication unchanged. We prove that it is invariant to how clients factorise their adapters, reduces to plain averaging when the client metrics coincide, and admits a convergence bound that accounts for the rank truncation. Extensive experiments on diverse LLM fine-tuning tasksExtensive experiments on diverse LLM fine-tuning tasks show that Fed3DLoRA outperforms existing methods in accuracy under joint rank and data heterogeneity, and that cross-layer consensus improves on its layer-wise counterpart in every setting tested show that Fed3DLoRA outperforms existing methods in accuracy under joint rank and data heterogeneity, and that cross-layer consensus improves on its layer-wise counterpart in every setting tested.