FedDOSA: One-Shot Federated LLM Tuning via Sparse Orthogonal Subspace Allocation

One-shot federated learning makes it affordable to adapt large language models across clients that cannot share data, since each client uploads its update only once. The single round, however, removes any chance to repair conflicts between locally trained updates, so whatever interference the aggregation introduces is permanent. We cast one-shot federated adaptation as a design problem over the subspace each client may train in, and show that its excess risk separates into capture, noise, and interference terms. The interference term acts through the layer output rather than the weights: updates that are orthogonal as matrices can still conflict in the function the layer computes. We prove that allocating clients disjoint output indices in a shared orthonormal basis makes their layer outputs orthogonal for every input, which removes the interference term exactly for a squared-loss linear layer, and that the overlap left by coordinate-level allocation is governed by how nearly the basis diagonalizes the activation covariance. These bounds make no distributional assumption on the learned coefficients: their randomness is each client's own index draw. We instantiate the design as FedDOSA, which trains sparse coefficients in a two-dimensional discrete cosine basis, and find on language-understanding and mathematical-reasoning benchmarks that its coordinate-level variant achieves a favourable communication–performance trade-off against existing one-shot methods.