On the Consistency of Federated LoRA under Secure Aggregation via Functional Sketching

Federated low-rank adaptation (LoRA) on heterogeneous devices faces two coupled aggregation inconsistencies. Independently averaging factors generally differs from averaging their products, while additive masking before local factor multiplication introduces non-canceling cross terms. We propose SAGE-LoRA, a functional-sketch framework that applies secure aggregation (SecAgg) to linear measurements of locally paired products rather than factor coordinates. Budget-aware paired blocks support heterogeneous local ranks, while fixed-width sketches decouple per-client measurement size from local rank and cohort size. An anchored right sketch identifies a cohort-level range, and a range-locked left sketch from the same surviving cohort enables fixed-rank reconstruction. Clients evaluate both sketches through factor chains before masking, without materializing dense updates. We establish exact measurement aggregation and sequential aggregate confidentiality under the universal SecAgg assumptions, and derive reconstruction and time-averaged stationarity bounds that distinguish approximation from optimization error. Across eight commonsense tasks with Qwen2.5-1.5B-Instruct and rank-8 adapters, SAGE-LoRA with SecAgg achieves 76.37% mean accuracy, within 0.66 percentage points of unprotected FlexLoRA and over 5 points above unprotected FLoRA and FSLoRA. Under the evaluated representations and precisions, logical communication is 67.3% and 60.1% lower than FLoRA and FSLoRA, respectively, including second-stage projection traffic. The paired mean accuracy change from plaintext is only 0.10 percentage points. On five GLUE tasks, plaintext SAGE-LoRA improves the macro score over FLoRA, FSLoRA, and FlexLoRA by 0.42–2.40 points. These results support SAGE-LoRA as a practical exploration of SecAgg for federated LoRA that reconciles product consistency, aggregate confidentiality, and compact communication.