FedReCall: Recalling Client-Specific Directions in Federated LoRA Fine-tuning

Low-rank adaptation (LoRA) enables communication-efficient federated fine-tuning of large language models, but standard FedLoRA averages the factors $A$ and $B$ separately even though the model update is determined by their product $\Delta W=sBA$. Under task heterogeneity, factor-wise aggregation can further dilute directions that are locally useful but not strongly shared across clients; we call this phenomenon directional dilution. We analyze this effect through the singular directions of client LoRA updates and find that low-strength directions are most vulnerable to dilution. Their repeated owner-specific recovery suggests client-specific adaptation signals rather than random noise. We propose FedReCall, a client-private plug-in that captures observable dilution gaps, selects reliable diluted directions into a private frozen cache, and recalls them through a low-rank bypass guarded by a single-batch loss probe when the client next participates. As a plug-in, FedReCall can be seamlessly integrated into existing FedLoRA methods without changing their aggregation rules, communication payloads, or trainable-parameter sets, consistently improving personalized adaptation and improving global evaluation metrics on most aggregators.