DualSpec: Exposing Disguised Federated Backdoor via Cross-Client Structure and Trajectory Residuals

Federated learning (FL) is vulnerable to backdoor attacks that manipulate client updates to implant backdoor features into the global model. Existing server-side defenses often rely on observable discrepancies in update magnitude, coordinate-wise statistics, or on comparisons against a fixed reference model in isolation, which may become unreliable against disguised malicious updates or adaptive trigger attacks. Other approaches rely on auxiliary or trusted datasets, introducing additional data requirements. We propose $\text{DualSpec}$, an FL backdoor defense that combines two complementary detection signals, without requiring auxiliary data. The $\text{Model-wise Cross-client}$ branch applies singular value decomposition (SVD) to per-layer normalized client updates to obtain a shared spectral space across clients, where suspicious groups are identified based on their directional cohesion and isolation. The $\text{Layer-wise Trajectory-residual}$ branch instead applies SVD to the global model's own recent update to construct a per-layer reference subspace, and compares each client's update against it using residual energy to identify updates that deviate from the recent trajectory. Both branches derive from spectral decomposition of model updates, which exposes their underlying structure beneath surface-level disguise. Finally, malicious updates detected from both branches are combined before aggregation to capture backdoor behavior at both the group and the individual level. Experiments across different datasets and IID/non-IID settings show that DualSpec consistently suppresses a diverse set of state-of-the-art backdoor attacks while preserving main task accuracy, and generalizes beyond convolutional architectures.