TACO: Text-Anchored Classification with Structural Covariance Transfer for One-Shot Federated Learning

Conventional federated learning (FL) requires multiple communication rounds and incurs high overhead, while One-shot FL (OFL) mitigates this burden by restricting interaction to a single round. However, most existing OFL methods either require clients to upload prototypes or mandate homogeneous client architectures. The former is prone to privacy leakage risks, while the latter greatly limits the scalability. The statistics-based OFL method constructs a classifier using per-class sufficient statistics derived from features extracted by a frozen pre-trained encoder, achieving partition invariance, that is, identical accuracy regardless of data distribution across clients. Nevertheless, current statistics-based OFL methods estimate each class independently, so that when per-class samples are scarce, covariances are regularized by shrinking toward an isotropic target that discards the covariance structure shared across classes. Since all classes pass through the same frozen pre-trained encoder, their covariances share common variation patterns, a source of structural information that isotropic shrinkage entirely wastes, leading to degraded performance as the number of classes grows. To address these limitations, we propose TACO (Text-Anchored Classification with Structural Covariance Transfer), a training-free OFL framework built on three components. First, Structural Covariance Transfer regularizes each class covariance toward the average of all other classes rather than a scaled identity, directly exploiting the shared variation patterns of the frozen pre-trained encoder, and its gains grow as per-class estimation becomes harder (up to $+5.57\%$ on Tiny-ImageNet). Second, dual-space QDA/LDA fusion uses text anchors generated by CLIP to construct a complementary text-space classifier and fuses it with the visual-space classifier, integrating discriminative information unavailable from either space alone. Third, confidence-weighted test-time augmentation addresses the bottleneck that training-side parameter estimation has limited room for improvement while per-sample feature noise at test time remains, providing orthogonal gains via multi-view confidence-weighted aggregation. Extensive experiments demonstrate that TACO outperforms or matches all SOTA OFL methods. The code of this work will be released upon acceptance.