acceptodds
Under review as a conference paper at ICLR 2027

TACO: Text-Anchored Classification with Structural Covariance Transfer for One-Shot Federated Learning

Abstract

Conventional federated learning (FL) requires multiple communication rounds and incurs high overhead, while One-shot FL (OFL) mitigates this burden by restricting interaction to a single round. However, most existing OFL methods either require clients to upload prototypes or mandate homogeneous client architectures. The former is prone to privacy leakage risks, while the latter greatly limits the scalability. The statistics-based OFL method constructs a classifier using per-class sufficient statistics derived from features extracted by a frozen pre-trained encoder, achieving partition invariance, that is, identical accuracy regardless of data distribution across clients. Nevertheless, current statistics-based OFL methods estimate each class independently, so that when per-class samples are scarce, covariances are regularized by shrinking toward an isotropic target that discards the covariance structure shared across classes. Since all classes pass through the same frozen pre-trained encoder, their covariances share common variation patterns, a source of structural information that isotropic shrinkage entirely wastes, leading to degraded performance as the number of classes grows. To address these limitations, we propose TACO (Text-Anchored Classification with Structural Covariance Transfer), a training-free OFL framework built on three components. First, Structural Covariance Transfer regularizes each class covariance toward the average of all other classes rather than a scaled identity, directly exploiting the shared variation patterns of the frozen pre-trained encoder, and its gains grow as per-class estimation becomes harder (up to on Tiny-ImageNet). Second, dual-space QDA/LDA fusion uses text anchors generated by CLIP to construct a complementary text-space classifier and fuses it with the visual-space classifier, integrating discriminative information unavailable from either space alone. Third, confidence-weighted test-time augmentation addresses the bottleneck that training-side parameter estimation has limited room for improvement while per-sample feature noise at test time remains, providing orthogonal gains via multi-view confidence-weighted aggregation. Extensive experiments demonstrate that TACO outperforms or matches all SOTA OFL methods. The code of this work will be released upon acceptance.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federated Learning, One-Shot Federated Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uFxaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors proposed a training-free OFL approach, TACO, where every client uses a frozen pretrained encoder to encode its own images, projecting the features onto the C class anchors (text embeddings of the class using CLIP). Each client then sends the sample count (N_c), feature sum (A_c), and scatter matrix (S_c) for both the visual feature space and projected text space back to the server. The server then aggregates these (sum) and compute the mean, covariance, and prior for each class. After OAS shrinkage, each class covariance is regularized towards the average of other classes' covariances, with a sample size dependent mixing weight (SCT in the paper). Authors also proposed 1) dual-space fusion, where QDA classifiers are built on both visual space and the projected text space, combining their scores, (if num of classes <= 20, a single visual-space LDA classifier is built instead) and 2) CTTA, where predictions are aggregated across augmented views of the test sample, weighted by prediction confidence.

Critical strengths
  1. paper is well-written and clear. OFL is a timely topic.

  2. application of SCT to OFL is novel and give substantial gains over standard OAS shrinkage

  3. evaluation is sufficiently thorough with many ablations and practical cost analyses, along with hyperparameter sensitivity

Critical weaknesses
  1. the main benefit from TACO seems to be from CTTA. Comparing the baselines in Table 1 and the ablation Table 2, TACO performs worse than some baselines without CTTA on CIFAR-10 and Tiny-ImageNet and only marginally better for CIFAR-100. would running CTTA on the baselines close the this performance gap?

  2. Line 366, the authors pair ResNet-18 with CLIP. These two representations are unaligned and taking their inner product (Eq 3) is hence an arbitrary linear projection.

  3. Line 52, the authors mention that uploading prototypes "poses a substantial risk of privacy leakage". However, TACO also uploads the prototypes (A_c / N_c), along with S_c, posing a higher risk of privacy leakage.

I am happy to raise my score if these concerns are addressed.

Questions for authors
  1. SCT is just a minor variation of regularized discriminant analysis (RDA) apart from considering the covariance of the class itself, right?

  2. Details on how the authors run the baselines is unclear. For instance, how many epochs was FAFI run on in Table 1? I assume is more than 50 epochs since the number in Table 17 for 50 epochs is very different from the number in Table 1.

  3. Line 388 and 1149, the authors mentioned that all hyperparameters are selected on CIFAR-100 but never detailed how this was selected? was there a separate validation set?

Post Rebuttal: the authors have addressed most of my concerns and i have raised my score accordingly. I would appreciate it if the authors could also report the baselines with CTTA in the other comparison tables (e.g. Tables 15,18,19,20). This would help readers separate the contributions of SCT and CTTA and underline where and why TACO falls behind existing baselines.

11 comments
AuthorsOpenReview ↗

(W1) Thank you for the careful and constructive review

W1: "the main benefit from TACO seems to be from CTTA. Comparing the baselines in Table 1 and the ablation Table 2, TACO performs worse than some baselines without CTTA on CIFAR-10 and Tiny-ImageNet and only marginally better for CIFAR-100. would running CTTA on the baselines close the this performance gap?"

Response to W1: While CTTA can certainly improve test accuracy, it is not the primary factor determining TACO's performance advantage. Instead, TACO's core improvement stems from SCT. Specifically, SCT directly improves the server-side classifier's estimation of class covariances, whereas CTTA merely performs a weighted average of predictions across multiple views of the same image during the testing phase. Thus, as a general test-time augmentation strategy, CTTA can be applied to any classifier. This difference in mechanism is also reflected in the ablation experiments in Table 2, where the introduction of SCT boosts accuracy by 3.99% and 5.14% on CIFAR-100 and Tiny-ImageNet, respectively, while stacking CTTA on top of this yields only an additional 0.85% and 1.62%.

To verify whether CTTA can bridge the gap between the baselines and TACO, we incorporated CTTA into the six partition-invariant baselines (AFL, GH-LDA, GH-QDA, GH-Proto-Hyper, DeepAFL, and SAFLe) published between 2025 and 2026, as listed in Table 1. These six methods include the strongest partition-invariant baseline for each backbone network and each dataset listed in Table 1. In terms of experimental setup, the hyperparameters for each baseline were consistent with those in Table 1, and the same CTTA as TACO was used. We then compared these baselines with TACO-Tune from Table 1 on two backbone networks, ResNet-18 and CLIP ViT-B/32, and the specific results are summarized in the table below.

Method (Test Accuracy %)CIFAR-10CIFAR-100Tiny-ImageNet
ResNet-18
AFL (CVPR'25) + CTTA83.4359.7255.95
GH-LDA (ICLR'26) + CTTA86.8565.3062.01
GH-QDA (ICLR'26) + CTTA84.8368.2860.65
GH-Proto-Hyper (ICLR'26) + CTTA86.8067.4163.70
DeepAFL (ICLR'26) + CTTA87.1268.0063.59
SAFLe (CVPR'26) + CTTA87.4968.7464.04
TACO-Tune (Table 1)89.2573.9367.62
CLIP ViT-B/32
AFL (CVPR'25) + CTTA92.4271.3265.82
GH-LDA (ICLR'26) + CTTA91.6370.9765.07
GH-QDA (ICLR'26) + CTTA90.0373.6466.35
GH-Proto-Hyper (ICLR'26) + CTTA92.0874.8470.25
DeepAFL (ICLR'26) + CTTA92.6576.4070.30
SAFLe (CVPR'26) + CTTA92.5575.1369.69
TACO-Tune (Table 1)93.3978.6571.48

The experimental results show that even when CTTA is applied, the accuracy of all baselines remains lower than that of TACO-Tune. Specifically, on the ResNet-18 backbone, the strongest baseline, SAFLe, still trails TACO-Tune by 1.76% to 5.19%, while on the CLIP ViT-B/32 backbone, the strongest baseline, DeepAFL, also trails by 0.74% to 2.25%. Combined with the ablation results in Table 2, this comparison further illustrates that TACO's primary gains stem from SCT, while CTTA plays only a supplementary role.

In addition to accuracy, TACO's communication overhead is also lower than that of SAFLe and DeepAFL. As shown in Table 11, on CIFAR-100 and Tiny-ImageNet, TACO's upload volume per client is 52.26 and 116.08 MiB, respectively, while SAFLe's is approximately 114 and 119 MiB, and DeepAFL's is 139.41 and 162.47 MiB. Furthermore, since the number of samples per class at the client is often far smaller than the feature dimension, the second-order statistics for that class form a low-rank matrix. Therefore, clients can instead upload smaller low-rank factors to achieve lower communication overhead. The server can precisely reconstruct the original second-order statistics from these low-rank factors, with both the classifier and accuracy remaining unchanged. The low-rank factors are computed from the second-order statistics themselves and do not contain per-sample features. Under a Dirichlet α=0.1\alpha = 0.1 partition with 5 clients, this approach reduces the upload size per client by more than half on average on the CIFAR-100 and Tiny-ImageNet datasets, with at most 18.03 and 38.85 MiB per client, respectively. Consequently, on these two datasets, TACO achieves higher accuracy with lower communication overhead than SAFLe and DeepAFL.

Finally, we have added the complete experimental results for each baseline combined with CTTA to Table 23 in Appendix O of the revised manuscript, and Section 4.2 now states that TACO-Tune still outperforms all partition-invariant baselines equipped with CTTA. Section 4.3 now also reports the share of the total improvement contributed by each component, with SCT accounting for 74% and 76% on CIFAR-100 and Tiny-ImageNet. The low-rank upload sizes above are reported in Table 12 of Appendix G.

AuthorsOpenReview ↗

(W2) Thank you for the careful and constructive review

W2: "Line 366, the authors pair ResNet-18 with CLIP. These two representations are unaligned and taking their inner product (Eq 3) is hence an arbitrary linear projection."

Response to W2: TACO was originally designed for CLIP and utilizes CLIP's paired visual and text encoders. Since CLIP's visual and text encoders are trained to align images and text, the text branch can leverage the text embeddings of class names to provide semantic information.

However, TACO does not need to rely on CLIP's paired visual and text encoders and can be generalized to other encoders. To verify this, we included experiments with the ResNet-18 encoder in our paper. The ResNet-18 experiments adopted the full configuration of the CLIP experiments, thus retaining the text branch. TACO can be generalized to ResNet-18 because its primary gains stem from SCT, which acts solely on statistics of visual features and does not require a paired text encoder. The experiments in Table 8, which do not use the text branch, show that on ResNet-18 features, the test accuracy of SCT is 0.70% to 6.15% higher than that of conventional OAS.

For further validation, we removed the text branch from ResNet-18, retaining only the visual branch (setting αf=0\alpha_f = 0). The remaining experimental settings were identical to those for TACO-Tune in Table 1. The average results across three random seeds are shown in the table below.

Method (ResNet-18, Test Accuracy %)CIFAR-10CIFAR-100Tiny-ImageNet
SAFLe (CVPR'26) + CTTA (table in W1)87.4968.7464.04
TACO-Tune (visual and text branches)89.1573.9767.21
TACO-Tune (visual branch only)89.2573.9367.62

When only the visual branch is retained, TACO-Tune's test accuracy is comparable to the results with both branches and remains 1.76% to 5.19% higher than SAFLe, the strongest baseline combined with CTTA, indicating that TACO's performance advantage on ResNet-18 does not depend on the text branch.

In the revised manuscript, we have replaced the ResNet-18 results in Table 1 and Section 4.2 with those obtained with only the visual branch retained, and have clarified in Section 4.1 that the text branch is applicable to image-text aligned encoders such as CLIP, while the ResNet-18 experiments serve to validate that TACO can be generalized to other encoders.

AuthorsOpenReview ↗

(W3) Thank you for the careful and constructive review

W3: "Line 52, the authors mention that uploading prototypes "poses a substantial risk of privacy leakage". However, TACO also uploads the prototypes (A_c / N_c), along with S_c, posing a higher risk of privacy leakage."

Response to W3: The AcA_c and NcN_c uploads in TACO contain class prototype information. However, uploading statistics computed from local data is a common practice in one-shot federated learning methods, and the visual statistics uploaded by TACO are identical to those in GH-QDA (ICLR'26) (Table 11). Furthermore, two commonly used privacy-preserving techniques, secure aggregation and differential privacy, can be directly applied to TACO.

First, the TACO server only requires the sum of all client statistics (Eq. (4)). Secure aggregation is a cryptographic protocol under which the server can only obtain the sum of all client uploads and cannot see the upload from any individual client. Therefore, TACO can be directly combined with secure aggregation, after which the server cannot obtain the statistics of any individual client, while the classifier and accuracy remain unchanged. In contrast, FedCOF requires the class means from each client to estimate the covariance, and FALCON requires the classifier and synthetic features from each client. Consequently, the FedCOF and FALCON servers must obtain uploads from individual clients.

Second, there is a fixed upper bound on the change in TACO's statistics caused by a single sample. (ε,δ)(\varepsilon, \delta)-differential privacy requires that the probability distribution of the server's output remain virtually unchanged when any training sample is added or removed. Thus, the server can hardly determine whether a particular sample was used in training. Here, ε\varepsilon controls the magnitude of the allowed variation in the probability distribution, while δ\delta is the small probability that this variation exceeds the allowed range. Differential privacy is commonly implemented by adding Gaussian noise to the results, whose standard deviation is jointly determined by ε\varepsilon, δ\delta, and the maximum change in the results caused by a single sample. In TACO, when a sample xx of class cc is added or removed, only the statistics of class cc change: the sample count NcN_c changes by 1, the feature sum AcA_c by ff, and the scatter matrix ScS_c by ff⊤ff^\top. Here, ff is the L2-normalized visual feature of sample xx (Eqs. (1) and (2)). Since ∣f∣=1\\|f\\| = 1, the norms of ff and ff⊤ff^\top both equal 1. As a result, the total change introduced by sample xx to the three statistics is 1+1+1=3\sqrt{1+1+1} = \sqrt{3}, which is independent of sample xx itself. Therefore, by adding Gaussian noise once to the sum of all client statistics, with its standard deviation determined by ε\varepsilon, δ\delta, and the change 3\sqrt{3}, TACO satisfies (ε,δ)(\varepsilon, \delta)-differential privacy.

Under the settings in Table 8, we compare the accuracy of TACO, GH-QDA, and DeepAFL (ICLR'26) using the same (ε,δ)(\varepsilon, \delta), where ε\varepsilon is set to 8 and 4 (a smaller ε\varepsilon gives stronger protection), and δ\delta is set to 10−510^{-5}.

Method (Table 8 Protocol, Test Accuracy %)εCIFAR-10CIFAR-100Tiny-ImageNet
GH-QDA (ICLR'26)889.4760.3859.83
DeepAFL (ICLR'26)882.281.1242.37
TACO890.4962.4160.50
GH-QDA (ICLR'26)489.0555.3955.64
DeepAFL (ICLR'26)485.481.2146.54
TACO489.8760.5658.99

As shown in the table, under both ε\varepsilon settings, TACO achieves higher accuracy than GH-QDA and DeepAFL on all three datasets. TACO uploads statistics only once, whereas DeepAFL trains a 20-layer network layer by layer and requires 41 uploads of statistics. Since (ε,δ)(\varepsilon, \delta) constrains the combined result of all uploads, the standard deviation of the Gaussian noise in each DeepAFL upload must be increased to 41≈6.4\sqrt{41} \approx 6.4 times that of a single upload. The Gaussian noise accumulates with the number of network layers, causing DeepAFL's accuracy on CIFAR-100 to drop close to random guessing. Furthermore, since the magnitude of the Gaussian noise is independent of the number of samples, on CIFAR-10, which has 5,000 samples per class, TACO's accuracy at ε=8\varepsilon = 8 matches the result without Gaussian noise (90.36% in Table 8), whereas on CIFAR-100 and Tiny-ImageNet, which have 500 samples per class, it is more affected by the noise.

In the revised version, we have removed the statements in the abstract and introduction that distinguish prototype-based methods from TACO based on privacy risks, corrected the statement in Section 3.1 regarding "not uploading prototypes," added TACO's compatibility with secure aggregation and the above differential privacy experiments to Appendix M, and updated the privacy statement in Appendix N.

Reviewer_1YGXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides a new training-free one-shot federated learning framework: TACO. Client do not need exchanging model parameters, they upload class-wise sufficient statistics computed in visual and text-anchored feature spaces. Then the server aggregates those statistics to build a global classifier in closed form. TACO consists of three components. First, Structural Covariance Transfer (SCT) exploits shared covariance structures across classes to improve covariance estimation. Second, dual-space QDA/LDA fusion incorporates text-anchor information to provide complementary discriminative cues. Third, confidence-weighted test-time augmentation (CTTA) reduces prediction noise by aggregating multiple augmented views. The paper provides theoretical analysis of cross-class covariance regularization and evaluates the method across multiple visual and language benchmarks, pretrained encoders, heterogeneous data partitions, and class-imbalance settings.

Post-rebuttal: We thank for author’s effort and response. Weakness 2 has not been addressed. We questioned the CTTA in table 2, but different experiment is provided. The improvement is not enough in the provided experiment.

Critical strengths

S1. TACO constructs the classifier using aggregated sufficient statistics, eliminating the need for gradient-based optimization or repeated communication. By leveraging shared frozen feature representations, it exhibits invariance to data distribution, offering significant advantages in FL. S2. TACO evaluates the key component across multiple datasets, pre-trained encoders, and classification settings. Ablation studies demonstrate that cross-class covariance regularization yields consistent performance improvements, while supplementary analyses provide valuable evidence regarding estimation

Critical weaknesses

W1. The SCT component adjusts the cross-class covariance mixing weights based on the sample counts of different categories; however, the benchmark datasets in the main text (CIFAR-10, CIFAR-100, and Tiny-ImageNet) feature balanced class distributions, meaning that same weights are applied to all classes within the datasets. No specific ablation studies were conducted to compare adaptive weighting against fixed coefficients to isolate the specific contribution of the former. Thus, the practical benefits and necessity of this adaptive design remain unclear. W2. The main benchmark performance gains yielded by the proposed text-space fusion are relatively limited, yet the method introduces additional communication and computational overhead. Also, the use of public text anchors appears to allow for the reconstruction of sufficient statistics in the text space from the visual statistics, which raising questions about the necessity of transmitting this information separately. Similarly, the main benchmark performance improvements from CTTA (extra inference computation and time overhead) also appear limited (close to 1 pp). Moreover, the lack of a detailed ablation study for CTTA makes it unclear to what extent the gains stem specifically from the proposed "confidence-weighted aggregation" method. In summary, I think the current ablation experiments are insufficient to demonstrate that these two components make a significant contribution.

Questions for authors

Q1. The paper sets "20 categories" as the threshold for switching between normal LDA and SCT-QDA, the rationale for this choice is not clearly explained. Could the authors further explains the considerations behind this threshold and its general applicability? Given that SCT is a core contribution of this study, clarifying why the default configuration disables SCT for CIFAR-10 while TACO-Tune enables it (claim on p. 22) would facilitate a better understanding of this design choice. Q2. The SCT method appears closely related to classical Regularized Discriminant Analysis (RDA), as both involve shrinking class-specific covariance estimates toward the covariance, also this relationship is particularly pronounced in class-balanced datasets. The authors are requested to clarify the key methodological differences between the two and provide a direct comparison with RDA.

12 comments
AuthorsOpenReview ↗

(W1-Part 1/2) We appreciate your time and valuable feedback

W1: "The SCT component adjusts the cross-class covariance mixing weights based on the sample counts of different categories; however, the benchmark datasets in the main text (CIFAR-10, CIFAR-100, and Tiny-ImageNet) feature balanced class distributions, meaning that same weights are applied to all classes within the datasets. No specific ablation studies were conducted to compare adaptive weighting against fixed coefficients to isolate the specific contribution of the former. Thus, the practical benefits and necessity of this adaptive design remain unclear."

Response to W1 (Part 1/2): We greatly appreciate your interest in the practical benefits of adaptive weights in Eq. (8). To address this, we have conducted an ablation study comparing adaptive weights with fixed coefficients on class-imbalanced merged tasks, based on theoretical reasoning.

Eq. (8) adjusts the cross-class mixing weights based on the number of samples, stemming from a theoretical analysis of covariance estimation error. Corollary 6 in Appendix C shows that, under the criterion of minimizing the mean squared error of covariance estimation, the optimal mixing weights for class cc take the fractional form γc/((Nc−1)+γc)\gamma_c / ((N_c - 1) + \gamma_c) and decrease inversely with the increase in the number of training samples NcN_c. Based on this theoretical property, Eq. (8) sets λc=γ/(Nc+γ)\lambda_c = \gamma / (N_c + \gamma) (where γ\gamma is a hyperparameter shared across all classes) to linearly combine the OAS estimate ΣcOAS\Sigma_c^{\text{OAS}} for class cc with the equally weighted average target Σccross\Sigma_c^{\text{cross}} of the remaining classes (Eq. (7)). When the number of samples across classes varies significantly, the covariance estimate for classes with sufficient samples is already relatively reliable, and a smaller λc\lambda_c allows it to retain its own estimate. Estimates for classes with few samples have larger variances, and a larger λc\lambda_c causes them to rely more heavily on Σccross\Sigma_c^{\text{cross}} to stabilize the estimate. In contrast, a fixed coefficient requires all classes to share the same trade-off weight, making it impossible to accommodate both types of requirements simultaneously. When the dataset is class-balanced, the λc\lambda_c for all classes is equal, and Eq. (8) degenerates into a single shared coefficient. Therefore, the improvement of SCT over OAS shown in Table 2 on CIFAR-10, CIFAR-100, and Tiny-ImageNet stems from the cross-class target Σccross\Sigma_c^{\text{cross}} itself.

AuthorsOpenReview ↗

(W1-Part 2/2) We appreciate your time and valuable feedback

Response to W1 (Part 2/2): To validate the effectiveness of per-class weights in class-imbalanced scenarios, we merged benchmarks with a large number of samples per class and those with a small number of samples per class into the same label space, constructing 4 merged tasks (AG News + CLINC150, AG News + Banking77, and CIFAR-10 + Flowers-102 with either 500 or 50 images per class), with the number of samples per class within the same task varying by a factor of 2.5 to 857. For image tasks, we used CLIP ViT-B/32 visual features, and for text tasks, we used DistilBERT features as in Table 14. The experiments compared a fixed coefficient λ\lambda with the γ\gamma of Eq. (8) within the same closed-form QDA baseline. For both methods, 10% of the training samples from each class were set aside as a validation set to select their respective hyperparameters. The models were then trained on the full training set and evaluated once on the test set. The table below reports the overall test accuracy for each task, as well as the test accuracy on the few-sample class subsets (CLINC150, Banking77, Flowers-102).

Metric / Method (Test Accuracy, %)AG News + CLINC150AG News + Banking77CIFAR-10 (500) + Flowers-102CIFAR-10 (50) + Flowers-102
Number of Samples per Class (Many / Few)30000 / 10030000 / 35–187500 / 2050 / 20
Fixed Coefficient (Overall)88.2086.7392.0290.28
Adaptive Weight (Eq. (8), Overall)90.2788.3792.3790.83
Fixed Coefficient (Few-Sample Classes)86.4480.3695.1494.91
Adaptive Weight (Eq. (8), Few-Sample Classes)92.0086.0795.8096.06

Experiments show that, across these 4 merged tasks, the overall accuracy and the accuracy on few-sample classes achieved with adaptive weights are both higher than those with fixed coefficients, with the overall accuracy improving by up to 2.07%. Breaking it down, the benefits of Eq. (8) are concentrated in few-sample classes: in the two text tasks with the widest range of sample sizes, the accuracy of CLINC150 and Banking77 increased from 86.44% and 80.36% to 92.00% and 86.07%, respectively, an improvement of 5.56% and 5.71%. On CIFAR-10 (50) + Flowers-102, the accuracy on Flowers-102 also increased from 94.91% to 96.06%, a gain of 1.15%. Meanwhile, the changes in accuracy for the many-sample classes (AG News and CIFAR-10) across the 4 tasks did not exceed 0.4%. These results are consistent with the theoretical expectations of Corollary 6, demonstrating that when there is a significant disparity in the number of samples across classes, the allocation of borrowing strength based on sample size as described in Eq. (8) can significantly improve the quality of covariance estimation and classification accuracy for few-sample classes while maintaining the original performance of many-sample classes.

We have included the above comparative experiments and theoretical explanations in Table 26 of Appendix P (lines 1984-1997), and explained in Section 3.2.3 that Eq. (8) (lines 292-302) degenerates into a shared coefficient on class-balanced datasets.

AuthorsOpenReview ↗

(W2-Part 1/2) We appreciate your time and valuable feedback

W2: "The main benchmark performance gains yielded by the proposed text-space fusion are relatively limited, yet the method introduces additional communication and computational overhead. Also, the use of public text anchors appears to allow for the reconstruction of sufficient statistics in the text space from the visual statistics, which raising questions about the necessity of transmitting this information separately. Similarly, the main benchmark performance improvements from CTTA (extra inference computation and time overhead) also appear limited (close to 1 pp). Moreover, the lack of a detailed ablation study for CTTA makes it unclear to what extent the gains stem specifically from the proposed "confidence-weighted aggregation" method. In summary, I think the current ablation experiments are insufficient to demonstrate that these two components make a significant contribution."

Response to W2 (Part 1/2): Thank you for your in-depth examination of text-space fusion and CTTA in terms of overhead and practical gains. Text fusion and CTTA were originally designed to address two scenarios: limited client uplink bandwidth and inconsistent reliability of test views. Under these two conditions, the two components can deliver significant performance improvements at minimal overhead.

First, regarding communication and computational overhead: since the text-space features in Eq. (3) are linear projections of visual features onto CC unit class anchors, when the server runs a text encoder to generate public anchors (as in Algorithm 1), the default configuration of TACO can directly project the aggregated visual statistics into text statistics on the server side. In this case, the client does not need to transmit text-branch data. It only needs to upload 50.29 MiB (CIFAR-100) and 100.59 MiB (Tiny-ImageNet) of visual statistics, while maintaining the same classification accuracy. This projection step accounts for only 5.1% and 5.0% of the server-side construction time, respectively, and eliminates the need to aggregate the uploaded text statistics. Therefore, the total server-side time remains essentially unchanged. The client is required to upload text statistics only when the server is not running a text encoder, and this accounts for just 3.8% (1.97 / 52.26 MiB) and 13.3% (15.49 / 116.08 MiB) of the default upload volume shown in Table 11 (Appendix G, lines 1412-1447). Conversely, when client upload bandwidth is extremely limited, the client can omit high-dimensional visual statistics and upload only low-dimensional text statistics, which is TACO-Lite in Appendix G. On CIFAR-100, TACO-Lite achieved 71.47% accuracy with an upload overhead of only 1.97 MiB, outperforming the linear-boundary method AFL (CVPR'25, 70.71% accuracy), which uploads 2.20 MiB, at a lower communication cost.

Second, when the quality of multiple cropped views of a test image varies, the visual space and text space exhibit clear complementarity. To verify this, we kept the remaining TACO settings at their default values and replaced the 10 views of each test image with randomly scaled and cropped views (crop areas of 0.25-1 or 0.08-1 of the original image, aspect ratios of 3/4–4/3, and random horizontal flips). In this case, the accuracy of individual test views ranged from 33.24% to 59.79%, with significant variations in reliability across different views. We compared the accuracy of using only the visual space, only the text space, and a fusion of both spaces on the same set of views (all using confidence-weighted averaging):

Table R1. Effect of text fusion (the three rows use the same set of 10 randomly scaled and cropped views, and the numbers in parentheses indicate the crop area as a proportion of the original image)

Test Accuracy (%)CIFAR-100 (0.25–1)CIFAR-100 (0.08–1)Tiny-ImageNet (0.25–1)Tiny-ImageNet (0.08–1)
Visual space only67.6361.6662.9660.01
Text space only66.4561.8859.8856.02
Fusion of both spaces (TACO)69.2464.4864.0061.25

As shown in Table R1, under these two cropping settings, the fusion of both spaces not only outperforms the visual-space-only approach by 1.04% to 2.82% but also outperforms the better-performing of the two individual spaces by 1.04% to 2.60% across all settings. This indicates that when view quality fluctuates significantly, the discriminative information from the two spaces can effectively complement each other.

Reviewer_6MfmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents TACO, a training-free approach for one-shot federated learning. The method aggregates class-level statistics from clients and constructs classifiers using both visual features and text-anchor representations. Its main technical component is SCT, which estimates class covariance matrices by shrinking them toward the average covariance of other classes. The authors provide a bias-variance analysis and evaluate the method on CIFAR-10, CIFAR-100, and Tiny-ImageNet.

Critical strengths

The main strength of this work is the covariance estimation strategy. Using information from other classes to regularize class-specific covariance matrices is reasonable, particularly when local training samples are limited. The experimental results generally support this design. In particular, the rotation experiment in Table 4 provides useful evidence that the improvement comes from the shared covariance structure rather than simply from stronger regularization. The ablation results also show consistent gains from SCT, especially on CIFAR-100 and Tiny-ImageNet (+3.79% and +5.57% over the OAS baseline). The paper is clearly written, and the comparison with existing methods is quite comprehensive.

Critical weaknesses

My main concern is the limited methodological novelty. SCT is closely related to classical covariance shrinkage estimators. Although the choice of a cross-class shrinkage target is reasonable and empirically effective, the difference from existing shrinkage methods needs to be better justified. Another issue is that the contributions of text-space fusion and test-time augmentation appear relatively small in the ablation results (0.1–1.6% in most settings). Most of the improvement comes from SCT. The ablation tables are honest about this. The paper should make this distinction clearer. I am also concerned about the practical cost of transmitting class-wise covariance matrices. The communication overhead increases quadratically with the feature dimension (the measured upload is roughly 20–50× that of linear-boundary methods), which may limit the applicability of the method in federated settings. Finally, the evaluation is mainly conducted on small image classification datasets. It is unclear whether the same conclusions hold for larger-scale problems.

Questions for authors

How was the shared shrinkage parameter y= 5000 selected? Have the authors evaluated the class-dependent weights suggested by the theoretical analysis? What happens when the number of classes becomes much larger, for example on ImageNet? Would covariance estimation become unstable in the text space? The improvements on CIFAR-10 are limited. Could the authors clarify under which data conditions SCT is expected to provide substantial benefits? Have the authors considered low-rank covariance approximations to reduce communication costs?

11 comments
AuthorsOpenReview ↗

Thank you for the thorough and insightful comments (1/8)

W1: "My main concern is the limited methodological novelty. SCT is closely related to classical covariance shrinkage estimators. Although the choice of a cross-class shrinkage target is reasonable and empirically effective, the difference from existing shrinkage methods needs to be better justified."

Response to W1: Thank you very much for detailing your concerns. Regarding the novelty of TACO, we are the first to observe in OFL that the class covariances extracted by the same frozen encoder share the main directions of variation, and we have ingeniously applied SCT to OFL, an approach that had not been considered by others previously. To better address your concerns, we provide the following detailed explanation:

As shown in Table 4 of the Appendix (lines 972-992), on CLIP ViT-B/32, ResNet-18, and DistilBERT, on average, 29% to 60% of the variance of each class falls within the top 20 principal directions of the average covariance of the other classes, whereas 20 randomly selected directions account for only 2% to 4%. Since the combined samples from the other classes far exceed those of the current class, the cross-class shared directions estimated from them are more reliable. Based on this observation, after OAS, SCT shrinks the covariance of each class toward the average of the covariances of the other classes, with each class weighted equally in this average. This cross-class and sample-size-independent construction of the shrinkage target distinguishes SCT from the two existing types of classical shrinkage methods.

The shrinkage target of the first type of classical shrinkage methods is constructed solely from in-class samples. For example, both OAS and Ledoit-Wolf use a scaled identity matrix determined by in-class samples as their target, and thus cannot provide cross-class shared directions. Table 8 of the Appendix (lines 1188-1201) compares SCT with six covariance estimation methods that use only in-class samples, including OAS and Ledoit-Wolf, under the same frozen encoder features and QDA classifier. SCT achieved the highest classification accuracy across all 8 settings, outperforming the next-best method by 2.35% to 5.78% in the 6 settings where the number of classes exceeds 20. These results demonstrate that incorporating shared directions provided by other classes is more effective at improving QDA's classification accuracy than constructing a shrinkage target based solely on in-class samples.

The second type of classical shrinkage methods incorporates statistics from other classes when constructing the shrinkage target. A representative method of this type, RDA, uses the pooled covariance of all classes as the shrinkage target. Since the pooled covariance is weighted by the sample size of each class, when there is a significant disparity in sample sizes across classes, RDA's shrinkage target is dominated by the classes with larger sample sizes. In contrast, SCT assigns equal weights to each class when constructing the cross-class shrinkage target, making it insensitive to differences in sample sizes across classes. To validate this distinction, we combined all classes from CIFAR-100, Food-101, Pets, DTD, and Aircraft into a 385-class classification task. We compared RDA and SCT using the same CLIP ViT-B/32 features and QDA classifier, with hyperparameters for both selected on a held-out validation set. In this task, CIFAR-100 and Food-101 account for only 52% of the classes but 90% of all training images. As a result, approximately 90% of RDA's shrinkage target comes from these two datasets, while the remaining three datasets account for about half of SCT's shrinkage target.

MethodOverallCIFAR-100Food-101PetsDTDAircraft
RDA79.7672.5986.7688.6670.4343.68
SCT81.0074.5687.1489.1873.7248.87

As shown in the table above, SCT achieves higher classification accuracy than RDA on all five datasets. Notably, on Aircraft, where each class has only about 67 training images, accuracy improved from 43.68% to 48.87%. This indicates that, under conditions where the number of samples per class varies significantly, a cross-class shrinkage target that assigns equal weights to each class can achieve higher classification accuracy than a pooled covariance weighted by sample count.

In Section 3.2.3 of the revised manuscript, we have clarified the differences between SCT and the two aforementioned types of classical shrinkage methods as well as RDA, and added relevant citations. We have also included the complete table and analysis of the 385-class experiment in Table 25 of Appendix P (lines 1944-1959).

AuthorsOpenReview ↗

Thank you for the thorough and insightful comments (2/8)

W2: "Another issue is that the contributions of text-space fusion and test-time augmentation appear relatively small in the ablation results (0.1–1.6% in most settings). Most of the improvement comes from SCT. The ablation tables are honest about this. The paper should make this distinction clearer."

Response to W2: Thank you very much for pointing this out. SCT is indeed our main contribution, and we have stated this clearly in the revised manuscript.

This paper addresses the issue of inaccurate class covariance estimation in one-shot federated learning when there are few samples per class. Existing methods shrink toward an isotropic target, thereby discarding the shared covariance structure across classes. Table 4 of the Appendix (lines 972-992) shows that the class covariances extracted by the same frozen encoder share the main directions of variation. SCT builds upon this finding by shrinking the covariance of each class toward the average of the covariances of the remaining classes. SCT directly addresses the problem this paper aims to solve. Therefore, it is the core contribution of this paper, and the improvement in accuracy stems primarily from SCT.

The ablation results in Table 2 corroborate this hierarchy. On CIFAR-100 and Tiny-ImageNet, the accuracy of visual-space QDA with OAS is 71.59% and 63.67%, respectively. TACO achieves a total improvement of 5.10% and 7.33% over this baseline. Specifically, SCT boosts accuracy to 75.38% and 69.24%, contributing 3.79% and 5.57%, respectively, which account for 74% and 76% of the total improvement. The text branch and CTTA further improve accuracy at different stages. The text branch constructs a classifier in the CC-dimensional text space using text embeddings of class names and fuses it with the classifier in the visual space, raising accuracy to 75.84% and 69.38% on top of SCT, an additional improvement of 0.46% and 0.14%, respectively. CTTA performs a confidence-weighted average of predictions for multiple views of the same image during the testing phase, further improving accuracy to 76.69% and 71.00%, representing additional gains of 0.85% and 1.62%, respectively. These results indicate that SCT contributes the majority of the accuracy improvement, while the text branch and CTTA provide consistent additional gains on top of SCT.

In the revised manuscript, we have adjusted the wording to reflect this hierarchy of contributions. In the abstract, the fourth paragraph of the introduction, and the list of contributions, we have removed the parallel listing of the three components and instead highlighted the shared covariance structure revealed in Table 4 and SCT as the core contributions of this paper, while describing the text branch and CTTA as designs that further improve accuracy on top of SCT. The first sentence of Section 4.3 already states that SCT is the largest source of improvement. We have added there the share of the total improvement contributed by each component.

AuthorsOpenReview ↗

Thank you for the thorough and insightful comments (3/8)

W3: "I am also concerned about the practical cost of transmitting class-wise covariance matrices. The communication overhead increases quadratically with the feature dimension (the measured upload is roughly 20–50× that of linear-boundary methods), which may limit the applicability of the method in federated settings."

Response to W3: Thank you very much for your concern regarding the actual communication overhead of per-class covariance matrices, as this overhead directly affects TACO's applicability in federated settings.

Several recent one-shot federated learning methods also require the upload of second-order statistics. Among them, TACO's client upload volume is smaller than that of SAFLe (CVPR'26) and DeepAFL (ICLR'26), while TACO's classification accuracy is higher than that of SAFLe and DeepAFL. Moreover, this upload volume can be further reduced in three ways.

Specifically, GH-QDA (ICLR'26), GH-Proto-Hyper (ICLR'26), SAFLe, and DeepAFL all upload second-order statistics from the client. Among these methods, SAFLe requires the upload of correlation blocks and cross-moments, while DeepAFL requires the upload of layer-wise statistics and multiple exchanges of statistics between layers. In contrast, TACO requires only a single upload. According to Table 11 of the Appendix (lines 1412-1447), on CIFAR-100 and Tiny-ImageNet, TACO and TACO-Tune in Table 1 of the main text (lines 378-414) both upload statistics of the visual and text spaces, with upload volumes of 52.26 MiB and 116.08 MiB per client, respectively. Under the same settings, SAFLe's upload volumes are approximately 114 MiB and 119 MiB, while DeepAFL's reach 139.41 MiB and 162.47 MiB. Therefore, among recent methods that also utilize second-order statistics, TACO-Tune has a smaller per-client upload volume than SAFLe and DeepAFL, while achieving higher classification accuracy.

Building on this, TACO's upload volume can be further reduced in three ways. First, for TACO with the default configuration, since the text-space statistics can be derived by the server from the aggregated visual statistics, the client can upload only the visual statistics. This modification reduces the per-client upload volume of the default configuration to 50.29 MiB and 100.59 MiB on the two datasets, respectively, while the classification accuracy remains unchanged. These volumes are the same as those of GH-QDA and GH-Proto-Hyper, which upload only per-class visual statistics. Second, building on the approach of uploading only visual statistics, the use of lossless low-rank factorization can further reduce the average upload size per client to approximately 27% of the original, without any loss in classification accuracy. For specific details and experimental data, please refer to our response to Q5 (regarding the question you asked: Have the authors considered low-rank covariance approximations to reduce communication costs?). Third, for scenarios with extremely limited communication, TACO-Lite in Appendix G achieves a per-client upload size of only 1.97 MiB on CIFAR-100 with a classification accuracy of 71.47%, whereas the linear-boundary method AFL (CVPR'25) has an upload size of 2.20 MiB and an accuracy of 70.71%. This indicates that on CIFAR-100, TACO-Lite can reduce communication overhead to a level below that of AFL while maintaining an accuracy advantage.

We have already pointed out the issue of communication overhead in the Limitations section of the original manuscript and mentioned that low-rank or sketch representations can reduce this overhead. In the revised manuscript, we have supplemented Appendix G (Table 12) and the Limitations section (Appendix N) with a comparison of communication overhead against the aforementioned baselines and a comprehensive analysis of the three compression schemes.

Comments on the paper

3 comments
AuthorsOpenReview ↗

To AC, SAC, PC and Reviewers

Dear AC, SAC, PC and Reviewers,

We have revised the manuscript according to the reviewers’ comments. All changes are highlighted in blue in the manuscript, and the revised manuscript has been updated on OpenReview. We have also updated and refined the rebuttal responses, with more detailed explanations where helpful, to facilitate review. We hope our responses address the reviewers’ concerns and would be happy to continue the discussion. We sincerely thank the reviewers for their careful reading and constructive suggestions, and thank the AC, SAC, and PC for their time and effort.

Best regards,

Authors

Dear reviewers,

Thank you for the reviews.

Authors have responded. You are encouraged to read the responses and let the authors know if any concerns remain.

Thank you.

Yours AC.

Dear AC,

Thank you for taking the time to follow up with the reviewers. We appreciate your help in facilitating the discussion.

If there are any points in our submission or our responses that you would like us to clarify, we would be happy to do so.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.