- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors proposed a training-free OFL approach, TACO, where every client uses a frozen pretrained encoder to encode its own images, projecting the features onto the C class anchors (text embeddings of the class using CLIP). Each client then sends the sample count (N_c), feature sum (A_c), and scatter matrix (S_c) for both the visual feature space and projected text space back to the server. The server then aggregates these (sum) and compute the mean, covariance, and prior for each class. After OAS shrinkage, each class covariance is regularized towards the average of other classes' covariances, with a sample size dependent mixing weight (SCT in the paper). Authors also proposed 1) dual-space fusion, where QDA classifiers are built on both visual space and the projected text space, combining their scores, (if num of classes <= 20, a single visual-space LDA classifier is built instead) and 2) CTTA, where predictions are aggregated across augmented views of the test sample, weighted by prediction confidence.
Critical strengths
-
paper is well-written and clear. OFL is a timely topic.
-
application of SCT to OFL is novel and give substantial gains over standard OAS shrinkage
-
evaluation is sufficiently thorough with many ablations and practical cost analyses, along with hyperparameter sensitivity
Critical weaknesses
-
the main benefit from TACO seems to be from CTTA. Comparing the baselines in Table 1 and the ablation Table 2, TACO performs worse than some baselines without CTTA on CIFAR-10 and Tiny-ImageNet and only marginally better for CIFAR-100. would running CTTA on the baselines close the this performance gap?
-
Line 366, the authors pair ResNet-18 with CLIP. These two representations are unaligned and taking their inner product (Eq 3) is hence an arbitrary linear projection.
-
Line 52, the authors mention that uploading prototypes "poses a substantial risk of privacy leakage". However, TACO also uploads the prototypes (A_c / N_c), along with S_c, posing a higher risk of privacy leakage.
I am happy to raise my score if these concerns are addressed.
Questions for authors
-
SCT is just a minor variation of regularized discriminant analysis (RDA) apart from considering the covariance of the class itself, right?
-
Details on how the authors run the baselines is unclear. For instance, how many epochs was FAFI run on in Table 1? I assume is more than 50 epochs since the number in Table 17 for 50 epochs is very different from the number in Table 1.
-
Line 388 and 1149, the authors mentioned that all hyperparameters are selected on CIFAR-100 but never detailed how this was selected? was there a separate validation set?
Post Rebuttal: the authors have addressed most of my concerns and i have raised my score accordingly. I would appreciate it if the authors could also report the baselines with CTTA in the other comparison tables (e.g. Tables 15,18,19,20). This would help readers separate the contributions of SCT and CTTA and underline where and why TACO falls behind existing baselines.
11 comments
(W2) Thank you for the careful and constructive review
W2: "Line 366, the authors pair ResNet-18 with CLIP. These two representations are unaligned and taking their inner product (Eq 3) is hence an arbitrary linear projection."
Response to W2: TACO was originally designed for CLIP and utilizes CLIP's paired visual and text encoders. Since CLIP's visual and text encoders are trained to align images and text, the text branch can leverage the text embeddings of class names to provide semantic information.
However, TACO does not need to rely on CLIP's paired visual and text encoders and can be generalized to other encoders. To verify this, we included experiments with the ResNet-18 encoder in our paper. The ResNet-18 experiments adopted the full configuration of the CLIP experiments, thus retaining the text branch. TACO can be generalized to ResNet-18 because its primary gains stem from SCT, which acts solely on statistics of visual features and does not require a paired text encoder. The experiments in Table 8, which do not use the text branch, show that on ResNet-18 features, the test accuracy of SCT is 0.70% to 6.15% higher than that of conventional OAS.
For further validation, we removed the text branch from ResNet-18, retaining only the visual branch (setting ). The remaining experimental settings were identical to those for TACO-Tune in Table 1. The average results across three random seeds are shown in the table below.
| Method (ResNet-18, Test Accuracy %) | CIFAR-10 | CIFAR-100 | Tiny-ImageNet |
|---|---|---|---|
| SAFLe (CVPR'26) + CTTA (table in W1) | 87.49 | 68.74 | 64.04 |
| TACO-Tune (visual and text branches) | 89.15 | 73.97 | 67.21 |
| TACO-Tune (visual branch only) | 89.25 | 73.93 | 67.62 |
When only the visual branch is retained, TACO-Tune's test accuracy is comparable to the results with both branches and remains 1.76% to 5.19% higher than SAFLe, the strongest baseline combined with CTTA, indicating that TACO's performance advantage on ResNet-18 does not depend on the text branch.
In the revised manuscript, we have replaced the ResNet-18 results in Table 1 and Section 4.2 with those obtained with only the visual branch retained, and have clarified in Section 4.1 that the text branch is applicable to image-text aligned encoders such as CLIP, while the ResNet-18 experiments serve to validate that TACO can be generalized to other encoders.
(W3) Thank you for the careful and constructive review
W3: "Line 52, the authors mention that uploading prototypes "poses a substantial risk of privacy leakage". However, TACO also uploads the prototypes (A_c / N_c), along with S_c, posing a higher risk of privacy leakage."
Response to W3: The and uploads in TACO contain class prototype information. However, uploading statistics computed from local data is a common practice in one-shot federated learning methods, and the visual statistics uploaded by TACO are identical to those in GH-QDA (ICLR'26) (Table 11). Furthermore, two commonly used privacy-preserving techniques, secure aggregation and differential privacy, can be directly applied to TACO.
First, the TACO server only requires the sum of all client statistics (Eq. (4)). Secure aggregation is a cryptographic protocol under which the server can only obtain the sum of all client uploads and cannot see the upload from any individual client. Therefore, TACO can be directly combined with secure aggregation, after which the server cannot obtain the statistics of any individual client, while the classifier and accuracy remain unchanged. In contrast, FedCOF requires the class means from each client to estimate the covariance, and FALCON requires the classifier and synthetic features from each client. Consequently, the FedCOF and FALCON servers must obtain uploads from individual clients.
Second, there is a fixed upper bound on the change in TACO's statistics caused by a single sample. -differential privacy requires that the probability distribution of the server's output remain virtually unchanged when any training sample is added or removed. Thus, the server can hardly determine whether a particular sample was used in training. Here, controls the magnitude of the allowed variation in the probability distribution, while is the small probability that this variation exceeds the allowed range. Differential privacy is commonly implemented by adding Gaussian noise to the results, whose standard deviation is jointly determined by , , and the maximum change in the results caused by a single sample. In TACO, when a sample of class is added or removed, only the statistics of class change: the sample count changes by 1, the feature sum by , and the scatter matrix by . Here, is the L2-normalized visual feature of sample (Eqs. (1) and (2)). Since , the norms of and both equal 1. As a result, the total change introduced by sample to the three statistics is , which is independent of sample itself. Therefore, by adding Gaussian noise once to the sum of all client statistics, with its standard deviation determined by , , and the change , TACO satisfies -differential privacy.
Under the settings in Table 8, we compare the accuracy of TACO, GH-QDA, and DeepAFL (ICLR'26) using the same , where is set to 8 and 4 (a smaller gives stronger protection), and is set to .
| Method (Table 8 Protocol, Test Accuracy %) | ε | CIFAR-10 | CIFAR-100 | Tiny-ImageNet |
|---|---|---|---|---|
| GH-QDA (ICLR'26) | 8 | 89.47 | 60.38 | 59.83 |
| DeepAFL (ICLR'26) | 8 | 82.28 | 1.12 | 42.37 |
| TACO | 8 | 90.49 | 62.41 | 60.50 |
| GH-QDA (ICLR'26) | 4 | 89.05 | 55.39 | 55.64 |
| DeepAFL (ICLR'26) | 4 | 85.48 | 1.21 | 46.54 |
| TACO | 4 | 89.87 | 60.56 | 58.99 |
As shown in the table, under both settings, TACO achieves higher accuracy than GH-QDA and DeepAFL on all three datasets. TACO uploads statistics only once, whereas DeepAFL trains a 20-layer network layer by layer and requires 41 uploads of statistics. Since constrains the combined result of all uploads, the standard deviation of the Gaussian noise in each DeepAFL upload must be increased to times that of a single upload. The Gaussian noise accumulates with the number of network layers, causing DeepAFL's accuracy on CIFAR-100 to drop close to random guessing. Furthermore, since the magnitude of the Gaussian noise is independent of the number of samples, on CIFAR-10, which has 5,000 samples per class, TACO's accuracy at matches the result without Gaussian noise (90.36% in Table 8), whereas on CIFAR-100 and Tiny-ImageNet, which have 500 samples per class, it is more affected by the noise.
In the revised version, we have removed the statements in the abstract and introduction that distinguish prototype-based methods from TACO based on privacy risks, corrected the statement in Section 3.1 regarding "not uploading prototypes," added TACO's compatibility with secure aggregation and the above differential privacy experiments to Appendix M, and updated the privacy statement in Appendix N.
(W1) Thank you for the careful and constructive review
Response to W1: While CTTA can certainly improve test accuracy, it is not the primary factor determining TACO's performance advantage. Instead, TACO's core improvement stems from SCT. Specifically, SCT directly improves the server-side classifier's estimation of class covariances, whereas CTTA merely performs a weighted average of predictions across multiple views of the same image during the testing phase. Thus, as a general test-time augmentation strategy, CTTA can be applied to any classifier. This difference in mechanism is also reflected in the ablation experiments in Table 2, where the introduction of SCT boosts accuracy by 3.99% and 5.14% on CIFAR-100 and Tiny-ImageNet, respectively, while stacking CTTA on top of this yields only an additional 0.85% and 1.62%.
To verify whether CTTA can bridge the gap between the baselines and TACO, we incorporated CTTA into the six partition-invariant baselines (AFL, GH-LDA, GH-QDA, GH-Proto-Hyper, DeepAFL, and SAFLe) published between 2025 and 2026, as listed in Table 1. These six methods include the strongest partition-invariant baseline for each backbone network and each dataset listed in Table 1. In terms of experimental setup, the hyperparameters for each baseline were consistent with those in Table 1, and the same CTTA as TACO was used. We then compared these baselines with TACO-Tune from Table 1 on two backbone networks, ResNet-18 and CLIP ViT-B/32, and the specific results are summarized in the table below.
The experimental results show that even when CTTA is applied, the accuracy of all baselines remains lower than that of TACO-Tune. Specifically, on the ResNet-18 backbone, the strongest baseline, SAFLe, still trails TACO-Tune by 1.76% to 5.19%, while on the CLIP ViT-B/32 backbone, the strongest baseline, DeepAFL, also trails by 0.74% to 2.25%. Combined with the ablation results in Table 2, this comparison further illustrates that TACO's primary gains stem from SCT, while CTTA plays only a supplementary role.
In addition to accuracy, TACO's communication overhead is also lower than that of SAFLe and DeepAFL. As shown in Table 11, on CIFAR-100 and Tiny-ImageNet, TACO's upload volume per client is 52.26 and 116.08 MiB, respectively, while SAFLe's is approximately 114 and 119 MiB, and DeepAFL's is 139.41 and 162.47 MiB. Furthermore, since the number of samples per class at the client is often far smaller than the feature dimension, the second-order statistics for that class form a low-rank matrix. Therefore, clients can instead upload smaller low-rank factors to achieve lower communication overhead. The server can precisely reconstruct the original second-order statistics from these low-rank factors, with both the classifier and accuracy remaining unchanged. The low-rank factors are computed from the second-order statistics themselves and do not contain per-sample features. Under a Dirichlet α=0.1 partition with 5 clients, this approach reduces the upload size per client by more than half on average on the CIFAR-100 and Tiny-ImageNet datasets, with at most 18.03 and 38.85 MiB per client, respectively. Consequently, on these two datasets, TACO achieves higher accuracy with lower communication overhead than SAFLe and DeepAFL.
Finally, we have added the complete experimental results for each baseline combined with CTTA to Table 23 in Appendix O of the revised manuscript, and Section 4.2 now states that TACO-Tune still outperforms all partition-invariant baselines equipped with CTTA. Section 4.3 now also reports the share of the total improvement contributed by each component, with SCT accounting for 74% and 76% on CIFAR-100 and Tiny-ImageNet. The low-rank upload sizes above are reported in Table 12 of Appendix G.