- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper targets a real bottleneck of federated fine-tuning: parameter-efficient methods reduce trainable parameters but clients still deploy and run the full pretrained backbone—the dominant cost on resource-constrained devices. The authors propose FedW2S, a federated weak-to-strong framework with lightweight weak models on clients and a pretrained strong model permanently on the server. Knowledge transfer uses (i) reliability-based filtering by confidence + top-2 margin and an entropy-driven selection budget; (ii) discrepancy-aware target mixing with a β coefficient from JS divergence and cached-server entropy; (iii) compatibility-aware strong-to-weak feedback gated by confidence margin × JS-based compatibility. The plug-in form L_k = L_base + λ L_S2W works on any base FL. On ImageNet / iWildCam / FMoW (DINO ViT-B/8 → ViT-L/14) and UltraFeedback (Qwen2.5-3B → 7B), FedW2S adds +1.5–2 pp on weak clients and +6–9 pp on strong server over 13 baselines.
Critical strengths
-
Conceptual clarity about the actual cost of federated LM adaptation. Separating trainable-parameter count from deployed-backbone forward/backward cost and proposing weak-to-strong as a way to address the second (which federated PEFT admittedly does not) is a well-articulated contribution. The headline numbers — ~0.19 GB / 0.01 TFLOPs on the client for ResNet-18 → ViT-L/14, with 87.15 % strong-side on ImageNet and 71.95 % on iWildCam — realise that promise.
-
Methodologically tight and modular. Each step of FedW2S is a small, auditable component that fits an explicit role: reliability filtering controls which public samples contribute, discrepancy-aware mixing controls how much of the weak teacher to absorb, and the compatibility-aware feedback gate controls when a server prediction is allowed to update the client. The decomposition is strictly additive (Eq. 11) and Table 1 demonstrates consistent +1.5–2 pp on the weak side and +6–9 pp on the strong side across six base FL methods. Ablation (Table 5) shows reliability filtering is the largest single component, target mixing is second, feedback is third.
Critical weaknesses
-
The end-to-end client-resource advantage is not established. Table 4 reports memory and FLOPs "per input" based on parameter estimation, but FedW2S also requires clients to predict on the public split and, after warm-up, to optimise a feedback loss over that split. In the vision experiments the public split is 40 % of the full dataset while private data are divided among 20 clients; this additional repeated public-data workload can materially change total client FLOPs, wall-clock, energy and storage / I-O. Table 9 similarly reports payload size but not the cost of generating the transmitted predictions or the cumulative communication over the full training run. Measured peak memory, client-hours / FLOPs per round and to convergence, wall-clock, energy if available, and total transmitted bytes are needed under the exact participation and public-data schedules. Without these results, "freeing clients" is demonstrated at the model-size / per-example level but not at the system level.
-
Performance attribution and statistical support are incomplete, especially for the small margins over strong baselines. Main vision tables provide point estimates averaged over three runs without standard deviations or confidence intervals, even though several claimed advantages are only ~0.1–0.6 percentage points. The language results have strongly overlapping uncertainty ranges. It is also unclear which Table 2 baselines have access to the same unlabelled public inputs and comparable server-side computation; a matched control should adapt the strong model on the same public set without client knowledge, and all relevant baselines should be compared under clearly reported data and compute budgets. The evidence supports "competitive performance," but not reliable superiority over the closest methods.
-
Heavy dependence on the unlabelled public dataset D_p with limited diagnosis of its distribution shift. The conclusion explicitly states that a large distribution shift between D_p and the client distributions degrades the method, and that weak-to-strong only works when the strong model has enough prior to absorb the weak supervision. Both failure modes matter in real deployments. The paper does not include a controlled shift-experiment (e.g. ImageNet as D_p for iWildCam / FMoW, general web text as D_p for UltraFeedback), nor the per-round reliability survival rate as D_p drifts. Without these, the failure mode is unquantified.
-
The cost-comparison baseline is not matched, and the trust in Table 2 rests on cost asymmetry. FedW2S's strong-side win (87.15 % on ImageNet vs FedIT 81.04 / FedSLM 79.24 / HSplitLoRA 76.63) is concurrent with the fact that the baselines deploy the strong backbone on the client and FedW2S does not. Table 3 reports the gap (~±15 GB / ~14 TFLOPs on the language task) but does not plot a client memory vs. strong-side accuracy Pareto frontier across methods, leaving open whether FedW2S is on the cost / accuracy frontier or simply the cost-asymmetric best.
-
The theoretical results are conditional identities / bounds with unknown target risks. Theorem 1 and Proposition 1 are conditional identities / bounds involving unknown target risks, fitting error and unselected-sample drift; they regulate the interpretation of the update but do not by themselves guarantee that FedW2S improves the learned model. The sufficient condition in Theorem 1 is not empirically verified in the evaluated runs, and the framework is per-round with no convergence / stability analysis of the bidirectional feedback loop.
Questions for authors
-
What are the measured end-to-end client costs per round and to convergence — including local private-data training, all public-set forward passes, feedback training, storage / I-O, and communication — for FedW2S and each resource-oriented baseline? How many public examples or minibatches does each participating client process in every stage?
-
In Tables 2 and 3, which baselines have access to exactly the same unlabeled public inputs and server-side compute budget? Could you add a matched strong-model control that uses the same public set and cached / self-distillation objective but no client predictions, as well as a compute / data-matched comparison?
-
Could you report per-run results, standard deviations or confidence intervals, and paired statistical comparisons for Tables 1, 2, 5, 6 and 7? Do the apparent 0.1–0.6 pp gains over the strongest baselines persist across seeds?
-
How robust is the reliability mechanism when the public distribution is shifted or when clients share a confidently wrong consensus? Can you also report the empirical terms in Theorem 1, particularly the selected-target gain, fitting error and unselected-sample drift, to show whether its sufficient condition is non-vacuous in the evaluated runs?
-
How sensitive is FedW2S to a distribution-shifted public dataset D_p? Please add a small ablation (or a new table) where D_p is drawn from a different domain than the client task (e.g. ImageNet as D_p for iWildCam / FMoW, general web-crawl text as D_p for UltraFeedback), together with the per-round reliability-filter survival rate.
-
Could you plot a client memory vs. strong-side accuracy Pareto frontier across FedW2S, FedIT, FedSA-LoRA, FlexLoRA, FLoRG, Fed-PLoRA, HSplitLoRA and FedSLM, both axes, language and vision? This would let the reader see whether FedW2S is genuinely on the cost / accuracy frontier or simply the cost-asymmetric best.
-
The compatibility gate a_{k,i} combines confidence margin (Eq. 8) and a JS-based compatibility factor (Eq. 9). Could you ablate the two individually and report the correlation between a_{k,i} and true usefulness (e.g. against a small held-out ground-truth public sample)?
9 comments
Response to Reviewer 9QZd (2/3)
Following up on the previous response Q3
Table R5: Standard deviations of strong model accuracy across vision tasks (original Table 2). | Method | ImageNet () | ImageNet () | iWildCam () | iWildCam () | FMoW () | FMoW () | |:---|:---:|:---:|:---:|:---:|:---:|:---:| | Local-Strong | | | | | | | | FedIT | | | | | | | | FedSA-LoRA | | | | | | | | FlexLoRA | | | | | | | | FLoRG | | | | | | | | Fed-PLoRA | | | | | | | | HSplitLoRA | | | | | | | | FedSLM | | | | | | | | FedW2S | | | | | | |
Table R6: Seed-wise paired performance differences between FedW2S and representative baselines. | Source | Dataset/setting | Comparison (A − B) | | | | | |:------:|:----------------|:-------------------|----:|----:|----:|----:| | Tab.1 | ImageNet (ϑ = 1, Weak) | FedW2S − FedMD | +1.25 | +1.83 | +0.97 | +1.36 | | Tab.2 | ImageNet (Avg.) | FedW2S − FLoRG | +0.34 | -0.17 | -0.47 | -0.10 | | Tab.2 | iWildCam (Avg.) | FedW2S − HSplitLoRA | +0.61 | +0.49 | +0.46 | +0.52 | | Tab.2 | FMoW (Avg.) | FedW2S − FlexLoRA | +0.63 | +0.38 | +0.52 | +0.51 | | Tab.3 | AlpacaEval 2 | FedW2S − HSplitLoRA | -0.74 | -0.12 | -0.37 | -0.41 | | Tab.3 | Arena-Hard | FedW2S − HSplitLoRA | +0.35 | +0.45 | -0.08 | +0.24 |
Q4&W5 Robustness of Reliability Filtering and Empirical Validation of Theorem 1.
The objective of reliability-based filtering is to reduce the influence of unreliable weak supervision, not to identify every incorrect prediction without ground-truth labels. Our existing Figure 9 provides empirical evidence that the proposed reliability score is positively associated with weak-label correctness across the three vision benchmarks.
To examine the practical implications of Theorem 1, we additionally measure the selected-target gain , fitting error , and unselected-sample drift at three training checkpoints on ImageNet, iWildCam, and FMoW. The results are shown in the Table R7. The results show that the average selected-target gain is positive across all three datasets, while the empirical Brier risk decreases at every evaluated checkpoint. These observations provide empirical evidence of beneficial knowledge transfer in the evaluated settings. Confidently incorrect client consensus and severe public-data distribution shifts remain potential failure modes rather than cases covered by unconditional guarantees.
Table R7: Empirical diagnostics of server-side knowledge transfer. | Dataset | | | | | |:---|:---:|:---:|:---:|:---:| | ImageNet | 0.2521 | 0.1874 | 0.0513 | -0.6698 | | iWildCam | 0.2967 | 0.3218 | 0.0157 | -0.3233 | | FMoW | 0.1956 | 0.5061 | 0.0592 | -0.1434 |
Q5&W3 Sensitivity to Public-Data Distribution Shift
To further investigate the sensitivity of FedW2S to public-data distribution shifts, we have conducted additional controlled experiments on iWildCam and FMoW, as reported in Table R1 (Item 1 of our General Response). Specifically, for iWildCam, clients hold private data from disjoint camera views, while the shifted public data are drawn from unseen camera views. For FMoW, clients hold private data from disjoint time periods, while the shifted public data are drawn from unseen periods. We maintain consistent private-data partitions, public-data sizes, and training budgets for fair comparisons.
The results in the table show that our method still demonstrates a certain competitive advantage. Our reliability filtering mechanism reduces the selection budget by increasing prediction uncertainty, thereby limiting the amount of potentially unreliable weakly supervised data transmitted to the server. We acknowledge that significant variations in distribution may still reduce knowledge transfer, particularly when weak predictions are confidently deemed incorrect. We discuss this limitation further in the revised manuscript and plan to conduct in-depth research on it in the future.
Response to Reviewer 9QZd (3/3)
Q6&W4 Client Memory–Performance Pareto Comparison
We have added client memory versus strong-model performance Pareto plots for both vision and language tasks in Figures 4 and 10 of this paper, covering FedW2S and all seven federated fine-tuning baselines. The results show that FedW2S lies on the Pareto frontier across the evaluated tasks. For visual tasks, FedW2S achieves an accuracy of 78.51% while using only 0.39 GB of peak GPU memory. By leveraging small model deployment, FedW2S reduces peak memory usage by over 90% compared to the best method FLoRG, while its performance is less than 0.3 points lower. For language tasks, FedW2S achieves good performance while requiring less memory than other methods.These results further highlight that FedW2S provides a favorable trade-off between client resource requirements and strong-model performance, rather than merely improving accuracy through increased client computation.
Q7 Ablation of the Compatibility-Aware Feedback Gate
To examine the individual contributions of the two factors, we conduct additional ablations by separately removing the confidence margin and the compatibility factor , while keeping all other training settings unchanged. The results of the elimination experiment are shown in the Table R8. Both factors independently improve performance over the no-feedback variant. Combining them achieves the best results, outperforming confidence-only feedback by 2.3/1.1 percentage points and compatibility-only feedback by 0.8/1.7 points on ImageNet/iWildCam, respectively. The confidence margin restricts feedback based on the server's relative confidence advantage, while the compatibility factor attenuates potentially conflicting guidance. Their combination enables selective absorption of server knowledge while mitigating interference with client-specific learning under non-IID distributions.
Table R8: Ablation study of confidence margin and compatibility factors. | Variant | Weight | ImageNet | iWildCam | |:---|:---:|:---:|:---:| | w/o Feedback | — | 81.2 | 68.9 | | Confidence Only | | 82.6 | 71.4 | | Compatibility Only | | 84.1 | 70.8 | | FedW2S (Full) | | 84.9 | 72.5 |
Thank you again for your constructive suggestions. Could you let us know whether these additions address your questions and concerns? We would be happy to discuss further and provide more details.
Response to Reviewer 9QZd (1/3)
Thank you for the helpful review. We carefully address your questions and comments below.
Q1&W1 End-to-End Client Resource Costs
Q2&W2 Public-Data Access and Matched Comparisons
Q3 Statistical Reliability and Small Performance Margins