- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes FedDOSA, a communication-efficient framework for one-shot federated LLM fine-tuning. Each client trains sparse coefficients in a shared two-dimensional DCT basis, and the server aggregates them in a single communication round. The authors theoretically analyze cross-client interference, highlighting that weight-space orthogonality does not necessarily guarantee layer-output orthogonality. Experiments on LLaMA and Qwen models demonstrate competitive performance with substantially fewer uploaded parameters.
Critical strengths
The first major strength is its theoretical analysis of interference in one-shot federated learning, particularly the distinction between weight space and layer output interference. This provides a meaningful perspective on why orthogonal parameter updates may still interfere and establishes conditions for eliminating such interference.
The second strength is its favorable communication performance trade-off, achieving competitive accuracy while substantially reducing the number of uploaded parameters compared to existing one-shot FL methods.
Critical weaknesses
The first major weakness is the insufficient justification for adopting an orthonormal 2D DCT basis in one-shot federated fine tuning. Although the authors demonstrate its theoretical validity and practical potential through experiments, the motivation for selecting DCT over other spectral fine tuning approaches remains unclear. Furthermore, the paper lacks sufficient ablation studies to determine whether the observed improvements originate from sparse parameter efficient fine tuning or from theproposed subspace allocation method.
The second weakness is the limited empirical experiments of theoretical analysis. Although the authors theoretically demonstrate that output disjoint allocation can eliminate layer output interference, the actual LLM experiments only employ scattered coordinate allocation. Consequently, it remains unclear whether the theoretically guaranteed reduction in interference translates into meaningful performance improvements in real LLMs.
Questions for authors
Q1. Output disjoint allocation eliminates layer output interference, but the LLM experiments only evaluate scattered coordinate allocation. Please provide evidence for performance improvements in the lower layers of actual LLMs.
Q2. How does output-index-disjoint allocation perform on actual LLMs, and how does its layer-output interference compare with the scattered-coordinate variant? This would help validate the connection between the theoretical analysis and empirical improvements.
Q3. How does FedDOSA compare with original-coordinate sparse fine-tuning and random orthogonal bases under matched budgets? These controls would clarify which components are responsible for the observed gains.
Q4. Can the main experimental results be reproduced with fully specified evaluation subsets and hyperparameter-selection protocols? This would improve confidence in the reported comparisons.
3 comments
I appreciate the authors for the response, and most of them have been resolved.
I am currently uncertain about the whether the novelty exceeds the bar of ICLR.
I have raised my score to 3 and lowered my confidence to 2.
Your suggestions have been helpful in improving our work and thank you for raising your score. We would also like to highlight the novelty of our work.
Methodological novelty. In one-shot federated learning, different clients may update the same parameter directions, causing interference during aggregation. We propose sparse orthogonal subspace allocation to reduce this overlap while keeping the number of trainable parameters small. Comparisons across scattered-coordinate, coordinate-disjoint, and output-index-disjoint allocation show that reducing weight-space overlap suppresses output interference, making independently trained client updates more compatible for single-round aggregation.
Theoretical novelty. We provide theoretical support for the orthogonal subspace allocation in our method by decomposing excess risk into capture, noise, and interference terms. Our theoretical analysis and empirical results support sparse orthogonal allocation as a way to reduce cross-client interference while retaining useful updates and limiting training noise. We further characterize residual output interference through covariance-dependent bounds.
We appreciate the reviewer's thoughtful comments and the opportunity to clarify the key aspects of our work.
W1 and Q3: Rationale for Choosing the DCT Basis
Thanks for this question. We clarify the role of an orthonormal frequency basis and the respective contributions of frequency-domain parameterization and subspace allocation.
Relation to Other Spectral Fine-Tuning Approaches. Our analysis emphasizes the role of an orthonormal frequency basis: disjoint coefficient supports ensure weight-space orthogonality, while disjoint output indices additionally guarantee layer-output orthogonality. DCT is one implementation. Section 4.2.2 compares DCT, FFT, and Hadamard transforms and reports similar performance, supporting our focus on client subspace allocation rather than a particular transform.
Comparison with Sparse Parameterizations. We further compare DCT with original-coordinate sparse fine-tuning and a random orthogonal basis under matched budgets in Table 1. DCT achieves the highest accuracy and lowest residual output overlap, while requiring less local training time than the random orthogonal basis (42.4 vs. 73.1 s). These results support DCT as a practical trade-off between accuracy and computational efficiency.
Overall, these experiments support three conclusions: (1) our framework operates in a shared orthonormal frequency basis, with DCT serving as one implementation; (2) frequency-domain parameterization improves adaptation over original-coordinate sparsity under matched settings; and (3) client-specific subspace allocation improves performance over fully overlapping supports at the same per-client budget.
W2, Q1 & Q2 Theoretical and Empirical Validation in Real LLMs
Thank you for highlighting the need to connect the theoretical guarantee with practical LLM performance. We provide additional experiments to (1) verify its zero layer-output interference guarantee in actual Transformer layers and (2) evaluate the downstream performance of output-disjoint allocation.
(1) Layer-output Interference. As shown in Table 2, we compare four allocation strategies on LLaMA3.2-1B at layers 0, 8, and 15, using 4,096 trainable coefficients per client. Coordinate-disjoint allocation achieves zero weight overlap but retains nonzero output overlap (0.00017, 0.00403, and 0.00292), whereas output-disjoint allocation eliminates both at all three layers. These measurements, including those in the lowest layer, empirically verify Theorem 3.4.
(2) Downstream performance. Output-index-disjoint allocation achieves the highest mean accuracy (45.7%) while eliminating measured layer-output interference. These results demonstrate the practical benefits of interference suppression.
Q4. Experimental Reproducibility
Yes. The main experimental results can be reproduced using the specified evaluation subsets and hyperparameter settings. We have further supplemented the experimental details. For MMLU-Pro evaluation, we randomly sample 1,000 questions from the official test set using three random seeds (0, 1, and 2). All methods are evaluated on identical subsets for each seed to ensure fair comparison. Following the baseline settings, we consistently adopt a learning rate of 10−2 and 2,000 local training steps across methods.