acceptodds
Under review as a conference paper at ICLR 2027

FedDOSA: One-Shot Federated LLM Tuning via Sparse Orthogonal Subspace Allocation

Abstract

One-shot federated learning makes it affordable to adapt large language models across clients that cannot share data, since each client uploads its update only once. The single round, however, removes any chance to repair conflicts between locally trained updates, so whatever interference the aggregation introduces is permanent. We cast one-shot federated adaptation as a design problem over the subspace each client may train in, and show that its excess risk separates into capture, noise, and interference terms. The interference term acts through the layer output rather than the weights: updates that are orthogonal as matrices can still conflict in the function the layer computes. We prove that allocating clients disjoint output indices in a shared orthonormal basis makes their layer outputs orthogonal for every input, which removes the interference term exactly for a squared-loss linear layer, and that the overlap left by coordinate-level allocation is governed by how nearly the basis diagonalizes the activation covariance. These bounds make no distributional assumption on the learned coefficients: their randomness is each client's own index draw. We instantiate the design as FedDOSA, which trains sparse coefficients in a two-dimensional discrete cosine basis, and find on language-understanding and mathematical-reasoning benchmarks that its coordinate-level variant achieves a favourable communication–performance trade-off against existing one-shot methods.

Primary area:
optimization
Keywords:
federated learning, one shot communication

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_uAcAOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FedDOSA, a communication-efficient framework for one-shot federated LLM fine-tuning. Each client trains sparse coefficients in a shared two-dimensional DCT basis, and the server aggregates them in a single communication round. The authors theoretically analyze cross-client interference, highlighting that weight-space orthogonality does not necessarily guarantee layer-output orthogonality. Experiments on LLaMA and Qwen models demonstrate competitive performance with substantially fewer uploaded parameters.

Critical strengths

The first major strength is its theoretical analysis of interference in one-shot federated learning, particularly the distinction between weight space and layer output interference. This provides a meaningful perspective on why orthogonal parameter updates may still interfere and establishes conditions for eliminating such interference.

The second strength is its favorable communication performance trade-off, achieving competitive accuracy while substantially reducing the number of uploaded parameters compared to existing one-shot FL methods.

Critical weaknesses

The first major weakness is the insufficient justification for adopting an orthonormal 2D DCT basis in one-shot federated fine tuning. Although the authors demonstrate its theoretical validity and practical potential through experiments, the motivation for selecting DCT over other spectral fine tuning approaches remains unclear. Furthermore, the paper lacks sufficient ablation studies to determine whether the observed improvements originate from sparse parameter efficient fine tuning or from theproposed subspace allocation method.

The second weakness is the limited empirical experiments of theoretical analysis. Although the authors theoretically demonstrate that output disjoint allocation can eliminate layer output interference, the actual LLM experiments only employ scattered coordinate allocation. Consequently, it remains unclear whether the theoretically guaranteed reduction in interference translates into meaningful performance improvements in real LLMs.

Questions for authors

Q1. Output disjoint allocation eliminates layer output interference, but the LLM experiments only evaluate scattered coordinate allocation. Please provide evidence for performance improvements in the lower layers of actual LLMs.

Q2. How does output-index-disjoint allocation perform on actual LLMs, and how does its layer-output interference compare with the scattered-coordinate variant? This would help validate the connection between the theoretical analysis and empirical improvements.

Q3. How does FedDOSA compare with original-coordinate sparse fine-tuning and random orthogonal bases under matched budgets? These controls would clarify which components are responsible for the observed gains.

Q4. Can the main experimental results be reproduced with fully specified evaluation subsets and hyperparameter-selection protocols? This would improve confidence in the reported comparisons.

3 comments

We appreciate the reviewer's thoughtful comments and the opportunity to clarify the key aspects of our work.


W1 and Q3: Rationale for Choosing the DCT Basis

Thanks for this question. We clarify the role of an orthonormal frequency basis and the respective contributions of frequency-domain parameterization and subspace allocation.

  1. Relation to Other Spectral Fine-Tuning Approaches. Our analysis emphasizes the role of an orthonormal frequency basis: disjoint coefficient supports ensure weight-space orthogonality, while disjoint output indices additionally guarantee layer-output orthogonality. DCT is one implementation. Section 4.2.2 compares DCT, FFT, and Hadamard transforms and reports similar performance, supporting our focus on client subspace allocation rather than a particular transform.

  2. Comparison with Sparse Parameterizations. We further compare DCT with original-coordinate sparse fine-tuning and a random orthogonal basis under matched budgets in Table 1. DCT achieves the highest accuracy and lowest residual output overlap, while requiring less local training time than the random orthogonal basis (42.4 vs. 73.1 s). These results support DCT as a practical trade-off between accuracy and computational efficiency.

Table 1. Comparison of different parameterization strategies

MetricDCTOriginal coordinatesRandom orthogonal
Accuracy (%) ↑45.438.741.8
Output overlap ↓0.010570.018520.01243
Local training time (s) ↓42.439.073.1
  1. Contribution of Subspace Allocation. With the DCT basis and per-client coefficient budget fixed, we compare independent scattered allocation with fully overlapping supports. Scattered allocation improves MMLU accuracy from 24.9% to 45.6%, supporting the effectiveness of client-specific subspace allocation. These gains reflect the combined effects of reduced coordinate collisions and increased global coefficient coverage.

Overall, these experiments support three conclusions: (1) our framework operates in a shared orthonormal frequency basis, with DCT serving as one implementation; (2) frequency-domain parameterization improves adaptation over original-coordinate sparsity under matched settings; and (3) client-specific subspace allocation improves performance over fully overlapping supports at the same per-client budget.


W2, Q1 & Q2 Theoretical and Empirical Validation in Real LLMs

Thank you for highlighting the need to connect the theoretical guarantee with practical LLM performance. We provide additional experiments to (1) verify its zero layer-output interference guarantee in actual Transformer layers and (2) evaluate the downstream performance of output-disjoint allocation.

(1) Layer-output Interference. As shown in Table 2, we compare four allocation strategies on LLaMA3.2-1B at layers 0, 8, and 15, using 4,096 trainable coefficients per client. Coordinate-disjoint allocation achieves zero weight overlap but retains nonzero output overlap (0.00017, 0.00403, and 0.00292), whereas output-disjoint allocation eliminates both at all three layers. These measurements, including those in the lowest layer, empirically verify Theorem 3.4.

Table 2. Comparison of weight-space and output-space interference across allocation strategies

AllocationWeight overlap, layers 0 / 8 / 15Output overlap, layers 0 / 8 / 15Accuracy(MMlU)
Full Overlap1 / 1 / 10.92518/0.84261/0.9381224.9
Independent scattered0.00051 / 0.00042 / 0.000400.01046 / 0.01425 / 0.0148845.4
Coordinate-disjoint0 / 0 / 00.00017 / 0.00403 / 0.0029245.4
Output-disjoint0 / 0 / 00 / 0 / 045.7

(2) Downstream performance. Output-index-disjoint allocation achieves the highest mean accuracy (45.7%) while eliminating measured layer-output interference. These results demonstrate the practical benefits of interference suppression.


Q4. Experimental Reproducibility

Yes. The main experimental results can be reproduced using the specified evaluation subsets and hyperparameter settings. We have further supplemented the experimental details. For MMLU-Pro evaluation, we randomly sample 1,000 questions from the official test set using three random seeds (0, 1, and 2). All methods are evaluated on identical subsets for each seed to ensure fair comparison. Following the baseline settings, we consistently adopt a learning rate of 10−210^{-2} and 2,000 local training steps across methods.

I appreciate the authors for the response, and most of them have been resolved.

I am currently uncertain about the whether the novelty exceeds the bar of ICLR.

I have raised my score to 3 and lowered my confidence to 2.

Your suggestions have been helpful in improving our work and thank you for raising your score. We would also like to highlight the novelty of our work.

Methodological novelty. In one-shot federated learning, different clients may update the same parameter directions, causing interference during aggregation. We propose sparse orthogonal subspace allocation to reduce this overlap while keeping the number of trainable parameters small. Comparisons across scattered-coordinate, coordinate-disjoint, and output-index-disjoint allocation show that reducing weight-space overlap suppresses output interference, making independently trained client updates more compatible for single-round aggregation.

Theoretical novelty. We provide theoretical support for the orthogonal subspace allocation in our method by decomposing excess risk into capture, noise, and interference terms. Our theoretical analysis and empirical results support sparse orthogonal allocation as a way to reduce cross-client interference while retaining useful updates and limiting training noise. We further characterize residual output interference through covariance-dependent bounds.

Reviewer_VYfwOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FedDOSA for one-shot federated LLM fine-tuning. Clients independently train sparse coefficients in a shared two-dimensional DCT basis and upload them once; the server merges coefficients through index-wise averaging. The analysis distinguishes weight-space orthogonality from layer-output orthogonality and characterizes interference under different allocation schemes. Experiments evaluate the scattered-coordinate variant on language-understanding and mathematical-reasoning benchmarks using LLaMA and Qwen backbones. The reported results suggest favorable performance with fewer uploaded parameters, with additional analyses of heterogeneity, client participation, and coefficient budgets.

Critical strengths
  1. The distinction between weight-space and layer-output orthogonality is useful. The analysis explains why disjoint parameter supports can still produce overlapping output perturbations, providing a clearer foundation for designing client allocations.

  2. The sparse-update protocol addresses an important communication constraint in federated LLM adaptation. The reported results across multiple backbones suggest that competitive performance may be achievable with substantially fewer uploaded parameters.

Critical weaknesses
  1. Incomplete experimental records. Section 4.1 and Appendix B.2 acknowledge missing evaluation subsets and learning-rate selection splits. Several ablations also lack essential configuration details. These omissions prevent verification of evaluation independence and experimental comparability, weakening confidence in the central communication–performance claims.

  2. Insufficient validation of the proposed mechanism. The exact output-orthogonality guarantee concerns disjoint output rows, whereas the LLM experiments use scattered-coordinate allocation. The activation statistics governing the remaining interference are not measured. Furthermore, the shared-support ablation also reduces the total number of distinct coefficients, confounding interference with model capacity. The experiments therefore do not establish that reduced functional interference explains the reported gains.

Questions for authors
  1. Clarification: Can you recover the exact evaluation subsets, the validation split used for learning-rate selection, and the missing configurations in Appendix B.2? If these records are unavailable, can the main comparisons be rerun under a fully documented protocol?

  2. Evidence supporting the current claims: Do measurements on trained LLMs demonstrate reduced output interference under the DCT parameterization? Can matched-budget allocation comparisons separate the effects of interference reduction from increased parameter coverage?

  3. Contribution clarification: Could the authors briefly discuss how FedDOSA relates to spectral adaptation methods FourierFT [1] and LoCA [2], and sparse zeroth-order federated adaptation such as Meerkat [3], particularly in parameter selection, optimization, and communication assumptions?

[1] Ziqi Gao et al. Parameter-Efficient Fine-Tuning with Discrete Fourier Transform. ICML 2024. Paper

[2] Zhekai Du et al. LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning. ICLR 2025. Paper

[3] Yide Ran et al. Mitigating Non-IID Drift in Zeroth-Order Federated LLM Fine-Tuning with Transferable Sparsity. ICLR 2026. Paper

4 comments

We thank the esteemed Reviewer for constructive and valuable feedback that has helped us refine our work.

W1 & Q1. Experimental Details and Statistical Reliability

We thank the reviewer for highlighting these concerns.

  • Experimental Details. For MMLU-Pro evaluation, we randomly sample 1,000 questions from the official test set using three random seeds (0, 1, and 2). All methods are evaluated on identical subsets for each seed to ensure fair comparison. Following the baseline settings, we consistently adopt a learning rate of 10−210^{-2} and 2,000 local training steps across methods. The learning rate is fixed across methods and is not selected using the evaluation subsets.

  • Statistical reliability. All main experiments are independently repeated with three random seeds. Table 1 already reports statistical significance tests for the performance improvements. We will explicitly document the random seeds and configurations used in the additional experiments to improve reproducibility.


W2 & Q2. Validation of the Proposed Mechanism

We provide additional LLM measurements and clarify the scope of the shared-support ablation.

(1) Layer-output orthogonality in trained LLMs. We compare scattered-coordinate, coordinate-disjoint, and output-index-disjoint allocation on LLaMA3.2-1B under matched coefficient budgets. Output-index-disjoint allocation yields zero measured output overlap at layers 0, 8, and 15, empirically verifying the prediction of Theorem 3.4 that allocating disjoint output indices eliminates pairwise layer-output interference. In addition, with both the per-client coefficient budget and the global number of distinct coefficients fixed, output-index-disjoint allocation achieves a mean accuracy of 45.7%, compared with 45.4% for coordinate-disjoint allocation. This improvement provides empirical support for the practical benefit of suppressing residual output interference beyond ensuring weight-space orthogonality. Overall, these results empirically validate Theorem 3.4’s output-orthogonality prediction and support the benefits of suppressing residual output interference.

Table 1. Comparison of weight-space and output-space interference across allocation strategies

AllocationWeight overlap, layers 0 / 8 / 15Output overlap, layers 0 / 8 / 15Accuracy(MMlU)
Full Overlap1 / 1 / 10.92518/0.84261/0.9381224.9
Independent scattered0.00051 / 0.00042 / 0.000400.01046 / 0.01425 / 0.0148845.4
Coordinate-disjoint0 / 0 / 00.00017 / 0.00403 / 0.0029245.4
Output-disjoint0 / 0 / 00 / 0 / 045.7

(2) Activation statistics and residual interference. We measure the off-diagonal activation covariance and examine its relationship in Table 2 with residual output overlap. After whitening, the off-diagonal covariance measure decreases from 0.995462 to 4.414.41 \times 10^{-6}, while coordinate-disjoint output overlap decreases from $0.011309$\pm$0.008742$ to $1.31 $\times 10^{-9}. These results support the covariance-mediated mechanism identified by our analysis.

Table 2. Effect of activation whitening on residual output interference in Transformer layers.

Input ConditionActivation Covariance (Off-Diagonal Measure)Coordinate-Disjoint Output Overlap
Original activations0.9954620.011309 ± 0.008742
Whitened activations4.41 × 10⁻⁶1.31 × 10⁻⁹

(3) Separating interference from coefficient coverage. We agree that shared-support allocation reduces the global number of distinct coefficients. Our additional comparison between coordinate-disjoint and output-index-disjoint allocation fixes both the per-client coefficient budget and the global number of distinct coefficients, thereby removing the coefficient-coverage confound. As shown in Table 1, output-index-disjoint allocation eliminates measured layer-output overlap and achieves a mean accuracy of 45.7%, compared with 45.4% for coordinate-disjoint allocation. Overall, this result is consistent with a benefit from suppressing residual output interference rather than increasing the number of distinct trainable coefficients.

Q3. Relation to Spectral Federated Adaptation and Experimental Comparison

Relation to Spectral Adaptation.

  • FourierFT FedDOSA builds on the sparse spectral parameterization of FourierFT and extends it to one-shot federated adaptation through client-specific subspace allocation. A direct federated implementation that assigns the same FourierFT support to every client corresponds to our fully overlapping baseline. FedDOSA instead controls cross-client coordinate overlap: disjoint supports ensure weight-space orthogonality, while independent sparse sampling limits expected collisions. Residual layer-output interference is further characterized through activation covariance.

  • LoCA addresses a different objective: it dynamically optimizes frequency locations to improve adaptation expressivity, whereas FedDOSA allocates client subspaces to control interactions during aggregation. Thus, our contribution lies in cross-client allocation and interference analysis rather than spectral parameterization alone.

  • Meerkat selects a static sparse subset of original model parameters using gradient statistics from pre-training data and performs zeroth-order optimization through forward-pass perturbations. Its federated protocol exchanges scalar projected-gradient information using shared random seeds and supports repeated synchronization to mitigate client drift. FedDOSA instead samples coefficients in a shared transform basis without gradient-based mask calibration, uses first-order local optimization, and uploads the learned coefficients once for final aggregation.

Accordingly, FedDOSA’s contribution centers on client subspace allocation and interference control under single-upload aggregation, rather than introducing spectral fine-tuning or sparse optimization alone.

Experimental Evidence. With per-client coefficient budget fixed and a singal global round, the following table shows FedDOSA achieves the highest accuracy among FourierFT, LoCA, and Meerkat under our evaluated settings. FourierFT and LoCA focus on spectral parameter selection for individual-model adaptation, while Meerkat combines sparse zeroth-order optimization with repeated federated synchronization. FedDOSA instead explicitly designs client subspaces for single-upload aggregation. These results support the value of accounting for cross-client update interactions when adapting sparse fine-tuning methods to one-shot FL.

MethodMMLUMMLU-Pro
FourierFT24.98.5
LoCA20.67.4
Meerkat31.210.1
Our45.614.7

Thank you for the matched-budget comparison. Holding both the per-client budget and the total number of distinct coefficients fixed addresses my concern about parameter coverage. The measurements on LLaMA 3.2-1B also show that output-index-disjoint allocation eliminates the measured overlap at all three tested layers.

I still have a question about the downstream effect. MMLU accuracy was 45.4% with coordinate-disjoint allocation and 45.7% with output-index-disjoint allocation. If available, could you share the differences for individual seeds? That would help me understand how consistently the reduction in overlap translates into an accuracy gain.

Could you also clarify which learning-rate protocol produced the results in Table 2? Section 4.1 describes selecting from four values, while the response says the rate was fixed at 0.01 across methods.

Thank you for your follow-up and suggestions. We clarify both points below.

1. Per-seed results.

SeedCoordinate-disjoint (%)Output-index-disjoint (%)Gain (pp)
045.345.8+0.5
145.645.9+0.3
245.345.4+0.1
Mean45.445.7+0.3

We use it as a zero-output-interference reference: coordinate-disjoint allocation retains only small residual output overlap and achieves accuracy close to this reference across all three seeds, supporting its practical effectiveness for one-shot aggregation.

2. Learning-rate protocol for Table 2.

Our method uses a fixed learning rate of 0.01, including in the Table 2 experiments. The search over {0.001,0.005,0.01,0.05}\{0.001, 0.005, 0.01, 0.05\} described in Section 4.1 applies to the baselines. Specifically, DENSE and Co-Boosting use 0.001 for GSM8K, 0.05 for MMLU, and 0.01 for MMLU-Pro. Both baselines were sensitive to the learning rate: excessively large values caused unstable optimization and incoherent outputs, whereas overly small values slowed convergence. We will correct this statement and explicitly document the method-specific learning rates in the revised manuscript.

Reviewer_P43QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FedDOSA, a communication-efficient one-shot federated fine-tuning framework for large language models based on sparse orthogonal subspace allocation. The method aims to mitigate cross-client update interference by learning and aggregating sparse coefficients in a shared orthonormal basis. The paper also provides a theoretical analysis of subspace allocation and interference, supported by experiments on multiple LLM backbones and downstream benchmarks. Experimental results demonstrate competitive performance with substantially reduced communication overhead compared with existing one-shot federated learning methods.

Critical strengths

FedDOSA demonstrates a favorable communication–performance trade-off across multiple LLM backbones and evaluation benchmarks. In Table 2, it reduces the reported per-client upload parameter count by approximately 5.8–14.2 times compared with the one-shot baselines while maintaining competitive or superior accuracy on several tasks. The additional matched-computation experiment on LLaMA3.2-1B also suggests that its performance advantage over FLoRA is not simply due to additional local computation. These results support the practical relevance of the proposed approach for communication-constrained federated adaptation.

Critical weaknesses
  1. The Introduction claims that parameter interference under Non-IID data can "severely distort the global model's ability to generalize." However, Proposition 3.1 does not establish that cross-client interaction terms are necessarily harmful or that eliminating them improves global risk. More importantly, Theorem 3.4 guarantees zero output interference only for disjoint output-index allocation, whereas the implemented method uses scattered-coordinate sampling. Without measuring the remaining output interference in real LLMs, it is difficult to determine whether the proposed method actually addresses the central problem motivating the paper.
  2. The experiments do not establish that the performance gains come from the proposed subspace allocation. Appendix B.3 acknowledges that comparisons with sparse updates in the original parameter space and random orthogonal bases "were not run on LLMs." Furthermore, Figure 2 changes both parameter overlap and the number of distinct trainable coefficients, making it impossible to isolate the effect of interference. The main baselines also differ in parameter budgets and adapted projections. Without these controlled comparisons, the reported gains cannot be confidently attributed to the proposed allocation strategy rather than differences in parameterization or model capacity.
Questions for authors
  1. The Introduction claims that one-shot FL faces "amplified spatial heterogeneity and parameter space collisions." What exactly does spatial heterogeneity refer to, and why should it be amplified in one-shot FL compared with standard multi-round FL? Furthermore, the authors state that local updates reflect "conflicting data characteristics" and "compete for overlapping regions in the parameter space." What theoretical or empirical evidence supports these claims, particularly the connection between Non-IID data distributions, parameter conflicts, and degraded generalization?

  2. The paper argues that interference should be measured at the layer output rather than in weight space. However, the implemented method uses randomly sampled sparse coordinates to reduce coefficient collisions, which does not guarantee output-space orthogonality. Could the authors explain why this strategy is expected to reduce functional interference and provide direct measurements of output-space interference in trained LLMs to support this claim?

  3. Section 3.1 acknowledges that the cross-client interference term can be either positive or negative. However, the proposed method aims to eliminate or reduce these interactions, although doing so does not necessarily imply a reduction in global risk. Could the authors establish the conditions under which eliminating these interaction terms improves the global objective, either through a theoretical guarantee or direct empirical evidence?

  4. The main experiments primarily compare against methods published in or before 2024, without sufficient coverage of recent developments in one-shot federated learning from 2025 to 2026. Furthermore, closely related approaches using orthogonal subspaces or directional constraints to mitigate model merging interference, such as OSRM and YOCO, are not adequately discussed. Could the authors clarify the methodological differences between FedDOSA and these approaches, and provide additional comparisons where their experimental settings are compatible?

7 comments

We appreciate the reviewer's thoughtful comments and the opportunity to clarify the key aspects of our work.

W1 & Q1. What does “spatial heterogeneity” mean, and how does it affect one-shot aggregation?

Thank you for requesting a precise justification. This connection can be characterized theoretically. By “spatial heterogeneity,” we mean cross-client statistical heterogeneity: differences in client data distributions induce different local objectives and learning directions. We will use this clearer terminology.

Let Fk(w)F_k(w) denote client kk’s population objective and F(w)=K−1∑kFk(w)F(w)=K^{-1}\sum_kF_k(w). A distributional description is $ g_k=\mu+\epsilon_k, $ where gk=∇Fk(w)g_k=\nabla F_k(w), μ\mu represents shared task structure, and the independent, centered client deviations have covariance CC. Then $ \mathbb{E}\left[K^{-1}\sum_k|g_k-\bar g|^2\right] =(1-1/K)\operatorname{tr}(C). $ Here, ϵk\epsilon_k describes between-client population-gradient variation, distinct from the gradient-estimation noise ξk\xi_k in Section 3.1.

The connection to global risk can be seen explicitly for heterogeneous quadratic objectives: $ F_k(w)=\tfrac12(w-a_k)^\top H_k(w-a_k),\qquad H_k\succ0. $ Converged local training followed by averaging returns aˉ=K−1∑kak\bar a=K^{-1}\sum_ka_k. However, with H=K−1∑kHkH=K^{-1}\sum_kH_k, the global optimum satisfies $ w^\star=H^{-1}K^{-1}\sum_kH_ka_k, $ and completing the square gives $ F(\bar a)-F(w^\star)=\tfrac12|\bar a-w^\star|_H^2. $ Thus, locally optimal models can yield positive global excess risk when heterogeneous client preferences and curvature make aˉ≠w⋆\bar a\ne w^\star.

One-shot FL does not increase the underlying distributional heterogeneity. Its difficulty is that independently learned updates must be reconciled in a single aggregation, without subsequent synchronization to revise the result. We will clarify “amplified” accordingly.

Our analysis further makes “parameter conflict” precise through the curvature-weighted interaction in Section 3.1. For a squared-loss linear layer, this equals $ \mathbb{E}\langle\Delta W_i x,\Delta W_j x\rangle. $ Therefore, overlapping parameter directions permit interactions, but their effect depends on the data covariance and the learned updates—not overlap alone. This is why our theory distinguishes weight-space orthogonality from output-space interference.


W2. Contribution of Subspace Allocation

Thank you for highlighting the need for controlled comparisons. We provide additional matched-budget LLM experiments to clarify the contribution of subspace allocation.

To isolate the contribution of frequency-domain parameterization beyond sparsity and orthogonality alone, we compare DCT, original-coordinate sparse updates, and a random orthogonal basis under matched allocation and coefficient budgets. Table 1 shows that DCT achieves the highest accuracy and lowest residual output overlap, while requiring less local training time than the random orthogonal basis (42.4 vs. 73.1 s). These results support DCT as a practical trade-off between accuracy and computational efficiency.

Table 1. Comparison of different parameterization strategies

MetricDCTOriginal coordinatesRandom orthogonal
Accuracy (%) ↑45.438.741.8
Output overlap ↓0.010570.018520.01243
Local training time (s) ↓42.439.073.1

Q4 Additional Experimental Comparisons

Thank you for highlighting these related approaches. We clarify the differences between FedDOSA, OSRM, and YOCO and provide additional controlled comparisons.

Methodological Comparison. Compared with OSRM and YOCO, FedDOSA offers a computationally lightweight mechanism for mitigating model merging interference. Specifically, FedDOSA achieves structural orthogonality through disjoint coordinate allocation in the DCT domain, without requiring additional data-dependent subspace estimation or directional supervision. In contrast, OSRM exploits activation statistics to construct constrained LoRA subspaces, while YOCO introduces directional supervision and regularization to mitigate client-update conflicts. These additional operations introduce communication overhead, whereas FedDOSA establishes parameter-level orthogonality directly through its coordinate allocation mechanism.

Experimental Validation. To further evaluate these methodological differences, we consider comparisons under matched backbone models, data partitions, and communication overhead. As shown in following Table, FedDOSA achieves higher accuracy than both OSRM and YOCO on two datasets, demonstrating its effectiveness under the matched budget.

MethodMMLUMMLU-Pro
OSRM38.6 ± 0.111.3 ± 0.7
YOCO26.6 ± 0.19.5 ± 4.4
Our45.4± 0.214.7 ± 1.5

Q2. Output-Space Interference: Analysis and Empirical Validation

Our analysis explains when sparse coordinate allocation reduces layer-output interaction and identifies the residual term that must be checked in trained LLMs. The key distinction is between coordinate collisions and covariance-mediated interactions across different coordinates.

1. Why sparse allocation can reduce output-space interaction.

Write ΔWk=CoFkCi⊤\Delta W_k=C_oF_kC_i^\top, and define the activation second moment in the input basis as Σ~=Ci⊤XX⊤Ci=DΣ+E.\widetilde\Sigma=C_i^\top XX^\top C_i=D_\Sigma+E. Orthonormality of the output basis gives $\langle\Delta W_iX,\Delta W_jX\rangle_F=\sum_{(u,v)\in S_i\cap S_j} \widetilde\Sigma_{vv}F_i(u,v)F_j(u,v) + \sum_{u\in U_i\cap U_j}(f_i^u)^\top E f_j^u.$

The first term requires clients to occupy the same coefficient coordinate. The second couples different input coordinates within a shared output row. Sparse allocation therefore controls the first source directly; the second depends on the off-diagonal activation covariance.

Let θk\theta_k and ρk\rho_k denote coefficient-energy fractions on collided coordinates and shared output rows. Cauchy–Schwarz yields ∣⟨ΔWiX,ΔWjX⟩F∣∥Fi∥F∥Fj∥F≤σmax⁡θiθj+∥E∥2ρiρj.\frac{\left\lvert\left\langle\Delta W_iX,\Delta W_jX\right\rangle_F\right\rvert}{\lVert F_i\rVert_F\lVert F_j\rVert_F}\le\sigma_{\max}\sqrt{\theta_i\theta_j}+\lVert E\rVert_2\sqrt{\rho_i\rho_j}. where σmax⁡=max⁡vΣ~vv\sigma_{\max}=\max_v\widetilde\Sigma_{vv}.

For independent uniform sampling, another client includes each coordinate with probability B/DB/D. Consequently, conditional on a nonzero locally learned coefficient array, its expected collided-energy fraction is B/DB/D. The probability of occupying a particular output row is $ q=1-\frac{\binom{D-d_i}{B}}{\binom DB}. $ Applying Cauchy–Schwarz again gives, for a fixed reference activation second moment, E[∣⟨ΔWiX,ΔWjX⟩F∣∥Fi∥F∥Fj∥F]≤σmax⁡B/D+∥E∥2q.\mathbb{E}\left[\frac{\lvert\langle\Delta W_iX,\Delta W_jX\rangle_F\rvert}{\lVert F_i\rVert_F\lVert F_j\rVert_F}\right]\le\sigma_{\max}B/D+\lVert E\rVert_2q.Zero-update pairs contribute zero. This argument requires independent support draws, but no independence or centering of learned coefficient entries.

Thus, the expected benefit has a precise condition: scattered allocation suppresses the collision component, while low residual interaction additionally requires covariance alignment or reduced output-row sharing. DCT offers a suitable basis for smooth correlated inputs; its alignment with trained LLM activations is the quantity to verify empirically.

2. Existing output-space measurements support the mechanism in the controlled setting.

The manuscript’s linear-layer simulation directly measures output overlap. Under AR(1) input covariance in the DCT basis, it reports:

AllocationNormalized absolute output overlapSigned cross term /R0R_0Risk /R0R_0
Shared support0.7690.0150.98
Scattered, independent0.0260.0150.87
Scattered, coordinated0.0060.0070.81
Output-row blocks, coordinated000.78

The output-overlap diagnostic is normalized by output norms, whereas the theorem normalizes by coefficient energy. They provide complementary views: the theorem bounds interaction magnitude relative to the updates, and the diagnostic measures output alignment.

Independent allocation reduces the reported output overlap. Its signed cross term remains equal to the shared-support value at the reported precision, illustrating the paper’s distinction between geometric overlap, signed risk interaction, and total risk.

Q2. Output-Space Interference: Analysis and Empirical Validation

3. Direct verification in trained LLMs. We conduct two experiments in real LLms layers to verify that coordinate-disjoint allocation can retain small but nonzero output interference and that this residual interference depends on activation covariance.

  • (i) Output Interference As shown in following Tab. 2, we evaluate three allocation strategies on LLaMA3.2-1B across layers 0, 8, and 15, with 4,096 trainable coefficients per client. Coordinate-disjoint allocation achieves zero weight overlap but retains small nonzero output overlap (0.00017, 0.00403, and 0.00292), whereas output-disjoint allocation eliminates both. These results confirm that parameter-space orthogonality does not necessarily imply output-space orthogonality. These results empirically support the distinction between parameter-space and output-space orthogonality.

Table 2. Comparison of weight-space and output-space interference across allocation strategies

AllocationWeight overlap, layers 0 / 8 / 15Output overlap, layers 0 / 8 / 15
Independent scattered0.00051 / 0.00042 / 0.000400.01046 / 0.01425 / 0.01488
Coordinate-disjoint0 / 0 / 00.00017 / 0.00403 / 0.00292
Output-disjoint0 / 0 / 00 / 0 / 0
  • (ii) Effect of Activation Covariance. We further investigate the relationship in following Table 3 between activation covariance and residual output interference. Under the original activations, the off-diagonal covariance measure is 0.995462, with an output overlap of 0.011309 ± 0.008742. After whitening, these values decrease to 4.41×10−64.41 \times 10^{-6} and 1.31×10−91.31 \times 10^{-9}, respectively. These findings are consistent with our theoretical analysis that residual functional interference depends on activation covariance, providing empirical support for its relevance to practical Transformer layers.

Table 3. Effect of activation whitening on residual output interference in Transformer layers.

Input ConditionActivation CovarianceCoordinate-Disjoint Output Overlap
Original activations0.9954620.011309 ± 0.008742
Whitened activations4.41 × 10⁻⁶1.31 × 10⁻⁹
Reviewer_uUEGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FedDOSA, a subspace adaptation method for one-shot federated learning. Grounded in theoretical analysis, the method trains sparse coefficients in a two-dimensional discrete cosine transform (2D-DCT) basis to effectively allocate training subspaces across clients and reduce interference during model aggregation, thereby achieving a favorable trade-off between communication overhead and model performance.

Critical strengths
  1. Formulates one-shot federated LLM adaptation as a trainable subspace allocation problem, providing a useful perspective on the distinction between weight-space and output-space orthogonality.
  2. Establishes theoretical guarantees for output-space orthogonality under disjoint output-index allocation and derives interference bounds without imposing distributional assumptions on the learned coefficients.
  3. Evaluates FedDOSA on language-understanding and mathematical-reasoning benchmarks. The experimental results demonstrate a favorable trade-off between communication efficiency and model performance compared with existing one-shot federated learning methods.
Critical weaknesses
  1. The theoretical guarantees are established under settings that differ from those used in the main experiments, while the theoretically supported design lacks sufficient validation at practical scales. Consequently, the applicability of the theoretical results and their relevance to the observed empirical gains remain insufficiently established.
Questions for authors
  1. Although the authors prove that disjoint output-index allocation eliminates layer-output interference, the main LLM experiments adopt scattered-coordinate allocation, for which this guarantee does not hold. The residual interference depends on the activation covariance, but its behavior is not empirically examined in the adapted Transformer layers. Moreover, the theoretical analysis considers a simplified loss and optimization setting, whereas the practical implementation uses cross-entropy training with 2,000 local steps and sums client updates rather than taking the convex combination considered in the analysis. These differences make it difficult to assess how well the theoretical results explain the observed performance.
  2. The output-disjoint allocation strategy is not sufficiently evaluated at practical model scales, and its per-client output-row budget is unclear. Under a balanced allocation across nine clients, each client would access only approximately one ninth of the output directions, potentially limiting adaptation capacity. Furthermore, the synthetic simulation uses a denser per-row coefficient allocation than the reported practical setting. A comparison between output-disjoint and scattered-coordinate allocation under matched coefficient budgets and client configurations would help clarify the practical trade-off between theoretical guarantees and model performance.
  3. The contribution of the DCT basis is not sufficiently isolated in the LLM experiments. Comparisons with sparse allocation in the original parameter space and with a random orthogonal basis would help establish whether the improvements arise from the DCT basis itself or from sparse subspace allocation more generally. The structured bases evaluated also exhibit similar performance. Furthermore, differences in trainable parameter counts and communication budgets complicate fair comparisons with existing methods. Additional matched-budget experiments and comparisons with closely related sparse-subspace and mask-based approaches would strengthen the empirical evidence for the proposed design.
  4. Several important experimental details are insufficiently documented, including the MMLU-Pro evaluation subset, the data split used for learning-rate selection, and random seeds for some additional experiments. These omissions make independent reproduction difficult. Moreover, without consistent uncertainty estimates across independent runs, it is difficult to assess the reliability of the smaller reported performance improvements.
5 comments

We thank the esteemed Reviewer for constructive and valuable feedback that has helped us refine our work.

W1 & Q1 & Q2: Theory Validation and Practical Performance

The analysis covers scattered allocation and the implemented index-wise aggregation. Below, we clarify the connection and derive how a simple allocation refinement strengthens the interference bound.

1. Why scattered coordinates can still interfere.

(1) Theoretical explanation.

Write a client’s update as ΔWk=CoFkCi⊤\Delta W_k=C_oF_kC_i^\top, where Co,CiC_o,C_i are orthonormal output and input bases. For a common activation batch XX, define Σ~=Ci⊤XX⊤Ci=DΣ+E.\widetilde\Sigma=C_i^\top XX^\top C_i=D_\Sigma+E. Orthonormality of CoC_o gives ⟨ΔWiX,ΔWjX⟩F=∑u(fiu)⊤Σ~fju,\langle\Delta W_iX,\Delta W_jX\rangle_F=\sum_u(f_i^u)^\top\widetilde\Sigma f_j^u, where fkuf_k^u contains client kk’s coefficients on output row uu.

This identity explains the distinction directly:

  • The diagonal part DΣD_\Sigma couples coefficients at the same coordinate.
  • The off-diagonal part EE also couples different input coordinates within the same output row.

Therefore, disjoint coordinates remove the first interaction, whereas disjoint output rows remove both. Applying Cauchy–Schwarz yields ∣⟨ΔWiX,ΔWjX⟩F∣≤∥Fi∥F∥Fj∥F[σmax⁡θiθj+∥E∥2ρiρj],\left\lvert\left\langle\Delta W_iX,\Delta W_jX\right\rangle_F\right\rvert\le\lVert F_i\rVert_F\lVert F_j\rVert_F\left[\sigma_{\max}\sqrt{\theta_i\theta_j}+\lVert E\rVert_2\sqrt{\rho_i\rho_j}\right],where σmax⁡=max⁡vΣ~vv\sigma_{\max}=\max_v\widetilde\Sigma_{vv}, and θk,ρk\theta_k,\rho_k are the coefficient-energy fractions on collided coordinates and shared output rows. This is the covariance-dependent guarantee for scattered allocation. The manuscript’s controlled simulation tests this distinction. Under AR(1) input covariance, it reports:

Table 1. Interference and risk across allocation strategies.

AllocationDCT CrossDCT R/R0R/R_0Random orthogonal CrossRandom orthogonal R/R0R/R_0
Scattered, independent0.0150.872.562.25
Scattered, coordinated0.0070.812.922.60
Row blocks, independent0.0130.830.370.79
Row blocks, coordinated00.7800.37

“Cross” is the signed interaction normalized by the zero-update excess risk R0R_0; comparisons are within each basis. These results demonstrate that coordinate disjointness can leave substantial interaction, whereas output-disjoint allocation eliminates it.

(2) Empirical validation in Real LLMs. We examine Theorem 3.4’s prediction and the benefits of interference suppression through output-interference experiments, and assess how residual interference depends on activation covariance.

  • (i) Output Interference. We compare scattered-coordinate, coordinate-disjoint, and output-index-disjoint allocation on LLaMA3.2-1B under matched coefficient budgets. Table 2 shouws that output-index-disjoint allocation yields zero measured output overlap at layers 0, 8, and 15, empirically verifying the prediction of Theorem 3.4 that allocating disjoint output indices eliminates pairwise layer-output interference. In addition, output-index-disjoint allocation achieves a mean accuracy of 45.7%, compared with 45.4% for coordinate-disjoint allocation. This improvement provides empirical support for the practical benefit of suppressing residual output interference beyond ensuring weight-space orthogonality. Overall, the results empirically validate Theorem 3.4’s output-orthogonality prediction and support the benefits of suppressing residual output interference.

Table 2. Comparison of weight-space and output-space interference across allocation strategies

AllocationWeight overlap, layers 0 / 8 / 15Output overlap, layers 0 / 8 / 15Accuracy(MMlU)
Full Overlap1 / 1 / 10.92518/0.84261/0.9381224.9
Independent scattered0.00051 / 0.00042 / 0.000400.01046 / 0.01425 / 0.0148845.4
Coordinate-disjoint0 / 0 / 00.00017 / 0.00403 / 0.0029245.4
Output-disjoint0 / 0 / 00 / 0 / 045.7
  • (ii) Effect of Activation Covariance. We further investigate the relationship in following Table 3 between activation covariance and residual output interference. Under the original activations, the off-diagonal covariance measure is 0.995462, with an output overlap of 0.011309 ± 0.008742. After whitening, these values decrease to 4.41×10−64.41 \times 10^{-6} and 1.31×10−91.31 \times 10^{-9}, respectively. These findings are consistent with our theoretical analysis that residual functional interference depends on activation covariance, providing empirical support for its relevance to practical Transformer layers.

Table 3. Effect of activation whitening on residual output interference in Transformer layers.

Input ConditionActivation CovarianceCoordinate-Disjoint Output Overlap
Original activations0.9954620.011309 ± 0.008742
Whitened activations4.41 × 10⁻⁶1.31 × 10⁻⁹

W1 & Q1 & Q2: Theory Validation and Practical Performance

2. Why the geometry covers the implemented aggregation and final updates.

The actual merge averages each coordinate over its occupants: $ \bar F=\sum_k\Gamma_kF_k, $ where Γk\Gamma_k is diagonal in the coefficient basis. Under disjoint supports, its occupied-coordinate weights equal one, so the merge becomes a sum.

Substituting Ak=ΓkFkA_k=\Gamma_kF_k into the identity above gives the interaction between the actual aggregated contributions. Since the averaging weights lie in [0,1][0,1], the relevant coefficient energies are bounded by those of FkF_k. Thus, the same upper bound applies using the original coefficient norms; a convex combination of complete client models is unnecessary.

These identities involve only the final matrices and a common activation batch, so they hold after arbitrary local training, including 2,000 steps. Their interpretation in global loss is a separate question. For final parameter contributions vkv_k, let u=∑kvku=\sum_kv_k. Taylor expansion gives

F(w+u)−F(w)=g⊤u+12∑kvk⊤Hvk+∑i<jvi⊤Hvj+R,F(w+u)-F(w)=g^\top u+\tfrac12\sum_kv_k^\top Hv_k+\sum_{i<j}v_i^\top Hv_j+R,

where g=∇F(w)g=\nabla F(w) and H=∇2F(w)H=\nabla^2F(w). If the Hessian is LHL_H-Lipschitz along the update segment, ∣R∣≤LH∥u∥3/6.|R|\le L_H\|u\|^3/6. Thus, the geometric identity is independent of local step count, while the quadratic risk approximation depends on update size and curvature variation.

For a squared-loss linear layer, the curvature interaction equals layer-output interaction exactly. For a cross-entropy Transformer, downstream and loss curvature also contribute. We will make this distinction explicit.

3. A simple refinement: use the manuscript’s independent row-block allocation.

Each client independently selects mm output rows and samples BB coefficients within those rows. This retains the sparse parameterization, coefficient budget, index-wise merge, and single upload.

Proposition. Fix a reference activation second moment. Suppose client supports are sampled independently of one another and the data, and each client learns its coefficients from its own support and local data. Then row-block allocation satisfies E[∣⟨Zi,Zj⟩F∣∥Fi∥F∥Fj∥F]≤σmax⁡BD+∥E∥2mdo,\mathbb{E}\left[\frac{\left\lvert\left\langle Z_i,Z_j\right\rangle_F\right\rvert}{\lVert F_i\rVert_F\lVert F_j\rVert_F}\right]\le\sigma_{\max}\frac{B}{D}+\lVert E\rVert_2\frac{m}{d_o},where D=dodiD=d_od_i, B≤mdiB\le md_i, and ZkZ_k is the output contribution after index-wise weighting. Zero-update pairs contribute zero. The normalization uses the original coefficient norms.

Derivation. Conditional on client ii’s coefficients, client jj’s independent draw includes a particular coordinate with probability Pr⁡[(u,v)∈Sj]=mdoBmdi=BD.\Pr[(u,v)\in S_j]=\frac m{d_o}\frac B{md_i}= \frac BD. For nonzero FiF_i, summing its fixed coefficient energies against these inclusion probabilities gives $\mathbb E[\theta_i\mid F_i]= \frac{\sum_{u,v}|F_i(u,v)|^2(B/D)} {|F_i|_F^2} =\frac BD.$ Likewise, a particular output row is selected with probability m/dom/d_o, so $ \mathbb E[\rho_i\mid F_i]=m/d_o. $

Defining the energy fractions as zero for zero updates gives Eθi≤B/D\mathbb E\theta_i\le B/D and Eρi≤m/do\mathbb E\rho_i\le m/d_o. Therefore, $ \mathbb E\sqrt{\theta_i\theta_j} \le\sqrt{\mathbb E\theta_i,\mathbb E\theta_j}\le B/D, $ and similarly Eρiρj≤m/do\mathbb E\sqrt{\rho_i\rho_j}\le m/d_o. Substitution into the deterministic bound proves the proposition. No independence or centering of the learned coefficient entries is required.

For scattered allocation, the same argument gives the shared-row factor $ q=1-\frac{\binom{D-d_i}{B}}{\binom DB}, $ because the ratio is the probability of missing every coordinate in a particular row. Row-block allocation therefore replaces qq by m/dom/d_o, while preserving the coordinate-collision factor B/DB/D.

At B=6000B=6000, an illustrative m=4m=4 gives:

ProjectionScattered shared-row factorRow-block shared-row factor
Query, do=3584d_o=3584Approximately 0.810.00112
Value, do=512d_o=512Approximately 10.00781

These are reductions in the bound’s covariance multiplier, not proportional accuracy predictions. The existing simulation uses eight rows; four rows is a proposed setting for LLM evaluation.

This refinement connects the implementation to the stronger allocation analysis already in the manuscript. Its trade-off is explicit: rank⁡(ΔWk)≤m\operatorname{rank}(\Delta W_k)\le m, so the number of rows controls adaptation capacity and interference at the same coefficient budget.

Q3. Rationale for Choosing the DCT Basis

1. Advantages of DCT DCT parameterization offers two practical advantages: improved adaptation accuracy over original-coordinate sparsity and computational efficiency over the sparse orthogonal basis.

  • (1) Lower output overlap and higher accuracy. Compared with sparse updates in the original parameter space, DCT distributes each coefficient update across multiple weights through frequency basis functions. Under matched settings, it yields lower residual output overlap and higher accuracy, suggesting that reduced interference between client updates contributes to better adaptation.

  • (2) Computational efficiency. The DCT implementation reduces the complexity of an nn-dimensional transform from �1��1� for a random orthogonal basis to �2��2�, enabling more efficient computation.

2. Empirical validation. We compare accuracy and training time on LLaMA3.2-1B under matched experimental settings with original-coordinate sparse fine-tuning and a random orthogonal basis. Table 4 shows that DCT improves accuracy and reduces residual output overlap over both sparse fine-tuning methods, while DCT provides the low local client training time (42.4 s). The result supports DCT as a practical trade-off between accuracy and computational efficiency.

Table 4. Comparison of different parameterization strategies

MetricDCTOriginal coordinatesRandom orthogonal
Accuracy (%) ↑45.438.741.8
Output overlap ↓0.010570.018520.01243
Local training time (s) ↓42.439.073.1

Q4. Experimental Details and Statistical Reliability

We thank the reviewer for highlighting these concerns.

  • Experimental Details For MMLU-Pro evaluation, we randomly sample 1,000 questions from the official test set using three random seeds (0, 1, and 2). All methods are evaluated on identical subsets for each seed to ensure fair comparison. Following the baseline settings, we consistently adopt a learning rate of 10−210^{-2} and 2,000 local training steps across methods. The learning rate is fixed according to the baseline protocol.

  • Uncertainty estimates All main experiments are conducted independently using three random seeds. For example, Table 1 already includes statistical significance analysis to assess the reliability of the reported performance improvements. We will further clarify the random seeds and experimental settings for additional experiments to improve reproducibility.

Comments on the paper

3 comments

Esteemed Reviewers,

We sincerely thank you for the time and effort you have devoted to evaluating our work.

We hope that our responses and revisions have adequately addressed your questions and concerns. If so, we would be grateful if you could kindly consider whether the current revisions might merit a reassessment of the score. Of course, should any technical quality points remain unclear or if you have further suggestions, we would be most happy to hear from you.

Best regards,

Authors

Dear Reviewers,

Thank you to those who have already contributed to the discussion.

As the discussion period draws to a close, I’d especially appreciate hearing from Reviewers Reviewer uUEG, Reviewer P43Q, Reviewer VYfw, and Reviewer uAcA, who have yet to join the discussion. Please post a brief reply to the authors indicating which concerns their response has addressed and which, if any, remain. If all your concerns have been resolved, a short confirmation is sufficient.

Thank you for your time and input.

Best,
Your AC

Dear Area Chair,

We are grateful for your active follow-up in encouraging reviewers to revisit our responses and continue the discussion. We sincerely thank all reviewers for their careful evaluation and valuable feedback.

1. We are encouraged by their recognition of:

  • Our subspace-allocation perspective. Reviewer uUEG recognizes the formulation of one-shot federated adaptation as a “trainable subspace allocation problem,” while Reviewer VYfw notes that our analysis provides “a clearer foundation for designing client allocations.”

  • Our theoretical analysis of interference. Reviewer uAcA describes the distinction between weight-space and layer-output interference as a “meaningful perspective.” Reviewer uUEG also acknowledges the output-orthogonality guarantees and interference bounds without distributional assumptions on learned coefficients.

  • Our communication–performance trade-off. All four reviewers recognize this strength. Reviewer P43Q highlights approximately 5.8–14.2× fewer uploaded parameters while maintaining competitive or superior accuracy.

We also appreciate Reviewer uAcA’s confirmation that most concerns have been resolved and their decision to raise the score to weak acceptance.


2. Summary of Rebuttal and Discussions

For convenience, we summarize the key questions and how we addressed them.

Key questionResponse
Theory–practice connection and empirical validation (uUEG, P43Q, VYfw, uAcA)1. We clarified applicability to index-wise aggregation and final trained updates, distinguishing layer-level identities from loss-level approximations. 2. We added LLM-layer overlap measurements and whitening experiments supporting the predicted orthogonality and covariance-dependent interference. 3. We provided sufficient conditions for expected-risk improvement in a quadratic setting.
Interference versus parameter coverage (P43Q, VYfw)Under matched per-client budgets and global coefficient coverage, coordinate-disjoint allocation exhibits small residual output overlap and accuracy close to the zero-output-interference reference across three seeds.
DCT justification (uUEG, uAcA)We added matched-budget comparisons with original-coordinate sparsity and random orthogonal bases. Accuracy, output-overlap, and training-time results support DCT as a practical choice.
Related approaches (P43Q, VYfw)We clarified methodological differences and added comparisons with FourierFT, LoCA, Meerkat, OSRM, and YOCO.
Experimental details (uUEG, VYfw, uAcA)We specified evaluation sampling and random seeds, reported per-seed accuracy differences, and corrected the learning-rate description to distinguish our fixed rate from baseline-specific settings.

3. Summary of Methodological and Theoretical Contributions

Methodological novelty. In one-shot federated learning, different clients may update the same parameter directions, causing interference during aggregation. We propose sparse orthogonal subspace allocation to reduce this overlap while keeping the number of trainable parameters small. Comparisons across scattered-coordinate, coordinate-disjoint, and output-index-disjoint allocation show that reducing weight-space overlap suppresses output interference, making independently trained client updates more compatible for single-round aggregation.

Theoretical novelty. We provide theoretical support for the orthogonal subspace allocation in our method by decomposing excess risk into capture, noise, and interference terms. Our theoretical analysis and empirical results support sparse orthogonal allocation as a way to reduce cross-client interference while retaining useful updates and limiting training noise. We further characterize residual output interference through covariance-dependent bounds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.