- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Paper Summary
- FedSALT addresses heterogeneous federated LoRA training by measuring how client updates diverge from a historical consensus output subspace.
- It converts factorization-invariant alignment scores into adaptive client- and module-specific proximal strengths, guiding subsequent local updates toward a momentum-based reference.
- Experiments on three heterogeneous benchmarks report improved average accuracy, supported by component ablations and an idealized convergence analysis.
Critical strengths
Critical Strengths
- Clear geometric motivation. The paper identifies stronger task differentiation in output-side than input-side update geometry and uses this observation to guide selective local regularization.
- Principled drift measurement. The alignment score operates on realized adapter updates, making it invariant to equivalent LoRA factorizations. Computation remains in low-rank form without constructing dense updates.
- Informative mechanism ablations. Input-side sensing, matched-mean uniform control, permuted assignments, and zero-centered targets help distinguish the contributions of geometry-aware allocation and momentum guidance.
Critical weaknesses
Critical Weaknesses
- The theory does not cover the implemented algorithm. The analysis assumes full participation, optimization directly in realized-adapter space, and exact product-space aggregation. The implementation instead trains LoRA factors and uses factor-wise FedAvg, leaving both factorization geometry and aggregation mismatch outside the guarantee.
- Empirical gains are modest and narrowly validated. Evaluation uses only Llama-3.2-1B, rank 16, and eight-client federations. Improvements over the strongest baseline are approximately 0.3, 0.5, and 0.3 percentage points across the three benchmarks; broader settings and paired statistical comparisons would strengthen the evidence.
- Practical trade-offs need further evaluation. Table 4 reports 1.50× communication and 6.3% runtime overhead relative to FedAvg. Sensitivity to controller parameters and reference ranks, together with accuracy comparisons under matched communication or time budgets, is needed to establish whether the gains justify the additional complexity.
Questions for authors
Please refer to weakness.
4 comments
Gentle Follow-up for Reviewer 1Lia on Our Rebuttal
Dear Reviewer 1Lia,
Thank you again for your review. As the discussion period closes today, we would like to briefly summarize our response.
- Theory. Full participation and exact averaging are the standard setting of federated optimization analyses (e.g., Li et al., ICLR 2020; Stich, ICLR 2019). Our analysis is carried out in the realized-adapter space, where FedSALT senses and controls drift, and because FedSALT changes only the local objective while keeping factor-wise FedAvg, the analysis isolates exactly its contribution.
- Strength of the gains. FedSALT is more accurate than every baseline on every seed, with statistically significant improvements, and remains the most accurate method with Qwen2.5-1.5B, Llama-3.2-3B, and 12 clients.
- Trade-offs. Under a matched wall-clock budget, FedSALT remains the most accurate method. It leaves the uplink unchanged, its total communication remains below SCAFFOLD, and it is robust to its controller parameters and reference rank.
We sincerely appreciate your time and would be grateful for any feedback on whether our response addresses your concerns. If it does, we would be grateful if you would kindly consider updating your rating.
Thank you for your response. I will keep my score.
Thank you
We thank the reviewer for reading our response.
We will incorporate the theoretical discussion, the paired comparison, the results with additional backbones and 12 clients, the matched time budget comparison, and the sensitivity analysis into the revised paper.
Rebuttal to Reviewer 1Lia
We thank the reviewer for highlighting the clear geometric motivation, the principled drift measurement, and the informative mechanism ablations.
Tables report client-average test accuracy (%) as mean ± standard deviation over three seeds.
W1
Standard analytical setting. Full participation and exact averaging of client updates are the standard setting of federated optimization analyses. For example, Li et al. [1, Theorem 1] analyze FedAvg under full participation with weighted model averaging, and Stich [2, Algorithm 1] analyzes local SGD with periodic exact averaging.
Analysis in the space where FedSALT acts. Both the factorization-invariant alignment score and the adaptive proximal objective are defined on the realized LoRA update, so the realized-adapter space is exactly where FedSALT senses and controls drift. In this space, Proposition 1 establishes that stronger proximal control contracts the leading off-subspace component of local drift, and Theorem 1 gives a nonconvex stationarity bound that explicitly quantifies selection bias, client heterogeneity, and momentum-anchor effects.
Factor-wise aggregation. FedSALT changes only the local objective and keeps standard factor-wise FedAvg aggregation, so the analysis isolates exactly the component that FedSALT contributes. We will add this discussion and the references to Section 4.
[1] Li et al., On the Convergence of FedAvg on Non-IID Data, ICLR 2020.
[2] Stich, Local SGD Converges Fast and Communicates Little, ICLR 2019.
W2
Paired comparison. The table below reports a paired comparison on MC-8 over three seeds. FedSALT is more accurate than every baseline on every seed, and the improvement is statistically significant against every baseline.
Additional backbones. The table below reports MC-8 with Qwen2.5-1.5B and Llama-3.2-3B under the protocol of Section 5. FedSALT remains the most accurate method on both backbones.
More clients. MC-12 extends MC-8 to 12 clients. FedSALT remains the most accurate method with more clients.
We will include the results with the additional backbones and with MC-12 in the revision.
W3
Matched time budget. We match wall-clock time on MC-12 by limiting FedSALT to the time FedAvg needs for its full training, computed from the per-round runtime in Table 4, while every baseline keeps its full training. Within this budget, FedSALT reaches the same accuracy as in the MC-12 table under W2 and remains the most accurate method.
Communication. FedSALT leaves the uplink unchanged, and its additional cost is a single downlink broadcast of one low-rank reference shared by all clients. Its total communication remains below SCAFFOLD, and its runtime overhead is smaller than that of FedProx, FedRot-LoRA, SCAFFOLD, and FedSA-LoRA (Table 4).
Sensitivity. The table below changes one controller parameter or the reference rank at a time on MC-8. FedSALT is robust to its controller parameters and reference rank.