- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies memory-efficient federated fine-tuning of large language models (LLMs), aiming to avoid backpropagation through the LLM backbone while retaining first-order optimization. The authors propose FoRA, which extracts detached multi-level representations from a frozen backbone and trains a lightweight Split-Bridge residual network. A global-reference regularization mechanism is further introduced to reduce local model drift under heterogeneous client data. The paper provides theoretical analyses of the proposed framework and evaluates it on classification and generation tasks. Experimental results show that FoRA achieves comparable or better performance than existing baselines while reducing client GPU memory usage and training time.
Critical strengths
-
The paper addresses an important challenge in federated LLM fine-tuning: reducing client-side training memory while preserving effective first-order optimization. The idea of separating the trainable adaptation modules from the frozen LLM backbone is well motivated and provides a practical direction for improving training efficiency.
-
The proposed framework combines multi-level representation extraction, residual adaptation, and global-reference regularization in a coherent way. The experiments cover different model sizes, classification and generation tasks, as well as client scalability and data heterogeneity.
Critical weaknesses
-
FoRA emphasizes that clients only upload auxiliary model parameters without transmitting intermediate activations, which is an important difference from some existing federated side-network methods. However, parameter communication does not necessarily lead to lower overall communication costs. The paper could provide a clearer analysis of when parameter communication is more efficient, considering factors such as auxiliary model size, training rounds, and local dataset size.
-
The paper highlights that FoRA avoids backbone backpropagation while retaining first-order optimization, and reports clear improvements over FedMeZO in both performance and training time. However, the two methods differ not only in their optimization strategies but also in their trainable architectures and parameterizations. Therefore, the current comparison does not clearly show whether the improvements mainly come from first-order optimization or from FoRA's multi-level representations and Split-Bridge residual design.
-
One of FoRA's main contributions is the combination of a Split branch and multiple Bridge branches for residual adaptation. However, this structure can be represented by a single wider adapter at a fixed parameter state. This suggests that the benefit of Split-Bridge may come more from its parameterization and optimization behavior than from greater representational capacity.
-
The paper is somewhat difficult to follow due to its dense notation, terminology, and method descriptions. In particular, Figure 1 is too small and contains many labels that are difficult to read, with some symbols appearing incorrectly rendered or garbled. Improving the clarity of the method description and the quality of the overview figure would make the paper easier to understand.
Questions for authors
-
Could the authors further analyze how auxiliary model size, the number of training rounds, and local dataset size affect the overall communication cost? It would be helpful to clarify under which conditions FoRA's parameter communication is more efficient than activation-based communication methods.
-
Could the authors compare first-order and zeroth-order optimization while keeping the same multi-level representations, auxiliary architecture, and parameter budget? Such a comparison would help clarify how much of FoRA's performance improvement comes from first-order optimization rather than differences in model architecture.
-
Could the authors further explain the key advantages of Split-Bridge over simpler multi-level side networks? In particular, it would be helpful to extend the comparison with single-branch or wider adapter architectures under similar parameter and computation budgets to more models and tasks, to better demonstrate the necessity of the factorized design.
4 comments
Rebuttal-2
Advantages of Split-Bridge (W3/Q3)
We choose the Split-Bridge structure instead of a simpler multi-level side network because it improves how multi-level corrections are learned and aggregated during federated training. We have analyzed this from both theoretical and empirical perspectives.
- Theoretical analysis. Compared with a simpler multi-level side network, Split-Bridge offers separate control over complementary corrections and their relative contributions during local learning and federated aggregation. Folding the Split-Bridge structure into a simpler multi-level side network preserves the representational capacity while absorbing the mixing weights into the wide adapter's output projections. Updating and averaging these merged projections generally differs from learning and aggregating the separate mappings and weights (Appendix C.4, Eq. (106)). Consequently, even from identical initial predictions, the two parameterizations can follow different local learning trajectories and produce different global corrections after aggregation.
- Existing evidence. We have tested the advantage of the Split-Bridge structure over a simpler multi-level side network on OPT-350M@MRPC (Table 10). FoRA improves Final-10 accuracy over Folded-Wide by 5.38 percentage points (75.68% versus 70.30%), with the same multi-level features, identical initial residual functions, comparable parameter counts (4.067M versus 3.944M), and identical peak GPU memory.
To further address the reviewer's concern, we extend the same Folded-Wide control (Appendix D.6) to OPT-2.7B@SST-2 under the Table 1 training protocol.
| Method | Best accuracy (%) | Best F1 (%) | Final-10 accuracy (%) | Peak GPU memory (GiB) |
|---|---|---|---|---|
| FoRA | 96.22 | 96.30 | 96.00 | 10.157 |
| Folded-Wide | 95.37 | 95.46 | 95.04 | 10.157 |
FoRA improves Final-10 accuracy by 0.96 percentage points at the same peak GPU memory, extending the Split-Bridge advantage to a second model and task.
Readability (W4)
We have tried our best to reproduce the rendering issue in the submitted PDF, but have not yet been able to do so after checking it in four commonly used tools: Chrome, Microsoft Edge, Microsoft Office, and VS Code with a PDF reader extension. Therefore, we believe that one of these tools may offer a useful alternative to display Figure 1 correctly. Any additional details about the affected symbols and viewing environment would help us reproduce the issue and work toward a solution.
Thank you for your time and thoughtful consideration.
Thank you for your response and clarifications. However, given the four-point rating scale and my overall assessment of the paper's quality, I will maintain my original score.
Dear Reviewer 1t2q,
We sincerely thank you for your time and the thorough, technically grounded feedback you provided throughout the discussion phase. Your four core concerns (communication cost scaling, first-order vs. architectural disentanglement, Split-Bridge design necessity, and manuscript readability) have directly helped us strengthen the work substantially.
To further consolidate the improvements we promised in the rebuttal, we will make three targeted revisions to the manuscript:
- Add a dedicated new subsection in the Complexity Analysis part, summarizing the three key conditions (small auxiliary model size, moderate communication rounds, large local dataset size) where FoRA's parameter-only communication outperforms activation-based federated methods, with the full theoretical scaling formula and two-task empirical results presented together for clarity.
- Place the controlled FO/ZO comparison table (fixed architecture across optimization strategies) in the main paper instead of the appendix, making it much easier for readers to distinguish the performance gains contributed by first-order optimization and the gains brought by the Split-Bridge multi-level design.
- Add the new OPT-2.7B@SST-2 Folded-Wide ablation result directly after the original OPT-350M comparison, to show that the advantage of the Split-Bridge structure generalizes across different model sizes and tasks under similar parameter and memory budgets.
We will also simplify the notation and reduce its density in the method section, improve Figure 1’s layout and label sizes and verify its rendering in commonly used PDF viewers, and add a one-paragraph overview at the start of Section 4 to help readers follow the framework more smoothly.
Your feedback has made this work much more rigorous and easier to follow. We really appreciate the careful and constructive review.
Sincerely,
The Authors
Rebuttal-1
We thank Reviewer 1t2q for the constructive feedback. Below, we address each concern.
Communication cost versus federated side-network methods (W1/Q1)
We have analyzed communication cost relative to existing federated side-network methods from both theoretical and empirical perspectives. Specifically, FoRA's traffic grows with auxiliary-model size and communication rounds, but not directly with local dataset size. Smaller auxiliary models and fewer rounds therefore favor FoRA, especially when larger local datasets make cumulative activation traffic exceed parameter traffic.
First-order optimization and architectural gains (W2/Q2)
FoRA's performance benefits from both first-order gradients and multi-level Split-Bridge adaptation. We compare LoRA-FO (FedIT), LoRA-ZO (FedMeZO), FoRA-FO, and FoRA-ZO on OPT-350M@MRPC following the training protocol used for Table 1 of the original paper. Each FO/ZO pair keeps the architecture, parameter count, initialization, local objective, and federated protocol fixed.
Results show: