acceptodds
Under review as a conference paper at ICLR 2027

FoRA: Forward-Only Residual Adaptation for Federated Fine-Tuning of Large Language Models

Abstract

Federated fine-tuning of large language models (LLMs) faces a tension between avoiding client-side backpropagation through the LLM backbone and retaining first-order optimization for effective adaptation. Existing methods either keep trainable modules inside the backbone, preserving first-order gradients at the cost of backbone backpropagation, or adopt zeroth-order optimization for forward-only execution at the cost of gradient-estimation error. We propose FoRA, a federated fine-tuning framework that keeps the LLM backbone forward-only while preserving first-order optimization for the trainable adaptation. FoRA detaches compact multi-level representations from a frozen backbone, learns a lightweight factorized residual with a direct Split branch and a globally weighted Bridge branch, and regularizes local training against a fixed global reference in function space. Our analysis verifies the execution-graph memory separation, characterizes the local Jacobian geometry of the auxiliary parameterization, characterizes the local dynamics induced by global-reference regularization under a fixed-tangent model, and establishes non-convex convergence for the federated protocol. Empirically, FoRA reduces peak allocated client GPU memory by 27.3%-59.2% over first-order baselines while achieving comparable quality, and outperforms FedMeZO by 6.99-18.37 points in Final-10 Accuracy on classification and 4.74 points in Final-10 ROUGE-L on generation, with 65.8%-74.7% less training time.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Federated Learning, Large Language Models, Parameter-Efficient Fine-Tuning, Memory-Efficient Training, First-Order Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1t2qOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies memory-efficient federated fine-tuning of large language models (LLMs), aiming to avoid backpropagation through the LLM backbone while retaining first-order optimization. The authors propose FoRA, which extracts detached multi-level representations from a frozen backbone and trains a lightweight Split-Bridge residual network. A global-reference regularization mechanism is further introduced to reduce local model drift under heterogeneous client data. The paper provides theoretical analyses of the proposed framework and evaluates it on classification and generation tasks. Experimental results show that FoRA achieves comparable or better performance than existing baselines while reducing client GPU memory usage and training time.

Critical strengths
  1. The paper addresses an important challenge in federated LLM fine-tuning: reducing client-side training memory while preserving effective first-order optimization. The idea of separating the trainable adaptation modules from the frozen LLM backbone is well motivated and provides a practical direction for improving training efficiency.

  2. The proposed framework combines multi-level representation extraction, residual adaptation, and global-reference regularization in a coherent way. The experiments cover different model sizes, classification and generation tasks, as well as client scalability and data heterogeneity.

Critical weaknesses
  1. FoRA emphasizes that clients only upload auxiliary model parameters without transmitting intermediate activations, which is an important difference from some existing federated side-network methods. However, parameter communication does not necessarily lead to lower overall communication costs. The paper could provide a clearer analysis of when parameter communication is more efficient, considering factors such as auxiliary model size, training rounds, and local dataset size.

  2. The paper highlights that FoRA avoids backbone backpropagation while retaining first-order optimization, and reports clear improvements over FedMeZO in both performance and training time. However, the two methods differ not only in their optimization strategies but also in their trainable architectures and parameterizations. Therefore, the current comparison does not clearly show whether the improvements mainly come from first-order optimization or from FoRA's multi-level representations and Split-Bridge residual design.

  3. One of FoRA's main contributions is the combination of a Split branch and multiple Bridge branches for residual adaptation. However, this structure can be represented by a single wider adapter at a fixed parameter state. This suggests that the benefit of Split-Bridge may come more from its parameterization and optimization behavior than from greater representational capacity.

  4. The paper is somewhat difficult to follow due to its dense notation, terminology, and method descriptions. In particular, Figure 1 is too small and contains many labels that are difficult to read, with some symbols appearing incorrectly rendered or garbled. Improving the clarity of the method description and the quality of the overview figure would make the paper easier to understand.

Questions for authors
  1. Could the authors further analyze how auxiliary model size, the number of training rounds, and local dataset size affect the overall communication cost? It would be helpful to clarify under which conditions FoRA's parameter communication is more efficient than activation-based communication methods.

  2. Could the authors compare first-order and zeroth-order optimization while keeping the same multi-level representations, auxiliary architecture, and parameter budget? Such a comparison would help clarify how much of FoRA's performance improvement comes from first-order optimization rather than differences in model architecture.

  3. Could the authors further explain the key advantages of Split-Bridge over simpler multi-level side networks? In particular, it would be helpful to extend the comparison with single-branch or wider adapter architectures under similar parameter and computation budgets to more models and tasks, to better demonstrate the necessity of the factorized design.

4 comments
AuthorsOpenReview ↗

Rebuttal-1

We thank Reviewer 1t2q for the constructive feedback. Below, we address each concern.

Communication cost versus federated side-network methods (W1/Q1)

We have analyzed communication cost relative to existing federated side-network methods from both theoretical and empirical perspectives. Specifically, FoRA's traffic grows with auxiliary-model size and communication rounds, but not directly with local dataset size. Smaller auxiliary models and fewer rounds therefore favor FoRA, especially when larger local datasets make cumulative activation traffic exceed parameter traffic.

  1. Theoretical analysis. Each participating FoRA client uploads swPaux+O(1)s_wP_{\mathrm{aux}}+O(1) bytes per round and downloads the same amount (Appendix B.3, Eqs. (29)-(30)), where PauxP_{\mathrm{aux}} is the auxiliary parameter count and sws_w is the number of bytes per parameter. With fixed client participation per round and transmission precision, these equations show:
    • Auxiliary-model size: FoRA's cumulative parameter traffic grows linearly with PauxP_{\mathrm{aux}}.
    • Training rounds: Each round exchanges one auxiliary state in each direction, so cumulative traffic grows linearly with the number of rounds.
    • Local dataset size: Neither expression depends on the number of local samples, so FoRA's communication cost is independent of local dataset size when the auxiliary model and participation schedule are fixed. Fed MobiLLM instead uploads activations and output residuals by minibatch, so its traffic grows with processed samples and their activation payload (Appendix D.5.1).
  2. Empirical evidence. Table 9 (Appendix D.5.1) confirms these scaling differences. On OPT-350M@MRPC, FoRA's 200 parameter-exchange rounds yield 15.150 GiB uplink, whereas the small dataset keeps Fed MobiLLM's activation traffic at 0.333 GiB. On OPT-2.7B@SST-2, the larger dataset increases Fed MobiLLM's activation uploads, while FoRA uses 50 rounds, reducing uplink by 38.1% (12.603 versus 20.350 GiB). However, FoRA achieves the best task performance in both settings, improving Final-10 accuracy over Fed MobiLLM by 9.86 and 6.07 percentage points, respectively.

First-order optimization and architectural gains (W2/Q2)

FoRA's performance benefits from both first-order gradients and multi-level Split-Bridge adaptation. We compare LoRA-FO (FedIT), LoRA-ZO (FedMeZO), FoRA-FO, and FoRA-ZO on OPT-350M@MRPC following the training protocol used for Table 1 of the original paper. Each FO/ZO pair keeps the architecture, parameter count, initialization, local objective, and federated protocol fixed.

MethodTrainable parameters (M)Final-10 accuracy (%)Training time (h)
LoRA-FO (FedIT)1.57473.7112.14
LoRA-ZO (FedMeZO)1.57468.6932.09
FoRA-FO4.06775.688.59
FoRA-ZO4.06772.2830.51

Results show:

  1. For first-order gradients: With the architecture fixed, switching from ZO to FO improves Final-10 accuracy by 5.02 percentage points for LoRA and 3.40 for FoRA, showing the benefit of first-order optimization in both architectures.
  2. For multi-level Split-Bridge adaptation: With the optimization strategy fixed, FoRA outperforms LoRA by 1.97 percentage points under FO and 3.59 under ZO. Table 3 further supports this design contribution: with FO unchanged, using only the last layer, removing Split, or removing Bridge lowers MRPC peak accuracy by 7.05, 7.57, and 4.95 percentage points, respectively.
AuthorsOpenReview ↗

Rebuttal-2

Advantages of Split-Bridge (W3/Q3)

We choose the Split-Bridge structure instead of a simpler multi-level side network because it improves how multi-level corrections are learned and aggregated during federated training. We have analyzed this from both theoretical and empirical perspectives.

  1. Theoretical analysis. Compared with a simpler multi-level side network, Split-Bridge offers separate control over complementary corrections and their relative contributions during local learning and federated aggregation. Folding the Split-Bridge structure into a simpler multi-level side network preserves the representational capacity while absorbing the mixing weights into the wide adapter's output projections. Updating and averaging these merged projections generally differs from learning and aggregating the separate mappings and weights (Appendix C.4, Eq. (106)). Consequently, even from identical initial predictions, the two parameterizations can follow different local learning trajectories and produce different global corrections after aggregation.
  2. Existing evidence. We have tested the advantage of the Split-Bridge structure over a simpler multi-level side network on OPT-350M@MRPC (Table 10). FoRA improves Final-10 accuracy over Folded-Wide by 5.38 percentage points (75.68% versus 70.30%), with the same multi-level features, identical initial residual functions, comparable parameter counts (4.067M versus 3.944M), and identical peak GPU memory.

To further address the reviewer's concern, we extend the same Folded-Wide control (Appendix D.6) to OPT-2.7B@SST-2 under the Table 1 training protocol.

MethodBest accuracy (%)Best F1 (%)Final-10 accuracy (%)Peak GPU memory (GiB)
FoRA96.2296.3096.0010.157
Folded-Wide95.3795.4695.0410.157

FoRA improves Final-10 accuracy by 0.96 percentage points at the same peak GPU memory, extending the Split-Bridge advantage to a second model and task.

Readability (W4)

We have tried our best to reproduce the rendering issue in the submitted PDF, but have not yet been able to do so after checking it in four commonly used tools: Chrome, Microsoft Edge, Microsoft Office, and VS Code with a PDF reader extension. Therefore, we believe that one of these tools may offer a useful alternative to display Figure 1 correctly. Any additional details about the affected symbols and viewing environment would help us reproduce the issue and work toward a solution.

Thank you for your time and thoughtful consideration.

Thank you for your response and clarifications. However, given the four-point rating scale and my overall assessment of the paper's quality, I will maintain my original score.

Dear Reviewer 1t2q,

We sincerely thank you for your time and the thorough, technically grounded feedback you provided throughout the discussion phase. Your four core concerns (communication cost scaling, first-order vs. architectural disentanglement, Split-Bridge design necessity, and manuscript readability) have directly helped us strengthen the work substantially.

To further consolidate the improvements we promised in the rebuttal, we will make three targeted revisions to the manuscript:

  • Add a dedicated new subsection in the Complexity Analysis part, summarizing the three key conditions (small auxiliary model size, moderate communication rounds, large local dataset size) where FoRA's parameter-only communication outperforms activation-based federated methods, with the full theoretical scaling formula and two-task empirical results presented together for clarity.
  • Place the controlled FO/ZO comparison table (fixed architecture across optimization strategies) in the main paper instead of the appendix, making it much easier for readers to distinguish the performance gains contributed by first-order optimization and the gains brought by the Split-Bridge multi-level design.
  • Add the new OPT-2.7B@SST-2 Folded-Wide ablation result directly after the original OPT-350M comparison, to show that the advantage of the Split-Bridge structure generalizes across different model sizes and tasks under similar parameter and memory budgets.

We will also simplify the notation and reduce its density in the method section, improve Figure 1’s layout and label sizes and verify its rendering in commonly used PDF viewers, and add a one-paragraph overview at the start of Section 4 to help readers follow the framework more smoothly.

Your feedback has made this work much more rigorous and easier to follow. We really appreciate the careful and constructive review.

Sincerely,

The Authors

Reviewer_TDJwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

FoRA is a federated fine-tuning framework for LLMs which can run the backbone using only forward pass. The framework can still updating the parameters using first-order gradients. During the updating, each client will takes the pooled hidden states from every b-th layer of the original frozen backbone. This can cuts the gradient, and then it trains a tiny residual correction component on the top of the frozen logits of the model output.

Critical strengths
  • In first-order PEFT it usually needs the backpropagation through the original backbone. In comparison, the zeroth-order methods aim to avoid the backpropagation. However such methods rely on relative noisy gradient estimation. The paper aims to address this tradeoff. The proposed method is simple and looks easy to reimplement. Specifically, the training stays on the client part, and only adapter parameters are communicated, and then follow the standard federated learning, the server will aggregate these parameters.

  • Evaluation is good. All baseline methods and the proposed methods both share partitions and client schedules. All results average three seeds.

  • The paper also includes sensitivity analysis which covers six hyperparameters, and scaling to 50 clients on three heterogeneity levels and different component ablations.

Critical weaknesses
  • Since the backbone of the model is frozen and the features blocks are detached, the proposed method in fact reduces to federated training with a tiny multi-layer adapter on top of the fixed features then combined with the correction which applied to the frozen logits of the model's output. Unfortunately this is a well-established pipeline that many existing works already describes, the novelty of the proposed framework seems limited.

  • In tasks like MRPC, SST-2 and Dolly, frozen features itself already work well. Also two of the three backbones are OPT models. It seems the paper did not discuss and tests reasoning, knowledge or code. In these tasks a residual on frozen representations maybe fall short of LoRA.

Questions for authors
  • Because of the backbone are frozen and the features accordingly are detached, each client can compute the features once only and then cache them. The question is will the backbone re-run at every local step? And if the features were cached, how would training time compare? Also how would FoRA differ from a federated multi-layer probe methods?

  • What is the per-round and total uplink for the proposed method FoRA compared with LoRA baseline in all three settings configured in the paper?

  • How does the proposed method FoRA performs on difficult tasks (such as the GSM8K, MMLU, code, or instruction following judged by an LLM or by humans) and on newer backbones? Can you discuss where does a frozen-representation residual could fall behind the LoRA?

6 comments
AuthorsOpenReview ↗

Rebuttal-1

We thank Reviewer TDJw for the constructive feedback. Below, we address each concern.

Novelty (W1)

FoRA is novel in both its research objective and algorithm design.

  1. Novelty in Research objective. FoRA studies how to preserve effective client-side first-order adaptation while removing the LLM's backward-memory cost and requiring no proxy or public data. By contrast, existing methods each leave a key cost unresolved: LoRA-based FL still incurs backbone backpropagation and its memory cost, Fed MobiLLM incurs activation communication that scales with local data, and FedProxy depends on a proxy built from public data instead of directly adapting the target LLM (Table 4, Appendix A).
  2. Novelty in Algorithm design. FoRA introduces two complementary designs to address how a memory-efficient side network can efficiently extract and use more task-relevant knowledge from a frozen LLM and how to control prediction drift during non-IID local training:
    • Split-Bridge (§4.3). Layer-specific Split adapters and separately parameterized Bridge banks learn complementary corrections from multiple depths. The key design is to retain the banks and their learned mixture weights as separate variables during both local updates and FedAvg. In Table 10 (Appendix D.6), directly disabling this structure by training a Folded-Wide side network lowers MRPC Final-10 accuracy from 75.68% to 70.30%, demonstrating the effectiveness of the Split-Bridge structure.
    • Global Reference (§4.4). We stabilize local residual learning by constraining each client's correction against the round's fixed global residual. This couples adaptation and drift control in prediction space, with the broadcast parameters providing both initialization and the reference. Without this constraint, shared initialization alone does not control subsequent prediction drift during local updates. Removing it lowers MRPC peak accuracy from 76.11% to 74.16% (Table 3).

Effectiveness on harder tasks (W2/Q3)

FoRA remains effective on harder tasks. It achieves the highest utility performance in all three original settings (Table 1) and the recommended GSM8K experiment. The only potential gap to LoRA would arise if the required adaptation exceeds what a lightweight residual can learn from existing multi-layer representations. Although such a performance limit remains theoretically possible, increasing the bottleneck width or number of Bridge banks may still help mitigate it by expanding residual capacity.

We extend the evaluation to a harder task (GSM8K) on a newer backbone (Qwen3-8B-base), using 10 clients with non-IID data, 50% participation per round, one local epoch, and a batch size of 2. We compare FoRA with the frozen backbone, FedIT (standard federated LoRA), and GMFL (second-best overall utility in Table 1). On this newer backbone and harder task, FoRA outperforms FedIT by 1.6 percentage points in exact match while reducing training time by 51.6% and peak GPU memory by 26.7%.

MethodPPL↓Exact Match (%)↑Training time (h)↓Peak training GPU memory (GiB)↓
Frozen Qwen3-8B-base1.65086.4N/AN/A
FedIT1.141992.285.0850.098
GMFL1.142591.674.5650.070
FoRA1.136393.841.1636.697
AuthorsOpenReview ↗

Rebuttal-2

Cached features (Q1)

The backbone is not re-run at every local step in our OPT experiments. Each participating client caches the frozen logits and multi-layer features once per communication round and reuses them across local updates (Appendix B.2-B.3). Under this cached setting, we include feature extraction and cache construction before local optimization in the measured local training time (Appendix D.8). With these costs included, FoRA takes 8.59 h versus FedIT's 12.14 h on MRPC and 126.87 h versus 361.09 h on SST-2, reducing training time by 29.2% and 64.9%, respectively (Table 1).

Differences from a federated multi-layer probe (Q1)

FoRA differs from a basic federated multi-layer probe in prediction, parameterization and aggregation, and the local training objective, as summarized below.

AspectFederated multi-layer probeFoRA
PredictionA learned head directly predicts from frozen multi-layer featuresA learned residual corrects the frozen backbone logits (§3.2)
Parameterization and aggregationTrain the prediction head and average its parameters with FedAvgLocally optimize Split adapters, Bridge banks, and mixture logits as separate parameters, then aggregate them with FedAvg (§4.3)
Local training objectiveTask lossTask loss with a penalty on local residual-output drift from the round's fixed global residual (§4.4)

Per-round and total uplink (Q2)

The main evaluation in our paper focused on client training memory, time, and task quality, with communication cost analyzed in Appendix B.3 and uplink reported in the side-network comparisons (Tables 9-10). We further extend the original uplink measurements to all LoRA baselines across the three settings, reporting per-round and total costs below. Results show that at the cost of increased uplink, FoRA accommodates more trainable parameters with lower training GPU memory and higher task quality. On MRPC, total uplink is 15.150 GiB versus FedIT's 5.862 GiB, while peak training GPU memory drops by 58.9% and Final-10 accuracy rises from 73.71% to 75.68% (Table 1). This trade-off supports high-quality training on clients with tighter memory budgets.

SettingMethodUplink per client per round (MiB)Total uplink, all clients (GiB)
OPT-350M@MRPCFedIT6.0035.862
RAVAN5.9375.798
GMFL[1.501,6.003]2.960
FoRA15.51315.150
OPT-2.7B@SST-2FedIT20.0204.887
RAVAN19.9894.880
GMFL[10.010,20.020]3.226
FoRA51.62112.603
Llama-3.1-8B@Dolly-15KFedIT26.06.348
RAVAN25.9466.334
GMFL[13.000,26.000]4.189
FoRA81.88519.991

Note: GMFL's per-round range reflects progressive freezing, and its total is calculated by summing uploads over the freezing schedule.

We sincerely appreciate your feedback!

Thank you for the detailed rebuttal and the additional results, which addressed some of my concerns in Q1 and Q2. For the GSM8K, the frozen backbone already reaches 86.4, so for me this task does not show the regime where frozen representations are insufficient. Is the gain over FedIT averaged over multiple seeds as I did not see variance. At last I did not see the revised version and how the authors addressed the questions. For the current stage, I will keep my score but happy to increase it if addressed.

Reviewer_BAvMOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

One of the common solutions to reduce the cost in federated fine-tuning is to adapt PEFT methods (mostly LoRA or its later variations) which require fewer parameter updates. While the the backbone model is still frozen when using LoRA, clients still need to store intermediate activations for back propagation, which is memory-heavy; conversely, existing forward-only zeroth-order (ZO) methods avoid back propagation but suffer from slow convergence. This paper proposes utilizing a different efficient fine-tuning method that keeps the base model completely frozen / forward-only and does not include it in the backward update graph at all, while still training the adaptation with exact first-order gradients. In this method (called FoRA), first we have a forward pass through the frozen LLM. Each client copies intermediate outputs from a few layers and detaches them from the network (to avoid gradient calculations on these layers). These detached features are then fed into a lightweight external side-network (the Split-Bridge module), which combines a direct per-layer adapter path with a weighted mixture of parallel adapter banks. Finally, the side-network's output is added directly to the frozen LLM's final predictions as a learned adjustment, which is regularized against the broadcast server model during local training and aggregated across clients via FedAvg. The authors accompany this framework with theoretical analyses, and evaluate it against PEFT and ZO baselines.

Critical strengths

1- The paper targets a real memory bottleneck in federated LLM fine-tuning with standard PEFT methods (like LoRA), keeping the base LLM forward-only while still allowing exact first-order gradient updates on the client.

2- Unlike prior server-assisted side-tuning approaches, FoRA keeps the trainable side-network on the client and does not require clients to upload any intermediate layer outputs to the server, which better preserves privacy.

3- The authors support their design choices through both theoretical analysis and ablation studies.

Critical weaknesses

1- While FoRA is more memory-efficient during training, all clients are left with an extra external module that cannot be merged into the model during inference. This can create extra runtime computation and latency overhead at deployment.

2- Since the updates are never directly applied to the model’s internal layers, it is not very clear how this type of training would work for complex multi-step reasoning or math tasks where the model actually needs to adapt its internal attention patterns.

3- The paper is hard to follow in several places (especially the Introduction and the Method/Theory sections).

4- Because side-network tuning outside a frozen backbone already exists in the centralized setting (e.g., Ladder Side-Tuning), the paper is missing a direct comparison against simply applying standard FedAvg or FedProx to an existing side-tuning method.

Questions for authors

1- How does FoRA scale with input sequence length?

2- How can we compare the number of trainable parameters in a LoRA-based method and FoRA? And what is a fair comparison if we want to fix the computation cost?

3- How does FoRA compare against simply applying standard FedAvg or FedProx directly to an existing side-tuning architecture like Ladder Side-Tuning?

4 comments
AuthorsOpenReview ↗

Rebuttal-1

We thank Reviewer BAvM for recognizing our work and for the constructive feedback. Below, we address each concern.

Extra overhead at deployment (W1)

FoRA's external side network does introduce a small deployment overhead. However, the overhead is minimal relative to the substantial training savings and performance gains in Table 1 of the original paper. We measured inference computation and latency for FoRA and FedIT in the three experimental settings used in the paper, with results below.

SettingMethodMean per-example latency (ms)Mean per-example inference computation (GFLOPs)
OPT-350M@MRPCFedIT67.45161.90
FoRA70.07161.91
OPT-2.7B@SST-2FedIT93.871310.90
FoRA102.261310.92
Llama-3.1-8B@Dolly-15KFedIT644.977393.33
FoRA679.937402.05

Note: GFLOPs denotes billions (10910^9) of floating-point operations.

Compared with FedIT, FoRA requires only 0.00618%, 0.00153%, and 0.118% more computation on MRPC, SST-2, and Dolly-15K, respectively, and adds only 2.62 ms (3.9%), 8.39 ms (8.9%), and 34.96 ms (5.4%) of latency per example in the same settings. These small increases in inference cost accompany substantial training savings of 29.2-64.9% in time and 27.3-58.9% in peak GPU memory (Table 1). For example, on MRPC, FoRA saves 3.55 training hours and improves Final-10 accuracy by 1.97 percentage points, with only 1.06 s of additional inference time over the full test set.

Effectiveness on complex tasks (W2)

FoRA adapts to complex multi-step reasoning and math tasks by correcting each next-token prediction without updating the backbone's internal attention parameters. Take multi-step reasoning as an example. First, the frozen backbone processes the question tokens together with the reasoning tokens generated so far, and the side network reads their detached multi-layer representations (§4.2, Eq. (3)). Next, Split-Bridge computes a hidden-space correction from these representations (§4.3, Eqs. (4)-(6)). Then, the frozen language-model head projects this correction into a vocabulary-space residual, which is added to the backbone logits to adjust the next-token distribution (§3.2, Eq. (2); Appendix D.3). Finally, the token selected from this corrected distribution extends the reasoning prefix for the next generation step, allowing FoRA to steer intermediate reasoning and thereby influence later steps and the final answer.

We additionally evaluate Qwen3-8B-base on GSM8K with 10 clients, a non-IID data distribution, 50% client participation, 1 local epoch, and a batch size of 2. We compare FoRA with the frozen backbone (naive baseline), FedIT (standard federated LoRA), and GMFL (second-best overall utility in Table 1). FoRA reaches 93.8% exact match, 7.4 percentage points above the frozen backbone, demonstrating effective adaptation to multi-step mathematical reasoning without updating backbone attention parameters.

MethodPPL↓Exact Match (%)↑Training time (h)↓Peak training GPU memory (GiB)↓
Frozen Qwen3-8B-base1.65086.4N/AN/A
FedIT1.141992.285.0850.098
GMFL1.142591.674.5650.070
FoRA1.136393.841.1636.697

Scalability with input sequence length (Q1)

FoRA scales to longer input sequences without changing its architecture or training procedure. We tested OPT-2.7B@SST-2 with input sequences padded to fixed lengths of 64, 128, 256, and 512 tokens (512 is four times RAVAN's GLUE length cap). Table 11 (Appendix D.8) shows that FoRA's peak client GPU memory increases only from 10.11 to 10.19 GiB as the maximum length rises from 64 to 512, while its memory reduction relative to FedIT grows from 24.83% to 40.86%.

We further report accuracy and training time for FedIT and FoRA under the same setting:

Maximum input lengthFedIT Final-10 Acc (%)FoRA Final-10 Acc (%)FedIT training time (h)FoRA training time (h)
6495.6495.79189.4676.43
12895.4195.64257.5091.73
25695.8996.00361.09126.87
51295.8796.06413.61162.09

These results demonstrate FoRA's scalability with input length. Its Final-10 accuracy remains between 95.64% and 96.06% across these lengths and exceeds FedIT by 0.11-0.23 percentage points at each length, while training time is reduced by 59.66%-64.86%.

AuthorsOpenReview ↗

Rebuttal-2

Compared with LST (W4/Q3)

Our original comparison focused on FL methods because LST studies centralized side tuning, whereas FoRA addresses side-network learning under non-IID local training and federated aggregation. Directly applying FedAvg to LST introduces non-IID local training and parameter aggregation without adapting its centralized architecture or objective to these conditions, compromising the accuracy of the resulting global model.

  1. Layer coordination under federated aggregation. FoRA's Split-Bridge computes corrections directly from fixed backbone features, reducing dependence on sequential trainable layers (§4.3, Eqs. (4)-(6)). By contrast, LST couples each side layer to the preceding trainable layer through a learned gate (LST §3.2), so dependencies learned on non-IID clients may be disrupted when FedAvg averages parameters.
  2. Client drift during local federated training. FoRA's Global Reference constrains drift during non-IID local updates by anchoring each local residual to the round's fixed global residual (§4.4, Eq. (7)). LST's centralized objective lacks this constraint, and FedAvg averages parameters only after local training, leaving prediction drift during local updates unregulated.

To address the reviewer's concern, we further apply standard FedAvg directly to LST under the OPT-350M@MRPC setting in Table 1.

MethodBest Acc (%)↑Best F1 (%)↑Final-10 Acc (%)↑Final-10 F1 (%)↑Training time (h)↓Peak allocated GPU memory (GiB)↓
LST+FedAvg73.7683.2671.3479.9114.212.034
FoRA76.1183.9975.6883.808.591.306

FoRA improves Final-10 Accuracy by 4.34 percentage points and Final-10 F1 by 3.89 percentage points, while reducing training time by 39.5% and peak allocated GPU memory by 35.8%.

Trainable parameter counts (Q2)

We can compare LoRA-based methods and FoRA by directly counting their trainable parameters during training. Appendix B.3 also gives FoRA's theoretical count for classification (Eqs. (17)-(18)). We report measured counts for the paper's three experimental settings below. The results show that FoRA accommodates more trainable parameters with substantially lower GPU memory and achieves higher utility. For example, on MRPC, FoRA reduces peak GPU memory by 58.9% and improves Final-10 accuracy by 1.97 percentage points over FedIT (Table 1).

SettingFedIT (M)RAVAN (M)GMFL (M)FoRA (M)
OPT-350M@MRPC1.5741.556[0.394,1.574]4.067
OPT-2.7B@SST-25.2485.240[2.624,5.248]13.532
Llama-3.1-8B@Dolly-15K6.8166.802[3.408,6.816]21.466

Note: GMFL is shown as a range because its trainable parameter count decreases with scheduled freezing.

Fixed-computation-cost comparison (Q2)

A fair comparison gives each method the same cumulative training budget, measured in FLOPs (floating-point operations). For example, one scalar addition or multiplication counts as 1 FLOP, and one multiply-add counts as 2.

We added a fixed-computation-cost comparison on OPT-350M@MRPC using the data split and federated training protocol from Table 1. For each method, we accumulated FLOPs across participating clients and any required server-side calculations. For each column below, we restrict every method's cumulative floating-point operation count to the listed threshold and report its highest accuracy (%) within that allowance, ensuring the same computation budget for all methods. FoRA achieves the highest accuracy at all three budgets.

Method15,000 TFLOPs30,000 TFLOPs45,000 TFLOPs
FedIT58.9164.8568.81
FedMeZO50.4950.7451.23
RAVAN70.2971.2871.28
GMFL68.5670.0470.79
FedProxy52.7253.9654.45
FoRA74.5076.2376.23

Note: TFLOPs = 101210^{12} FLOPs.

Presentation (W3)

Thanks for the suggestion. We will improve readability in the revision.

We believe these revisions improve the clarity of our work. We sincerely thank the reviewer again for the valuable feedback.

Reviewer_BAvMOpenReview ↗

Rebuttal

I thank the authors for their efforts during the rebuttal. They have answered all of my questions clearly, and I have no further questions. I will maintain my score of 3.

Dear Reviewer BAvM,

We sincerely appreciate your constructive feedback and the time invested in reviewing our work. We are delighted that our rebuttal addressed your concerns. Your acknowledgment of our revisions is highly encouraging, and we are grateful for the opportunity to improve the manuscript.

Thank you again for your valuable insights, which have significantly strengthened our paper. We look forward to incorporating these refinements in the revised version.

Sincerely,

The Authors

Comments on the paper

1 comment

As only a few days remain in the author-reviewer discussion period, I encourage reviewers to review the authors' rebuttal and actively participate in the discussion. In particular, please clarify which concerns have been addressed and identify any remaining concerns or questions that may require further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.